Image processing method, electronic device, image processing system and chip system

Abstract

The embodiments of the present application provide an image processing method, an electronic device, an image processing system, and a chip system, which relate to the field of image processing technology and can solve the problem of poor image processing effect of a feature analysis network model in a multi-task deep neural network model. The first device extracts feature information of an image to be processed through a feature extraction network model; the first device identifies the extracted feature information to obtain identification information of the feature information; the first device sends the feature information of the image to be processed and the identification information of the feature information to the second device; after the second device receives the feature information and the corresponding identification information sent by the first device, it selects the feature analysis network model corresponding to the identification information to process the received feature information to obtain an image processing result.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular, to an image processing method, an electronic device, an image processing system, and a chip system. Background Art

[0002] With the development of deep learning theory, image processing based on deep neural network models has also developed rapidly. For example, the features of an image can be extracted through a deep neural network model, and then the features of the image can be analyzed to complete the image processing. Image processing can include: object detection, semantic segmentation, panoramic segmentation and image classification.

[0003] From a functional perspective, deep neural network models based on deep learning can be divided into two parts: feature extraction network model and feature analysis network model. The feature extraction network model is used to extract image features; the feature analysis network model is used to analyze and process image features to complete the corresponding image processing tasks. In the multi-task image processing process, in order to reduce the parameters of the deep neural network model and reduce the amount of tasks for deep neural network model training, multiple different feature analysis network models can share the same feature extraction network model. However, the image features used by multiple different feature analysis network models may be different. The image features extracted by sharing the same feature extraction network model cannot take into account each feature analysis network model, resulting in poor image processing effect of the feature analysis network model. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, an electronic device, an image processing system and a chip system to solve the problem of poor image processing effect of a feature analysis network model in a multi-task deep neural network model.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] The first aspect provides an image processing method, including: a first device extracts feature information of an image to be processed by using at least one pre-stored feature extraction network model; the first device identifies the extracted feature information to obtain identification information of the feature information; the first device sends the feature information of the image to be processed and the identification information of the feature information to a second device to instruct the second device to select a feature analysis network model corresponding to the identification information to process the feature information.

[0007] In an embodiment of the present application, after the first device identifies the extracted feature information to obtain the identification information, it can instruct the second device where there are multiple feature analysis network models to select the feature analysis network model corresponding to the identification information to process the received feature information according to the identification information. In this method, the feature analysis network model in the second device can correspond to the feature extraction network models in multiple first devices, or correspond to multiple feature extraction network models in the first device. When there are multiple first devices or there are multiple feature extraction network models in the first device, the multiple feature analysis network models in the second device can determine which feature analysis network model to input the feature information into to complete the corresponding image processing task according to the identification information, thereby avoiding the problem of poor image processing effect caused by the feature information of the same feature extraction network model being unable to meet the needs of multiple feature analysis network models at the same time.

[0008] In a possible implementation manner of the first aspect, the first device identifies the extracted feature information, and obtaining the identification information of the feature information includes: the first device obtains the identification of the feature extraction network model that extracts the feature information; the first device uses the identification of the feature extraction network model that extracts the feature information as the identification information of the feature information.

[0009] In some examples, the identifier of the feature extraction network model that extracts the feature information can be used as the identification information of the feature information. The second device can determine the feature extraction network model that extracts the feature information based on the identification information of the feature information, thereby selecting a suitable feature analysis network model to analyze the received feature information to complete the corresponding image processing task.

[0010] In a possible implementation of the first aspect, the first device identifies the extracted feature information, and the identification information of the feature information obtained includes: the first device obtains the identification of the output level of the feature information, wherein the output level of the feature information is the level at which the feature information is output in a feature extraction network model that extracts the feature information; the first device uses the identification of the output level of the feature information as the identification information of the feature information.

[0011] In some examples, the identification of the output level of the feature information can be used as the identification information of the feature information. The second device can determine the output level of the feature information according to the identification information of the feature information, thereby selecting a suitable feature analysis network model to analyze the received feature information to complete the corresponding image processing task.

[0012] In a possible implementation manner of the first aspect, the first device identifies the extracted feature information, and the identification information of the feature information includes: the first device obtains the identification of a feature extraction network model for extracting the feature information; the first device obtains the identification of an output level of the feature information, wherein the output level of the feature information is the level at which the feature information is output in the feature extraction network model for extracting the feature information; the first device uses the identification of the feature extraction network model for extracting the feature information and the identification of the output level of the feature information as the identification information of the feature information.

[0013] In some examples, the identifier of the feature extraction network model for extracting feature information and the identifier of the output level of the feature information can be used as the identification information of the feature information. The second device can determine the feature extraction network model and the output level of the feature information according to the identification information of the feature information, thereby selecting a suitable feature analysis network model to analyze the received feature information to complete the corresponding image processing task.

[0014] In practical applications, any of the above-listed methods for generating identification information can be selected according to actual needs, thereby improving the flexibility of the application of the embodiments of the present application.

[0015] The second aspect provides an image processing method, including: a second device obtains feature information of an image to be processed and identification information of the feature information sent by a first device connected to the second device; the second device determines a feature analysis network model for processing the feature information based on the identification information of the feature information; the second device inputs the feature information of the image to be processed into the determined feature analysis network model to obtain an image processing result.

[0016] In a possible implementation of the second aspect, the second device determines a feature analysis network model for processing the feature information based on identification information of the feature information, including: the second device obtains a correspondence between the identification information and the feature analysis network model; and the second device uses, based on the correspondence, the feature analysis network model corresponding to the identification information of the feature information as the feature analysis network model for processing the feature information.

[0017] In a possible implementation of the second aspect, the identification information of the feature information includes: an identification of a feature extraction network model that extracts the feature information; and / or an identification of an output level of the feature information, wherein the output level of the feature information is a level at which the feature information is output in the feature extraction network model that extracts the feature information.

[0018] A third aspect provides an electronic device, including:

[0019] A feature information extraction unit, used to extract feature information of the image to be processed by using at least one pre-stored feature extraction network model;

[0020] An identification information generating unit, used to identify the extracted characteristic information and obtain identification information of the characteristic information;

[0021] An information sending unit is used to send feature information of an image to be processed and identification information of the feature information to the second device, so as to instruct the second device to select a feature analysis network model corresponding to the identification information to process the feature information.

[0022] A fourth aspect provides an electronic device, including:

[0023] An information acquisition unit, used to acquire feature information of an image to be processed and identification information of the feature information sent by a connected first device;

[0024] A model determination unit, configured to determine a feature analysis network model for processing the feature information according to identification information of the feature information;

[0025] The image processing unit is used to input the feature information of the image to be processed into a determined feature analysis network model to obtain an image processing result.

[0026] The fifth aspect provides an electronic device, comprising a processor, the processor being used to run a computer program stored in a memory to implement any method of the first aspect of the present application.

[0027] A sixth aspect provides an electronic device, comprising a processor, the processor being used to run a computer program stored in a memory to implement any method of the second aspect of the present application.

[0028] The seventh aspect provides an image processing system, comprising at least one electronic device provided by the fifth aspect and at least one electronic device provided by the sixth aspect.

[0029] The eighth aspect provides a chip system, including a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement any method of the first aspect and / or any method of the second aspect of the present application.

[0030] The ninth aspect provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, it implements any method of the first aspect and / or any method of the second aspect of the present application.

[0031] The tenth aspect provides a computer program product. When the computer program product is run on a device, the device executes any method in the first aspect and / or any method in the second aspect.

[0032] It can be understood that the beneficial effects of the second to tenth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application;

[0034] Figure 2 An example diagram of an image processing method provided in an embodiment of the present application;

[0035] Figure 3 Another example diagram of the image processing method provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the hardware structure of an electronic device for executing an image processing method provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a processing process in which a first device and a second device perform an image processing method provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of a flow chart of a first device executing an image processing method provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of the correspondence between identification information and a feature analysis network model in the image processing method provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of another correspondence between identification information and a feature analysis network model in the image processing method provided in an embodiment of the present application;

[0041] Fig. 9 A schematic diagram of another correspondence between identification information and a feature analysis network model in the image processing method provided in an embodiment of the present application;

[0042] Fig.10 A schematic diagram of a flow chart of a second device executing an image processing method provided in an embodiment of the present application;

[0043] Fig.11 A schematic block diagram of the functional architecture modules of the first device and the second device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0045] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0046] It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0047] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0048] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0049] The embodiments of the present application can be applied to multiple image processing task scenarios. The image processing process includes an image feature extraction process and an image feature analysis process. Figure 1 , Figure 1 This is an application scenario of the image processing method provided in the embodiment of the present application. Figure 1As shown, corresponding to a cloud platform that performs an image feature analysis process, there may be multiple cameras that perform an image feature extraction process. Only three cameras are shown in the figure. More or fewer cameras may be set in actual applications. In one example of this application scenario, these cameras are set on different roads. In another example of this application scenario, these cameras are set in a factory, for example, in a factory workshop, office, entrance and exit gate, and garage entrance and exit. For these cameras, feature extraction network models may be stored in some or all of the cameras. After the camera captures video and / or images, the feature extraction network model in the camera can extract feature information of the image frame in the video and / or feature information of the image. The camera sends the feature information to the cloud platform, and the cloud platform selects an appropriate feature analysis network model to process the received feature information based on the image processing task to be executed.

[0050] The above application scenarios can be Figure 2 An example of an image processing method shown is to perform image processing, Figure 2 In the example shown, multiple feature analysis network models (for example, the image classification network model, target detection network model, and semantic segmentation network model in the diagram) share a feature extraction network model. The feature extraction network model can be loaded in each camera device, and multiple feature analysis network models can be loaded in the cloud platform. Multiple feature analysis network models in the cloud platform share feature information extracted by a feature extraction network model in the camera device. The camera device and the cloud platform can establish a communication connection wirelessly.

[0051] If you follow Figure 2 The image processing method shown in the figure may have the following problems when performing image processing:

[0052] (1) In order to meet the needs of feature analysis network models in the cloud platform, it may be necessary to update the feature extraction network model in the camera device. When the feature extraction network model in the camera device needs to be updated, the cost of updating is very high due to the large number of cameras and their wide distribution (such as cameras on urban roads). Figure 2 The image processing method of the example shown has poor flexibility when being deployed and updated during application.

[0053] (2) In order to adapt to the needs of the newly added feature analysis network model in the cloud platform, when a new feature extraction network model with better performance is added to the camera device, it is very likely that the original feature analysis network model will be unable to recognize the feature information extracted by the newly added feature extraction network model, resulting in some feature analysis network models in the cloud platform being unable to complete the image processing task or having poor image processing effects. In other words, it cannot be guaranteed that the same feature extraction network model can be fully suitable for all feature analysis network models.

[0054] (3) For different image processing tasks in the cloud platform, the feature analysis network model used for target detection may require a high-performance feature extraction network model; the feature analysis network model used for image classification does not need to adopt a high-performance feature extraction network model. In order to meet the high requirements, all feature analysis network models adopt a high-performance feature analysis network model, which leads to large amount of calculation each time the camera device extracts features and serious memory consumption.

[0055] To solve Figure 2 The above problems in the example shown can also be solved by Figure 3 An example of an image processing method is shown in FIG. Figure 3 As shown, each camera device can be set with different feature extraction network models according to the application occasion. For example, the images collected by the camera device set at the entrance and exit of the factory garage are mainly used for vehicle detection. Therefore, the camera device set at the entrance and exit of the garage can load a feature extraction network model suitable for vehicle detection. The images collected by the camera device set at the factory entrance and exit gate are mainly used for face recognition. Therefore, the camera device set at the factory entrance and exit gate can load a feature extraction network model suitable for face recognition. The cloud platform can load a feature analysis network model suitable for vehicle detection and a feature extraction network model suitable for face recognition. For the cloud platform, it may receive feature information sent by the camera device set at the factory entrance and exit gate, and it may also receive feature information sent by the camera device set at the entrance and exit of the factory garage. In order to facilitate the cloud platform to input feature information into a suitable feature analysis network model to complete the corresponding image processing task, after extracting the feature information of the image from the camera device, identification information can be generated for the feature information. For example, a rule is pre-set: the identification of the feature extraction network model suitable for vehicle detection is 0, and the identification of the feature extraction network model suitable for face recognition is 1. After the camera device obtains the feature information, it can also generate identification information for the feature information based on the identification of the feature extraction network model. The camera device sends the feature information and the identification information of the feature information to the cloud platform. The cloud platform can identify the identification information of the received feature information, and then select a suitable feature analysis network model to process the received feature information according to the identification information of the received feature information, thereby obtaining an image processing result.

[0056] As an example, corresponding to the cloud platform that executes the image feature analysis process, there are three camera devices. The feature extraction network model in the first camera device is model 1, the extracted feature information is feature information A, and the corresponding identification information is 00. The feature extraction network model in the second camera device is model 2, the extracted feature information is feature information B, and the corresponding identification information is 01. The feature extraction network model in the third camera device is model 3, the extracted feature information is feature information C, and the corresponding identification information is 10.

[0057] Three feature analysis network models may be stored in the cloud platform: model α for image classification tasks, model β for object detection tasks, and model γ for semantic segmentation tasks.

[0058] Among them, the corresponding relationship between the camera device, the feature extraction network model stored in the camera device, the identification information, the image processing task and the feature analysis network model stored in the cloud platform is:

[0059] The first camera device—model 1—identification information 00—image classification task—model α;

[0060] The second camera device - model 2 - identification information 01 - target detection task - model β;

[0061] The third camera device—model 3—identification information 10—semantic segmentation task—model γ.

[0062] In one example, these cameras autonomously send the extracted feature information and identification information to the cloud platform. According to the above correspondence, the cloud platform can input the feature information with identification information 00 into the model α corresponding to the identification information 00 to complete the image classification task; according to the above correspondence, the cloud platform can input the feature information with identification information 01 into the model β corresponding to the identification information 01 to complete the target detection task; according to the above correspondence, the cloud platform can input the feature information with identification information 10 into the model γ corresponding to the identification information 10 to complete the semantic segmentation task.

[0063] In another example, when the user needs to perform an image classification task, the execution instruction of the image classification task is sent to the cloud platform. Through the above correspondence, the cloud platform sends a feature extraction instruction to the first camera device corresponding to the image classification task. The first camera device inputs the captured image into model 1, obtains feature information, and generates identification information 00 of the feature information. The first camera device sends the feature information and identification information 00 to the cloud platform. After receiving the feature information and identification information 00, the cloud platform inputs the feature information into the model α corresponding to the identification information 00 to complete the image classification task.

[0064] In another example, when the user needs to perform a semantic segmentation task, the user determines through the above correspondence that the image feature extraction process is performed through the third camera device. The user sends an execution instruction to the third camera device. The third camera device obtains the feature information of the image to be processed through the model 3 and generates identification information 10 of the feature information. The third camera device sends the feature information and identification information 10 to the cloud platform. After the cloud platform receives the feature information and identification information 10, the cloud platform inputs the feature information into the model γ corresponding to the identification information 10 to complete the semantic segmentation task.

[0065] In the above three examples, each camera device may store one feature extraction network model. In practical applications, each camera device may also store multiple feature extraction network models.

[0066] according to Figure 3 In the image processing method shown, when the network model needs to be updated, a new feature extraction network model can be added to one or more camera devices that need to be updated or the new feature extraction network model can replace the old feature extraction network model, and a unique identifier of the feature extraction network model can be set accordingly. It is not necessary to add a new feature extraction network model to each camera device that has a network connection relationship with the cloud platform or replace the old feature extraction network model with a new feature extraction network model. Figure 3 The image processing method shown increases the flexibility and scalability of network updates or deployments. For ease of description, the electronic device where the feature extraction network model is located is recorded as the first device, and the steps performed by the first device are recorded as the image feature extraction process. The electronic device where the feature analysis network model is located is recorded as the second device, and the steps performed by the second device are recorded as the image feature analysis process. The first device and the second device jointly complete the image processing method.

[0067] in addition, Figure 1 In the application scenario shown, the image processing system includes multiple first devices and one second device. In actual applications, the image processing system may also include one first device and one second device; the image processing system may also include one first device and multiple second devices, or the image processing system may also include multiple first devices and multiple second devices. When there are multiple second devices, the first device may determine to send to the corresponding second device based on the identification information of the feature information.

[0068] As an example, when the image processing system includes: multiple first devices and multiple second devices, the correspondence between the first device, the feature extraction network model stored in the first device, the identification information, the image processing task, the second device, and the feature analysis network model stored in the second device is:

[0069] The first camera device—model 1—identification information 00—image classification task—cloud platform 1—model α;

[0070] The first camera device—model 2—identification information 01—target detection task—server 2—model β;

[0071] Second camera device—model 3—identification information 10—semantic segmentation task—cloud platform 1—model γ.

[0072] In this example, after the first camera device extracts the feature information of the image through model 1, it generates identification information 00, and sends the feature information and identification information 00 to cloud platform 1, and cloud platform 1 inputs the feature information of identification information 00 into model α; after the first camera device extracts the feature information of the image through model 2, it generates identification information 01, and sends the feature information and identification information 01 to server 2, and server 2 inputs the feature information of identification information 01 into model β.

[0073] Of course, in this example, the image processing task can also be performed in the following manner: the first camera device extracts a set of feature information of the image through model 1 and generates identification information 00; extracts a set of feature information of the image through model 2 and generates identification information 01. The first camera device sends both sets of feature information and the corresponding identification information to cloud platform 1 and server 2. Cloud platform 1 selects the feature information of identification information 00 from the two sets of feature information received and inputs it into model α. Server 2 selects the feature information of identification information 01 from the two sets of feature information received and inputs it into model β.

[0074] Based on the above examples, it can be understood that the image processing system may include at least one second device, wherein the second device may store multiple feature analysis network models or one feature analysis network model.

[0075] It should be noted that the above application scenarios and the corresponding camera devices and cloud platforms are only used as examples of the first device and the second device. In actual applications, the first device may be an electronic device other than the camera device, and the second device may be an electronic device other than the cloud platform.

[0076] As an example, an image processing method provided in an embodiment of the present application can be applied to a first device, and the first device can be an electronic device with a camera, such as a camera device, a mobile phone, a tablet computer, etc. Of course, the first device may not have a camera, but receive images or videos sent by other electronic devices with cameras. An image processing method provided in an embodiment of the present application can also be applied to a second device, and the second device can be an electronic device with image feature analysis capabilities, such as a cloud platform, a server, a computer, a notebook, a mobile phone, etc. Of course, in practical applications, the first device and the second device can be the same electronic device, for example, both the first device and the second device can be mobile phones. The image feature extraction process and the image feature analysis process are both executed in the processor of the mobile phone, or the image feature extraction process is executed in the first processor of the mobile phone, and the image feature analysis process is executed in the second processor of the mobile phone. At least one first device and a second device constitute an image processing system.

[0077] Figure 4 The schematic diagram of the structure of an electronic device is shown. The electronic device can be used as a first device to perform an image feature extraction process in an image processing method, can be used as a second device to perform an image feature analysis process in an image processing method, and can also be used as an electronic device to perform an image feature extraction process and an image feature analysis process in an image processing method. The electronic device 400 may include a processor 410, an external memory interface 420, an internal memory 421, a universal serial bus (USB) interface 430, a charging management module 440, a power management module 441, a battery 442, an antenna 1, an antenna 2, a mobile communication module 450, a wireless communication module 460, an audio module 470, a speaker 470A, a receiver 470B, a microphone 470C, an earphone interface 470D, a sensor module 480, a button 490, a motor 491, an indicator 492, a camera 493, a display screen 494, and a subscriber identification module (SIM) card interface 495, etc. The sensor module 480 may include a pressure sensor 480A, a gyroscope sensor 480B, an air pressure sensor 480C, a magnetic sensor 480D, an acceleration sensor 480E, a distance sensor 480F, a proximity light sensor 480G, a fingerprint sensor 480H, a temperature sensor 480J, a touch sensor 480K, an ambient light sensor 480L, a bone conduction sensor 480M, etc.

[0078] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 400. In other embodiments of the present application, the electronic device 400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0079] The processor 410 may include one or more processing units, for example: the processor 410 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, the processor 410 is used to perform the image feature extraction process in the image processing method in the embodiment of the present application, for example, the following steps 601 to 603, and / or perform the image feature analysis process in the image processing method in the embodiment of the present application, for example, the following steps 1001 to 1003.

[0080] The controller may be the nerve center and command center of the electronic device 400. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0081] The processor 410 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. The memory may store instructions or data that the processor 410 has just used or cyclically used. If the processor 410 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 410, and thus improves the efficiency of the system.

[0082] In some embodiments, the processor 410 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0083] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 410 may include multiple groups of I2C buses. The processor 410 may be coupled to the touch sensor 480K, the charger, the flash, the camera 493, etc. through different I2C bus interfaces. For example: the processor 410 may be coupled to the touch sensor 480K through the I2C interface, so that the processor 410 communicates with the touch sensor 480K through the I2C bus interface to realize the touch function of the electronic device 400.

[0084] The I2S interface can be used for audio communication. In some embodiments, the processor 410 can include multiple I2S buses. The processor 410 can be coupled to the audio module 470 via the I2S bus to achieve communication between the processor 410 and the audio module 470.

[0085] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 470 and the wireless communication module 460 can be coupled via a PCM bus interface.

[0086] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.

[0087] The MIPI interface can be used to connect the processor 410 with peripheral devices such as the display screen 494 and the camera 493. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 410 and the camera 493 communicate via the CSI interface to realize the shooting function of the electronic device 400. The processor 410 and the display screen 494 communicate via the DSI interface to realize the display function of the electronic device 400.

[0088] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 410 with the camera 493, the display screen 494, the wireless communication module 460, the audio module 470, the sensor module 480, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0089] The USB interface 430 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 430 can be used to connect a charger to charge the electronic device 400, and can also be used to transfer data between the electronic device 400 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0090] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 400. In other embodiments of the present application, the electronic device 400 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0091] The charging management module 440 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 440 may receive charging input from a wired charger through the USB interface 430. In some wireless charging embodiments, the charging management module 440 may receive wireless charging input through a wireless charging coil of the electronic device 400. While the charging management module 440 is charging the battery 442, it may also power the electronic device through the power management module 441.

[0092] The power management module 441 is used to connect the battery 442, the charging management module 440 and the processor 410. The power management module 441 receives input from the battery 442 and / or the charging management module 440, and supplies power to the processor 410, the internal memory 421, the external memory, the display screen 494, the camera 493, and the wireless communication module 460. The power management module 441 can also be used to monitor parameters such as battery capacity, battery cycle number, and battery health status (leakage, impedance).

[0093] In some other embodiments, the power management module 441 may also be disposed in the processor 410. In some other embodiments, the power management module 441 and the charging management module 440 may also be disposed in the same device.

[0094] The wireless communication function of the electronic device 400 can be implemented through the antenna 1, the antenna 2, the mobile communication module 450, the wireless communication module 460, the modem processor and the baseband processor.

[0095] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 400 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0096] The mobile communication module 450 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 400. The mobile communication module 450 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 450 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 450 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0097] In some embodiments, at least some functional modules of the mobile communication module 450 may be disposed in the processor 410. In some embodiments, at least some functional modules of the mobile communication module 450 and at least some functional modules of the processor 410 may be disposed in the same device.

[0098] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 470A, a receiver 470B, etc.), or displays an image or video through a display screen 494. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 410 and be set in the same device as the mobile communication module 450 or other functional modules.

[0099] The wireless communication module 460 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 400. The wireless communication module 460 can be one or more devices integrating at least one communication processing module. The wireless communication module 460 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 410. The wireless communication module 460 can also receive the signal to be sent from the processor 410, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0100] In some embodiments, the antenna 1 of the electronic device 400 is coupled to the mobile communication module 450, and the antenna 2 is coupled to the wireless communication module 460, so that the electronic device 400 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0101] The electronic device 400 implements the display function through a GPU, a display screen 494, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 410 may include one or more GPUs, which execute program instructions to generate or change display information.

[0102] The display screen 494 is used to display images, videos, etc. The display screen 494 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 400 may include 1 or N display screens 494, where N is a positive integer greater than 1.

[0103] The electronic device 400 can realize the shooting function through ISP, camera 493, video codec, GPU, display screen 494 and application processor.

[0104] ISP is used to process the data fed back by camera 493. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 493.

[0105] The camera 493 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 400 may include 1 or N cameras 493, where N is a positive integer greater than 1.

[0106] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 400 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0107] Video codecs are used to compress or decompress digital videos. The electronic device 400 may support one or more video codecs. Thus, the electronic device 400 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0108] NPU is a neural network (NN) computing processor, which can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of electronic device 400, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0109] In an embodiment of the present application, the NPU or other processor can be used to perform operations such as face detection, face tracking, face feature extraction, and image clustering on face images in a video stored in the electronic device 400; perform operations such as face detection, face feature extraction, and other operations on face images in pictures stored in the electronic device 400, and cluster the pictures stored in the electronic device 400 based on the facial features of the pictures and the clustering results of face images in the video.

[0110] The external memory interface 420 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 400. The external memory card communicates with the processor 410 through the external memory interface 420 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0111] The internal memory 421 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 410 executes various functional applications and data processing of the electronic device 400 by running the instructions stored in the internal memory 421. The internal memory 421 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area may store data created during the use of the electronic device 400 (such as audio data, a phone book, etc.).

[0112] In addition, the internal memory 421 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0113] The electronic device 400 can implement audio functions such as music playing and recording through the audio module 470, the speaker 470A, the receiver 470B, the microphone 470C, the headphone jack 470D, and the application processor.

[0114] The audio module 470 is used to convert digital audio signals into analog audio signals for output, and is also used to convert analog audio inputs into digital audio signals. The audio module 470 can also be used to encode and decode audio signals. In some embodiments, the audio module 470 can be arranged in the processor 410, or some functional modules of the audio module 470 can be arranged in the processor 410.

[0115] The speaker 470A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 400 can listen to music or listen to a hands-free call through the speaker 470A.

[0116] The receiver 470B, also called a "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 400 receives a call or a voice message, the voice can be received by placing the receiver 470B close to the human ear.

[0117] Microphone 4270C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 470C with his mouth to input the sound signal into the microphone 470C. The electronic device 400 can be provided with at least one microphone 470C. In other embodiments, the electronic device 400 can be provided with two microphones 470C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 400 can also be provided with three, four or more microphones 470C.

[0118] The earphone interface 470D is used to connect a wired earphone and can be a USB interface 430 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0119] The pressure sensor 480A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 480A can be set on the display screen 494. There are many types of pressure sensors 480A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 480A, the capacitance between the electrodes changes. The electronic device 400 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 494, the electronic device 400 detects the intensity of the touch operation based on the pressure sensor 480A. The electronic device 400 can also calculate the position of the touch based on the detection signal of the pressure sensor 480A.

[0120] The gyroscope sensor 480B can be used to determine the motion posture of the electronic device 400. In some embodiments, the angular velocity of the electronic device 400 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 480B. The gyroscope sensor 480B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 480B detects the angle of the electronic device 400 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 400 through reverse movement to achieve anti-shake. The gyroscope sensor 480B can also be used for navigation and somatosensory game scenes.

[0121] The air pressure sensor 480C is used to measure air pressure. In some embodiments, the electronic device 400 calculates the altitude through the air pressure value measured by the air pressure sensor 480C to assist in positioning and navigation.

[0122] The magnetic sensor 480D includes a Hall sensor. The electronic device 400 can use the magnetic sensor 480D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 400 is a flip phone, the electronic device 400 can detect the opening and closing of the flip cover according to the magnetic sensor 480D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0123] The acceleration sensor 480E can detect the magnitude of the acceleration of the electronic device 400 in all directions (generally three axes). When the electronic device 400 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0124] The distance sensor 480F is used to measure the distance. The electronic device 400 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 400 can use the distance sensor 480F to measure the distance to achieve fast focusing.

[0125] The proximity light sensor 480G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 400 emits infrared light outward through the light emitting diode. The electronic device 400 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 400. When insufficient reflected light is detected, the electronic device 400 can determine that there is no object near the electronic device 400. The electronic device 400 can use the proximity light sensor 480G to detect that the user holds the electronic device 400 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 480G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0126] Ambient light sensor 480L is used to sense the brightness of ambient light. Electronic device 400 can adaptively adjust the brightness of display screen 494 according to the perceived brightness of ambient light. Ambient light sensor 480L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 480L can also cooperate with proximity light sensor 480G to detect whether electronic device 400 is in a pocket to prevent accidental touch.

[0127] The fingerprint sensor 480H is used to collect fingerprints. The electronic device 400 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0128] The temperature sensor 480J is used to detect temperature. In some embodiments, the electronic device 400 uses the temperature detected by the temperature sensor 480J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 480J exceeds a threshold, the electronic device 400 reduces the performance of the processor located near the temperature sensor 480J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 400 heats the battery 442 to avoid abnormal shutdown of the electronic device 400 due to low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 400 performs a boost on the output voltage of the battery 442 to avoid abnormal shutdown caused by low temperature.

[0129] The touch sensor 480K is also called a "touch panel". The touch sensor 480K can be set on the display screen 494, and the touch sensor 480K and the display screen 494 form a touch screen, also called a "touch screen". The touch sensor 480K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 494. In other embodiments, the touch sensor 480K can also be set on the surface of the electronic device 400, which is different from the position of the display screen 494.

[0130] The bone conduction sensor 480M can obtain vibration signals. In some embodiments, the bone conduction sensor 480M can obtain vibration signals of vibrating bones of the human body. The bone conduction sensor 480M can also contact the human body's pulse to receive blood pressure beating signals.

[0131] In some embodiments, the bone conduction sensor 480M can also be set in the earphone to form a bone conduction earphone. The audio module 470 can parse the voice signal based on the vibration signal of the vocal bone obtained by the bone conduction sensor 480M to realize the voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 480M to realize the heart rate detection function.

[0132] The key 490 includes a power key, a volume key, etc. The key 490 may be a mechanical key or a touch key. The electronic device 400 may receive key input and generate key signal input related to user settings and function control of the electronic device 400.

[0133] Motor 491 can generate vibration prompts. Motor 491 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 494, motor 491 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0134] Indicator 492 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0135] The SIM card interface 495 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 400 by inserting it into the SIM card interface 495 or pulling it out from the SIM card interface 495. The electronic device 400 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 495 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 495 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 495 can also be compatible with different types of SIM cards. The SIM card interface 495 can also be compatible with external memory cards. The electronic device 400 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 400 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 400 and cannot be separated from the electronic device 400.

[0136] It should be noted that, if the second device is a server, the server includes a processor and a communication interface.

[0137] In the embodiments of the present application, the specific structure of the execution subject of the image feature extraction process and the image feature analysis process is not particularly limited, as long as the program that records the code of the image feature extraction process and / or the image feature analysis process in the image processing method of the embodiments of the present application can be run to communicate according to the image feature extraction process and / or the image feature analysis process in the image processing method of the embodiments of the present application. For example, the execution subject of an image processing method provided in the embodiments of the present application may be a functional module in the first device that can call and execute the program, or a device applied in the first device, such as a chip; the execution subject of an image processing method provided in the embodiments of the present application may be a functional module in the second device that can call and execute the program, or a device applied in the second device, such as a chip.

[0138] In the above application scenarios and corresponding examples, multiple camera devices (each camera device loads a feature extraction network model) corresponding to a cloud platform complete an image processing task. In order to have a clearer understanding of the present application, the subsequent embodiments are described as an example of one camera device among multiple camera devices and a corresponding cloud platform completing an image processing task. The camera device can load one feature extraction network model or multiple feature extraction network models.

[0139] See also Figure 5 , Figure 5An example diagram of a first device and a second device executing an image processing method provided in an embodiment of the present application, in which the first device loads a feature analysis network model, the first device extracts feature information of an image to be processed through the feature analysis network model, the first device generates identification information of the feature information, the first device sends the feature information and the identification information to the second device, the second device loads an image classification network model, a target detection network model, and a semantic segmentation network model, the second device identifies the identification information of the received feature information, and selects a feature analysis network model corresponding to the identification information based on the identification information of the received feature information, and inputs the feature information into the selected feature analysis network model, thereby completing the corresponding image processing task.

[0140] See also Figure 6 , Figure 6 A flowchart of an image processing method provided in an embodiment of the present application is shown in the figure. The method is applied to a first device and includes:

[0141] Step 601: The first device extracts feature information of an image to be processed by using at least one pre-stored feature extraction network model.

[0142] In the embodiment of the present application, the feature extraction network model may include: VGG model, ResNet model, Inception model, etc. It may also be other feature extraction network models other than the models listed above, and the embodiment of the present application does not limit this.

[0143] The feature information of the image includes a feature map obtained after the feature extraction network model in the first device processes the image to be processed.

[0144] Step 602: The first device identifies the extracted feature information to obtain identification information of the feature information.

[0145] In an embodiment of the present application, the feature extraction network model in the first device is used to extract feature information of the image, and the feature analysis network model in the second device is used to perform corresponding image processing tasks on the image based on the feature information of the image. However, there may be multiple image analysis network models in the second device to complete different image processing tasks. As an example, there are A model, B model and C model in the second device, and the A model is used to obtain image classification results based on the feature information of the image, the B model is used to obtain target detection results based on the feature information of the image, and the C model is used to obtain semantic segmentation results based on the feature information of the image. After the image feature extraction network model in the first device extracts the feature information of the image, the feature information of the image can be identified according to the image processing task to be performed, so that after the second device receives the feature information of the image, it can determine whether to select the A model, the B model or the C model for subsequent image processing tasks according to the identification information.

[0146] In practical applications, the identification information of the feature information can be determined according to the image processing task to be performed. For example, the identification information of model A used in the second device to obtain image classification results based on the feature information of the image is 00, the identification information of model B used in the second device to obtain target detection results based on the feature information of the image is 01, and the identification information of model C used in the second device to obtain semantic segmentation results based on the feature information of the image is 11. In the first device, a suitable feature extraction network model can be selected according to the image processing task to be performed on the image to be processed. Then, the extracted feature information is identified according to the image processing task to be performed. Of course, in practical applications, feature information can also be extracted and identified in other ways. Please refer to the subsequent Figures 7 to 9 Related description in .

[0147] When the image processing task for the processed image is image classification, the identification information of the feature information can be set to 00. When the image processing task for the processed image is target detection, the identification information of the feature information can be set to 01. When the image processing task for the processed image is semantic segmentation, the identification information of the feature information can be set to 11.

[0148] In practical applications, identification information of feature information may also be generated based on other information, and details may be referred to in subsequent descriptions.

[0149] It should be noted that the above example uses "0" and "1" as identification characters to form identification information. In actual applications, other forms of identification characters can also be used to form identification information.

[0150] Step 603: The first device sends feature information of the image to be processed and identification information of the feature information to the second device to instruct the second device to select a feature analysis network model corresponding to the identification information to process the feature information.

[0151] In the embodiment of the present application, when the first device and the second device are not the same device, the first device sends the feature information of the image to be processed and the identification information of the feature information to the second device. When the first device and the second device are the same device, the following situations may exist:

[0152] Case 1: The feature extraction network model is located in the first processor, and the feature analysis network model is located in the second processor.

[0153] The first device sending the feature information of the image to be processed and the identification information of the feature information to the second device includes: the first processor of the first device sending the feature information of the image to be processed and the identification information of the feature information to the second processor of the first device.

[0154] Case 2: The feature extraction network model and the feature analysis network model are located in the same processor.

[0155] The first device sends feature information and identification information of the image to be processed to the second device, including: the feature extraction function module of the first device sends the feature information and identification information of the image to be processed to the feature analysis function module of the first device, wherein the feature extraction function module stores the feature extraction network model, and the feature analysis function module stores the feature analysis network model.

[0156] In the embodiment of the present application, the first device generates identification information for the feature information of the image to be processed to instruct the second device to select the corresponding feature analysis network model according to the identification information to complete the corresponding image processing task after receiving the identification information of the feature information, so that the feature information obtained by the feature analysis network model in the second device is the matching feature information, thereby improving the problem of poor multi-task image processing effect in the second device.

[0157] As another embodiment of the present application, the first device may obtain identification information of the feature information in the following manner:

[0158] The first device obtains an identifier of a feature extraction network model for extracting feature information;

[0159] The first device uses the identifier of the feature extraction network model that extracts the feature information as the identifier information of the feature information.

[0160] In an embodiment of the present application, at least one feature extraction network model is stored in the first device, an identifier is pre-set for each feature extraction network model, and identification information of the feature information can be generated based on the identifier of the feature extraction network model that obtains the feature information. Since a second device may have a network connection with multiple first devices, that is, the second device may receive feature information sent by multiple first devices. Therefore, even if there is only one feature extraction network model in the first device, a unique identifier needs to be set for the feature extraction network model.

[0161] See also Figure 7As an example of this embodiment, if the image processing task in the second device corresponds to three models, model A, model B and model C, wherein the image processing tasks corresponding to models A, B and C have different requirements for feature information. Then, in order to meet the requirements of models A, B and C for feature information, the number of feature extraction network models in the first device can be set to 3: model 1, model 2 and model 3. Model 1 can meet the image processing task of model B, model 2 can meet the image processing task of model C, and model 3 can meet the image processing task of model A. Of course, in addition to the above-mentioned satisfaction relationship, there may be other situations. For example, the feature information obtained by model 3 can meet the requirements of model B in addition to the requirements of model A described above. However, the process of extracting feature information by model 3 occupies more memory than the process of extracting feature information by model 1, resulting in a waste of resources. In this case, in order to avoid the problem of large memory usage and waste of resources, the correspondence between the identification information and the feature analysis network model is set to Figure 7 The corresponding relationship shown is that the feature information extracted by model 3 is input into model A, and the feature information extracted by model 1 is input into model B.

[0162] When the user wants to perform image processing tasks corresponding to model A, the user can Figure 7 According to the correspondence shown, the image to be processed is input into the model 3 in the first device, and the model 3 in the first device outputs the feature information of the image to be processed and the identification information 10 of the feature information. The first device sends the feature information of the image to be processed and the identification information 10 of the feature information to the second device. After the second device receives the feature information and the corresponding identification information 10, the second device determines that model A is the target model according to the correspondence, and inputs the feature information into model A, thereby obtaining the image processing result desired by the user.

[0163] As another embodiment of the present application, the first device may obtain identification information of the feature information in the following manner:

[0164] The first device obtains an identifier of an output level of the feature information, wherein the output level of the feature information is a level of outputting the feature information in a feature extraction network model for extracting the feature information;

[0165] The first device uses the identifier of the output level of the feature information as the identifier information of the feature information.

[0166] In an embodiment of the present application, the feature extraction network model may have multiple levels. For example, there may be multiple convolutional layers, multiple pooling layers, and fully connected layers in the structure of the feature extraction network model. These levels exist in the form of convolutional layers, pooling layers, convolutional layers, pooling layers, ..., and fully connected layers. There may be the following relationship between these levels: the output of the previous level is the input of the next level, and finally the feature information of the output image of a level is obtained. However, in practical applications, a feature analysis network model not only requires the output of the last level, but also the output of one or more levels in the middle; it is possible that a feature analysis network model does not need the output of the last level, but requires the output of one or more levels in the middle. Therefore, in order to meet the needs of the feature analysis network model in the second device, a specific level of the feature extraction network model in the first device can be set as the feature information of the output level output image; and the corresponding identification information is generated according to the output level of the output feature information, so that the second device can select a suitable feature analysis network model.

[0167] For the feature extraction network model, the layers that can output the feature information of the image (such as the convolutional layer, pooling layer, and fully connected layer in the above example) can all be used as the output layers of the feature information.

[0168] The above example of the feature extraction network model is only used to illustrate the output level in the feature extraction network model. The above example of the feature extraction network model does not limit the structure of the feature extraction network model. For example, the above feature extraction network model can also be VGG, DenseNet, or a feature extraction network model with a feature pyramid structure.

[0169] See also Figure 8 As an example of this embodiment, if the image processing task in the second device corresponds to two feature analysis network models: model A and model B, the image processing task corresponding to model A has the following feature information requirements: feature information output from output level 2 to output level 5 of model 1, and the image processing task corresponding to model B has the following feature information requirements: feature information output from output level 3 to output level 5 of model 1. Then, the feature extraction network model in the first device has four output levels: output level 2, output level 3, output level 4, and output level 5. The identification information corresponding to the feature information of output level 2 to output level 5 of model 1 can be set to 0, and the identification information corresponding to the feature information of output level 3 to output level 5 of model 1 can be set to 1.

[0170] Of course, in practical applications, the first device may also include other feature extraction network models. The embodiment of the present application only uses model 1 to illustrate how to use the identifier of the output level as the identifier information of the feature information.

[0171] As another embodiment of the present application, when there are at least two feature extraction network models in the first device, and at least one of the feature extraction network models includes multiple output levels, the first device can obtain identification information of the feature information in the following manner:

[0172] The first device obtains an identifier of a feature extraction network model for extracting feature information;

[0173] The first device obtains an identifier of an output level of the feature information, wherein the output level of the feature information is a level of outputting the feature information in a feature extraction network model for extracting the feature information;

[0174] The first device uses the identifier of the feature extraction network model that extracts the feature information and the identifier of the output layer of the feature information as the identification information of the feature information.

[0175] See also Fig. 9 As an example of this embodiment, there are two feature extraction network models in the first device: model 1 and model 2. The identifier of model 1 can be 0, and the identifier of model 2 can be 1. Model 1 corresponds to 1 output level; model 2 corresponds to 4 output levels: output level 2, output level 3, output level 4 and output level 5. The identifiers of output levels 2 to 4 of model 2 are 0, and the identifiers of output levels 3 to 5 of model 2 are 1. Correspondingly, the identifier information of the feature information obtained by model 1 is 0X (which can be 00 or 01), the identifier information of the feature information obtained by output levels 2 to 4 of model 2 is 10, and the identifier information of the feature information obtained by output levels 3 to 5 of model 2 is 11. Based on the requirements of each feature analysis network model in the second device for feature information, the following corresponding relationships exist: 00—model A, 10—model B, 11—model C, or 01—model A, 10—model B, 11—model C.

[0176] When the user needs to perform an image processing task corresponding to model B on the image to be processed, the user searches for the corresponding relationship and determines the feature information that needs to be obtained from output levels 2 to 4 of model 2 in the first device. The user can control output levels 2 to 4 of model 2 to output the feature information on the first device side. The user can also send instructions to the first device through the second device on the second device side to control output levels 2 to 4 of model 2 in the first device to output the feature information. After output levels 2 to 4 of model 2 in the first device output the feature information, an identifier 1 of an extraction model of the feature information can be generated, an identifier 0 of the output level of the feature information can be generated, and 10 can be used as the identification information of the feature information. The first device sends the feature information and identification information 10 to the second device. The second device inputs the feature information into model B according to the identification information 10 to obtain the image processing result.

[0177] In practical applications, whether the identifier of the feature extraction network model that extracts feature information is used as the identification information of the feature information, or the identifier of the output level of the feature information is used as the identification information of the feature information, or both are used as the identification information of the feature information, it can be set according to actual conditions.

[0178] As another embodiment of the present application, in order to have a clearer understanding of the identification information of the feature information, the following rules may be set:

[0179] The fields corresponding to the identification information are divided into: a first field for representing a feature extraction network model for extracting feature information and a second field for representing an output level of the feature information.

[0180] The first field occupies m bytes, and the second field occupies n bytes. In practical applications, the value of m can be determined according to the number of feature extraction network models in the first device, and the value of n can be determined according to the number of output forms of the feature extraction network model.

[0181] In order to make the embodiment of the present application scalable during implementation, m and n can be set relatively large. For example, m is set to 4 bytes, which can cover up to 2 4 feature extraction network models. Setting n to 4 bytes can cover up to 2 4A feature extraction network model with an output form. The output form represents a set of output levels that output feature information. For example, a set of output levels corresponding to a feature extraction network model, {output level 1}, {output level 3}, {output level 2 to output level 4}, and {output level 3 to output level 5}, are 4 different output forms. In practical applications, m and n can also be other values, and this application does not limit this. In the case where there are more feature analysis network models on the second device side, there is sufficient identification information to form a one-to-one correspondence with the feature analysis network model.

[0182] When the first device sends identification information of feature information to the second device, the first field and the second field can be used as a whole field, and there are m consecutive bytes in the whole field to represent the feature extraction network model for extracting feature information, and there are n consecutive bytes to represent the output level of the feature information. Of course, the whole field can also contain bytes representing other meanings, and the embodiment of the present application does not limit the total number of bytes of the whole field.

[0183] The above examples are only used for illustrative purposes. In actual applications, the first field and the second field can be used as two completely independent fields. For example, the first independent field includes the first field and may also include at least one byte to distinguish whether the current independent field is an identifier of the extraction model or an identifier of the output level. The second independent field includes the second field and may also include at least one byte to distinguish whether the current independent field is an identifier of the extraction model or an identifier of the output level. The first device sends the first independent field and the second independent field as identification information of the feature information to the second device.

[0184] For example, the first byte in the first independent field is 0, indicating that the m consecutive bytes stored in the independent field are the identifier of the extraction model, and the independent field also includes m connected bytes for representing the feature extraction network model for extracting feature information. The first byte in the second independent field is 1, indicating that the n consecutive bytes stored in the independent field are the identifier of the output level, and the independent field also includes n connected bytes for representing the output level of the feature information.

[0185] The above-mentioned method for generating identification information is only used as an example. In practical applications, other generation methods may be used, and the embodiments of the present application do not limit this.

[0186] See also Fig.10 , Fig.10 A flowchart of an image processing method provided in an embodiment of the present application is shown in the figure. The method is applied to a second device. The method includes:

[0187] Step 1001: A second device obtains feature information of an image to be processed and identification information of the feature information sent by a first device connected to the second device.

[0188] Step 1002: The second device determines a feature analysis network model for processing the feature information according to identification information of the feature information.

[0189] Step 1003: The second device inputs feature information of the image to be processed into a determined feature analysis network model to obtain an image processing result.

[0190] In an embodiment of the present application, the second device needs to cooperate with the first device to complete the image processing task, the first device completes the image feature extraction task, and the second device analyzes the feature information extracted by the first device to obtain the image processing result. Therefore, there is a connection between the first device and the second device, which can be a wired connection or a wireless connection. As mentioned above, the feature information extracted by the first device contains identification information, and the identification information is used to instruct the second device to select a suitable feature analysis network model to complete the corresponding image processing task. After the second device obtains the identification information of the feature information of the image to be processed, it needs to determine the feature analysis network model for processing the feature information based on the identification information.

[0191] Corresponds to Figure 7 , 8 In the method for obtaining identification information of characteristic information in the embodiment shown in 9, the identification information of the characteristic information obtained by the second device includes:

[0192] Identification of a feature extraction network model for extracting feature information;

[0193] and / or,

[0194] An identifier of an output level of feature information, wherein the output level of feature information is a level of output feature information in a feature extraction network model that extracts feature information.

[0195] As another embodiment of the present application, the second device determines, according to the identification information of the feature information, a feature analysis network model for processing the feature information, including:

[0196] The second device obtains the correspondence between the identification information and the feature analysis network model;

[0197] According to the corresponding relationship, the second device uses the feature analysis network model corresponding to the identification information of the feature information as the feature analysis network model for processing the feature information.

[0198] It should be noted that the correspondence may include not only the correspondence between identification information and feature analysis network models, but also the correspondence between other information. Figure 3In the example shown, the described correspondence is the correspondence between the first device, the feature extraction network model stored in the first device, the identification information, the image processing task, and the feature analysis network model stored in the second device.

[0199] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0200] The embodiment of the present application can divide the functional units of the first device and the second device according to the above method example. For example, each functional unit can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation. The following is an example of dividing each functional unit corresponding to each function:

[0201] Reference Fig.11 , the first device 1110 includes:

[0202] A feature information extraction unit 1111 is used to extract feature information of the image to be processed by using at least one pre-stored feature extraction network model;

[0203] The identification information generating unit 1112 is used to identify the extracted characteristic information and obtain identification information of the characteristic information;

[0204] The information sending unit 1113 is used to send the feature information of the image to be processed and the identification information of the feature information to the second device, so as to instruct the second device to select the feature analysis network model corresponding to the identification information to process the feature information.

[0205] As another embodiment of the present application, the identification information generating unit 1112 is further configured to:

[0206] Obtaining an identifier of a feature extraction network model for extracting feature information; and using the identifier of the feature extraction network model for extracting feature information as identifier information of the feature information.

[0207] As another embodiment of the present application, the identification information generating unit 1112 is further configured to:

[0208] Obtaining an identifier of an output level of feature information, wherein the output level of feature information is a level of output feature information in a feature extraction network model for extracting feature information; and using the identifier of the output level of feature information as identification information of the feature information.

[0209] As another embodiment of the present application, the identification information generating unit 1112 is further configured to:

[0210] Obtaining an identifier of a feature extraction network model for extracting feature information;

[0211] Obtaining an identifier of an output level of feature information, wherein the output level of feature information is a level of output feature information in a feature extraction network model for extracting feature information;

[0212] The identifier of the feature extraction network model for extracting the feature information and the identifier of the output layer of the feature information are used as the identification information of the feature information.

[0213] It should be noted that the information interaction, execution process and other contents between the units in the above-mentioned first device are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0214] join Fig.11 , the second device 1120 includes:

[0215] The information acquisition unit 1121 is used to acquire feature information of the image to be processed and identification information of the feature information sent by the connected first device;

[0216] A model determination unit 1122, configured to determine a feature analysis network model for processing the feature information according to identification information of the feature information;

[0217] The image processing unit 1123 is used to input feature information of the image to be processed into a determined feature analysis network model to obtain an image processing result.

[0218] As another embodiment of the present application, the model determination unit 1122 is further configured to:

[0219] The correspondence between the identification information and the feature analysis network model is obtained; and according to the correspondence, the feature analysis network model corresponding to the identification information of the feature information is used as the feature analysis network model for processing the feature information.

[0220] As another embodiment of the present application, the identification information of the characteristic information includes:

[0221] Identification of a feature extraction network model for extracting feature information;

[0222] and / or,

[0223] An identifier of an output level of feature information, wherein the output level of feature information is a level of output feature information in a feature extraction network model that extracts feature information.

[0224] It should be noted that the information interaction, execution process, etc. between the units in the above-mentioned second device are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0225] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units is used as an example. In practical applications, the above-mentioned functions can be assigned to different functional units as needed, that is, the internal structure of the first device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0226] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0227] The embodiment of the present application also provides a computer program product. When the computer program product is executed on a first device, the first device can implement the steps in the above-mentioned method embodiments.

[0228] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0229] The embodiment of the present application also provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0230] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0231] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0232] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: The first device extracts feature information of the image to be processed by using at least one pre-stored feature extraction network model; The first device identifies the extracted feature information to obtain identification information of the feature information, wherein the identification information of the feature information includes: an identification of a feature extraction network model for extracting the feature information, and / or an identification of an output level of the feature information, wherein the output level of the feature information is a level at which the feature information is output in the feature extraction network model for extracting the feature information; The first device sends the feature information of the image to be processed and identification information of the feature information to the second device to instruct the second device to select a feature analysis network model corresponding to the identification information to process the feature information.

2. The method according to claim 1, characterized in that The identification information of the feature information includes: an identification of a feature extraction network model for extracting the feature information, the first device identifies the extracted feature information, and obtaining the identification information of the feature information includes: The first device obtains an identifier of a feature extraction network model for extracting the feature information; The first device uses the identifier of the feature extraction network model that extracts the feature information as the identifier information of the feature information.

3. The method according to claim 1, characterized in that The identification information of the feature information includes: an identification of an output level of the feature information, the first device identifies the extracted feature information, and obtaining the identification information of the feature information includes: The first device obtains an identifier of an output level of the characteristic information; The first device uses the identifier of the output level of the feature information as the identifier information of the feature information.

4. The method according to claim 1, characterized in that The identification information of the feature information includes: an identification of a feature extraction network model for extracting the feature information, and an identification of an output level of the feature information. The first device identifies the extracted feature information, and obtaining the identification information of the feature information includes: The first device obtains an identifier of a feature extraction network model for extracting the feature information; The first device obtains an identifier of an output level of the characteristic information; The first device uses an identifier of a feature extraction network model that extracts the feature information and an identifier of an output layer of the feature information as identification information of the feature information.

5. An image processing method, characterized in that: include: The second device obtains feature information of the image to be processed sent by the first device connected to the second device and identification information of the feature information, wherein the identification information of the feature information includes: an identification of a feature extraction network model for extracting the feature information; and / or an identification of an output level of the feature information, wherein the output level of the feature information is a level at which the feature information is output in the feature extraction network model for extracting the feature information; The second device determines, according to the identification information of the feature information, a feature analysis network model for processing the feature information; The second device inputs the feature information of the image to be processed into a determined feature analysis network model to obtain an image processing result.

6. The method according to claim 5, characterized in that The second device determines, according to the identification information of the feature information, a feature analysis network model for processing the feature information, including: The second device obtains a correspondence between the identification information and the feature analysis network model; The second device uses, according to the corresponding relationship, the feature analysis network model corresponding to the identification information of the feature information as the feature analysis network model for processing the feature information.

7. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to run a computer program stored in a memory to implement the method according to any one of claims 1 to 4.

8. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to run a computer program stored in a memory to implement the method according to claim 5 or 6.

9. An image processing system, characterized in that: include: At least one electronic device as claimed in claim 7 and at least one electronic device as claimed in claim 8.

10. A chip system, characterized in that: The chip system includes a processor coupled to a memory, and the processor is used to run a computer program stored in the memory to implement the method according to any one of claims 1 to 4 and / or the method according to claim 5 or 6.

Images