Image processing method and electronic device

Abstract

Embodiments of the present application provide an image processing method and an electronic device, and relate to the technical field of terminals. The method comprises: obtaining an original image and inputting the original image into a first model, wherein the first model comprises a first encoder and a plurality of convolution adapters, and the plurality of convolution adapters respectively correspond to respective image processing algorithms. In the first model, the original image is input into the first encoder to obtain first image features of the original image; the first image features are input into the plurality of convolution adapters respectively to obtain second image features output by each convolution adapter; and each second image feature is decoded to obtain a plurality of target images output by the first model. In this way, the convolution adapters corresponding to the respective image processing algorithms can be set, and the image features extracted by the first encoder can be shared, so as to reduce power consumption.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method and an electronic device. Background Technology

[0002] With the continuous development of electronic device technology, most electronic devices now have the function of taking pictures. In order to improve the imaging effect of the images taken by electronic devices, one or more image processing algorithms can be added to the picture processing process of electronic devices.

[0003] Currently, each image processing algorithm employs its own independent algorithmic workflow framework. These frameworks extract image features from the same image separately, and then apply different processing algorithms based on those extracted features. This results in high power consumption during the image processing workflow. Summary of the Invention

[0004] This application provides an image processing method and an electronic device, applied in the field of terminal technology, to reduce power consumption caused by image processing in the photo-taking process.

[0005] Firstly, embodiments of this application propose an image processing method. The method includes:

[0006] The original image is acquired and input into the first model, which includes a first encoder and multiple convolution adapters, each corresponding to a different image processing algorithm.

[0007] In the first model, the original image is input to the first encoder to obtain the first image features of the original image; and,

[0008] The first image features are input into multiple convolutional adapters to obtain the second image features output by each convolutional adapter; and,

[0009] Each second image feature is decoded separately to obtain multiple target images output by the first model. These multiple target images are obtained by processing the original image with multiple image processing algorithms.

[0010] By setting up convolution adapters for each image processing algorithm and sharing the image features extracted by the first encoder, the high power consumption caused by each image processing algorithm extracting features from the original image can be avoided, thereby reducing the power consumption caused by image processing algorithms in the image processing process.

[0011] In one possible implementation, the first encoder includes N encoding units, where N is an integer greater than or equal to 1;

[0012] In the first model, the original image is input into the first encoder to obtain the first image features of the original image:

[0013] In the first model, the original image is input into the first coding unit in the first encoder to obtain the first intermediate feature output by the first coding unit;

[0014] The first intermediate feature output by the i-th coding unit is input into the (i+1)-th coding unit in the first encoder to obtain the first intermediate feature output by the (i+1)-th coding unit. The value of i ranges from 1 to N-1.

[0015] In the case where i takes the value of N-1, the first intermediate feature output by the Nth coding unit is the first image feature.

[0016] In this way, during the process of the first encoder extracting image features from the original image, the feature extraction process is carried out sequentially by each coding unit, and each coding unit performs further feature extraction based on the output of the previous coding unit. This can improve the comprehensiveness and richness of the finally extracted image features.

[0017] In one possible implementation, for any one of a plurality of convolution adapters, the convolution adapter includes M processing units, where M is an integer greater than or equal to N;

[0018] The first image features are input into multiple convolutional adapters to obtain the second image features output by each convolutional adapter, including:

[0019] In the first model, the first intermediate feature output by the j-th coding unit and / or the second intermediate feature output by the (j-1)-th processing unit are input to the j-th processing unit in the convolutional adapter to obtain the second intermediate feature output by the j-th processing unit, where the value of j ranges from 1 to M.

[0020] Where j takes the value M, the second intermediate feature output by the Mth processing unit is the second image feature.

[0021] In one possible implementation, the first intermediate feature output by the j-th encoding unit and / or the second intermediate feature output by the (j-1)-th processing unit are input to the j-th processing unit in the convolutional adapter to obtain the second intermediate feature output by the j-th processing unit, including:

[0022] When j is 1, the first intermediate feature output by the j-th coding unit is input into the j-th processing unit in the convolutional adapter to obtain the second intermediate feature output by the j-th processing unit.

[0023] When the value of j is in the range of 2 to N, the first intermediate feature output by the j-th coding unit and the second intermediate feature output by the (j-1)-th processing unit are input into the j-th processing unit in the convolution adapter to obtain the second intermediate feature output by the j-th processing unit.

[0024] When the value of j is greater than N and less than or equal to M, the second intermediate feature output by the (j-1)th processing unit is input into the jth processing unit in the convolution adapter to obtain the second intermediate feature output by the jth processing unit.

[0025] In this way, during the process of the convolutional adapter adapting the modeling processing of the first image features to the corresponding algorithm, on the one hand, it can perform modeling processing based on the intermediate features output by the coding unit of the corresponding level, and on the other hand, it can perform modeling processing based on the results output by the previous processing unit. This allows the convolutional adapter to simultaneously fuse image features from different levels and the feature modeling results processed by the previous processing unit during the feature modeling process, thereby improving the comprehensiveness and accuracy of the image features output by the convolutional adapter.

[0026] In one possible implementation, for any one of the M processing units, the processing unit includes a plurality of first convolutional units and a first fusion unit, the plurality of first convolutional units correspond to different convolutional kernels, and the first convolutional units are used to perform convolutional processing.

[0027] The first intermediate feature output by the j-th coding unit and / or the second intermediate feature output by the (j-1)-th processing unit are input into the j-th processing unit in the convolutional adapter to obtain the second intermediate feature output by the j-th processing unit, including:

[0028] The first intermediate feature output by the j-th coding unit and / or the second intermediate feature output by the (j-1)-th processing unit are input into the j-th processing unit in the convolutional adapter.

[0029] In the j-th processing unit, the first intermediate feature and / or the second intermediate feature are respectively input into multiple first convolutional units to obtain the first feature vectors output by the multiple first convolutional units respectively; wherein, the processing of the multiple first convolutional units is in parallel;

[0030] The first feature vectors output by multiple first convolutional units are input into the first fusion unit so that the first fusion unit outputs the second intermediate feature.

[0031] In this way, by using multiple first convolutional units with different kernels to perform convolutional processing on the first intermediate features in parallel across different dimensions, the efficiency of convolutional processing can be effectively improved and its richness expanded. On the other hand, by fusing the various first feature vectors, the final output second intermediate feature is a result of fusing the processing of each convolutional unit, so that the different dimensions of processing of each convolutional unit can be reflected in the output second intermediate feature.

[0032] In one possible implementation, the first fusion unit includes a first fusion layer, a first weight layer, and a first weighting layer;

[0033] In the first fusion unit, the first fusion layer is used to fuse the first feature vectors output by multiple first convolutional units to obtain a first fusion vector, which includes multiple channels.

[0034] The first weighting layer is used to determine the weights of each of the multiple channels in the first fusion vector.

[0035] The first weighting layer is used to weight the multiple channels of the first fusion vector based on their respective weights, so as to output the second intermediate feature.

[0036] This allows for the retention of more important features and the influence of more vital signs during the modeling of the second image features, thereby improving the accuracy of the final modeled second image features. This accuracy is reflected in the ability to model the intermediate features required for algorithm processing.

[0037] In one possible implementation, each first convolutional unit is also used to downsample the first intermediate feature and / or the second intermediate feature.

[0038] In one possible implementation, the first model also includes decoding units corresponding to the multiple convolutional adapters respectively;

[0039] Each second image feature is decoded separately to obtain multiple target images output by the first model, including:

[0040] For any one of the multiple convolution adapters, the second image feature output by the convolution adapter is input to the corresponding decoding unit to obtain the decoded image feature output by the corresponding decoding unit;

[0041] Multiple target images are generated and output based on the decoded image features output by multiple decoding units.

[0042] In one possible implementation, for any one of the multiple decoding units, the decoding unit includes multiple second convolution units, a second fusion unit, and an upsampling unit. The multiple second convolution units correspond to different convolution kernels, and the second convolution units are used to perform convolution processing.

[0043] The second image feature output from the convolution adapter is input to the corresponding decoding unit to obtain the decoded image feature output by the corresponding decoding unit, including:

[0044] The second image feature output by the convolution adapter is input into the corresponding decoding unit;

[0045] In the decoding unit, the second image features are input into multiple second convolutional units to obtain the second feature vectors output by the multiple second convolutional units respectively; wherein, the processing of the multiple second convolutional units is in parallel;

[0046] The second feature vectors output by multiple second convolutional units are input into the second fusion unit to obtain the fused feature vector output by the second fusion unit.

[0047] The fused feature vector is input into the upsampling unit so that the upsampling unit outputs the decoded image features.

[0048] In this implementation, multiple second convolutional units, each with different kernels, perform parallel convolutional processing on the second image features across different dimensions. This effectively improves the efficiency of convolutional processing and expands its richness. Furthermore, the various second feature vectors are fused to obtain a fused feature vector that combines the results of each convolutional unit. Subsequent upsampling of this fused feature vector restores the feature dimensions, resulting in the decoded feature vector.

[0049] In one possible implementation, the second fusion unit includes a second fusion layer, a second weighting layer, and a second weighting layer;

[0050] In the second fusion unit, the second fusion layer is used to fuse the second feature vectors output by multiple second convolution units to obtain a second fusion vector, which includes multiple channels.

[0051] The second weighting layer is used to determine the weights for each of the multiple channels in the second fusion vector.

[0052] The second weighting layer is used to weight the multiple channels of the second fusion vector based on their respective weights to obtain the fused feature vector.

[0053] In this way, during the decoding process of feature vectors, more important features and the degree of influence of more important features can be retained, thereby improving the accuracy of the processing results reflected by the final decoded feature vectors. This accuracy is reflected in the ability to reflect the result features required by the algorithm.

[0054] In one possible implementation, the method also includes:

[0055] The first model is trained based on the training data to obtain the trained first model.

[0056] The training data includes training images and corresponding labeled images.

[0057] In one possible implementation, the training process includes:

[0058] The training image is input into the first model to obtain the predicted image output by the first model;

[0059] The loss value is determined based on the labeled image corresponding to the predicted image and the training image;

[0060] Based on the loss value, the model parameters of multiple convolutional adapters in the first model, as well as the model parameters of the decoding unit used to perform the decoding process, are adjusted.

[0061] In this way, by setting the parameters of the first encoder to be fixed after pre-training is completed and not participating in the parameter adjustment during the subsequent model training process, only the parameters of the convolution adapter and the decoding unit are adjusted, so that when the first model is applied for image processing, the convolution adapters of each downstream algorithm can extract features through the first encoder at one time, and the features output by each convolution adapter will not flow back to the first encoder, so as to ensure that the feature extraction effect of the first encoder is not affected.

[0062] Secondly, embodiments of this application provide an image processing apparatus, which may be an electronic device, or a chip or chip system within an electronic device. The image processing apparatus may include a display unit and a processing unit.

[0063] When the image processing apparatus is an electronic device, the display unit therein can be a display screen. The display unit is used to perform the display step so that the electronic device implements an image processing method described in the first aspect or any possible implementation of the first aspect.

[0064] When the image processing apparatus is an electronic device, the processing unit may be a processor. The image processing apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation thereof.

[0065] When the image processing device is a chip or chip system within an electronic device, the processing unit can be a processor. The processing unit executes instructions stored in a storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0066] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory for storing code instructions, and the processor for running the code instructions to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0067] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0068] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0069] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0070] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0071] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0072] Figure 1 A flowchart illustrating the image processing procedure provided in the embodiments of this application. Figure 1 ;

[0073] Figure 2 A flowchart illustrating the image processing procedure provided in the embodiments of this application. Figure 2 ;

[0074] Figure 3 A flowchart illustrating the image processing procedure provided in the embodiments of this application. Figure 3 ;

[0075] Figure 4 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application;

[0076] Figure 5 This is a schematic diagram of the software structure of a terminal device provided in an embodiment of this application;

[0077] Figure 6 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 ;

[0078] Figure 7 A schematic diagram of the structure of the first model provided in the embodiments of this application;

[0079] Figure 8 This is a schematic diagram of the structure of the processing unit provided in the embodiments of this application;

[0080] Figure 9 This is a schematic diagram of the structure of the decoding unit provided in an embodiment of this application;

[0081] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0082] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0083] 1. Target Detection

[0084] Object detection is a technique in computer vision that aims to automatically identify and accurately locate target objects from images or videos. Its core objective is to simultaneously accomplish two sub-tasks: classification, which determines the category of an object in an image; and localization, which determines the object's position in the image, typically by regressing to predict the coordinates of its bounding box. Object detection not only requires recognizing different objects in an image but also considering factors such as the object's scale, shape, and spatial location.

[0085] 2. Semantic segmentation

[0086] Semantic segmentation is a technique in computer vision that aims to assign a semantic label to each pixel in an image to indicate its category. This means that an image is segmented into multiple semantic regions, each with the same semantic label, such as classifying each pixel in an image as "sky," "trees," or "buildings."

[0087] 3. ResNet

[0088] ResNet (Residual Network) is a deep neural network architecture. Its core idea is to solve the gradient vanishing and gradient exploding problems in deep network training through residual connections, so that the network can be trained to be deeper and have stronger performance.

[0089] 4. InceptionNet

[0090] InceptionNet is a deep convolutional neural network architecture that uses multiple convolutional kernels of different sizes to process input data in parallel, and then concatenates these feature maps to obtain a richer feature representation. This approach increases both the width of the network and the diversity of features, enabling the model to better handle complex visual tasks.

[0091] 5. MobileNet

[0092] MobileNet is a lightweight convolutional neural network (CNN) architecture specifically designed to run on mobile and embedded devices. Its goal is to improve computational efficiency while maintaining relatively high model accuracy, making it suitable for resource-constrained environments (such as those with low computational power and memory limitations).

[0093] 6. Other terms

[0094] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0095] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0096] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0097] 5. Electronic equipment

[0098] The electronic devices in this application embodiment may include handheld devices, vehicle-mounted devices, etc., with **functions. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, wearable devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0099] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0100] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0101] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0102] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0103] To better understand the technical solution of this application, the relevant technologies involved in this application will be further described in detail below.

[0104] With the continuous development of electronic device technology, most electronic devices now have a camera function. To improve the imaging effect of images captured by electronic devices, one or more image processing algorithms can be added to the camera's image processing flow. These image processing algorithms can include, for example, object detection and semantic segmentation algorithms. Object detection algorithms can be further subdivided into text detection, portrait detection, and other algorithms, while semantic segmentation algorithms can be further subdivided into portrait segmentation, multi-class semantic segmentation, and other algorithms.

[0105] By performing object detection on the captured image during the image processing workflow, and then applying specific image processing to the detected objects, the final image quality can be improved. Similarly, performing semantic segmentation on the captured image, and then applying different image processing to different semantic regions of the image, can also improve the final image quality.

[0106] In actual implementation, the specific image processing algorithms included in the electronic device's photo processing flow, and the specific methods used to implement these image processing algorithms, can be selected according to actual needs, and this application does not limit them in this regard.

[0107] In current implementations, the image processing algorithms included in the photo-taking process of electronic devices typically use their own independent algorithm frameworks. The following section combines... Figure 1 and Figure 2 This implementation will be introduced. Figure 1 A flowchart illustrating the image processing procedure provided in the embodiments of this application. Figure 1 , Figure 2 A flowchart illustrating the image processing procedure provided in the embodiments of this application. Figure 2 .

[0108] like Figure 1 As shown, when an electronic device detects a user's input to take a picture, it can, for example, respond to the shooting operation by acquiring an image to obtain... Figure 1 Image 101 is shown. After obtaining image 101, it can be processed according to the relevant algorithms included in the image processing flow, thereby outputting the processed image corresponding to each algorithm.

[0109] Reference Figure 1 For example, image 101 can be input into the algorithm units of Algorithm A, Algorithm B, and Algorithm C respectively, so that image processing is performed on image 1 according to Algorithm A, Algorithm B, and Algorithm C respectively. Then, the algorithm unit of Algorithm A can output processing result 1 for image 1, the algorithm unit of Algorithm B can output processing result 2 for image 1, and the algorithm unit of Algorithm C can output processing result 3 for image 1.

[0110] For example, the processing result can be in the form of an image, meaning that the processing result of algorithm A is already integrated into the image output by the algorithm. For instance, the image output by the algorithm may contain semantic segmentation results or object detection results, etc. Alternatively, the processing result can also be in the form of information, where the information output by the algorithm can indicate the processing result of the algorithm for image 1. For example, the information output by the algorithm may indicate the semantic segmentation result of image 1 or the object detection result of image 1, etc. This embodiment does not limit the specific form of the processing result.

[0111] Then, based on the processing result 1 corresponding to Algorithm A, image 102 can be generated, where image 102 is the image containing the processing effect of Algorithm A. Similarly, based on the processing result 2 corresponding to Algorithm B, image 103 can be generated, where image 103 is the image containing the processing effect of Algorithm B. Algorithm C is similar.

[0112] The above combination Figure 1 In the implementation described, algorithms A, B, and C operate in parallel, meaning they can process image 1 simultaneously and output their respective results. Referring to the illustration, it's also clear that the input to these algorithms is the original acquired image 1. In another implementation, the multiple algorithms within the image processing flow can also exist sequentially, meaning the input to a downstream algorithm depends on the output of an upstream algorithm. This can be further illustrated below. Figure 2 To understand.

[0113] like Figure 2 As shown, for example, an electronic device acquires an image in response to a shooting operation, thereby obtaining... Figure 2 The image 201 is shown below. Then, image 201 is processed using algorithms A, B, and C, but the difference is that in this example, the inputs to algorithms B and C depend on the output of algorithm A, so algorithm A needs to be executed before algorithms B and C.

[0114] Reference Figure 2 For example, image 101 can be input into the algorithm unit of algorithm A to perform processing of image 1 by algorithm A. Then, the algorithm unit of algorithm A can output the processing result 1 after processing image 1. Similar to the above description, the processing result 1 can be in the form of an image or in the form of information.

[0115] When the processing result 1 is in image form, it can be used as input to the algorithm units of Algorithm B and Algorithm C, enabling Algorithm B and Algorithm C to process the image accordingly. Alternatively, when the processing result 1 is in information form, the initially acquired image 1 and the processing result 1 can be input to the algorithm units of Algorithm B and Algorithm C respectively. Alternatively, the intermediate image resulting from the fusion of the initially acquired image 1 and the processing result 1 can be input to the algorithm units of Algorithm B and Algorithm C, allowing Algorithm B and Algorithm C to process the image accordingly.

[0116] Similar to the above description, the algorithm unit of algorithm B can output processing result 2, and the algorithm unit of algorithm C can output processing result 3. Subsequently, based on processing result 2 and processing result 3, images 202 and 203 can be obtained respectively. Image 202 is the image containing the processing effect of algorithm B, and image 203 is the image containing the processing effect of algorithm C.

[0117] In the above Figure 1 and Figure 2 Based on the introduction, it can be understood that the multiple image processing algorithms included in the photo processing flow can be in a serial relationship or in a parallel relationship. The specific algorithm relationship can be set according to actual needs, and this embodiment does not limit it.

[0118] It is understandable that in the image processing workflow, the input data for multiple parallel algorithms is consistent. For example, it could be the initial image captured at the beginning, or an intermediate image that incorporates the processing results of upstream algorithms. Furthermore, the internal data processing procedures of most image processing algorithms in the image processing workflow are similar. Typically, features are extracted from the input image, and then corresponding algorithmic processing is performed based on the extracted features to output the processing result.

[0119] You can refer to Figure 3 To understand, Figure 3 A flowchart illustrating the image processing procedure provided in the embodiments of this application. Figure 3 .

[0120] like Figure 3 As shown, algorithms A, B, and C are parallel processes, meaning that the input to all three algorithms is image 101 as shown in the figure. For example, within the algorithm unit of algorithm A, feature extraction can be performed on the input image 101 to obtain image feature a1. Algorithm A then performs corresponding data analysis and processing based on the extracted image feature a1 to obtain the processing result 1 output by algorithm A.

[0121] Similarly, within the algorithm unit of Algorithm B, image features b1 can be extracted from the input image 101. Then, Algorithm A performs corresponding data processing on image features b1 to obtain the processing result 2 output by Algorithm B. Likewise, within the algorithm unit of Algorithm C, image features c1 can be extracted from the input image 101. Then, Algorithm C performs corresponding data processing on image features c1 to obtain the processing result 3 output by Algorithm C.

[0122] The subsequent processing methods for processing results 1, 2, and 3 are the same as described above. Figure 1 The descriptions are similar, so I won't repeat them here.

[0123] Reference Figure 3 As the introduction suggests, multiple parallel algorithm units each employ an independent algorithmic workflow framework. Within these frameworks, each unit extracts image features from the same image and then applies different processing algorithms based on those features. It's understandable that the performance and power consumption of each algorithm are independent. However, the total power consumption in the image processing stage is the sum of the power consumption of each algorithm. Therefore, if each algorithm unit performs similar feature extraction on the same image, the resulting total power consumption after the image processing workflow is completed will be high.

[0124] To address the technical problems described above, this application proposes an image processing method. In this method, a shared base encoder is set up for multiple parallel algorithm units. The image feature extraction work required by multiple algorithm units is all performed by this base encoder. Then, the image features output by the base encoder are provided to the respective subsequent processing units of each algorithm, so that these subsequent processing units output the processing results corresponding to the algorithm based on the image features. This saves the overhead caused by each algorithm unit needing to perform image feature extraction, effectively reducing the power consumption of image processing.

[0125] The image processing method provided in this application will be described below with reference to specific embodiments. The image processing method of this application can be executed by an electronic device equipped with a processor, or by a chip, chip system, or processor that supports the implementation of the image processing method by the electronic device. It can also be executed by a logic module or software capable of implementing all or part of the functions of the electronic device; this application does not impose specific limitations in this regard. The image processing method of this application will be described in detail below using an electronic device as the execution subject as an example.

[0126] Electronic devices can be, for example, terminal devices. The following section will first combine... Figure 4 and Figure 5A brief introduction to the terminal equipment.

[0127] For example, Figure 4 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application.

[0128] Figure 4 This is a schematic diagram of the terminal device provided in an embodiment of this application. The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0129] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0130] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors. In one implementation, the relevant data processing in the image processing method described in this application may be performed by processor 110, for example.

[0131] Furthermore, the terminal device can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194, and application processor.

[0132] The software system of a terminal device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This application uses the layered architecture Android system as an example to illustrate the software structure of the terminal device.

[0133] For example, Figure 5 This is a schematic diagram of the software structure of a terminal device provided in an embodiment of this application.

[0134] like Figure 5 As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android operating system can be divided into five layers, from top to bottom: applications, application framework, hardware abstraction layer (HAL), kernel, and hardware layer.

[0135] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the application framework layer. For example... Figure 5 As shown, the application package can include applications such as camera and gallery. Applications can include system applications and third-party applications.

[0136] The application framework layer provides APIs and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 5 As shown, the application framework layer can include a camera access interface and a view system. The camera access interface provides the application programming interface and framework for the camera application. The view system is responsible for rendering the application's interface and handling events. The view system can include visual controls, such as various controls for taking photos.

[0137] like Figure 5As shown, the HAL layer may include a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer can provide virtual hardware for the camera device. The camera algorithm library may include runtime code and data implementing the image processing methods provided in the embodiments of this application. For example, the image processing model of the embodiments of this application can be deployed in the camera algorithm library to perform a series of image algorithm processing on the acquired images to obtain the final output image.

[0138] The kernel layer is the layer between hardware and software. For example... Figure 5 As shown, this kernel layer may include one or more of the following: camera device driver, digital signal processor driver, and graphics processor driver. The camera device driver is used to drive the camera's sensor to acquire images. The digital signal processor driver is used to drive the digital signal processor to process images. The graphics processor driver is used to drive the graphics processor to process images.

[0139] The hardware layer may include hardware such as cameras, displays, digital signal processors, and graphics signal processors.

[0140] It should be understood that in some embodiments, layers that perform the same function may be called by other names, or layers that can perform the functions of multiple layers may be considered as one layer, or layers that can perform the functions of multiple layers may be divided into multiple layers. This application does not impose such limitations.

[0141] It should be noted that the embodiments of this application are only illustrated using the Android operating system as an example. In other operating systems (such as Windows operating system, iOS operating system, etc.), as long as the functions implemented by each functional module are similar to those in the embodiments of this application, the solution of this application can also be implemented.

[0142] The technical solutions of the embodiments of this application and how the technical solutions of the embodiments of this application solve the above-mentioned technical problems will be described in detail below with reference to the accompanying drawings and specific examples. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0143] Figure 6 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 .

[0144] like Figure 6 As shown, the method includes:

[0145] S601. Input the original image into the first model.

[0146] The original image can be, for example, an image initially captured by a camera, or an intermediate image output after processing by one or more algorithms. This embodiment does not limit the specific implementation of the original image, but it can be understood that the original image is the common input image corresponding to multiple parallel image processing algorithms.

[0147] In this embodiment, the first model is a model for implementing multiple image processing algorithms, and these multiple image processing algorithms are the parallel image processing algorithms described above. For example, regarding the above... Figure 1 The examples introduced can be understood as follows: the first model is used to implement image processing algorithms A, B, and C. And regarding the above... Figure 2 The example provided can be understood as follows: the first model is used to implement the image processing algorithms B and C. Figure 2 In the example shown, algorithm A can correspond to the other processing models, for example.

[0148] In actual implementation, the specific image processing algorithms covered by the first model, and the corresponding models for each algorithm, can be set according to the actual algorithm partitioning requirements. This application does not impose any restrictions on this. The following is an introduction using the first model to implement parallel algorithms A, B, and C as examples.

[0149] S602. In the first model, the original image is input to the first encoder to obtain the first image features of the original image.

[0150] In one implementation, the first model may include a first encoder, which is used to extract features from an image.

[0151] Reference Figure 6 After the original image is input into the first model, within the first model, the original image can first be input into the first encoder, so that the first encoder outputs the first image features extracted from the original image. For example, the first encoder can be any pre-trained image feature extractor, such as ResNet, InceptionNet, MobileNet, etc. The choice of model can be determined based on the accuracy, performance, and power consumption of the specific task. The pre-training method can be generative pre-training or supervised pre-training, etc. This application does not limit the specific choice of the first encoder, as long as the first encoder can perform feature extraction on the image.

[0152] S603. Input the first image features into multiple convolution adapters respectively to obtain the second image features output by each convolution adapter.

[0153] The first model also includes multiple convolutional adapters, each corresponding to a different image processing algorithm. These adapters are used to align the first image features with the features required by the corresponding algorithm task, thereby outputting intermediate image features to provide semantic information that better meets the algorithm's processing requirements for downstream processing units.

[0154] Reference Figure 6 Within the first model, after the first encoder outputs the first image features, these features can be further input into convolution adapter 1 corresponding to algorithm A, convolution adapter 2 corresponding to algorithm B, and convolution adapter 3 corresponding to algorithm C. These convolution adapters are used to model the semantic information of their respective image processing algorithms, thereby outputting their respective second image features. These second image features reflect the semantic information required by each image processing algorithm.

[0155] For example in Figure 6 In the example, convolution adapter 1 processes the first image feature and outputs a second image feature A1, which reflects the semantic information required by algorithm A. Similarly, convolution adapter 2 processes the first image feature and outputs a second image feature B1, which reflects the semantic information required by algorithm B. Convolution adapter 3 works similarly, outputting a second image feature C1 that reflects the semantic information required by algorithm C.

[0156] S604. Input the second image features output by the convolution adapter into the corresponding decoding unit to obtain the decoded image features output by the corresponding decoding unit.

[0157] Furthermore, the first model also includes decoding units corresponding to multiple convolution adapters. The decoding units are used to decode the image features output by the intermediate convolution adapters, thereby outputting the final decoded image features for inferring the final image processing result.

[0158] Reference Figure 6 For example, the second image feature A1 output by convolution adapter 1 can be input to decoding unit 1 corresponding to algorithm A, so that decoding unit 1 outputs decoded image feature A2. Similarly, the second image feature B1 output by convolution adapter 2 can be input to decoding unit 2 corresponding to algorithm B, so that decoding unit 2 outputs decoded image feature B2. Finally, the second image feature C1 output by convolution adapter 3 can be input to decoding unit 3 corresponding to algorithm C, so that decoding unit 3 outputs decoded image feature C2.

[0159] It is understood that the decoding unit is used to decode image features. The decoding process may include convolution and upsampling to restore the image features. This application does not limit the specific implementation of the decoding unit, which can be selected according to actual needs.

[0160] S605. Generate and output the target image based on the decoded image features output by multiple decoding units respectively.

[0161] After obtaining the decoded image features corresponding to each image processing algorithm, multiple target images can be generated based on these features. In one implementation, for example, image reconstruction can be performed separately based on the multiple decoded image features to obtain the target images corresponding to each image processing algorithm. It can be understood that the target images are obtained by the image processing algorithms after processing the original image.

[0162] Can be combined Figure 6 To understand, such as Figure 6 As shown, for example, based on the decoded image features A2 output by the decoding unit 1 corresponding to algorithm A, a target image 1 can be output, where the target image 1 is the image obtained after the original image has been processed by algorithm A. For example, if algorithm A is a semantic segmentation algorithm, then the target image 1 contains the semantic segmentation processing result.

[0163] Similarly, based on the decoded image feature B2 output by decoding unit 2 corresponding to algorithm B, target image 2 can be output, where target image 2 is the image obtained after processing the original image by algorithm B. Based on the decoded image feature C2 output by decoding unit 3 corresponding to algorithm C, target image 3 can be output, where target image 3 is the image obtained after processing the original image by algorithm C. In actual implementation, the specific implementations of algorithms A, B, and C can be set according to actual needs.

[0164] In this embodiment, multiple image processing algorithms share a unified basic encoder, namely the first encoder, for extracting image features. Simultaneously, multiple independent convolutional adapters are used to predict results for different downstream algorithms. For example, as described above, each image processing algorithm has its own dedicated adapter and decoding unit based on a convolutional network to model the semantic information required for the tasks of different image processing algorithms and infer the final processing result. This effectively reduces the performance and power consumption of image processing algorithms during the image capture process.

[0165] Based on the above embodiments, the internal structure of the first encoder and the convolution adapter will be described in further detail below. Figure 7A schematic diagram of the structure of the first model provided in the embodiments of this application.

[0166] like Figure 7 As shown, the first encoder 700 may include multiple encoding units. Figure 7 The example specifically includes four coding units, namely coding unit 1, coding unit 2, coding unit 3, and coding unit 4 as shown in the figure. The coding units are used to perform image feature extraction operations.

[0167] In one implementation, during the process of the first encoder extracting image features from the original image, the feature extraction process can be carried out sequentially through each coding unit, and each coding unit performs further feature extraction based on the output of the previous coding unit. This can improve the comprehensiveness and richness of the finally extracted image features.

[0168] Reference Figure 7 For example, the original image can be input into the first encoding unit, i.e. encoding unit 1, in the first encoder 700 to obtain the first intermediate feature output by encoding unit 1.

[0169] Then, the first intermediate feature output by coding unit 1 can be input into the second coding unit, namely coding unit 2, to obtain the first intermediate feature output by coding unit 2, and so on for subsequent coding units.

[0170] In other words, the first intermediate feature output by the i-th coding unit can be input into the (i+1)-th coding unit in the first encoder to obtain the first intermediate feature output by the (i+1)-th coding unit, where i ranges from 1 to N-1. N can be understood as the number of coding units in the first encoder.

[0171] After the last encoding unit outputs the first intermediate feature, the first image feature output by the first encoder can be obtained.

[0172] exist Figure 7 In the example, the first encoder contains four encoding units. In actual implementation, the number of encoding units contained in the first encoder can be arbitrarily set according to actual needs.

[0173] And in Figure 7 The diagram also illustrates the internal structure of convolution adapter 701 corresponding to algorithm A, convolution adapter 702 corresponding to algorithm B, and convolution adapter 703 corresponding to algorithm C. It can be understood that the internal structure of each convolution adapter is similar. Therefore, the following explanation will take convolution adapter 701 as an example.

[0174] like Figure 7 As shown, the convolution adapter may include multiple processing units. Figure 7 The example specifically includes four processing units, namely processing unit 1_A, processing unit 2_A, processing unit 3_A, and processing unit 4_A as shown in the figure. The processing units in the convolution adapter 701 are used to perform feature modeling processing related to algorithm A.

[0175] In one implementation, during the modeling process of the convolution adapter 701 adapting to the algorithm A for the first image features, on the one hand, modeling can be performed based on the intermediate features output by the corresponding level of the coding unit, and on the other hand, modeling can be performed based on the results output by the previous processing unit. This allows the convolution adapter to simultaneously fuse image features from different levels and the feature modeling results processed by the previous processing unit during the feature modeling process, thereby improving the comprehensiveness and accuracy of the image features output by the convolution adapter.

[0176] Reference Figure 7 In this context, encoding unit 1 and processing unit 1_A can be considered to be at the same level, and processing unit 1_A is the first processing unit. Therefore, the first intermediate feature output by encoding unit 1 can be input into processing unit 1_A, which in turn causes processing unit 1_A to output the modeled second intermediate feature.

[0177] Furthermore, encoding unit 2 and processing unit 2_A can be considered to be at the same level. Therefore, the first intermediate feature output by encoding unit 2 and the second intermediate feature output by the previous processing unit 1_A can be input into processing unit 2_A, so that processing unit 2_A outputs the modeled second intermediate feature. The implementation of processing units 3_A and 4_A is similar and will not be described in detail here.

[0178] It is understandable that during the first model's processing of the original image, the processing of the encoding unit and the processing unit can be understood as parallel. For example, after the encoding unit 1 outputs the first intermediate feature, the first intermediate feature will be simultaneously input to the encoding unit 2 and the processing unit 1_A. At this time, the encoding unit 2 and the processing unit 1_A can perform data processing at the same time, thereby outputting their respective processing results. The subsequent processing is similar.

[0179] The number of processing units contained within the convolutional adapter can be represented, for example, as M, and the number of encoding units contained within the first encoder can be represented, for example, as N. The value of M can be equal to N, meaning the number of processing units and the number of encoding units can be the same. Alternatively, the value of M can be greater than N, meaning the number of processing units can be greater than the number of encoding units.

[0180] exist Figure 7 In the example, the case described is that the number of processing units inside the convolution adapter is the same as the number of encoding units inside the first encoder. Therefore, the input to the last processing unit can be the output of the last encoding unit and the second-to-last processing unit.

[0181] However, when M is greater than N, some processing units will not have corresponding coding units. For example, for the N+1 to Mth processing units, since the maximum number of coding units is N, these processing units will not have corresponding coding units.

[0182] For this part of the processing unit, for example, the second intermediate feature output by the previous processing unit can be used as input to output the second intermediate feature.

[0183] The above combination Figure 7 The internal implementation of the first encoder and convolutional adapter is introduced. Through this model architecture, multi-level encoding and modeling processing of image features can be achieved to improve the accuracy and effectiveness of the first and second intermediate features.

[0184] And in Figure 7 The diagram also illustrates the decoding unit located downstream of the convolutional adapter. The implementation of the decoding unit is similar to that described above and will not be repeated here. Expandably, within the first model, further processing units can be included downstream of the decoding unit. For example, for segmentation algorithms, a masking module can be added to improve the processing performance of the segmentation algorithm. This application does not limit the specific model architecture within the first model; it can be extended according to actual needs.

[0185] Based on the above embodiments, the internal structure of the processing unit and the decoding unit will be further described in detail below. Figure 8 This is a schematic diagram of the structure of the processing unit provided in the embodiments of this application. Figure 9 This is a schematic diagram of the structure of the decoding unit provided in an embodiment of this application.

[0186] In one implementation, the processing unit may include, for example, multiple convolutional units and a fusion unit. In this embodiment, the convolutional unit within the processing unit is referred to as the first convolutional unit, and the fusion unit within the processing unit is referred to as the first fusion unit. The convolutional unit is used to perform convolution processing on the feature vectors, and the fusion unit is used to fuse the feature vectors.

[0187] As can be determined from the above description, the input to the processing unit may include a first intermediate feature and / or a second intermediate feature. For ease of explanation, the following description uses the first intermediate feature as the input to the processing unit as an example; other input methods are similar and will not be repeated here.

[0188] like Figure 8 As shown, for example, a processing unit may include four first convolutional units: first convolutional unit 1, first convolutional unit 2, first convolutional unit 3, and first convolutional unit 4. Each of these first convolutional units can correspond to a different convolutional kernel. In a specific example, the convolutional kernels corresponding to these five first convolutional units might be 1×1, 3×3, 7×7, and 11×11, respectively. In actual implementation, the convolutional kernels corresponding to the convolutional units can be set according to actual needs.

[0189] Optionally, for convolutional kernels larger than the preset size, depthwise separable convolution techniques can be added to the corresponding convolutional units to reduce the computational load of convolution processing. The preset size can be set according to actual needs; for example, 5×5 can be used as the preset size.

[0190] Reference Figure 8 It is certain that within the processing unit, the processing of the first intermediate feature by multiple first convolutional units is parallel. Therefore, by setting multiple first convolutional units to correspond to different convolutional kernels, convolution processing can be performed on the first intermediate feature from multiple different dimensions.

[0191] Furthermore, within each first convolutional unit, in addition to performing convolution processing on the first intermediate features, downsampling processing can also be performed on the first intermediate features to adjust the dimension of the feature vector during the modeling process.

[0192] Optionally, the processing unit may also include a pooling unit, which performs pooling operations on the feature vectors, whereby the pooling operation can also achieve downsampling. (See reference...) Figure 8 It is certain that the processing of the E-cell unit is parallel to that of the convolutional unit.

[0193] In actual implementation, the number of first convolutional units and pooling units contained in a processing unit can be selected according to actual needs, and this embodiment does not impose any restrictions on this.

[0194] Reference Figure 8 After inputting the first intermediate feature into multiple first convolutional units and pooling units, for example, the first feature vectors output by multiple first convolutional units and pooling units can be obtained, wherein the processing of multiple first convolutional units and pooling units is parallel.

[0195] Then, for example, the first feature vectors output by multiple first convolutional units and pooling units can be input into the first fusion unit in the processing unit, so that the first fusion unit can perform fusion processing on multiple first feature vectors, thereby outputting the second intermediate feature corresponding to the current processing unit.

[0196] In this implementation, multiple first convolutional units, each with a different kernel, perform parallel convolutional processing on the first intermediate features in different dimensions. This effectively improves the efficiency of convolutional processing and expands its richness. Furthermore, the various first feature vectors are then fused, ensuring that the final output second intermediate feature is a fusion of the processing from each convolutional unit. This allows the different dimensions of processing from each convolutional unit to be reflected in the output second intermediate feature.

[0197] Here we will briefly introduce the internal implementation of the first fusion unit.

[0198] In one implementation, the first fusion unit may include, for example, a first fusion layer, a first weighting layer, and a first weighting layer.

[0199] In the first fusion unit, the first fusion layer is used to fuse the first feature vectors output by multiple first convolutional units to obtain a first fusion vector, which may include multiple channels.

[0200] Subsequently, the first weighting layer is used to determine the corresponding weights for each of the multiple channels in the first fusion vector. For example, the first weighting layer may process the features of each channel in the first fusion vector separately based on the channel attention mechanism, so as to map the features of each channel to a certain range (e.g., 0-1), thereby obtaining the features corresponding to each channel.

[0201] Then, the first weighted layer further weights the multiple channels of the first fusion vector based on their respective weights, thereby outputting the second intermediate feature. This allows for the preservation of more important features and the influence of more vital characteristics during the modeling of the second image feature, improving the accuracy of the final modeled second image feature. This accuracy is reflected in the ability to model the intermediate features required for algorithm processing.

[0202] The above combination Figure 8 The network structure described can be understood as the network structure within a processing unit, or it can be configured within a processing unit to include multiple... Figure 8 The network structures shown are arranged sequentially.

[0203] Next, let's combine... Figure 9 The implementation of the decoding unit is described.

[0204] In one implementation, the decoding unit may include, for example, multiple convolutional units and a fusion unit. In this embodiment, the convolutional unit within the decoding unit is referred to as the second convolutional unit, and the fusion unit within the decoding unit is referred to as the second fusion unit. The convolutional unit is used to perform convolution processing on the feature vectors, and the fusion unit is used to fuse the feature vectors.

[0205] As can be determined from the above description, the input to the decoding unit is the second image feature output by the convolution adapter.

[0206] like Figure 9 As shown, for example, the decoding unit may include four second convolutional units: second convolutional unit 1, second convolutional unit 2, second convolutional unit 3, and second convolutional unit 4. Each of these second convolutional units can correspond to a different convolutional kernel. The implementation of the second convolutional units is similar to that of the first convolutional units described above, except that downsampling processing is not performed within the second convolutional units. Further details will not be elaborated here.

[0207] Reference Figure 9 After inputting the second image features into multiple second convolutional units, for example, we can obtain the second feature vectors output by the multiple second convolutional units respectively, where the processing of the multiple second convolutional units is in parallel.

[0208] Then, for example, the second feature vectors output by multiple second convolutional units can be input into the second fusion unit in the decoding unit, so that the second fusion unit can fuse the multiple second feature vectors and output the fused feature vector.

[0209] Furthermore, the image features are downsampled in the processing unit. In order to restore the image features, an upsampling unit can be set in the decoding unit. By inputting the fused feature vector into the upsampling unit, the decoded image features can be restored.

[0210] In this implementation, multiple second convolutional units, each with different kernels, perform parallel convolutional processing on the second image features across different dimensions. This effectively improves the efficiency of convolutional processing and expands its richness. Furthermore, the various second feature vectors are fused to obtain a fused feature vector that combines the results of each convolutional unit. Subsequent upsampling of this fused feature vector restores the feature dimensions, resulting in the decoded feature vector.

[0211] The internal implementation of the second fusion unit is similar to that of the first fusion unit. For example, the second fusion unit may include a second fusion layer, a second weight layer, and a second weighting layer.

[0212] In the second fusion unit, the second fusion layer is used to fuse the second feature vectors output by multiple second convolution units to obtain a second fusion vector, which includes multiple channels.

[0213] The second weighting layer is used to determine the weights for each of the multiple channels in the second fusion vector.

[0214] The second weighting layer is used to weight the multiple channels of the second fusion vector based on their respective weights to obtain the fused feature vector.

[0215] In this way, during the decoding process of feature vectors, more important features and the degree of influence of more important features can be retained, thereby improving the accuracy of the processing results reflected by the final decoded feature vectors. This accuracy is reflected in the ability to reflect the result features required by the algorithm.

[0216] The above combination Figure 9 The network structure described can be understood as the network structure inside a decoding unit, or it can be set up to include multiple [network structures] within a decoding unit. Figure 9 The network structures shown are arranged sequentially.

[0217] Based on the above introduction, the training process of the first model will be briefly introduced below.

[0218] In one implementation, for example, a first model can be trained using training data to obtain a trained first model. The training data includes training images and corresponding labeled images. The labeled images can also be understood as the images output by the first model after processing the training images, according to the correct expected model.

[0219] During model training, for example, training images can be input into a first model to obtain predicted images output by the first model. Then, a loss value is determined based on the labeled images corresponding to the predicted and training images. Subsequently, based on the loss value, the model parameters of multiple convolutional adapters in the first model, as well as the model parameters of the decoding units used to perform the decoding process, are adjusted.

[0220] This approach improves the adaptability of the convolutional adapter and decoding unit to their respective algorithms. It's understood that the technical solution of this application employs a pre-training method for the first encoder. After pre-training, the first encoder is integrated into the first model. Pre-training the first encoder separately allows it to focus more on the accuracy of extracted image features, without considering its compatibility with specific algorithms. Furthermore, subsequent model training and updates eliminate the need to update the first encoder, ensuring that the image features output by the first encoder remain universal and not biased towards the features required by a particular algorithm. This enhances the adaptability and universality of the first encoder with the convolutional adapters corresponding to subsequent algorithms.

[0221] In other words, the parameters of the first encoder are fixed after pre-training and do not participate in the parameter adjustment during subsequent model training. This ensures that during the image processing stage using the first model, the convolutional adapters of each downstream algorithm can extract features through the first encoder at once, and the features output by each convolutional adapter will not flow back to the first encoder, thus ensuring that the feature extraction effect of the first encoder is not affected.

[0222] In practice, model training can be completed on the terminal device, or it can be completed in advance on the server or cloud device and then directly deployed to the terminal device.

[0223] In summary, the technical solution provided in this application effectively reduces the power consumption of image processing algorithms by setting up convolutional adapters for multiple algorithms to share the image features output by the first encoder. Furthermore, by pre-training the first encoder, it ensures that the image features output by the first encoder can meet the needs of each algorithm. Therefore, the technical solution provided in this application achieves an effective balance between the accuracy and power consumption of image processing algorithms.

[0224] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0225] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0226] The image processing method of the present application embodiments has been described above. The apparatus for performing the above method provided in the present application embodiments is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the present application embodiments can perform the steps in the above image processing method.

[0227] The image processing method provided in this application can be applied to electronic devices with shooting and data processing functions. The electronic device includes a terminal device; the specific device form of the terminal device can be referred to the above description, and will not be repeated here.

[0228] In one implementation, this application provides an electronic device. Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0229] like Figure 10 As shown, the electronic device 100 includes: a processor 1001 and a memory 1002; the memory 1002 stores computer execution instructions; the processor 1001 executes the computer execution instructions stored in the memory 1002, causing the electronic device 100 to perform the above-described method.

[0230] When the memory 1002 is set up independently, the electronic device also includes a bus 1003 for connecting the memory 1002 and the processor 1001.

[0231] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0232] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0233] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0234] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0235] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0236] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: The original image is acquired and input into a first model, which includes a first encoder and multiple convolutional adapters, each of which corresponds to a different image processing algorithm. In the first model, the original image is input to the first encoder to obtain the first image features of the original image; as well as, The first image features are respectively input into the plurality of convolutional adapters to obtain the second image features output by each of the convolutional adapters; and, Each of the second image features is decoded to obtain multiple target images output by the first model. The multiple target images are images obtained by processing the original image with multiple image processing algorithms.

2. The method according to claim 1, characterized in that, The first encoder includes N encoding units, where N is an integer greater than or equal to 1; In the first model, the original image is input to the first encoder to obtain the first image features of the original image: In the first model, the original image is input to the first encoding unit in the first encoder to obtain the first intermediate feature output by the first encoding unit; The first intermediate feature output by the i-th coding unit is input into the (i+1)-th coding unit in the first encoder to obtain the first intermediate feature output by the (i+1)-th coding unit, where the value of i ranges from 1 to N-1. Where i takes the value of N-1, the first intermediate feature output by the Nth encoding unit is the first image feature.

3. The method according to claim 2, characterized in that, For any one of the plurality of convolution adapters, the convolution adapter includes M processing units, where M is an integer greater than or equal to N; The step of inputting the first image features into the plurality of convolutional adapters respectively to obtain the second image features output by each of the convolutional adapters includes: In the first model, the first intermediate feature output by the j-th encoding unit and / or the second intermediate feature output by the (j-1)-th processing unit are input to the j-th processing unit in the convolutional adapter to obtain the second intermediate feature output by the j-th processing unit, wherein the value of j ranges from 1 to M. Wherein, when j takes the value M, the second intermediate feature output by the Mth processing unit is the second image feature.

4. The method according to claim 3, characterized in that, The step of inputting the first intermediate feature output by the j-th encoding unit and / or the second intermediate feature output by the (j-1)-th processing unit into the j-th processing unit in the convolutional adapter to obtain the second intermediate feature output by the j-th processing unit includes: When j is 1, the first intermediate feature output by the j-th encoding unit is input to the j-th processing unit in the convolutional adapter to obtain the second intermediate feature output by the j-th processing unit; When the value of j is in the range of 2 to N, the first intermediate feature output by the j-th encoding unit and the second intermediate feature output by the (j-1)-th processing unit are input to the j-th processing unit in the convolution adapter to obtain the second intermediate feature output by the j-th processing unit. When the value of j is greater than N and less than or equal to M, the second intermediate feature output by the (j-1)th processing unit is input to the jth processing unit in the convolution adapter to obtain the second intermediate feature output by the jth processing unit.

5. The method according to claim 3 or 4, characterized in that, For any one of the M processing units, the processing unit includes a plurality of first convolution units and a first fusion unit, the plurality of first convolution units correspond to different convolution kernels, and the first convolution unit is used to perform convolution processing; The step of inputting the first intermediate feature output by the j-th encoding unit and / or the second intermediate feature output by the (j-1)-th processing unit into the j-th processing unit in the convolutional adapter to obtain the second intermediate feature output by the j-th processing unit includes: The first intermediate feature output by the j-th encoding unit and / or the second intermediate feature output by the (j-1)-th processing unit are input into the j-th processing unit in the convolutional adapter; In the j-th processing unit, the first intermediate feature and / or the second intermediate feature are respectively input into the plurality of first convolutional units to obtain the first feature vectors output by the plurality of first convolutional units respectively; wherein, the processing of the plurality of first convolutional units is in parallel; The first feature vectors output by the plurality of first convolutional units are input into the first fusion unit so that the first fusion unit outputs the second intermediate feature.

6. The method according to claim 5, characterized in that, The first fusion unit includes a first fusion layer, a first weight layer, and a first weighting layer; In the first fusion unit, the first fusion layer is used to fuse the first feature vectors output by the plurality of first convolution units respectively to obtain a first fusion vector, which includes multiple channels. The first weighting layer is used to determine the weights corresponding to each of the multiple channels in the first fusion vector; The first weighting layer is used to weight the multiple channels of the first fusion vector based on the weights corresponding to each of the multiple channels of the first fusion vector, so as to output the second intermediate feature.

7. The method according to claim 5 or 6, characterized in that, Each of the first convolutional units is further configured to perform downsampling processing on the first intermediate feature and / or the second intermediate feature.

8. The method according to any one of claims 1-7, characterized in that, The first model also includes decoding units corresponding to the plurality of convolutional adapters respectively; The step of decoding each of the second image features to obtain multiple target images output by the first model includes: For any one of the plurality of convolution adapters, the second image feature output by the convolution adapter is input to the corresponding decoding unit to obtain the decoded image feature output by the corresponding decoding unit; Based on the decoded image features output by the multiple decoding units, the multiple target images are generated and output.

9. The method according to claim 8, characterized in that, For any one of the plurality of decoding units, the decoding unit includes a plurality of second convolution units, a second fusion unit and an upsampling unit, the plurality of second convolution units correspond to different convolution kernels, and the second convolution units are used to perform convolution processing; The step of inputting the second image features output by the convolution adapter to the corresponding decoding unit to obtain the decoded image features output by the corresponding decoding unit includes: The second image features output by the convolution adapter are input into the corresponding decoding unit; In the decoding unit, the second image features are respectively input into the plurality of second convolutional units to obtain the second feature vectors output by the plurality of second convolutional units respectively; wherein, the processing of the plurality of second convolutional units is in parallel; The second feature vectors output by the plurality of second convolutional units are respectively input into the second fusion unit to obtain the fused feature vector output by the second fusion unit; The fused feature vector is input to the upsampling unit so that the upsampling unit outputs the decoded image features.

10. The method according to claim 9, characterized in that, The second fusion unit includes a second fusion layer, a second weighting layer, and a second weighting layer; In the second fusion unit, the second fusion layer is used to fuse the second feature vectors output by the plurality of second convolution units respectively to obtain a second fusion vector, which includes multiple channels; The second weighting layer is used to determine the weights for each of the multiple channels in the second fusion vector. The second weighting layer is used to weight the multiple channels of the second fusion vector based on the weights corresponding to each of the multiple channels of the second fusion vector, so as to obtain the fused feature vector.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: The first model is trained based on the training data to obtain the trained first model. The training data includes training images and labeled images corresponding to the training images.

12. The method according to claim 11, characterized in that, The training process includes: The training image is input into the first model to obtain the predicted image output by the first model; The loss value is determined based on the labeled image corresponding to the predicted image and the training image; Based on the loss value, the model parameters of the plurality of convolutional adapters in the first model, as well as the model parameters of the decoding unit used to perform the decoding process, are adjusted.

13. An electronic device, characterized in that, The electronic device includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 12.

14. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 12.

Images