An image processing method and an electronic device

Abstract

This application provides an image processing method and an electronic device for providing feature maps with rich edge information, which helps improve the accuracy of recognition or prediction tasks. The method includes: acquiring a target image; inputting the target image into a pre-trained processing model to obtain a target feature map corresponding to the target image; wherein the processing model is used to determine the target feature map based on a gradient energy feature map and a diffusion convolution corresponding to the target image; and performing a visual task based on the target feature map, the visual task including a recognition task or a detection task.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and provides an image processing method and an electronic device. Background Technology

[0002] Image processing is the foundation of computer vision. It typically involves extracting features from images and using these extracted feature maps for tasks such as recognition or prediction. High-quality feature maps are beneficial for improving recognition or prediction accuracy. Related techniques increase the number of channels in the feature map to enhance feature diversity. However, this process can easily lead to focusing on a single piece of information, causing other information to be smoothed out, resulting in the loss of useful information in the target image, such as edge details. Summary of the Invention

[0003] This application provides an image processing method and an electronic device for providing feature maps with rich edge information, which helps to improve the accuracy of recognition or prediction tasks.

[0004] In a first aspect, embodiments of this application provide an image processing method, which can be executed by an electronic device, including: Acquire the target image; The target image is input into a pre-trained processing model to obtain a target feature map corresponding to the target image; wherein, the processing model is used to determine the target feature map based on the gradient energy feature map and diffusion convolution corresponding to the target image; Based on the target feature map, a visual task is performed, which includes a recognition task or a detection task.

[0005] In one possible implementation, the image processing method provided in this application includes a first feature processing unit; the first feature processing unit includes a feature encoding subunit and a feature processing subunit. The feature encoding subunit receives the target image, performs feature encoding on the target image to obtain a first feature map, and inputs it to the feature processing subunit; The feature processing subunit has a first attention layer and a first image semantic parsing layer; the first attention layer receives the first feature map and outputs a first processing result to the first image semantic parsing layer, the first image semantic parsing layer receives the first processing result and outputs a second feature map, the second feature map being used to determine the gradient energy feature map.

[0006] In one possible implementation, the image processing method provided in this application further includes at least one second feature processing unit; the at least one second feature processing unit is used to determine the target feature map based on the gradient energy feature map and diffusion convolution corresponding to the target image.

[0007] In one possible implementation, the image processing method provided in this application includes at least N second feature processing units, where N is a positive integer greater than or equal to 2; each second feature processing unit has a second attention layer and a second image semantic parsing layer; wherein the first second feature processing unit performs the following process: The second attention layer of the first second feature processing unit receives the second feature map and outputs the second processing result to the second image semantic parsing layer of the first second feature processing unit; the second image semantic parsing layer of the first second feature processing unit determines the gradient energy feature map corresponding to the second processing result based on the second processing result, and determines the first thermal diffusion feature map based on the gradient energy feature map corresponding to the second processing result and the diffusion convolution, and inputs it to the second second feature processing unit.

[0008] In one possible implementation, the image processing method provided in this application, the Kth second feature processing unit performs the following process, where K is greater than 1 and is a positive integer less than or equal to N: The second attention layer of the Kth second feature processing unit receives the K-1th thermal diffusion feature map provided by the (K-1)th second feature processing unit and outputs the third processing result to the second image semantic parsing layer of the Kth second feature processing unit; the second image semantic parsing layer of the Kth second feature processing unit determines the gradient energy feature map corresponding to the third processing result based on the third processing result, and determines the Kth thermal diffusion feature map based on the gradient energy feature map corresponding to the third processing result and the diffusion convolution, and inputs it to the (K+1)th second feature processing unit; The Nth thermal diffusion feature map determined by the Nth second feature processing unit is the target feature map.

[0009] In one possible implementation, the image processing method provided in this application includes a second image semantic parsing layer of the Kth second feature processing unit comprising a first expert branch and a second expert branch. The determination of the Kth thermal diffusion feature map based on the gradient energy feature map and diffusion convolution corresponding to the third processing result includes: The first expert branch determines the first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution; The second expert branch determines a second thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the second diffusion convolution, wherein the size of the second diffusion convolution is different from the size of the first diffusion convolution; Based on the contribution weight parameters corresponding to each expert branch and the heat diffusion feature sub-map determined by each expert branch, the Kth heat diffusion feature map is determined.

[0010] In one possible implementation, the image processing method provided in this application further includes a third expert branch in the second image semantic parsing layer of the Kth second feature processing unit; The step of determining the Kth thermal diffusion feature map based on the gradient energy feature map and diffusion convolution corresponding to the third processing result further includes: The third expert branch determines the third thermal diffusion feature map based on the gradient energy feature map and sparse convolution corresponding to the third processing result.

[0011] In one possible implementation, the image processing method provided in this application, wherein the first expert branch determines a first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution, including: Based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution, the first diffusion feature map is determined. Based on the target image, an isotropic image is determined; The isotropic image is multiplied with the first diffusion feature map to obtain the second diffusion feature map; The second diffusion feature map, the isotropic image, and the third processing result are summed to obtain the first thermal diffusion feature sub-map.

[0012] In one possible implementation, the image processing method provided in this application, wherein the first expert branch determines a first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution, including: Based on the gradient energy feature map corresponding to the third processing result, the horizontal gradient feature map and the vertical gradient feature map are determined. Based on the horizontal gradient feature map and the vertical gradient feature map, the gradient angle feature map is determined; For each of the preset multiple expansion directions, a third diffusion feature map corresponding to each expansion direction is determined based on the gradient angle feature map, the similarity factor corresponding to each expansion direction, and the first diffusion convolution. Based on the weights corresponding to each expansion direction and the third diffusion feature map corresponding to each expansion direction, the first thermal diffusion feature sub-map is determined.

[0013] Secondly, this application also provides an electronic device, including a memory and a processor; The memory stores computer instructions; The processor executes the computer instructions to perform the image processing method as described in the first aspect and any possible implementation thereof.

[0014] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram illustrating a possible application scenario provided in the embodiments of this application; Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a processing model provided in an embodiment of this application; Figure 4A , Figure 4B , Figure 4C These are schematic diagrams of a second feature processing unit provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the working process of an electronic device provided in an embodiment of this application; Figure 6 This is a schematic diagram of a processing model provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0017] It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A connected to B can represent: A and B directly connected, or A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0018] Figure 1 This example illustrates one application scenario. An electronic device can be configured with a pre-trained processing model and task modules. The processing model has the function of generating feature maps, also known as extracting feature maps from images. The electronic device can have the ability to perform visual tasks. Visual tasks can be understood as image-based tasks. Visual tasks can be implemented as image-based recognition tasks or image-based prediction tasks. The module in the electronic device that performs the visual task is denoted as the task module. The electronic device inputs a target image into the processing model, and the processing model outputs the feature map corresponding to the target image, denoted as the target feature map. The task module performs the visual task based on the target feature map. In some possible application scenarios, the processing model and task module can be integrated together. The processing model and task module can be treated as a single unit.

[0019] Figure 2 An image processing method is illustrated, which can be applied to electronic devices. The image processing method may include the following steps: S201, acquire the target image.

[0020] S202, the target image is input into a pre-trained processing model to obtain a target feature map corresponding to the target image; wherein, the processing model is used to determine the target feature map based on the gradient energy feature map and diffusion convolution corresponding to the target image.

[0021] The processing model has the ability to extract feature maps from images. The model outputs a feature map of the target image for performing visual tasks. For simplicity, the output feature map used for performing visual tasks is denoted as the target feature map.

[0022] The processing model utilizes the gradient energy feature map of the target image to highlight edge and texture variations, providing underlying support for high-level semantic reasoning. By combining the gradient energy feature map with diffusing convolution, the model can enhance edge information within the target feature map.

[0023] S203, based on the target feature map, perform a visual task, which includes a recognition task or a detection task.

[0024] Electronic devices can improve the accuracy of visual tasks by using target feature maps with enhanced edge information to perform visual tasks.

[0025] In one possible implementation, please refer to Figure 3 The processing model includes a first feature processing unit; the first feature processing unit includes a feature encoding subunit and a feature processing subunit. The feature encoding subunit receives the target image, performs feature encoding on the target image to obtain a first feature map, and inputs it to the feature processing subunit; The feature processing subunit has a first attention layer and a first image semantic parsing layer; the first attention layer receives the first feature map and outputs a first processing result to the first image semantic parsing layer, the first image semantic parsing layer receives the first processing result and outputs a second feature map, the second feature map being used to determine the gradient energy feature map.

[0026] In practice, the feature encoding subunit in the first feature processing unit performs feature encoding processing on the target image to obtain the first feature map. The feature encoding subunit may have pixel block integrated encoding capabilities.

[0027] The feature processing subunit has an attention layer and an image semantic parsing layer. For ease of distinction, the attention layer in the feature processing subunit is denoted as the first attention layer, and the image semantic parsing layer is denoted as the first image semantic parsing layer. The first attention layer can execute a preset attention mechanism to extract key information from the first feature map.

[0028] The first image semantic parsing layer can simulate the human visual cognitive process, gradually transforming raw pixel data into semantically meaningful feature representations. The first image semantic parsing layer can use a convolutional kernel size of 1. A convolutional layer of 1 is used to perform image semantic parsing to enhance feature representation capabilities.

[0029] After processing the first processing result, the first image semantic parsing layer outputs a second feature map. The electronic device can determine the gradient energy feature map corresponding to the target image based on the second feature map.

[0030] The processing model can have a multi-level structure, and may further include at least one second feature processing unit; the at least one second feature processing unit is used to determine the target feature map based on the gradient energy feature map corresponding to the target image and a diffusion convolution. Each second feature processing unit may have an attention layer and an image semantic parsing layer.

[0031] At least one second feature processing unit and the aforementioned feature processing subunits constitute a multi-level structure. At least one second feature processing unit may include one second feature processing unit. Alternatively, at least one second feature processing unit may include multiple second feature processing units.

[0032] In one possible design, the processing model has multiple second feature processing units. That is, the processing model includes N second feature processing units, where N is a positive integer greater than or equal to 2; each second feature processing unit has a second attention layer and a second image semantic parsing layer; wherein the first second feature processing unit performs the following process: The second attention layer of the first second feature processing unit receives the second feature map and outputs the second processing result to the second image semantic parsing layer of the first second feature processing unit; the second image semantic parsing layer of the first second feature processing unit determines the gradient energy feature map corresponding to the second processing result based on the second processing result, and determines the first thermal diffusion feature map based on the gradient energy feature map corresponding to the second processing result and the diffusion convolution, and inputs it to the second second feature processing unit.

[0033] The N second feature processing units are connected in a preset order, which can be 1, 2, 3, ..., N. In two adjacent second feature processing units, the output of the preceding unit serves as the input of the following unit. The output of the Nth second feature processing unit is the target feature map.

[0034] The attention layer in the second feature processing unit is denoted as the second attention layer. The second attention layer can execute a preset attention mechanism to extract key information from the feature map input to the second attention layer. Optionally, the second attention layer can have the same configuration as the first attention layer. In this embodiment, the processing procedure of the second image semantic parsing layer in the second feature processing unit differs from that of the aforementioned first image semantic parsing layer.

[0035] The second attention layer of the first second feature processing unit receives the second feature map, performs an attention mechanism on the second feature map, and obtains a second processing result. The second processing result is input to the second image semantic parsing layer of the first second feature processing unit. Based on the input second processing result, the second image semantic parsing layer of the first second feature processing unit determines the gradient energy feature map corresponding to the second processing result, and then, based on the gradient energy feature map corresponding to the second processing result and a diffusion convolution, determines the first thermal diffusion feature map and inputs it to the (K+1)th second feature processing unit.

[0036] For any other second feature processing unit, taking the Kth second feature processing unit as an example, K is greater than 1 and is a positive integer less than or equal to N.

[0037] The second attention layer of the Kth second feature processing unit receives the (K-1)th thermal diffusion feature map provided by the (K-1)th second feature processing unit and outputs the third processing result to the second image semantic parsing layer of the Kth second feature processing unit; the second image semantic parsing layer of the Kth second feature processing unit determines the gradient energy feature map corresponding to the third processing result based on the third processing result, and determines the Kth thermal diffusion feature map based on the gradient energy feature map corresponding to the third processing result and the diffusion convolution, and inputs it to the next second feature processing unit; The Nth thermal diffusion feature map determined by the Nth second feature processing unit is the target feature map.

[0038] The (K-1)th second feature processing unit determines the (K-1)th thermal diffusion feature map and inputs it into the Kth second feature processing unit. The second attention layer of the Kth second feature processing unit performs an attention mechanism on the (K-1)th thermal diffusion feature map to obtain a third processing result. Based on the third processing result, the second image semantic parsing layer of the Kth second feature processing unit determines the gradient energy feature map corresponding to the third processing result. Based on the gradient energy feature map corresponding to the third processing result and a diffusion convolution, it determines the Kth thermal diffusion feature map and inputs it into the Kth second feature processing unit. The following describes the process by which the second image semantic parsing layer of the Kth second feature processing unit determines the gradient energy feature map corresponding to the third processing result based on the third processing result, and determines the Kth thermal diffusion feature map based on the gradient energy feature map corresponding to the third processing result and the diffusion convolution.

[0039] The processing model provided in this application is a multi-level heterogeneous expert model. Each second feature processing unit's second image semantic parsing layer is configured with an expert module, which includes multiple parallel expert branches. Please refer to... Figure 4A The second image semantic parsing layer of the Kth second feature processing unit includes a first expert branch and a second expert branch; The determination of the Kth thermal diffusion feature map based on the gradient energy feature map and diffusion convolution corresponding to the third processing result includes: The first expert branch determines the first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution; The second expert branch determines a second thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the second diffusion convolution, wherein the size of the second diffusion convolution is different from the size of the first diffusion convolution; Based on the contribution weight parameters corresponding to each expert branch and the heat diffusion feature sub-map determined by each expert branch, the Kth heat diffusion feature map is determined.

[0040] The first expert branch uses a first-order diffraction convolution, and the second expert branch uses a second-order diffraction convolution. The size of the second-order diffraction convolution is different from that of the first-order diffraction convolution. For example, the kernel size of the second-order diffraction convolution can be 3. 3. The kernel size of the first diffusion convolution can be 1. 1.

[0041] The third processing result output by the second attention layer in the Kth second feature processing unit is input into each expert branch in the second image semantic parsing layer.

[0042] The first expert branch can determine the gradient energy feature map corresponding to the third processing result based on the third processing result. The first expert branch then determines the first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution kernel.

[0043] Similarly, the second expert branch can determine the gradient energy feature map corresponding to the third processing result based on the point processing result. The second expert branch then determines the second thermal diffusion feature sub-map based on the gradient energy corresponding to the third processing result and the second diffusion convolution kernel.

[0044] In the pre-trained processing model, the contribution weight parameters corresponding to each expert branch in the second image semantic parsing layer of each second feature processing unit are configured.

[0045] When the electronic device invokes the Kth second feature processing unit, it can determine the Kth thermal diffusion feature map based on the contribution weight parameters corresponding to each expert branch and the thermal diffusion feature sub-maps determined by each expert branch, thus obtaining the Kth thermal diffusion feature image determined by the second feature processing unit. In this example, when the electronic device invokes the Kth second feature processing unit, it can determine the Kth thermal diffusion feature image based on the contribution weight parameters corresponding to the first expert branch, the contribution weight parameters corresponding to the second expert branch, the first thermal diffusion feature sub-map, and the second thermal diffusion feature sub-map.

[0046] The contribution weight parameters for expert branches can include contribution parameters and depth weight parameters.

[0047] The processing model includes a pre-defined load loss function for calculating expert branches. Regarding contribution parameters, the contribution parameters of each expert branch reflect the proportion of that expert branch's load contribution to the load loss calculation at the same depth. Because the configurations of each expert branch (such as kernel size) are different, their feature emphases differ at the same depth. The contribution parameters of each expert branch can also reflect the proportion of that expert branch's feature emphase in the load loss calculation at the same depth.

[0048] Regarding the depth weight parameter, it reflects the differences in feature depth. The depth weight parameter of an expert branch reflects the proportion of that expert branch's load in load loss calculation at different depths. The depth parameter of an expert branch can also reflect the proportion of that expert branch's feature emphasis in load loss calculation at different depths.

[0049] The combination of contribution parameters and depth emphasis parameters can be used to control the different feature emphases at different depths.

[0050] In some examples, please combine Figure 4B The second image semantic parsing layer of the Kth second feature processing unit further includes a third expert branch; The step of determining the Kth thermal diffusion feature map based on the gradient energy feature map and diffusion convolution corresponding to the third processing result further includes: The third expert branch determines the third thermal diffusion feature map based on the gradient energy feature map and sparse convolution corresponding to the third processing result.

[0051] When the electronic device invokes the Kth second feature processing unit, it can determine the Kth thermal diffusion feature map based on the contribution weight parameters corresponding to each expert branch and the thermal diffusion feature sub-maps determined by each expert branch, thus obtaining the Kth thermal diffusion feature image determined by the second feature processing unit. In this example, when the electronic device invokes the Kth second feature processing unit, it can determine the Kth thermal diffusion feature image based on the contribution weight parameters corresponding to the first expert branch, the contribution weight parameters corresponding to the second expert branch, the contribution weight parameters of the third expert branch, the first thermal diffusion feature sub-map, the second thermal diffusion feature sub-map, and the third thermal diffusion feature sub-map.

[0052] In some examples, please combine Figure 4C The second image semantic parsing layer of the Kth second feature processing unit further includes a fourth expert branch; The step of determining the Kth thermal diffusion feature map based on the gradient energy feature map and diffusion convolution corresponding to the third processing result further includes: The third expert branch determines the fourth thermal diffusion feature map based on the gradient energy feature map corresponding to the third processing result and conventional convolution.

[0053] When the electronic device calls the Kth second feature processing unit, it can determine the Kth thermal diffusion feature map based on the contribution weight parameters corresponding to each expert branch and the thermal diffusion feature sub-maps determined by each expert branch, thus obtaining the Kth thermal diffusion feature image determined by the second feature processing unit. In this example, when the electronic device calls the Kth second feature processing unit, it can determine the Kth thermal diffusion feature image based on the contribution weight parameters corresponding to the first expert branch, the second expert branch, the third expert branch, the fourth expert branch, the first thermal diffusion feature sub-map, the second thermal diffusion feature sub-map, the third thermal diffusion feature sub-map, and the fourth thermal diffusion feature sub-map.

[0054] Optionally, the processing model may include multiple second feature processing units, and the number of expert branches in each second feature processing unit may be the same or different. For example, the number of expert branches in second feature processing unit P1 may be different from the number of expert branches in second feature processing unit P2.

[0055] Optionally, the combination of expert branches in each second feature processing unit can be the same. The combination of expert branches in each second feature processing unit can also be different. For example, there may be a second feature processing unit P3 with three expert branches: a first expert branch, a second expert branch, and a third expert branch; and there may be a second feature processing unit P4 with three expert branches: a first expert branch, a second expert branch, and a fourth expert branch.

[0056] Based on the image processing method provided in any of the foregoing embodiments, in one possible implementation, the first expert branch determines a first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution. The package can perform the following process: Based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution, the first diffusion feature map is determined. Based on the third processing result, an isotropic image is determined; The isotropic image is multiplied with the first diffusion feature map to obtain the second diffusion feature map; The second diffusion feature map, the isotropic image, and the third processing result are summed to obtain the first thermal diffusion feature sub-map.

[0057] Please combine Figure 5 Let me first introduce the process of determining the gradient energy feature map, and take the determination of the gradient energy feature map corresponding to the third processing result as an example.

[0058] First, gradient features are extracted from the third processing result using both the horizontal Sobel operator (sobel_h) and the vertical Sobel operator (sobel_v). For example, sobel_h = sobel_v= The above calculations yield the horizontal and vertical gradient images corresponding to the third processing result. Secondly, based on physical principles, the gradient energy image determination process is as follows: .in, Represents the horizontal gradient image and It represents the longitudinal gradient image. It should be noted that in the adaptive convolutional layer structure, the Sobel operator should be used as the initialization parameter for the convolution kernel. As training progresses, the convolutional network can adaptively adjust its alignment to obtain more valuable gradient information.

[0059] In this field, the Perona-Malik model (PM model) can be used to fuse gradient information into transmission coefficients and influence the diffusion process of an image. The PM model is... ,in, Characterize image gradient, Characterizing edge stopping functions. Commonly used edge stopping functions have values ​​close to 1 in high gradient regions and greater than 1 in low gradient regions, which means they improve the diffusion ability in smooth regions and suppress the diffusion ability in edge regions.

[0060] This application uses the gradient energy feature map corresponding to the third processing result and the first diffusion convolution to determine the first diffusion feature map. This application determines the diffusion image features through the following process: ,in This represents the first diffuse convolution. That is In time coefficient The generation result is obtained by performing a first diffusion convolution on the gradient energy image to obtain a first diffusion feature map.

[0061] The process of determining the first diffusion convolution can be as follows: Design a stacked convolutional layer structure. A stacked convolutional layer structure can handle convolutional layers with a size of bs. cin h The gradient energy image of w is adaptively generated by regression of G, which generates cin. cout kernelsize The kernel size is the convolutional kernel. Here, bs represents the number of training images in the same batch, cin represents the number of channels in the feature image output by the previous feature processing unit, h represents the horizontal length of the image, w represents the vertical length of the image, and cout represents the number of channels in the feature image output by the current feature processing unit.

[0062] The stacked convolutional layer structure is pre-configured. Inputting the gradient energy feature map into the stacked convolutional structure yields an initial diffusing convolutional kernel. To increase stability, the initial diffusing convolutional kernel is summed with a preset matrix to obtain the first diffusing convolution, where the preset matrix is ​​implemented as a "residual connection" with a center of 1 and surrounding zeros.

[0063] The electronic device can use randomly initialized convolutional layers to convolve the third processing result to obtain an isotropic image, that is, an isotropic diffusion result. The isotropic image is then multiplied with the first diffusion feature map to obtain a second diffusion feature map.

[0064] Then, using the isotropic image as a residual term, the second diffusion feature map is summed with the isotropic image to obtain a preliminary thermal diffusion image. ;in, This represents the second diffusion feature map, and the gradient-guided diffusion weight image (which can be called the edge-preserving enhancement image). This refers to the isotropic image.

[0065] The second diffusion feature map is summed with the isotropic image, and then summed with the third processing result to obtain the first thermal diffusion feature sub-map. This achieves residual connection and output normalization activation from the third processing result to the initial thermal diffusion image. The first thermal diffusion feature sub-map has better structural stability, which is beneficial to improving the learning ability of the processing model in the early stage and reducing the training difficulty.

[0066] In another possible implementation, the first expert branch determines a first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution. The package can perform the following process.

[0067] Based on the gradient energy feature map corresponding to the third processing result, the horizontal gradient feature map and the vertical gradient feature map are determined. Based on the horizontal gradient feature map and the vertical gradient feature map, the gradient angle feature map is determined; For each of the preset multiple expansion directions, a third diffusion feature map corresponding to each expansion direction is determined based on the gradient angle feature map, the similarity factor corresponding to each expansion direction, and the first diffusion convolution. Based on the weights corresponding to each expansion direction and the third diffusion feature map corresponding to each expansion direction, the first thermal diffusion feature sub-map is determined.

[0068] First, gradient features are extracted from the third processing result using both the horizontal Sobel operator (sobel_h) and the vertical Sobel operator (sobel_v). For example, sobel_h = sobel_v= The above operations yield the horizontal gradient image and the vertical gradient image corresponding to the third processing result. The horizontal gradient image corresponding to the third processing result is also known as the horizontal gradient feature map. The vertical gradient image corresponding to the third processing result is also known as the vertical gradient feature map.

[0069] The size of the gradient energy image is bs cin h w. When determining the gradient angle feature map based on the horizontal and vertical gradient feature maps, the electronic device combines the corresponding positions of the horizontal and vertical gradient feature maps into vectors to obtain the gradient angle feature map. Specifically, the value m of the target position in the gradient angle feature map is determined based on the first value a at the target position in the horizontal gradient feature map and the second value b at the target position in the vertical gradient feature map, with the following numerical relationship: The target location is any location in the gradient angle feature map, where the gradient angle feature map has a size of bs. cin h w 2.

[0070] The electronic device performs a process for each of the preset multiple expansion directions, and determines a third diffusion feature map corresponding to each expansion direction based on the gradient angle feature map, the similarity factor corresponding to each expansion direction, and the first diffusion convolution.

[0071] Among them, the third diffusion feature map corresponding to any expansion direction .in, The extended direction factor, which characterizes the filtering based on this extended direction vector, has the same size as the third processing result. The similarity factor representing the extension direction factor and the gradient angle feature map in the vertical direction is approximated by multiplying the gradient angle feature map vertically and the direction factor. This characterizes the first diffusion convolution, also known as gradient diffusion convolution. Weighting coefficients representing the perspective.

[0072] Assuming there are 8 pre-defined expansion directions, for each position in each expansion direction, the product of the gradient angle normal matrix and the neighborhood direction vector is calculated as the direction factor for that expansion direction. The direction factor of the expansion direction can also be used as the weight corresponding to that expansion direction.

[0073] The electronic device can determine the first thermal diffusion feature sub-map based on the weights corresponding to each expansion direction and the third diffusion feature map corresponding to each expansion direction. The electronic device then performs a weighted superposition of the weights corresponding to each expansion direction and the third diffusion feature map corresponding to each expansion direction to obtain a preliminary thermal diffusion image out_feat=sum(dir_out).

[0074] The electronic device sums the preliminary thermal diffusion image with the third processing result to obtain the first thermal diffusion feature sub-map, realizing the residual connection and output normalization activation from the third processing result to the preliminary thermal diffusion image. The first thermal diffusion feature sub-map has better structural stability, which is beneficial to improving the learning ability of the processing model in the early stage and reducing the training difficulty.

[0075] Similarly, the second expert branch determines the second thermal diffusion feature map operation process based on the gradient energy feature map and the second diffusion convolution corresponding to the third processing result, which is similar to the processing process of the first expert branch based on the gradient energy feature map and the first diffusion convolution corresponding to the third processing result.

[0076] The electronic device can replace the "first diffusion convolution" with the "second diffusion convolution" in the operation of the first expert branch to realize the operation of the second expert branch. The operation process of the second expert branch will not be described in detail in this application.

[0077] In addition, this application provides a training process for the processing model. Please refer to... Figure 6 The example will be given by describing a processing model that includes a first feature processing unit and three second feature processing units.

[0078] Each second feature processing unit may include a second image semantic parsing layer that can be parallelized with multiple expert branches. Each second feature processing unit may include at least two of the following expert branches: Any one of the following four expert branches: The first type is expert branch 1; in expert branch 1, the kernel size is 1. The standard convolution with a 1:1 branch and the expert branch with a 1:1 branch can integrate feature map channels and align them with traditional vision solutions to ensure model effectiveness; The second type is expert branch 2; in expert branch 2, the kernel size is 1. The first diffusing convolution in step 1, and the expert branch 2, can guide and integrate the feature map through gradient information of the feature image; The third type is expert branch 3; in expert branch 3, the kernel size is 3. The second diffuse convolution of 3, expert branch 3, can perform edge-preserving feature parsing of the feature image through anisotropic encoding; The fourth type is expert branch 4; in expert branch 4, the kernel size is 5. Sparse convolution with 5 and expert branch 4 can improve the low-cost information integration of feature maps in the receptive field.

[0079] In some examples, the second feature processing unit may include expert branch 2 and expert branch 3. In other examples, the second feature processing unit may include two expert branches 1 and two expert branches 3. In still other examples, the second feature processing unit may include expert branch 1, expert branch 2, expert branch 3, and expert branch 4.

[0080] The first feature processing unit includes a feature encoding subunit and a feature processing subunit (block1). The three second feature processing units are, in order, a shallow feature processing unit (block2), a medium-level feature processing unit (block3), and a deep feature processing unit (block4).

[0081] During training, shallow and deep features have different focuses, which may lead to poor adaptability of a single diffusing convolution configuration. Therefore, it is necessary to distinguish between feature levels and the internal structure of diffusing convolution. Shallow features mainly focus on local image features, with high detail preservation, while deep features mainly focus on global image features, with flatter regions of detail.

[0082] Diffusion convolution can include gradient kernels and diffusion kernels. Gradient kernels primarily focus on extracting gradient energy or angular information from feature maps. The Sobel operator has strong edge information extraction capabilities; however, for deep feature maps, the high strength of the Sobel operator can easily cause initial convergence difficulties or make it difficult to obtain effective information based on noisy feature maps. Diffusion kernels mainly smooth and integrate gradient energy. If high-intensity feature smoothing is applied to shallow feature maps, it can cause the shallow features to be over-smoothed and become ineffective, and it is also not conducive to mining deep information.

[0083] Therefore, when training the model, the shallow feature processing unit, the medium-level feature processing unit, and the deep feature processing unit are initialized and configured.

[0084] In the second image semantic parsing layer of the shallow feature processing unit, the gradient kernel of the diffusing convolution structure is initialized with the Sobel operator, and the diffusing kernel is initialized as follows: This design allows for the use of strong gradient kernel initial values ​​to preserve and extract edge and detail information from shallow features, while employing weak diffusion kernels to minimize the negative effects of gradient smoothing and provide channel integration functionality.

[0085] In the second image semantic parsing layer of the mid-level feature processing unit, the gradient kernel of the diffusing convolution structure is initialized with a Gaussian kernel with a mean of 0.5 and a variance of 0.1, while the diffusing kernel is initialized to... This design reduces the strength of the initial gradient kernel, increasing the model's learning ability, while retaining a weaker diffusion smoothing state to improve the information content of the features.

[0086] In the second image semantic parsing layer of the deep feature processing unit, the gradient kernel of the diffusing convolution structure is initialized with a Gaussian kernel with a mean of 0.5 and a variance of 0.1, while the diffusing kernel is initialized to... This design reduces the strength of the initial gradient kernel while enhancing the central strength of the diffusion result to ensure the information strength of the deep feature maps in the multi-level output.

[0087] And set up a loss calculation method based on load balancing: First, the topk function is used to obtain the randomly assigned gating weight matrix from the input data to each expert branch, with a size of bs. ne h w. Here, bs represents the number of training image data in the same batch, ne represents the number of expert branches, h represents the horizontal length of the feature map, and w represents the vertical length of the feature map. It can be seen that the gating weight matrix is ​​a 4-dimensional matrix, usually denoted as 0-dimensional, 1-dimensional, 2-dimensional, and 3-dimensional, which correspond to bs, ne, h, and w respectively.

[0088] For the gating weight matrix of any expert branch, the load of that expert branch is obtained by summing the values ​​of the 0-dimensional, 2-dimensional, and 3-dimensional components. The load of this expert branch represents its actual computational cost. This load is obtained by weighting the contribution parameter and the depth weight parameter to obtain the actual load value used to calculate the loss. During training, this actual load value is used to update the model's gating weight matrix iteratively through gradient forward propagation.

[0089] Alternatively, the contribution of expert branches can be set based on the kernel size of the expert branches, and the contribution weight ratio can be set according to the depth of the expert branches. Finally, the actual load can be obtained by multiplying the normalized contribution by the depth weight.

[0090] The target load is configured as a matrix with all elements equal to 1. The loss between the target load and the actual load is calculated based on the smoothL1 average loss function, and this loss is fed back into the model training process.

[0091] The contribution weight parameters of expert branches generally include contribution parameters and depth weight parameters. When configuring the contribution weight parameters of expert branches in each second feature processing unit, the depth weight coefficient of expert branches in shallow feature processing units can be configured to 2, and the depth weight coefficient of expert branches in deep feature processing units can be configured to 0.5. Alternatively, the kernel size can be greater than or equal to 3. The contribution parameter of the expert branch of 3 is configured to be 1.2, and the kernel size is less than 3. The expert branch contribution parameter for 3 is configured to be 0.8.

[0092] For example, the contribution parameter for expert branch 4 is configured to 1.2. The contribution parameter for expert branch 3 is configured to 1.2.

[0093] In some examples, such as Figure 6 As shown, the shallow feature processing unit includes two expert branches, namely expert branch 2 and expert branch 3. The contribution weight parameters of expert branch 3 are configured as contribution parameter 1.2 and depth weight parameter 2; the contribution weight parameters of expert branch 2 are contribution parameter 0.8 and depth weight parameter 2.

[0094] The mid-level feature processing unit includes two expert branches, namely expert branch 2 and expert branch 3. The contribution weight parameters of expert branch 3 are configured as contribution parameter 1.2 and depth weight parameter 1; the contribution weight parameters of expert branch 2 are configured as contribution parameter 0.8 and depth weight parameter 0.5.

[0095] The deep feature processing unit includes two expert branches, namely expert branch 2 and expert branch 3. The contribution weight parameters of expert branch 3 are configured as follows: contribution parameter 1.2 and depth weight parameter 0.5; the contribution weight parameters of expert branch 2 are configured as follows: contribution parameter 0.8 and depth weight parameter 0.5.

[0096] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0097] Regarding the apparatus in the above embodiments, the specific manner in which each unit executes the request has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0098] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0099] Having introduced the permission control method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0100] Figure 7 This is a block diagram illustrating an electronic device 500 according to an exemplary embodiment, the device comprising: Processor 510; Memory 520 is used to store executable instructions of processor 510; The processor 510 is configured to execute instructions to implement the steps in any of the image processing methods provided in the embodiments of this application or any operation performed by the aforementioned electronic device.

[0101] In an exemplary embodiment, a storage medium including operations is also provided, such as a memory 520 including operations, the operations of which can be executed by a processor 510 of an electronic device 500 to complete the above-described method. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), compact disk read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0102] Memory generally includes main memory and secondary storage. Main memory can be RAM, ROM, and cache, etc. Secondary storage can be hard disk, optical disk, USB flash drive, floppy disk, or tape drive, etc. Memory is used to store computer programs and other data. The computer programs include application programs, and the other data may include data generated after the operating system or application is run, including system data (such as operating system configuration parameters) and user data. In the embodiments of this application, program instructions are stored in memory, and the processor executes the program instructions in memory to implement the image processing method described above.

[0103] Based on the same inventive concept, this application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing methods provided in the various optional implementations of the above embodiments.

[0104] In some possible implementations, various aspects of the access control method provided in this application may also be implemented in the form of a program product, which includes a computer program that, when run on a computer device, causes the computer device to perform the steps of the image processing method according to the various exemplary embodiments of this application described above.

[0105] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] The program product of the embodiments of this application may be in the form of a CD-ROM and include program code, and may run on a computing device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0107] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device. Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0108] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized in that, The method includes: Acquire the target image; The target image is input into a pre-trained processing model to obtain a target feature map corresponding to the target image; wherein, the processing model is used to determine the target feature map based on the gradient energy feature map and the diffusion convolution corresponding to the target image, wherein the processing model is configured with a stacked convolutional layer structure, the stacked convolutional layer structure adaptively generating the convolution kernel of the diffusion convolution from the gradient energy feature map; Based on the target feature map, a visual task is performed, which includes a recognition task or a detection task.

2. The method as described in claim 1, characterized in that, The processing model includes a first feature processing unit; the first feature processing unit includes a feature encoding subunit and a feature processing subunit. The feature encoding subunit receives the target image, performs feature encoding on the target image to obtain a first feature map, and inputs it to the feature processing subunit; The feature processing subunit has a first attention layer and a first image semantic parsing layer; the first attention layer receives the first feature map and outputs a first processing result to the first image semantic parsing layer, the first image semantic parsing layer receives the first processing result and outputs a second feature map, the second feature map being used to determine the gradient energy feature map.

3. The method as described in claim 2, characterized in that, The processing model further includes at least one second feature processing unit; the at least one second feature processing unit is used to determine the target feature map based on the gradient energy feature map and diffusion convolution corresponding to the target image.

4. The method as described in claim 3, characterized in that, The at least one second feature processing unit comprises N second feature processing units, where N is a positive integer greater than or equal to 2; each second feature processing unit has a second attention layer and a second image semantic parsing layer; wherein, the first second feature processing unit performs the following process: The second attention layer of the first second feature processing unit receives the second feature map and outputs the second processing result to the second image semantic parsing layer of the first second feature processing unit; the second image semantic parsing layer of the first second feature processing unit determines the gradient energy feature map corresponding to the second processing result based on the second processing result, and determines the first thermal diffusion feature map based on the gradient energy feature map corresponding to the second processing result and the diffusion convolution, and inputs it to the second second feature processing unit.

5. The method as described in claim 4, characterized in that, The Kth second feature processing unit performs the following process, where K is greater than 1 and is a positive integer less than or equal to N: The second attention layer of the Kth second feature processing unit receives the (K-1)th thermal diffusion feature map provided by the (K-1)th second feature processing unit and outputs the third processing result to the second image semantic parsing layer of the Kth second feature processing unit; when K is less than N, the second image semantic parsing layer of the Kth second feature processing unit determines the gradient energy feature map corresponding to the third processing result based on the third processing result, and determines the Kth thermal diffusion feature map based on the gradient energy feature map corresponding to the third processing result and the diffusion convolution, and inputs it to the (K+1)th second feature processing unit; When K equals N, the Nth thermal diffusion feature map determined by the Nth second feature processing unit is the target feature map.

6. The method as described in claim 5, characterized in that, The second image semantic parsing layer of the Kth second feature processing unit includes a first expert branch and a second expert branch; The determination of the Kth thermal diffusion feature map based on the gradient energy feature map and diffusion convolution corresponding to the third processing result includes: The first expert branch determines the first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution; The second expert branch determines a second thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the second diffusion convolution, wherein the size of the second diffusion convolution is different from the size of the first diffusion convolution; Based on the contribution weight parameters corresponding to each expert branch and the heat diffusion feature sub-map determined by each expert branch, the Kth heat diffusion feature map is determined.

7. The method as described in claim 6, characterized in that, The second image semantic parsing layer of the Kth second feature processing unit also includes a third expert branch; The step of determining the Kth thermal diffusion feature map based on the gradient energy feature map and diffusion convolution corresponding to the third processing result further includes: The third expert branch determines the third thermal diffusion feature map based on the gradient energy feature map and sparse convolution corresponding to the third processing result.

8. The method as described in claim 6, characterized in that, The first expert branch determines a first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution, including: Based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution, the first diffusion feature map is determined. Based on the target image, an isotropic image is determined; The isotropic image is multiplied with the first diffusion feature map to obtain the second diffusion feature map; The second diffusion feature map, the isotropic image, and the third processing result are summed to obtain the first thermal diffusion feature sub-map.

9. The method as described in claim 6, characterized in that, The first expert branch determines a first thermal diffusion feature sub-map based on the gradient energy feature map corresponding to the third processing result and the first diffusion convolution, including: Based on the gradient energy feature map corresponding to the third processing result, the horizontal gradient feature map and the vertical gradient feature map are determined. Based on the horizontal gradient feature map and the vertical gradient feature map, the gradient angle feature map is determined; For each of the preset multiple expansion directions, a third diffusion feature map corresponding to each expansion direction is determined based on the gradient angle feature map, the similarity factor corresponding to each expansion direction, and the first diffusion convolution. Based on the weights corresponding to each expansion direction and the third diffusion feature map corresponding to each expansion direction, the first thermal diffusion feature sub-map is determined.

10. An electronic device, characterized in that, Including memory and processor; The memory stores computer instructions; The processor executes the computer instructions to perform the image processing method as described in any one of claims 1-9.

Images