Image processing method and related device

Abstract

The embodiment of the present application discloses an image processing method, which is applied to the field of artificial intelligence, including: obtaining an input feature map of an image to be processed, the input feature map including a first input sub-feature map and a second input sub-feature map, the resolution of the first input sub-feature map being higher than the resolution of the second input sub-feature map; performing feature fusion processing on the input feature map through a target network to obtain an output feature map, in which the features of the first input sub-feature map are fused into the features of the second input sub-feature map in a low-to-high-level manner in the target network; performing target detection on the image to be processed according to the output feature map to obtain a target detection result. In this solution, the low-level features are fused into the high-level features in a low-to-high-level manner to compensate for the sparsity problem of the high-level features, thereby improving the accuracy of target detection and solving the problem that additive neural networks cannot be applied to target detection.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method and related devices. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a science about how to use cameras / camcorders and computers to obtain the data and information of the objects we need. Figuratively speaking, it is to install eyes (cameras / camcorders) and brains (algorithms) on computers to replace human eyes to identify, track and measure targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0003] Target detection, also known as target extraction, is a type of image segmentation based on target geometry and statistical features. It focuses on specific object targets in the image and requires obtaining both the category information and location information of this specific object target. Target detection is used in many scenarios, such as unmanned security systems, intelligent transportation systems, and intelligent monitoring systems. Target detection is not difficult for humans. It is easy to locate and classify the target objects in the image by perceiving different color modules in the image. However, for computers, facing the RGB pixel matrix, it is difficult to directly obtain the abstract concept of the object from the image and locate its position. In addition, sometimes multiple objects are mixed with a messy background, making target detection even more difficult.

[0004] At present, deep learning based on convolutional neural networks has a good performance in processing computer vision tasks. However, this type of convolutional neural network usually includes a large number of multiplication operations, resulting in high power consumption of the network. Therefore, in the related art, an additive neural network that replaces the multiplication operation with an addition operation is proposed to reduce the power consumption of network calculation. However, the additive neural network in the related art is mainly used in technical fields such as image classification or image restoration. When the current additive neural network is applied to target detection, it is usually difficult to obtain effective detection results. Summary of the invention

[0005] The present application provides an image processing method, which, in the stage of feature fusion, fuses low-level features into high-level features from low layers to high layers, compensates for the sparsity problem of high-level features, thereby improving target detection accuracy and solving the problem that additive neural networks cannot be applied to target detection.

[0006] In a first aspect, the present application provides an image processing method, comprising: a terminal obtains an input feature map of an image to be processed, the input feature map comprising a first input sub-feature map and a second input sub-feature map, and the resolution of the first input sub-feature map is higher than the resolution of the second input sub-feature map. Wherein, the image to be processed is an image on which target detection needs to be performed. The image to be processed may be an image captured by a camera of the terminal, or the image to be processed may be an image obtained from inside the terminal (for example, an image stored in an album of the terminal, or an image obtained by the terminal from the cloud). Accordingly, the input feature map of the image to be processed may be a feature map of the image. Alternatively, the input feature map may be a feature map obtained after being processed by a backbone network in a target detection network, and the backbone network may be an additive neural network.

[0007] The terminal performs feature fusion processing on the input feature map through a target network to obtain an output feature map, wherein the target network is an additive neural network. The target network includes multiple branches for performing feature fusion processing. The multiple branches include a first branch and a second branch, wherein the first branch is used to perform feature extraction processing on the first input sub-feature map to obtain a first sub-feature map and a first output sub-feature map, and the first sub-feature map is an intermediate feature obtained by the first branch.

[0008] The second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, and the output feature map includes the first output sub-feature map and the second output sub-feature map.

[0009] Finally, the terminal performs target detection on the image to be processed according to the output feature map to obtain a target detection result.

[0010] In this scheme, when using additive neural networks to perform target detection, when performing multi-scale feature fusion, the low-level features are fused to the high-level features from low to high levels to make up for the sparsity problem of high-level features, thereby improving the target detection accuracy and solving the problem that additive neural networks cannot be applied to target detection.

[0011] Optionally, in a possible implementation, the feature extraction process in the target network is an addition operation or a subtraction operation. The addition operation refers to: the feature extraction kernel is added to the elements at the corresponding position in the input image to obtain a target matrix, and the absolute values ​​of the elements in the target matrix are taken and then summed. The subtraction operation refers to: the feature extraction kernel is subtracted from the elements at the corresponding position in the input image to obtain a target matrix, and the absolute values ​​of the elements in the target matrix are taken and then summed. In other words, the target matrix can be obtained by performing matrix addition or matrix subtraction between its corresponding input sub-feature map and the feature extraction kernel.

[0012] Among them, the role of the feature extraction kernel in image processing is equivalent to a filter that extracts specific information from the input image matrix. The feature extraction kernel can essentially be a weight matrix. During the convolution operation on the image, the weight matrix is ​​usually processed horizontally on the input image one pixel after another (or two pixels after two pixels... depending on the value of the stride), thereby completing the work of extracting specific features from the image.

[0013] For example, the feature extraction process performed by the first branch on the first input sub-feature graph may be the above-mentioned addition operation or subtraction operation. The feature extraction process performed by the second branch on the second input sub-feature graph and the first sub-feature graph may also be the above-mentioned addition operation or subtraction operation.

[0014] Optionally, in a possible implementation, the second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, including: the second branch is used to add the second input sub-feature map and the first sub-feature map after downsampling processing to obtain a second sub-feature map, and perform feature extraction processing on the second sub-feature map to obtain the second output sub-feature map. Since the resolution of the first sub-feature map obtained by the first branch processing is greater than the second input sub-feature map in the second branch, the first sub-feature map can be downsampled so that the resolution of the first sub-feature map is the same as that of the second input sub-feature map. Then, the second input sub-feature map and the first sub-feature map after downsampling processing are added to obtain the second sub-feature map.

[0015] That is to say, for the first branch processing the first input sub-feature map with a higher resolution, the first branch passes the intermediate features obtained by processing the first input sub-feature map to the second branch processing the second input sub-feature map with a lower resolution. That is, the low-level features are fused to the high-level features in a low-to-high-level manner, thereby compensating for the sparsity of the high-level features processed in the second branch.

[0016] Optionally, in a possible implementation, the second branch performs feature extraction processing on the second sub-feature map, including: the second branch processes the second sub-feature map through a convolution layer (e.g., an addition filter layer) and an activation function to obtain a third sub-feature map, and the convolution layer may include one or more feature extraction kernels. The second branch adds the second sub-feature map and the third sub-feature map to obtain the second output sub-feature map.

[0017] In this scheme, in order to address the expressive power defects of the additive feature extraction kernel itself, a skip connection structure including convolutional layers and activation functions is designed to improve the expressive power of the convolutional units in the target network, thereby enhancing the feature learning ability of the target network.

[0018] Optionally, in a possible implementation, the second branch is also used to perform feature extraction processing on the second sub-feature map to obtain a fourth sub-feature map, and the fourth sub-feature map is an intermediate feature obtained by the second branch; the first branch performs feature extraction processing on the first input sub-feature map, including: the first branch performs feature extraction processing on the first sub-feature map to obtain the first sub-feature map; the first branch adds the first sub-feature map and the fourth sub-feature map after upsampling processing to obtain a fifth sub-feature map; the first branch performs feature extraction processing on the fifth feature map to obtain the first output sub-feature map.

[0019] In this scheme, the low-level features are first fused into the high-level features from the low-level to the high-level, and then the processed high-level features are fused into the low-level features, thereby introducing richer semantic information into the low-level features, improving the feature fusion effect, and further improving the target detection accuracy of the network.

[0020] Optionally, in a possible implementation, the first branch performs feature extraction processing on the fifth feature map to obtain the first output sub-feature map, including: the first branch processes the fifth sub-feature map through a convolution layer and an activation function to obtain a sixth sub-feature map; the first branch adds the fifth sub-feature map and the sixth sub-feature map to obtain the first output sub-feature map.

[0021] Optionally, in a possible implementation, the gradient of the input feature map is determined by a difference between a weight of the target network and the input feature map.

[0022] Optionally, in a possible implementation, the gradient of the input feature map is obtained by performing a sign operation on the difference between the weight of the target network and the input feature map.

[0023] In this scheme, based on the characteristics that additive neural networks usually have multiple branches and multiple loss functions, a more accurate gradient is obtained by providing a gradient calculation method including a sign operation, which is beneficial to improving the final target detection accuracy.

[0024] According to a second aspect of the present application, there is provided an image processing device, comprising: an acquisition unit and a processing unit; the acquisition unit is used to acquire an input feature map of an image to be processed, the input feature map comprising a first input sub-feature map and a second input sub-feature map, and the resolution of the first input sub-feature map is higher than the resolution of the second input sub-feature map; the processing unit is used to perform feature fusion processing on the input feature map through a target network to obtain an output feature map, wherein the target network is an additive neural network, the target network comprises a first branch and a second branch, the first branch is used to perform feature extraction processing on the first input sub-feature map to obtain a first sub-feature map and a first output sub-feature map, the first sub-feature map is an intermediate feature obtained by the first branch, the second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, the output feature map comprises the first output sub-feature map and the second output sub-feature map; the processing unit is also used to perform target detection on the image to be processed according to the output feature map to obtain a target detection result.

[0025] Optionally, in a possible implementation, the second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, including: the second branch is used to add the second input sub-feature map and the first sub-feature map after downsampling processing to obtain a second sub-feature map, and perform feature extraction processing on the second sub-feature map to obtain the second output sub-feature map.

[0026] Optionally, in a possible implementation, the second branch performs feature extraction processing on the second sub-feature map, including: the second branch processes the second sub-feature map through a convolution layer and an activation function to obtain a third sub-feature map; the second branch adds the second sub-feature map and the third sub-feature map to obtain the second output sub-feature map.

[0027] Optionally, in a possible implementation, the second branch is also used to perform feature extraction processing on the second sub-feature map to obtain a fourth sub-feature map, and the fourth sub-feature map is an intermediate feature obtained by the second branch; the first branch performs feature extraction processing on the first input sub-feature map, including: the first branch performs feature extraction processing on the first sub-feature map to obtain the first sub-feature map; the first branch adds the first sub-feature map and the fourth sub-feature map after upsampling processing to obtain a fifth sub-feature map; the first branch performs feature extraction processing on the fifth feature map to obtain the first output sub-feature map.

[0028] Optionally, in a possible implementation, the first branch performs feature extraction processing on the fifth feature map to obtain the first output sub-feature map, including: the first branch processes the fifth sub-feature map through a convolution layer and an activation function to obtain a sixth sub-feature map; the first branch adds the fifth sub-feature map and the sixth sub-feature map to obtain the first output sub-feature map.

[0029] Optionally, in a possible implementation, the gradient of the input feature map is determined by a difference between a weight of the target network and the input feature map.

[0030] Optionally, in a possible implementation, the gradient of the input feature map is obtained by performing a sign operation on the difference between the weight of the target network and the input feature map.

[0031] The third aspect of the present application provides an image processing device, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect is implemented. For the steps in each possible implementation method of the processor executing the first aspect, the details can be referred to the first aspect, and will not be repeated here.

[0032] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0033] A fifth aspect of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect.

[0034] In a sixth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first aspect.

[0035] The seventh aspect of the present application provides a chip system, which includes a processor for supporting a server or a threshold value acquisition device to implement the functions involved in the first aspect, for example, sending or processing the data and / or information involved in the above method. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of chips, or it can include chips and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0037] Figure 2 A schematic diagram of a convolutional neural network provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a convolutional neural network provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0040] Figure 5 A schematic diagram for comparing high-level features provided in an embodiment of the present application;

[0041] Figure 6 It is a structural schematic diagram of a feature fusion network in the related technology;

[0042] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of the structure of a target network provided in an embodiment of the present application;

[0044] Fig. 9 A schematic diagram of a jump connection structure provided in an embodiment of the present application;

[0045] Fig.10 A schematic diagram of a network architecture provided in an embodiment of the present application;

[0046] Fig.11 A schematic diagram of the structure of a target network provided in an embodiment of the present application;

[0047] Fig.12 A schematic diagram of the structure of a target detection network provided in an embodiment of the present application;

[0048] Fig.13 A schematic diagram for comparing detection results of a network provided in an embodiment of the present application;

[0049] Fig.14 Another comparative schematic diagram of a network detection result provided in an embodiment of the present application;

[0050] Fig.15 A schematic diagram of theoretical power consumption comparison of a network provided in an embodiment of the present application;

[0051] Fig.16 A schematic diagram of performing target detection on an image provided in an embodiment of the present application;

[0052] Fig.17 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0053] Fig.18 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0054] Fig.19 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0056] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0057] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0058] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0059] (1) Infrastructure.

[0060] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0061] (2)Data.

[0062] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0063] (3) Data processing

[0064] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0065] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0066] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0067] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0068] (4) General ability.

[0069] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0070] (5) Smart products and industry applications.

[0071] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0072] The following describes the method provided by this application from the perspectives of model training and model application:

[0073] The model training method provided in the embodiment of the present application can be specifically applied to data training, machine learning, deep learning and other data processing methods to perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc. on the training data, and finally obtain a trained neural network model (such as the target neural network model in the embodiment of the present application); and the target neural network model can be used for model reasoning, specifically, the input data can be input into the target neural network model to obtain output data.

[0074] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0075] (1) Neural network.

[0076] A neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:

[0077] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0078] (2) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network includes a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal (for example, the first convolution layer and the second convolution layer in this embodiment). In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Shared weights can be understood as the way of extracting image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in one part can also be used in another part. So for all positions on the image, we can use the same learned image information. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0079] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0080] Specifically, Figure 2 As shown, the convolutional neural network (CNN) 100 may include an input layer 1110 , a convolutional layer / pooling layer 1120 , wherein the pooling layer is optional, and a neural network layer 1130 .

[0081] Among them, the structure composed of the convolution layer / pooling layer 1120 and the neural network layer 1130 can be the first convolution layer and the second convolution layer described in the present application, the input layer 1110 is connected to the convolution layer / pooling layer 1120, the convolution layer / pooling layer 1120 is connected to the neural network layer 1130, the output of the neural network layer 1130 can be input to the activation layer, and the activation layer can perform nonlinear processing on the output of the neural network layer 1130.

[0082] Convolutional layer / pooling layer 1120. Convolutional layer: Figure 2 The convolution layer / pooling layer 1120 shown may include layers 121-126 as shown in the example. In one implementation, layer 121 is a convolution layer, layer 122 is a pooling layer, layer 123 is a convolution layer, layer 124 is a pooling layer, layer 125 is a convolution layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolution layers, layer 123 is a pooling layer, layers 124 and 125 are convolution layers, and layer 126 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0083] Taking the convolution layer 121 as an example, the convolution layer 121 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions are also of the same dimension. The extracted feature maps with the same dimensions are then merged to form the output of the convolution operation.

[0084] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0085] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0086] Pooling layer: Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 2 The layers 121-126 illustrated in 1120 may be a convolutional layer followed by a pooling layer, or multiple convolutional layers may be followed by one or more pooling layers.

[0087] Neural network layer 1130: After being processed by the convolution layer / pooling layer 1120, the convolution neural network 100 is not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer 1120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 1130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 1130 may include multiple hidden layers (such as Figure 2 131, 132 to 13n) and the output layer 1140 shown, the parameters contained in the multi-layer hidden layers can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0088] After the multiple hidden layers in the neural network layer 1130, that is, the last layer of the entire convolutional neural network 100 is the output layer 1140, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (such as Figure 2 The propagation from 1110 to 1140 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 1140 to 1110 is called back propagation) and then the weight values ​​and biases of the aforementioned layers will begin to be updated to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0089] It should be noted that if Figure 2 The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, Figure 3 The multiple convolutional layers / pooling layers shown are operated in parallel, and the features extracted respectively are input to the full neural network layer 1130 for processing.

[0090] (3) Deep neural networks.

[0091] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0092] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0093] (4) Loss function

[0094] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0095] (5) Back propagation algorithm.

[0096] Convolutional neural networks can use the error back propagation (S21P000217) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial super-resolution model are updated by back-propagating the error loss information, so that the error loss converges. The back-propagation algorithm is a back-propagation movement dominated by error loss, which aims to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0097] (6) Linear operation.

[0098] Linearity refers to the proportional, linear relationship between quantities. Mathematically, it can be understood as a function whose first-order derivative is a constant. Linear operations can be, but are not limited to, addition operations, null operations, identity operations, convolution operations, batch normalization (BN) operations, and pooling operations. Linear operations can also be called linear mappings. Linear mappings need to meet two conditions: homogeneity and additivity. If either condition is not met, it is nonlinear.

[0099] Among them, homogeneity means f(ax)=af(x); additivity means f(x+y)=f(x)+f(y); for example, f(x)=ax is linear. It should be noted that x, a, and f(x) here are not necessarily scalars, but can be vectors or matrices to form a linear space of any dimension. If x and f(x) are n-dimensional vectors, when a is a constant, they are equivalent to satisfying homogeneity, and when a is a matrix, they are equivalent to satisfying additivity. Relatively speaking, a function graph that is a straight line does not necessarily conform to a linear mapping. For example, f(x)=ax+b neither satisfies homogeneity nor additivity, so it belongs to a nonlinear mapping.

[0100] In the embodiment of the present application, a combination of multiple linear operations may be referred to as a linear operation, and each linear operation included in a linear operation may also be referred to as a sub-linear operation.

[0101] (7) Target detection.

[0102] Target detection, also known as target extraction, is a kind of image segmentation based on the geometric and statistical features of the target. It combines the segmentation and recognition of the target into one. Its accuracy and real-time performance are important capabilities of the entire system. Especially in complex scenes, when multiple targets need to be processed in real time, automatic target extraction and recognition are particularly important. The essence of target detection is the positioning of multiple targets, that is, to locate multiple target objects in the image, including classification and positioning. For example, when performing target detection on a certain image, the result is several animals, and the position of each animal corresponds to a rectangular box. With the development of computer technology and the widespread application of computer vision principles, the research on real-time tracking of targets using computer image processing technology has become increasingly popular. Dynamic real-time tracking and positioning of targets has wide application value in intelligent transportation systems, intelligent monitoring systems, military target detection, and positioning of surgical instruments in medical navigation surgery.

[0103] Figure 4 is a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 4 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user can input data to the I / O interface 112 through a client device 140 .

[0104] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0105] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

[0106] Optionally, the client device 140 , for example, may be a control unit in an autonomous driving system, or a functional algorithm module in a mobile phone terminal. For example, the functional algorithm module may be used to implement related tasks.

[0107] It is worth noting that the training device 120 can generate corresponding target models / rules (such as the target neural network model in this embodiment) based on different training data for different goals or different tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0108] exist Figure 4 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific form can be display, sound, action and other specific methods. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0109] It is worth noting that Figure 4 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 4 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0110] At present, deep learning based on convolutional neural networks has a good performance in processing computer vision tasks. However, this type of convolutional neural network usually includes a large number of multiplication operations, resulting in high power consumption of the network. Therefore, the related art proposes an additive neural network that replaces the multiplication operation with the addition operation to reduce the power consumption of network calculation. Because in hardware calculation, the calculation efficiency of the addition operation is more efficient than the multiplication operation and the power consumption is lower. Therefore, by using the addition operation to replace the multiplication operation in the network, the power consumption of network calculation can be effectively reduced. The additive neural network can achieve lossless accuracy in image classification tasks while effectively reducing the power consumption of the network.

[0111] However, additive neural networks in related technologies are mainly used in technical fields such as image classification or image restoration. When current additive neural networks are used in target detection, it is usually difficult to obtain effective detection results.

[0112] Generally speaking, the target detection network usually includes a backbone network, a feature fusion network and a prediction head. The backbone network is used to extract features of different scales from the input image, such as different low-level features and high-level features. Among them, the resolution of low-level features is large, the location information contained is richer, but the semantic information is weaker; on the contrary, the resolution of high-level features is small, and the semantic information is richer. The feature fusion network is used to perform multi-scale feature fusion on the features output by the backbone network, thereby enhancing the expressive power of the features. The prediction head is used to perform target detection based on the fused features output by the feature fusion network to obtain the target detection result.

[0113] The inventors have found that when the backbone network uses an additive neural network, the high-level features extracted by the backbone network tend to show a more obvious sparse characteristic compared with the multiplicative neural network. Figure 5 , Figure 5 A schematic diagram for comparing high-level features provided in an embodiment of the present application. Figure 5 As shown, Figure 5 (a) in the figure represents the high-level features extracted by the multiplication neural network. Figure 5 (b) in the figure represents the high-level features extracted by the additive neural network. Obviously, the high-level features extracted by the additive neural network have a relatively obvious sparse characteristic. This sparse characteristic is mainly caused by the addition operation in the additive neural network.

[0114] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a feature fusion network structure in the related art. Figure 6As shown, the feature fusion network in the related art includes multiple branches, the input of each branch is the feature extracted by the backbone network, and the input of each branch is different. From branch 1 to branch 4, the resolution of the input feature gradually increases, the input feature of branch 1 is the high-level feature with the smallest resolution, and the input feature of branch 4 is the low-level feature with the largest resolution. In the feature fusion process, the feature fusion network fuses the high-level features to the low-level features in a high-level to low-level manner. In this way, when the additive neural network is used as the backbone network, the high-level features extracted by the backbone network have relatively obvious sparse features, so when the high-level features are fused to the low-level features, the low-level features will be damaged, which will ultimately affect the accuracy of target detection.

[0115] In view of this, an embodiment of the present application provides an image processing method based on an additive neural network. In the stage of feature fusion, low-level features are fused to high-level features from low layers to high layers to compensate for the sparsity problem of high-level features, thereby improving target detection accuracy and solving the problem that additive neural networks cannot be applied to target detection.

[0116] The image processing method provided in the embodiments of the present application can be applied to photography, video recording, smart cities, human-computer interaction, and other scenarios requiring target detection.

[0117] It should be understood that the image in the embodiments of the present application may be a static image (or a static picture) or a dynamic image (or a dynamic picture). For example, the image in the embodiments of the present application may be a video or a dynamic picture, or the image in the embodiments of the present application may also be a static picture or a photo. For ease of description, the present application will uniformly refer to static images or dynamic images as images in the following embodiments.

[0118] The method of the embodiment of the present application can be specifically applied to an image recognition scenario, and this scenario is introduced in detail below.

[0119] Image Recognition:

[0120] After a user captures an image through a terminal such as a mobile phone or tablet computer, the terminal can recognize the image captured by the user to obtain the object contained in the image. For example, the terminal can automatically recognize that the captured object is a flower, an animal, a car, etc. Furthermore, the terminal can also mark the recognized object with a rectangular frame to achieve target detection.

[0121] It should be understood that the image recognition introduced above is only a specific scenario in which the method of the embodiment of the present application is applied. The method of the embodiment of the present application is not limited to the above-mentioned scenario when applied. The method of the embodiment of the present application can be applied to any scenario where target detection is required, and this is not limited in the embodiment of the present application.

[0122] The feature extraction kernel mentioned in this embodiment plays a role in image processing equivalent to a filter that extracts specific information from an input image matrix. The feature extraction kernel is a weight matrix. During the convolution operation on the image, the weight matrix is ​​usually processed one pixel by one pixel (or two pixels by two pixels... depending on the value of the stride) along the horizontal direction on the input image to complete the work of extracting specific features from the image.

[0123] See also Figure 7 , Figure 7 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. Figure 7 As shown, the image processing method includes the following steps 701 to 703.

[0124] Step 701: Obtain an input feature map of an image to be processed, wherein the input feature map includes a first input sub-feature map and a second input sub-feature map, and a resolution of the first input sub-feature map is higher than a resolution of the second input sub-feature map.

[0125] Among them, the image to be processed is an image that needs to perform target detection. The image to be processed can be an image captured by the terminal through a camera, or the image to be processed can also be an image obtained from the inside of the terminal (for example, an image stored in the terminal's photo album, or an image obtained by the terminal from the cloud). Accordingly, the input feature map of the image to be processed can be a feature map of the image. Alternatively, the input feature map can be a feature map obtained after processing by the backbone network in the target detection network, and the backbone network can be an additive neural network.

[0126] The additive neural network in the embodiment of the present application may include one or more additive filter layers, and may also include Figure 2 The input layer, pooling layer, hidden layer or output layer and other layers shown in the embodiment of the present application are not limited to this. The additive neural network may include multiple additive filter layers, and each additive filter layer may include one or more feature extraction cores. That is, the additive neural network may include multiple feature extraction cores. Accordingly, the image to be processed can be subjected to multiple feature extraction processes by the multiple feature extraction cores to obtain an input feature map, and the output feature map includes multiple input sub-feature maps.

[0127] The feature extraction processing of the image to be processed by the feature extraction core may refer to an addition operation or a subtraction operation.

[0128] The addition operation means that the feature extraction kernel is added to the elements at the corresponding position in the input image to obtain a target matrix, and the absolute values ​​of the elements in the target matrix are taken and then summed.

[0129] Optionally, if the target matrix is ​​obtained by subtracting the elements at the corresponding positions in the feature extraction kernel and the input image, the feature extraction processing operation can also be called a subtraction operation. In this case, the absolute values ​​of the elements in the target matrix are taken and then summed to obtain the result of the feature extraction operation. In other words, in the embodiment of the present application, the target matrix is ​​obtained by performing matrix addition or matrix subtraction between the corresponding input sub-feature map and the feature extraction kernel.

[0130] Step 702: Perform feature fusion processing on the input feature map through the target network to obtain an output feature map.

[0131] Wherein, the target network is an additive neural network. The convolution process performed by the feature extraction kernel in the target network for performing feature extraction process is the above-mentioned addition operation or subtraction operation.

[0132] The target network includes a plurality of branches for performing feature fusion processing. The plurality of branches include a first branch and a second branch, the first branch is used to perform feature extraction processing on the first input sub-feature map to obtain a first sub-feature map and a first output sub-feature map, and the first sub-feature map is an intermediate feature obtained by the first branch.

[0133] Optionally, when the first branch is the lowest layer branch, the first sub-feature map obtained by the first branch may be a feature map obtained by performing feature extraction processing on the first input sub-feature map through a feature extraction core; the first sub-feature map may also be a feature map obtained by the first branch without performing feature extraction processing, that is, the first sub-feature map may be the same as the first input sub-feature map. When the first branch is not the lowest layer branch, the first sub-feature map obtained by the first branch may be obtained by fusing the first input sub-feature map with feature maps transmitted by other branches.

[0134] The second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, and the output feature map includes the first output sub-feature map and the second output sub-feature map.

[0135] Optionally, the second branch performs feature extraction processing on the second input sub-feature map and the first sub-feature map, which may be to add the second input sub-feature map and the first sub-feature map after downsampling processing to obtain the second sub-feature map, and perform feature extraction processing on the second sub-feature map to obtain the second output sub-feature map. Since the resolution of the first sub-feature map obtained by the first branch is greater than the second input sub-feature map in the second branch, the first sub-feature map may be downsampled so that the resolution of the first sub-feature map is the same as that of the second input sub-feature map. Then, the second input sub-feature map and the first sub-feature map after downsampling processing are added to obtain the second sub-feature map.

[0136] That is to say, for the first branch processing the first input sub-feature map with a higher resolution, the first branch passes the intermediate features obtained by processing the first input sub-feature map to the second branch processing the second input sub-feature map with a lower resolution. That is, the low-level features are fused to the high-level features in a low-to-high-level manner, thereby compensating for the sparsity of the high-level features processed in the second branch.

[0137] It is understandable that, in addition to the first branch and the second branch, the target network may also include other branches for performing feature fusion, such as the third branch, the fourth branch, etc. In practical applications, the number of branches in the target network can be determined according to actual needs, for example, the target network includes 3-7 branches, and this embodiment does not specifically limit the number of branches included in the target network.

[0138] Exemplarily, when the target network further includes a third branch, the input feature map further includes a third input sub-feature map, and the third branch is used to perform feature extraction processing on the third input sub-feature map and the intermediate features obtained by processing the second branch to obtain a third output sub-feature map. In other words, the second branch also passes the intermediate features obtained by processing the second input sub-feature map to the third branch, further compensating for the sparsity of the high-level features processed in the third branch.

[0139] For example, see Figure 8 , Figure 8 A schematic diagram of the structure of a target network provided in an embodiment of the present application. Figure 8 As shown, the target network includes a first branch, a second branch, a third branch and a fourth branch. The inputs of the first branch, the second branch, the third branch and the fourth branch are the first input sub-feature graph, the second input sub-feature graph, the third input sub-feature graph and the fourth input sub-feature graph respectively. Figure 8In the figure, the feature extraction cores in the dotted box (i.e., feature extraction core 1, feature extraction core 2, feature extraction core 3, and feature extraction core 4) represent optional feature extraction cores. In practical applications, the target network may include the feature extraction cores in the dotted box, or the target network may not include the feature extraction cores in the dotted box.

[0140] In the first branch, the first branch may perform convolution processing on the first input sub-feature map through feature extraction core 1 to obtain the first sub-feature map, and pass the first sub-feature map to the second branch. Then, the first branch continues to perform convolution processing on the first sub-feature map through feature extraction core 2 to obtain the first output sub-feature map. Optionally, the first branch may also directly pass the first input sub-feature map to the second branch, and perform convolution processing on the first input sub-feature map through feature extraction core 2 to obtain the first output sub-feature map.

[0141] In the second branch, the second branch may be to perform convolution processing on the second input sub-feature map through the feature extraction core 3, add the second input sub-feature map after the feature extraction processing and the first sub-feature map after the downsampling processing to obtain the second sub-feature map, and pass the first sub-feature map to the third branch. Then, the third branch continues to perform convolution processing on the second sub-feature map through the feature extraction core 4 to obtain the second output sub-feature map. Optionally, the second branch may also directly add the second input sub-feature map and the first sub-feature map after the downsampling processing to obtain the second sub-feature map, and perform convolution processing on the obtained second sub-feature map through the feature extraction core 4 to obtain the second output sub-feature map.

[0142] Similarly, the third branch obtains the features transmitted by the second branch, and performs feature fusion processing to obtain a third output sub-feature map. The fourth branch obtains the features transmitted by the third branch, and performs feature fusion processing to obtain a fourth output sub-feature map. The processing of the third branch and the fourth branch will not be repeated here.

[0143] In a possible embodiment, a skip connection structure may be introduced into a branch of the target network to enhance the feature learning capability of the target network. The skip connection structure introduced into the target network is introduced below by taking the second branch as an example.

[0144] For example, see Fig. 9 , Fig. 9A schematic diagram of a skip connection structure provided in an embodiment of the present application. The second branch performs feature extraction processing on the second sub-feature map, including: the second branch processes the second sub-feature map through a convolution layer (e.g., an additive filter layer) and an activation function to obtain a third sub-feature map, and the convolution layer may include one or more feature extraction kernels. The second branch adds the second sub-feature map and the third sub-feature map to obtain the second output sub-feature map. That is, in Fig. 9 In the embodiment, the skip connection structure includes a convolutional layer and an activation function. The second sub-feature map is processed by the skip connection structure to obtain a second output sub-feature map.

[0145] In this embodiment, in view of the defects in the expressive power of the additive feature extraction kernel itself, a skip connection structure is designed to improve the expressive power of the convolutional units in the target network, thereby enhancing the feature learning ability of the target network.

[0146] For example, for the feature extraction kernel in the multiplication neural network (i.e., the multiplication feature extraction kernel), the input and output of the multiplication feature extraction kernel can be made the same by involving the weight of the multiplication feature extraction kernel. However, for the additive feature extraction kernel, the input and output of the additive feature extraction kernel must be different. Therefore, compared with the multiplication neural network, the expression ability of the additive feature extraction kernel has certain defects, which easily leads to the poor feature learning ability of the additive neural network.

[0147] After the skip connection structure is introduced, the input of the skip connection structure can be set to 0 through the feature extraction kernel and activation function in the skip connection structure, so that the final output of the skip connection structure is the same as the input of the skip connection structure. Fig. 9 In the example, by adjusting the weights of the convolution layer and the activation function, the third sub-feature map obtained by processing the second sub-feature map through the convolution layer and the activation function can be made to be 0. In this way, after adding the second sub-feature map to the third sub-feature map, the output second output sub-feature map is the same as the second sub-feature map.

[0148] In a possible embodiment, the second branch is also used to perform feature extraction processing on the second sub-feature map to obtain a fourth sub-feature map, and the fourth sub-feature map is the intermediate feature obtained by the second branch. Wherein, when the second branch is not the branch of the highest layer, the second branch performs feature extraction processing on the second sub-feature map, which may be: the second branch performs processing on the second sub-feature map through a feature extraction core, and adds the second sub-feature map after the feature extraction processing to the feature map transmitted by the upper layer branch and after the upsampling processing, to obtain the fourth sub-feature map. When the second branch is the branch of the highest layer, the second branch performs feature extraction processing on the second sub-feature map, which may be: the second branch performs processing on the second sub-feature map through a feature extraction core, to obtain the fourth sub-feature map.

[0149] The first branch performs feature extraction processing on the first input sub-feature map, including: the first branch performs feature extraction processing on the first sub-feature map to obtain the first sub-feature map; the first branch adds the first sub-feature map and the fourth sub-feature map after upsampling processing to obtain a fifth sub-feature map; the first branch performs feature extraction processing on the fifth feature map to obtain the first output sub-feature map.

[0150] For example, see Fig.10 , Fig.10 A schematic diagram of a network architecture provided in an embodiment of the present application. Fig.10 As shown, the first branch and the second branch both include three feature extraction cores. In the first branch, the first input sub-feature map is processed by feature extraction core 1, and the processed first sub-feature map is passed to the second branch; the second branch processes the second input sub-feature map by feature extraction core 4, and adds the processed second input sub-feature map to the first sub-feature map after downsampling to obtain the second sub-feature map. The second branch processes the second sub-feature map by feature extraction core 5 to obtain the fourth sub-feature map, and passes the fourth sub-feature map to the first branch. The first branch processes the first sub-feature map by feature extraction core 2, and adds the processed first sub-feature map to the fourth sub-feature map passed by the second branch to obtain the fifth sub-feature map. Finally, the first branch processes the fifth sub-feature map by feature extraction core 3 to obtain the first output sub-feature map. The second branch processes the fourth sub-feature map by feature extraction core 6 to obtain the second output sub-feature map.

[0151] In this embodiment, the low-level features are first fused into the high-level features from the low-level to the high-level, and then the processed high-level features are fused into the low-level features, thereby introducing richer semantic information into the low-level features, improving the feature fusion effect, and further improving the target detection accuracy of the network.

[0152] Optionally, a skip connection structure may also be introduced into the first branch to enhance the feature learning capability of the first branch.

[0153] Exemplarily, the first branch performs feature extraction processing on the fifth feature map to obtain the first output sub-feature map, including: the first branch processes the fifth sub-feature map through a convolution layer and an activation function to obtain a sixth sub-feature map; the first branch adds the fifth sub-feature map and the sixth sub-feature map to obtain the first output sub-feature map.

[0154] Step 703: Perform target detection on the image to be processed according to the output feature map to obtain a target detection result.

[0155] Optionally, after obtaining the output feature map, the output feature map can be processed by a detection head to obtain a target detection result. The detection head can be an additive neural network, which includes one or more additive filter layers, each of which can include one or more feature extraction cores. The detection head processes the output feature map through the additive filter layer to obtain a final target detection result.

[0156] For example, see Fig.11 , Fig.11 A schematic diagram of the structure of a target network provided in an embodiment of the present application. Fig.11 As shown in Figure 2, the target network includes multiple branches, each of which includes a feature extraction core, two addition structures, and two skip connection structures, where the activation function in the skip connection structure is Fig.11 For one of the branches in the target network, after the branch processes the input sub-feature map through the feature extraction core, the obtained sub-feature map is added to the sub-feature map transmitted by the lower branch and processed by downsampling, and the added sub-feature map is obtained. Figure 1 The added sub-feature map is processed by the first skip connection structure and then added to the sub-feature map transmitted by the upper branch and processed by upsampling to obtain the added sub-feature map. Figure 2 Finally, the added sub-features Figure 2 After being processed by the second skip connection structure, the output sub-feature map is obtained.

[0157] For example, see Fig.12 , Fig.12 A schematic diagram of the structure of a target detection network provided in an embodiment of the present application. Fig.12 As shown, the target detection network includes a backbone network, a target network and a detection head. The backbone network is used to process the image to be processed to obtain an input sub-feature map with decreasing resolution. Then, the input sub-feature maps with different resolutions are input into different branches of the target network, and the branches in the target network implement multi-scale feature fusion to obtain multiple output sub-feature maps. Finally, multiple detection heads process the multiple output sub-feature maps respectively to obtain corresponding sub-target detection results. The sub-target detection results obtained by different detection heads perform mutual processing, such as performing deduplication processing (i.e. removing the detected duplicate targets) to obtain the final target detection result.

[0158] It is understandable that in the training process of the target detection network, it is usually necessary to obtain the gradient of the input (that is, the gradient of the output relative to the input), and optimize the target detection network according to the obtained gradient when using the back propagation algorithm, and finally obtain a trained target detection network.

[0159] In a possible embodiment, the gradient of the input feature map may be determined by the difference between the weight of the target network and the input feature map. Exemplarily, the gradient of the input feature map may be obtained by performing a sign-taking operation on the difference between the weight of the target network and the input feature map. In short, for any feature extraction kernel in the target network, the gradient of the input of the feature extraction kernel may be obtained by performing a sign-taking operation on the difference between the weight of the feature extraction kernel and the input of the feature extraction kernel.

[0160] Specifically, for the target network as an additive neural network, the output Y of the feature extraction core in the target network is calculated by the input X of the feature extraction core and the weight F of the feature extraction core. Exemplarily, the process of obtaining the output Y of the feature extraction core in the target network is shown in Formula 1.

[0161]

[0162] Wherein, |(·)| represents the absolute value operation, Σ(·) represents the summation operation, Y(m,n,t) is the feature map output by the feature extraction kernel, Y(m,n,t) is the element of the mth row, nth column and tth page in the feature map output by the feature extraction kernel, X(m+i,n+j,k) is the element of the ith row, jth column and kth page in the input feature map of the feature extraction kernel, F(i,j,k,t) is the element of the ith row, jth column and kth page in the feature extraction kernel, t is the number of channels of the feature extraction kernel, d is the number of rows of the feature extraction kernel, Cin is the number of channels of the input feature map, and d, Cin, i, j, k, m, n, t are all integers.

[0163] Specifically, the gradient of the input of the feature extraction kernel can be obtained based on Formula 2.

[0164]

[0165] in, is the gradient of the input of the feature extraction kernel, sign(.) represents the sign-taking operation, F is the weight of the feature extraction kernel, and X is the input of the feature extraction kernel. For the sign-taking operation represented by sign(.), when the value of FX is negative, the value obtained after the sign-taking operation is -1; when the value of FX is not negative, the value obtained after the sign-taking operation is 1.

[0166] Since there are usually multiple branches and multiple loss functions in an additive neural network, such as the regression prediction loss function and classification loss function that include the detection box position, there will be a large cumulative error when the gradient obtained by the traditional gradient calculation method is used for network optimization. However, the gradient calculation method provided in the embodiment of the present application can obtain a more accurate gradient, which is conducive to improving the final target detection accuracy.

[0167] For example, the traditional gradient calculation method usually uses the HardTanh function to obtain the gradient, as shown in Formula 3.

[0168]

[0169] in, is the gradient of the input of the feature extraction kernel, HT(.) represents the sign-taking operation, F is the weight of the feature extraction kernel, and X is the input of the feature extraction kernel. Specifically,

[0170] Assume that there is a two-layer additive neural network, the input is X, and after one additive filter, X1 is obtained. X1 is subjected to two additive filters to obtain Y1 and Y2. Among them, X1, Y1 and Y2 are specifically shown in Formula 4.

[0171] X 1 =-‖XF 1 ‖,

[0172] Y 1 =-||X 1 -F 2 ||,

[0173] Y 2 =-||X 1 -F 3 ||, Formula 4

[0174] Assuming that the losses calculated for outputs Y1 and Y2 are L1 and L2 respectively, the total loss is L=L1+L2, and the gradient of L with respect to F1 is specifically shown in Formula 5.

[0175]

[0176] If the traditional gradient calculation method is used to calculate the gradient, the gradient of F1 is specifically as shown in Formula 6.

[0177]

[0178] However, combined with Formula 5, it can be seen that the actual gradient of F1 should be as shown in Formula 7.

[0179]

[0180] In this case, there is a large error between the gradient obtained by the traditional gradient calculation method and the actual gradient, which can easily affect the final target detection accuracy.

[0181] In order to verify the beneficial effects of the image processing method provided in the embodiment of the present application, the image processing method was verified on the standard COCO dataset in the embodiment of the present application.

[0182] See also Fig.13 , Fig.13 A schematic diagram of a comparison of network detection results provided in an embodiment of the present application. Fig.13 As shown, the gradient calculation method (L1 Gradients) for additive neural network proposed in this embodiment can bring about a 0.4 average accuracy (mean average precision, mAP) improvement. Among them, mAP is an accuracy measurement indicator in the field of object detection.

[0183] See also Fig.14 , Fig.14 Another comparative schematic diagram of the detection results of a network provided in an embodiment of the present application. Fig.14 As shown, the target network (R-PAFPN) proposed in this embodiment can bring an improvement of 1.6-1.7mAP compared with the traditional feature fusion network (FPN and PAFPN) structure.

[0184] See also Fig.15 , Fig.15 The following is a schematic diagram of a theoretical power consumption comparison of a network provided in an embodiment of the present application. Fig.15 As shown, the target detection network (AdderDet) including the above-mentioned target network proposed in this embodiment can achieve 37.8mAP on the COCO dataset, which is only 1.2mAP lower than the standard multiplication convolutional neural network structure (FCOS), but the amount of multiplication calculation is greatly reduced, so the theoretical power consumption can be greatly reduced.

[0185] See also Fig.16 , Fig.16 A schematic diagram of performing target detection on an image provided by an embodiment of the present application. Fig.16 As shown, the image processing method proposed in this embodiment can obtain better detection results.

[0186] See also Fig.17 , Fig.17 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig.17As shown, an image processing device provided by an embodiment of the present application includes: an acquisition unit 1701 and a processing unit 1702; the acquisition unit 1701 is used to acquire an input feature map of an image to be processed, the input feature map includes a first input sub-feature map and a second input sub-feature map, and the resolution of the first input sub-feature map is higher than the resolution of the second input sub-feature map; the processing unit 1702 is used to perform feature fusion processing on the input feature map through a target network to obtain an output feature map, wherein the target network is an additive neural network, the target network includes a first branch and a second branch, the first branch is used to perform feature extraction processing on the first input sub-feature map to obtain a first sub-feature map and a first output sub-feature map, the first sub-feature map is an intermediate feature obtained by the first branch, and the second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, and the output feature map includes the first output sub-feature map and the second output sub-feature map; the processing unit 1702 is also used to perform target detection on the image to be processed according to the output feature map to obtain a target detection result.

[0187] Optionally, in a possible implementation, the second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, including: the second branch is used to add the second input sub-feature map and the first sub-feature map after downsampling processing to obtain a second sub-feature map, and perform feature extraction processing on the second sub-feature map to obtain the second output sub-feature map.

[0188] Optionally, in a possible implementation, the second branch performs feature extraction processing on the second sub-feature map, including: the second branch processes the second sub-feature map through a convolution layer and an activation function to obtain a third sub-feature map; the second branch adds the second sub-feature map and the third sub-feature map to obtain the second output sub-feature map.

[0189] Optionally, in a possible implementation, the second branch is also used to perform feature extraction processing on the second sub-feature map to obtain a fourth sub-feature map, and the fourth sub-feature map is an intermediate feature obtained by the second branch; the first branch performs feature extraction processing on the first input sub-feature map, including: the first branch performs feature extraction processing on the first sub-feature map to obtain the first sub-feature map; the first branch adds the first sub-feature map and the fourth sub-feature map after upsampling processing to obtain a fifth sub-feature map; the first branch performs feature extraction processing on the fifth feature map to obtain the first output sub-feature map.

[0190] Optionally, in a possible implementation, the first branch performs feature extraction processing on the fifth feature map to obtain the first output sub-feature map, including: the first branch processes the fifth sub-feature map through a convolution layer and an activation function to obtain a sixth sub-feature map; the first branch adds the fifth sub-feature map and the sixth sub-feature map to obtain the first output sub-feature map.

[0191] Optionally, in a possible implementation, the gradient of the input feature map is determined by a difference between a weight of the target network and the input feature map.

[0192] Optionally, in a possible implementation, the gradient of the input feature map is obtained by performing a sign operation on the difference between the weight of the target network and the input feature map.

[0193] Next, an execution device provided by an embodiment of the present application is introduced. Fig.18 , Fig.18 This is a schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device 1800 can be specifically a mobile phone, a tablet, a laptop, a smart wearable device, a server, etc., which is not limited here. Among them, the execution device 1800 can be deployed with Fig.18 The data processing device described in the corresponding embodiment is used to implement Fig.18 The data processing function in the corresponding embodiment. Specifically, the execution device 1800 includes: a receiver 1801, a transmitter 1802, a processor 1803 and a memory 1804 (wherein the number of the processor 1803 in the execution device 1800 can be one or more, Fig.18 In the example of FIG. 1803 , the processor 1803 may include an application processor 18031 and a communication processor 18032. In some embodiments of the present application, the receiver 1801, the transmitter 1802, the processor 1803 and the memory 1804 may be connected via a bus or other means.

[0194] The memory 1804 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1803. A portion of the memory 1804 may also include a non-volatile random access memory (NVRAM). The memory 1804 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0195] The processor 1803 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0196] The method disclosed in the above embodiment of the present application can be applied to the processor 1803, or implemented by the processor 1803. The processor 1803 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1803. The above processor 1803 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1803 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1804, and the processor 1803 reads the information in the memory 1804 and completes the steps of the above method in combination with its hardware.

[0197] The receiver 1801 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1802 can be used to output digital or character information through the first interface; the transmitter 1802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1802 can also include a display device such as a display screen.

[0198] In one embodiment of the present application, the processor 1803 is configured to execute Figure 7 The image processing method executed by the execution device in the corresponding embodiment.

[0199] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0200] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0201] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the image processing method described in the above embodiment, or so that the chip in the training device executes the image processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0202] For details, please refer to Fig.19 , Fig.19 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 1900, NPU 1900 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1903, which is controlled by the controller 1904 to extract matrix data from the memory and perform multiplication operations.

[0203] In some implementations, the operation circuit 1903 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1903 is a two-dimensional systolic array. The operation circuit 1903 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1903 is a general-purpose matrix processor.

[0204] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1902 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1901 and performs matrix operations with matrix B. The partial results or final results of the obtained matrix are stored in the accumulator 1908.

[0205] The unified memory 1906 is used to store input data and output data. The weight data is directly transferred to the weight memory 1902 through the direct memory access controller (DMAC) 1905. The input data is also transferred to the unified memory 1906 through the DMAC.

[0206] BIU stands for Bus Interface Unit 1913 , which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1909 .

[0207] The bus interface unit 1913 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1909 to obtain instructions from the external memory, and is also used for the storage unit access controller 1905 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0208] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1906 or to transfer weight data to the weight memory 1902 or to transfer input data to the input memory 1901.

[0209] The vector calculation unit 1907 includes multiple operation processing units, and further processes the output of the operation circuit 1903 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0210] In some implementations, the vector calculation unit 1907 can store the processed output vector to the unified memory 1906. For example, the vector calculation unit 1907 can apply a linear function; or a nonlinear function to the output of the operation circuit 1903, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1907 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1903, for example, for use in a subsequent layer in a neural network.

[0211] An instruction fetch buffer 1909 connected to the controller 1904 is used to store instructions used by the controller 1904;

[0212] Unified memory 1906, input memory 1901, weight memory 1902 and instruction fetch memory 1909 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0213] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0214] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0215] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0216] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0217] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. An image processing method, characterized in that: include: Acquire an input feature map of an image to be processed, wherein the input feature map includes a first input sub-feature map and a second input sub-feature map, and a resolution of the first input sub-feature map is higher than a resolution of the second input sub-feature map; Performing feature fusion processing on the input feature map through a target network to obtain an output feature map, wherein the target network is an additive neural network, the target network includes a first branch and a second branch, the first branch is used to perform feature extraction processing on the first input sub-feature map to obtain a first sub-feature map and a first output sub-feature map, the first sub-feature map is an intermediate feature obtained by the first branch, the second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, and the output feature map includes the first output sub-feature map and the second output sub-feature map; Performing target detection on the image to be processed according to the output feature map to obtain a target detection result.

2. The method according to claim 1, characterized in that The second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, including: The second branch is used to add the second input sub-feature map and the first sub-feature map after downsampling processing to obtain a second sub-feature map, and perform feature extraction processing on the second sub-feature map to obtain the second output sub-feature map.

3. The method according to claim 2, characterized in that The second branch performs feature extraction processing on the second sub-feature map, including: The second branch processes the second sub-feature map through a convolution layer and an activation function to obtain a third sub-feature map; The second branch adds the second sub-feature map and the third sub-feature map to obtain the second output sub-feature map.

4. The method according to claim 2 or 3, characterized in that: The second branch is further used to perform feature extraction processing on the second sub-feature map to obtain a fourth sub-feature map, where the fourth sub-feature map is an intermediate feature obtained by the second branch; The first branch performs feature extraction processing on the first input sub-feature map, including: The first branch performs feature extraction processing on the first sub-feature map to obtain the first sub-feature map; The first branch adds the first sub-feature map and the fourth sub-feature map after upsampling to obtain a fifth sub-feature map; The first branch performs feature extraction processing on the fifth sub-feature map to obtain the first output sub-feature map.

5. The method according to claim 4, characterized in that The first branch performs feature extraction processing on the fifth sub-feature map to obtain the first output sub-feature map, including: The first branch processes the fifth sub-feature map through a convolution layer and an activation function to obtain a sixth sub-feature map; The first branch adds the fifth sub-feature map and the sixth sub-feature map to obtain the first output sub-feature map.

6. The method according to any one of claims 1 to 3, characterized in that: The gradient of the input feature map is determined by the difference between the weights of the target network and the input feature map.

7. The method according to claim 6, characterized in that The gradient of the input feature map is obtained by performing a sign operation on the difference between the weight of the target network and the input feature map.

8. An image processing device, characterized in that: include: Acquisition unit and processing unit; The acquisition unit is used to acquire an input feature map of the image to be processed, the input feature map includes a first input sub-feature map and a second input sub-feature map, and the resolution of the first input sub-feature map is higher than the resolution of the second input sub-feature map; The processing unit is used to perform feature fusion processing on the input feature map through a target network to obtain an output feature map, wherein the target network is an additive neural network, the target network includes a first branch and a second branch, the first branch is used to perform feature extraction processing on the first input sub-feature map to obtain a first sub-feature map and a first output sub-feature map, the first sub-feature map is an intermediate feature obtained by the first branch, the second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, and the output feature map includes the first output sub-feature map and the second output sub-feature map; The processing unit is also used to perform target detection on the image to be processed according to the output feature map to obtain a target detection result.

9. The device according to claim 8, characterized in that The second branch is used to perform feature extraction processing on the second input sub-feature map and the first sub-feature map to obtain a second output sub-feature map, including: The second branch is used to add the second input sub-feature map and the first sub-feature map after downsampling processing to obtain a second sub-feature map, and perform feature extraction processing on the second sub-feature map to obtain the second output sub-feature map.

10. The device according to claim 9, characterized in that The second branch performs feature extraction processing on the second sub-feature map, including: The second branch processes the second sub-feature map through a convolution layer and an activation function to obtain a third sub-feature map; The second branch adds the second sub-feature map and the third sub-feature map to obtain the second output sub-feature map.

11. The device according to claim 8 or 9, characterized in that The second branch is further used to perform feature extraction processing on the second sub-feature map to obtain a fourth sub-feature map, where the fourth sub-feature map is an intermediate feature obtained by the second branch; The first branch performs feature extraction processing on the first input sub-feature map, including: The first branch performs feature extraction processing on the first sub-feature map to obtain the first sub-feature map; The first branch adds the first sub-feature map and the fourth sub-feature map after upsampling to obtain a fifth sub-feature map; The first branch performs feature extraction processing on the fifth sub-feature map to obtain the first output sub-feature map.

12. The device according to claim 11, characterized in that The first branch performs feature extraction processing on the fifth sub-feature map to obtain the first output sub-feature map, including: The first branch processes the fifth sub-feature map through a convolution layer and an activation function to obtain a sixth sub-feature map; The first branch adds the fifth sub-feature map and the sixth sub-feature map to obtain the first output sub-feature map.

13. The device according to any one of claims 8 to 10, characterized in that: The gradient of the input feature map is determined by the difference between the weights of the target network and the input feature map.

14. The device according to claim 13, characterized in that The gradient of the input feature map is obtained by performing a sign operation on the difference between the weight of the target network and the input feature map.

15. An image processing device, characterized in that: The image processing apparatus comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the image processing apparatus executes the method according to any one of claims 1 to 7.

16. A computer storage medium, characterized in that: The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 7.

17. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 7.

Images