Feature map processing method and apparatus, storage medium, and electronic device

Abstract

The application discloses a feature map processing method and device, a storage medium and an electronic device. The method comprises the following steps: obtaining an original input feature map in a target convolutional neural network, wherein the dimension of the original input feature map is c x h x w, c is the channel number of the original input feature map, h is the feature map height of the original input feature map, w is the feature map width of the original input feature map, c, h and w are positive integers greater than or equal to 1; performing a 1x1 convolution operation on the original input feature map in the dimensions of the channel number, the feature map height and the feature map width to obtain a first target output feature map, wherein the dimension of the first target output feature map is c x h x w; and performing a target operation in the target convolutional neural network on the first target output feature map. The application solves the technical problem of a large amount of calculation in the convolution processing of the feature map in the related art.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and more specifically, to a method and apparatus for processing feature maps, a storage medium, and an electronic device. Background Technology

[0002] In pattern recognition tasks, machine learning methods are typically used to extract features from the target object, and then the recognition result is output based on the feature comparison. For example, in image processing, a convolutional neural network is used to convolve the feature map of the original image, and the recognition result is output based on the information received by a neuron on the convolutional feature map.

[0003] In related technologies, convolution processing involves fixing the convolution kernel along the channel dimension of the feature map and then sliding it along the height and width dimensions of the feature map. Assuming the kernel size is k*k, it can only acquire information within a k*k range of the previous feature map at a time, resulting in very limited image information. However, in many image recognition processes, sufficient contextual information is often required to identify the target image within the original image. The above convolution processing method cannot meet these requirements.

[0004] To address the aforementioned issues, related technologies typically employ methods such as increasing the size of the convolutional kernel or stacking convolutional layers to acquire more contextual information. For example, when the image information obtained using a 3×3 convolutional kernel is insufficient, the kernel is enlarged to 5×5 or 7×7; or multiple convolutions (stacked convolutional layers) are performed using a 3×3 kernel to expand the acquired contextual information. However, while both methods satisfy the requirement of acquiring a certain range of information, they undoubtedly increase the computational load of the network, resulting in the technical problem of high computational cost during feature map convolution processing.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a method and apparatus for processing feature maps, a storage medium, and an electronic device, to at least solve the technical problem of high computational load in the process of convolution processing of feature maps in related technologies.

[0007] According to one aspect of the present invention, a method for processing a feature map is provided, comprising: obtaining an original input feature map in a target convolutional neural network, wherein the original input feature map has a dimension of c×h×w, c is the number of channels in the original input feature map, h is the feature map height of the original input feature map, w is the feature map width of the original input feature map, and c, h, and w are positive integers greater than or equal to 1; performing a 1×1 convolution operation on the original input feature map in the dimensions of the number of channels, the feature map height, and the feature map width to obtain a first target output feature map, wherein the first target output feature map has a dimension of c×h×w; and performing a target operation in a target convolutional neural network on the first target output feature map.

[0008] Optionally, the above-described 1×1 convolution operation on the original input feature map in the dimensions of channel number, feature map height, and feature map width to obtain the first target output feature map includes: transposing the original input feature map into a first input feature map, wherein the dimension of the first input feature map is h×c×w; using a first set of convolution kernels to convolve the first input feature map in the dimension of feature map height to obtain a first output feature map, wherein the dimension of each convolution kernel in the first set of convolution kernels is h×1×1, the number of convolution kernels in the first set of convolution kernels is h, and the dimension of the first output feature map is h×c×w; transposing the first output feature map into a second input feature map, wherein the second input feature map... The dimensions of the first input feature map are w×c×h. A second set of convolutional kernels is used to convolve the second input feature map along the width dimension to obtain the second output feature map. Each convolutional kernel in the second set has a dimension of w×1×1, and the number of kernels in the second set is w. The dimensions of the second output feature map are w×c×h. The second output feature map is transposed into a third input feature map, with dimensions c×h×w. A third set of convolutional kernels is used to convolve the third input feature map along the channel number dimension to obtain the first target output feature map. Each convolutional kernel in the third set has a dimension of c×1×1, and the number of kernels in the third set is c.

[0009] Optionally, the above-described 1×1 convolution operation on the original input feature map in the dimensions of channel number, feature map height, and feature map width to obtain the first target output feature map includes: transposing the original input feature map into a first input feature map, wherein the dimension of the first input feature map is h×c×w; convolving the first input feature map with a first set of convolution kernels in the dimension of feature map height to obtain a first output feature map, wherein each convolution kernel in the first set of convolution kernels has a dimension of h×1×1, the number of convolution kernels in the first set of convolution kernels is h, and the dimension of the first output feature map is h×c×w; transposing the original input feature map into a second input feature map, wherein the dimension of the second input feature map is w×c×h; and convolving the second input feature map with a second set of convolution kernels in the dimension of feature map width to obtain a second output feature map, wherein the second set of convolution kernels... Each convolutional kernel in the first set has a dimension of w×1×1, the number of convolutional kernels in the second set is w, and the dimension of the second output feature map is w×c×h. The original input feature map is convolved using a third set of convolutional kernels in the dimension of the number of channels to obtain a third output feature map. Each convolutional kernel in the third set has a dimension of c×1×1, the number of convolutional kernels in the third set is c, and the dimension of the third output feature map is c×h×w. The first output feature map is transposed into a first transposed feature map with a dimension of c×h×w, and the second output feature map is transposed into a second transposed feature map with a dimension of c×h×w. The first transposed feature map, the second transposed feature map, and the third output feature map are concatenated to form a first concatenated feature map, which has a dimension of 3c×h×w. The first concatenated feature map is then subjected to dimensionality reduction to obtain the first target output feature map.

[0010] Optionally, the above-mentioned dimensionality reduction processing of the first spliced ​​feature map to obtain the first target output feature map includes: using a fourth set of convolutional kernels to convolve the first spliced ​​feature map in the dimension of the number of channels to obtain the first target output feature map, wherein the dimension of each convolutional kernel in the fourth set of convolutional kernels is 3c×1×1, and the number of convolutional kernels in the fourth set of convolutional kernels is c.

[0011] Optionally, the above-mentioned target operation in the target convolutional neural network is performed on the first target output feature map, including: performing a superposition operation on the original input feature map and the first target output feature map.

[0012] Optionally, the above-mentioned superposition operation on the original input feature map and the first target output feature map includes: convolving the original input feature map with a third set of convolutional kernels in the dimension of the number of channels to obtain a second target output feature map, wherein each convolutional kernel in the third set of convolutional kernels has a dimension of c×1×1, the number of convolutional kernels in the third set of convolutional kernels is c, and the dimension of the second target output feature map is c×h×w; concatenating the first target output feature map and the second target output feature map to form a second concatenated feature map, wherein the dimension of the second concatenated feature map is 2c×h×w; and performing dimensionality reduction processing on the second concatenated feature map to obtain a third target output feature map, wherein the dimension of the third target output feature map is c×h×w.

[0013] Optionally, the above-mentioned dimensionality reduction processing of the second spliced ​​feature map to obtain the third target output feature map includes: using the fifth group of convolution kernels to convolve the second spliced ​​feature map in the dimension of the number of channels to obtain the third target output feature map, wherein the dimension of each convolution kernel in the fifth group of convolution kernels is 2c×1×1, and the number of convolution kernels in the fifth group of convolution kernels is c.

[0014] Optionally, the above-mentioned superposition operation of the original input feature map and the first target output feature map further includes: adding the features of the original input feature map and the third target output feature map to obtain a fourth target output feature map, wherein the dimension of the fourth target output feature map is c×h×w.

[0015] Optionally, after performing the superposition operation on the original input feature map and the first target output feature map, the method further includes: performing a 1×1 convolution operation on the third target output feature map in the dimensions of the number of channels, the height of the feature map, and the width of the feature map to obtain a fourth target output feature map, wherein the dimension of the fourth target output feature map is c×h×w.

[0016] Optionally, the above feature map processing method further includes: using a third set of convolution kernels to convolve the third target output feature map in the dimension of the number of channels to obtain the fifth target output feature map, wherein the dimension of the fifth target output feature map is c×h×w; concatenating the fourth target output feature map and the fifth target output feature map to form a third concatenated feature map, wherein the dimension of the third concatenated feature map is 2c×h×w; and performing dimensionality reduction processing on the third concatenated feature map to obtain the sixth target output feature map, wherein the dimension of the sixth target output feature map is c×h×w.

[0017] Optionally, the above feature map processing method further includes: adding the features of the sixth target output feature map to the original input feature map to obtain the seventh target output feature map, wherein the dimension of the seventh target output feature map is c×h×w.

[0018] Optionally, the above feature map processing method further includes: when the original input feature map is the input feature map corresponding to the target image and the target convolutional neural network is used to identify the target object in the target image, the first target output feature map is used to represent the context information associated with the target object in the target image.

[0019] Optionally, the above feature map processing method further includes: when the original input feature map is the input feature map corresponding to the target image, the target convolutional neural network is used to identify the target object in the target image, and the first target output feature map is used to represent the context information associated with the target object in the target image, the feature map obtained by performing a superposition operation on the original input feature map and the first target output feature map is used to represent the information after fusing the target object's own information and context information.

[0020] According to another aspect of the present invention, a feature map processing apparatus is also provided, comprising: an acquisition unit, configured to acquire an original input feature map in a target convolutional neural network, wherein the original input feature map has a dimension of c×h×w, c is the number of channels of the original input feature map, h is the feature map height of the original input feature map, w is the feature map width of the original input feature map, and c, h, and w are positive integers greater than or equal to 1; a first processing unit, configured to perform a 1×1 convolution operation on the original input feature map in the dimensions of the number of channels, the feature map height, and the feature map width to obtain a first target output feature map, wherein the first target output feature map has a dimension of c×h×w; and a second processing unit, configured to perform a target operation in the target convolutional neural network on the first target output feature map.

[0021] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the above-described feature map processing method at runtime.

[0022] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute a processing method for the feature map via the computer program.

[0023] In this embodiment of the invention, by performing 1×1 convolution operations on the original input feature map in the dimensions of channel number, feature map height, and feature map width, information in each dimension is obtained, so that each feature point on the output feature map has global information. This solves the problem of large computational load caused by stacking a large number of convolutional layers to obtain more contextual information, and achieves the technical effect of enhancing the ability to obtain contextual information and reducing computational load. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 This is a schematic diagram illustrating an application scenario of an optional feature map processing method according to an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of an optional feature map processing method according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of an optional feature map processing method according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of a method for processing feature maps in a serial mode according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of a method for processing feature maps in parallel mode according to an embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram of an optional convolution operation on the original input feature map of dimension 3×5×5 in the dimension of the number of channels according to an embodiment of the present invention.

[0031] Figure 7 This is a schematic diagram of performing a convolution operation on the original input feature map with dimensions of 3×5×5 in the dimension of the number of channels in related technologies;

[0032] Figure 8 This is a schematic diagram of an optional dimensionality reduction process for the first spliced ​​feature map according to an embodiment of the present invention;

[0033] Figure 9 This is a flowchart of a first optional method for performing a superposition operation on the original input feature map and the first target output feature map according to an embodiment of the present invention;

[0034] Figure 10 This is a schematic diagram of an optional dimensionality reduction process for the first spliced ​​feature map according to an embodiment of the present invention;

[0035] Figure 11 This is a flowchart illustrating a second optional method for performing a superposition operation between the original input feature map and the first target output feature map according to an embodiment of the present invention.

[0036] Figure 12 This is a flowchart illustrating a third optional method for performing a superposition operation between the original input feature map and the first target output feature map according to an embodiment of the present invention.

[0037] Figure 13 This is a flowchart illustrating a fourth optional method for performing a superposition operation between the original input feature map and the first target output feature map according to an embodiment of the present invention.

[0038] Figure 14 This is a flowchart of a fifth optional method for performing a superposition operation on the original input feature map and the first target output feature map according to an embodiment of the present invention;

[0039] Figure 15 This is an overall flowchart of an optional feature map processing method according to an embodiment of the present invention;

[0040] Figure 16 This is a schematic diagram of the structure of an optional feature map processing device according to an embodiment of the present invention;

[0041] Figure 17 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] According to one aspect of the present invention, a method for processing feature maps is provided. As an optional implementation, the above-described feature map processing method may be applied, but is not limited to, to applications such as... Figure 1 The application scenarios shown are as follows. In, for example... Figure 1In the application scenario shown, terminal device 102 may be, but is not limited to, products such as mobile phones, in-vehicle systems, drones, and robots. Terminal device 102 may, but is not limited to, communicate with server 106 via network 104. Server 106 may, but is not limited to, perform operations on database 108, such as write data operations or read data operations. The aforementioned terminal device 102 may, but is not limited to, include a human-computer interaction screen, a processor, and a memory. The aforementioned human-computer interaction screen may, but is not limited to, be used to display the recognition results of the terminal device (e.g., ...). Figure 1 The image recognition result display interface shown above, along with the human-computer interaction screen, is also used to provide a human-computer interaction interface to receive human-computer interaction operations for users to use image processing software to recognize target objects. The processor may, but is not limited to, respond to the aforementioned human-computer interaction operations, execute corresponding operations, or generate corresponding instructions and send the generated instructions to the server 106. The memory is used to store relevant processing data, such as training samples, feature points, and feature point distribution maps.

[0045] As an optional approach, the following steps in the above feature map processing method can be executed on the terminal device 102: Step S102, obtain the original input feature map in the target convolutional neural network, wherein the dimension of the original input feature map is c×h×w; Step S104, perform a 1×1 convolution operation on the original input feature map in the dimensions of the number of channels, the height of the feature map, and the width of the feature map to obtain the first target output feature map; then, as in steps S106-108, the server 106 sends the first target output feature map to the terminal device 102 through the network 104; Step S110, the terminal device 102 receives the first target output feature map and performs the target operation in the target convolutional neural network on the first target output feature map.

[0046] By employing the above method, the problem of excessive computation caused by stacking a large number of convolutional layers to obtain more contextual information can be solved, achieving the technical effect of enhancing the ability to obtain contextual information and reducing computational load. In particular, due to the reduced computational load, even when the hardware computing power of the terminal device 102 is low, the above-described feature map processing method can still be executed. This allows the terminal device 102 to perform corresponding tasks using a convolutional neural network employing the above method, such as image recognition of images acquired by the terminal device 102, audio recognition of audio acquired by the terminal device 102, text recognition of text acquired by the terminal device 102, or various combinations of the above tasks.

[0047] As an optional example, this embodiment does not limit the execution subject of the above steps S102 to S110. For example, the above steps S102 to S110 can all be executed on server 106.

[0048] According to one aspect of the present invention, a method for processing feature maps is provided. Figure 2 This is a flowchart of a feature map processing method according to an embodiment of the present invention, the process including the following steps:

[0049] Step S202: Obtain the original input feature map in the target convolutional neural network. The original input feature map has a dimension of c×h×w, where c is the number of channels in the original input feature map, h is the feature map height of the original input feature map, and w is the feature map width of the original input feature map. c, h, and w can be positive integers greater than or equal to 1. Alternatively, c, h, and w can also be positive integers greater than or equal to 2.

[0050] As an optional example, the above-mentioned original input feature map includes, but is not limited to, feature values ​​obtained by transforming the pixel values ​​of each pixel in the original image, such as: Figure 3 As shown, the pixel values ​​of each pixel in image 302, which includes ships, seagulls, and seawater, are normalized to form a result like... Figure 4 The original input feature map is shown.

[0051] Step S204: Perform a 1×1 convolution operation on the original input feature map in the dimensions of the number of channels, the height of the feature map, and the width of the feature map to obtain the first target output feature map, wherein the dimensions of the first target output feature map are c×h×w.

[0052] As an optional approach, in step S204, by performing multi-dimensional convolution operations (which can be understood as multi-directional convolution) on the original input feature map, the receptive field corresponding to the feature points on the first target output feature map can be made larger, while generating less computation. The aforementioned receptive field can be understood as the range of information received by the feature points on the feature map in the input feature map (or understood as the "input image"). That is, the larger the receptive field, the more feature values ​​of each feature point on the output feature map are obtained by calculating the feature values ​​of more feature points in the input feature map.

[0053] Step S206: Perform target operations in the target convolutional neural network on the first target output feature map.

[0054] As an optional approach, in step S206, the aforementioned target operation may include, but is not limited to, performing a superposition operation on the original input feature map and the first target output feature map. It may also include performing other operations in the target convolutional neural network on the first target output feature map or the result of the superposition operation, such as operations performed on pooling layers, operations performed on fully connected layers, etc. Figure 3Taking the image recognition scenario shown as an example, in order to identify whether object 300 is a car or a boat in the input original image 302, the original input feature map corresponding to the original image 302 can be obtained through step S202, and the first target output feature map can be obtained through step S204. Then, through step S206, the pooling layer operation and the fully connected layer operation are performed on the first target output feature map output in step S204 to obtain the estimated recognition result, which indicates whether object 300 is a car or a boat. The target operation described above is only an example, and this embodiment does not limit it. Different target operations can be determined according to different network structures in the target convolutional neural network, or different tasks performed by the target convolutional neural network (e.g., object recognition, action recognition, prediction of subsequent actions, etc.).

[0055] In this embodiment, when performing convolution operations in a convolutional neural network, convolution operations can be performed in various dimensions, so that each feature point on the output feature map has a larger receptive field. For example, each feature point corresponds to global information, thereby solving the problem of large computational load caused by stacking a large number of convolutional layers to obtain more contextual information, and achieving the technical effect of enhancing the ability to obtain contextual information and reducing computational load.

[0056] As an alternative example, suppose a convolutional neural network is used to perform image recognition on objects in an image, specifically, such as Figure 3 As shown, the original image 302 includes target object 300, seagulls, seawater, the sun, etc., among which the identified target object is 300, and the target object 300 is quite similar in shape to a car. In this embodiment, the specific steps for image recognition of target object 300 are as follows:

[0057] S1, Input the original image 302 into the convolutional neural network 304;

[0058] S2, in convolutional neural network 304, performs the following sub-steps:

[0059] S21, transform the pixel values ​​corresponding to each pixel in the original image 302 to form an input feature map 306, wherein the dimension of the input feature map is c×h×w, c is the number of channels of the input feature map 306, h is the feature map height of the input feature map 306, and w is the feature map width of the input feature map 306.

[0060] S22, perform a 1×1 convolution operation on the input feature map 306 in the dimensions of the number of channels c, the height of the feature map h, and the width of the feature map w to obtain the output feature map 308. Each feature point on the output feature map 308 has a larger receptive field, for example, corresponding to the global information in the input feature map. In this way, the output feature map 308 can have (or represent) sufficient contextual information.

[0061] S3. Based on the output feature map 308, the target object 300 is identified, and the identification result "ship" is displayed on the interface of the terminal device.

[0062] It should be noted that in step S22 above, the contextual information obtained by performing 1×1 convolution operations on the input feature map 306 in multiple dimensions includes, but is not limited to, image information of objects other than the target object 300, such as seawater, seagulls, and sails. Using the contextual information obtained in this application, it can be identified that the target object 300 traveling in the seawater is not a "car" but a "boat".

[0063] In related technologies, the convolution kernel is typically fixed on the channel dimension of the input feature map 306 and then slides along the height and width dimensions of the feature map. This "one-way" convolution operation can only obtain the feature information of the target object 300, but cannot obtain sufficient contextual information. Furthermore, relying solely on the feature information of the target object 300, and only from the obtained feature information of the shape and wheels of the target object 300, the output recognition result might be "car".

[0064] Understandably, in related technologies, in order for the feature points on the output feature map 308 to receive more feature information from the original image 302, the receptive field is usually expanded by stacking convolutional layers, for example, as... Figure 7 As shown, after one 3×3 convolution, the receptive field size is 3×3. After two 3×3 convolutions, the receptive field expands to 5×5. The receptive field can be understood as the range of information received by a feature point on the feature map from the input feature map (or the "input image"). In other words, the larger the receptive field, the more feature values ​​of each feature point in the output feature map are derived from the feature values ​​of more feature points in the input feature map. Therefore, a larger receptive field allows for the acquisition of more contextual information.

[0065] However, it is easy to see that while performing "unidirectional" convolution on the 306-channel dimension of the input feature map in related techniques can expand the receptive field, it requires a large amount of computation. In addition, since the convolution kernel is fixed on the channel dimension of the feature map, only a limited amount of contextual information can be obtained through multiple convolution operations.

[0066] By utilizing the above embodiments provided in this application, a 1×1 convolution operation is performed on the number of channels, height, and width of the original input feature map, thereby obtaining a larger receptive field with less computation, that is, obtaining more contextual information, thus solving the technical problem of large computational load in the process of convolution processing of feature maps in related technologies.

[0067] As an optional implementation method, such as Figure 4 As shown, convolution operations (which can be understood as multi-directional convolution operations) can be performed on the original input feature map in multiple dimensions through a concatenation method. For example, the above-mentioned 1×1 convolution operation on the original input feature map in the dimensions of channel number, feature map height, and feature map width yields the first target output feature map, including:

[0068] The original input feature map is transposed into a first input feature map, where the dimension of the first input feature map is h×c×w; the first input feature map is convolved with a first set of convolutional kernels along the high dimension of the feature map to obtain a first output feature map, where each convolutional kernel in the first set of convolutional kernels has a dimension of h×1×1, the number of convolutional kernels in the first set of convolutional kernels is h, and the dimension of the first output feature map is h×c×w;

[0069] The first output feature map is transposed into a second input feature map, where the second input feature map has dimensions w×c×h. The second input feature map is convolved with a second set of convolutional kernels along the width dimension of the feature map to obtain a second output feature map, where each convolutional kernel in the second set of convolutional kernels has dimensions w×1×1, the number of convolutional kernels in the second set of convolutional kernels is w, and the dimension of the second output feature map is w×c×h.

[0070] The second output feature map is transposed into a third input feature map, where the dimension of the third input feature map is c×h×w; the third input feature map is convolved with the third set of convolution kernels in the dimension of the number of channels to obtain the first target output feature map, where the dimension of each convolution kernel in the third set of convolution kernels is c×1×1, and the number of convolution kernels in the third set of convolution kernels is c.

[0071] Specifically, such as Figure 4 As shown, the specific steps to obtain the first target output feature map are as follows:

[0072] (1) Convolution on the high dimension of the feature map

[0073] Assuming the original input feature map has dimensions c×h×w, transposing the original input feature map yields a first input feature map n with dimensions h×c×w. c1 The first set of convolutional kernels x1 is used to process the first input feature map n.c1 Convolution is performed along the height h dimension of the feature map to obtain, as follows: Figure 4 The first output feature map f shown in (a) c1 .

[0074] The first set of convolutional kernels x1 includes h convolutional kernels of dimension h×1×1. The first input feature map n is processed by the h h×1×1 convolutional kernels. c1 Convolution is performed along the height h dimension of the feature map to obtain h 1×c×w output features. Then, the h 1×c×w output feature maps are concatenated to obtain the following: Figure 4 The first output feature map f shown in (a) c1 , where f c1 The dimensions are h×c×w, and the region connected by the dashed lines in the figure is the first output feature map f. c1 The receptive field, that is, the first output feature map f c1 The range of information received by each feature point on the first input feature map.

[0075] It is easy to understand that by performing a convolution operation on the first input feature map using h convolution kernels with dimensions h×1×1, the first output feature map f is made more efficient. c1 Each feature point on the map contains information from the original input feature map in the higher dimension of the feature map.

[0076] (2) Convolution along the width dimension of the feature map

[0077] The first output feature map f obtained by convolution operation in step (1) c1 After transposing, we obtain the second input feature map n with dimensions w×c×h. c2 The second set of convolutional kernels x2 is used to process the second input feature map n. c2 Perform convolution along the width w of the feature map to obtain, as follows: Figure 4 The second output feature map f shown in (b) c2 .

[0078] The second set of convolutional kernels x2 includes w convolutional kernels of dimension w×1×1. These w w×1×1 convolutional kernels are used to process the second input feature map n. c2 Convolution is performed along the width w of the feature map to obtain w 1×c×h output features. Then, the w 1×c×h output feature maps are concatenated to obtain the following result: Figure 4 The second output feature map f shown in (b) c2 , where f c2 The dimensions are w×c×h, and the region connected by the dashed lines in the figure is the second output feature map f. c2 The receptive field, that is, the second output feature map f c2The range of information received by each feature point on the original input feature map.

[0079] It should be noted that, due to the second input feature map n c2 Through the first output feature map f c1 Therefore, the second input feature map n is obtained by transposing the input map. c2 The feature points on the map contain the first output feature map f. c1 All information in the high h dimension of the feature map. In addition, the second output feature map n c2 Each feature point also possesses information along the width dimension of the feature map. From such... Figure 4 The second output feature map f shown in (b) c2 The dashed lines connecting the regions clearly show that the second output feature map f... c2 The receptive field in the middle is significantly larger than that in the first output feature map f. c1 The receptive field, that is, the second output feature map f c2 The range of information received by each feature point on the original input feature map is greater than that of the first output feature map f. c1 The range of information received by each feature point on the original input feature map.

[0080] (3) Convolution in the dimension of channel number

[0081] Based on the same principle, the second output feature map f obtained by the convolution operation in step (2) is further processed. c2 After transposing, we obtain the third input feature map n with dimensions c×h×w. c3 The third set of convolutional kernels x3 is used to process the third input feature map n. c3 Performing convolution along the dimension of channel number c yields the following result: Figure 4 The third output feature map f shown in (c) c3 .

[0082] The third group of convolutional kernels x3 includes c convolutional kernels with dimensions c×1×1. These c c×1×1 convolutional kernels are used to process the third input feature map n. c3 Convolution is performed along the dimension of channel number c to obtain c 1×h×w output features. Then, the c 1×h×w output features are concatenated to obtain, as shown below. Figure 4 The third output feature map f shown in (c) c3 , where f c3 The dimensions are c×h×w, and the region connected by the dashed lines in the figure is the third output feature map f. c3 The receptive field, that is, the third output feature map f c3 The range of information received by each feature point on the original input feature map.

[0083] like Figure 4 As shown in (c), each feature point on the third output feature map possesses global information from the original image. This is because the third input feature map n c3 Through the second output feature map f c2 Therefore, the third input feature map n is obtained by transposing the input map. c3 The feature points on the map contain the second output feature map f. c2 Information in the dimensions of feature map height h and feature map width w. In addition, the third output feature map n c3 Each feature point on the map also possesses information in the dimension of the number of channels, c. Therefore, the third output feature map f c3 The receptive field in the middle is significantly larger than that in the second output feature map f. c2 and the first output feature map f c1 The feeling of being wild.

[0084] Analysis of the embodiments of this application shows that, in the process of performing multi-directional convolution on the original input feature map in the dimensions of feature map height, feature map width, and number of channels, the output feature map (such as the first output feature map) in the previous dimension (such as the dimension of feature map height) is used as the input for the convolution processing in the next dimension. The context information obtained after convolution processing in each dimension is concatenated, so that each feature point on the final first target output feature map receives sufficient feature information from the original input feature map, thereby improving the ability to obtain context information and reducing the amount of network computation.

[0085] As an optional implementation method, such as Figure 5 As shown, convolution operations (which can be understood as multi-directional convolution operations) can be performed on the original input feature map in parallel. For example, the above-mentioned 1×1 convolution operation on the original input feature map in the dimensions of channel number, feature map height, and feature map width yields the first target output feature map, including:

[0086] The original input feature map is transposed into a first input feature map, where the dimension of the first input feature map is h×c×w; the first input feature map is convolved with a first set of convolutional kernels along the high dimension of the feature map to obtain a first output feature map, where each convolutional kernel in the first set of convolutional kernels has a dimension of h×1×1, the number of convolutional kernels in the first set of convolutional kernels is h, and the dimension of the first output feature map is h×c×w;

[0087] The original input feature map is transposed into a second input feature map, where the second input feature map has dimensions w×c×h. The second input feature map is then convolved with a second set of convolutional kernels along the width dimension of the feature map to obtain a second output feature map, where each convolutional kernel in the second set of convolutional kernels has dimensions w×1×1, the number of convolutional kernels in the second set of convolutional kernels is w, and the dimension of the second output feature map is w×c×h.

[0088] The original input feature map is convolved using a third set of convolutional kernels in the dimension of the number of channels to obtain a third output feature map. Each convolutional kernel in the third set of convolutional kernels has a dimension of c×1×1, the number of convolutional kernels in the third set of convolutional kernels is c, and the dimension of the third output feature map is c×h×w.

[0089] The first output feature map is transposed into a first transposed feature map with dimensions c×h×w, the second output feature map is transposed into a second transposed feature map with dimensions c×h×w, and the first transposed feature map, the second transposed feature map and the third output feature map are concatenated into a first concatenated feature map, wherein the dimension of the first concatenated feature map is 3c×h×w.

[0090] The first spliced ​​feature map is dimensionality reduced to obtain the first target output feature map.

[0091] Specifically, such as Figure 5 As shown, the specific steps to obtain the first target output feature map are as follows:

[0092] (1) Convolution on the high dimension of the feature map

[0093] Assuming the original input feature map has dimensions c×h×w, transposing the original input feature map yields a first input feature map n with dimensions h×c×w. b1 The first set of convolutional kernels x1 is used to process the first input feature map n. b1 Convolution is performed along the height h dimension of the feature map to obtain, as follows: Figure 5 The first output feature map f of the 502 indicator shown in (a) b1 .

[0094] The first set of convolutional kernels x1 includes h convolutional kernels of dimension h×1×1. The first input feature map n is processed by the h h×1×1 convolutional kernels. b1 Convolution is performed along the height h dimension of the feature map to obtain h 1×c×w output features. Then, the h 1×c×w output features are concatenated to obtain, as shown below. Figure 5 As shown in (a), 502 (i.e., the first output feature map f) b1 ), where f b1 The dimensions are h×c×w, and the region connected by the dashed lines in the figure is the first output feature map f. b1The receptive field, that is, the first output feature map f b1 The range of information received by each feature point on the original input feature map.

[0095] It is easy to understand that by performing a convolution operation on the first input feature map using h convolution kernels with dimensions h×1×1, the first output feature map f can be made more efficient. b1 Each feature point on the feature map contains information about the input features in the higher dimensions of the feature map.

[0096] (2) Convolution along the width dimension of the feature map

[0097] The original input feature map is transposed to obtain a second input feature map n with dimensions w×c×h. b2 The second set of convolutional kernels x2 is used to process the second input feature map n. b2 Perform convolution along the width w of the feature map to obtain, as follows: Figure 5 The second output feature map f of the 504 indicator shown in (b) b2 .

[0098] The second set of convolutional kernels x2 includes w convolutional kernels of dimension w×1×1. These w w×1×1 convolutional kernels are used to process the second input feature map n. b2 Convolution is performed on the height and width dimensions of the feature map to obtain w 1×c×h output features. Then, the w 1×c×h output features are stacked to obtain the following result: Figure 5 The 504 (second output feature map f) shown in (b) b2 ), where f b2 The dimensions are w×c×h, and the region connected by the dashed lines in the figure is the second output feature map f. b2 The receptive field, that is, the second output feature map f b2 The range of information received by each feature point on the original input feature map.

[0099] It should be noted that, due to the second input feature map n b2 It is obtained by transposing the original input feature map. Therefore, the second output feature map f after convolution is... b2 The feature points on the map contain information along the width dimension of the feature map, which can be referenced. Figure 5 The second output feature map f shown in (b) b2 The area connected by dashed lines.

[0100] (3) Convolution in the dimension of channel number

[0101] Using the third set of convolutional kernels x3, the original input feature map n of dimension c×h×w is processed. b3 Performing convolution along the dimension of channel number c yields the following result: Figure 5 The third output feature map f of the 506 indicator shown in (c) b3 .

[0102] The third group of convolutional kernels x3 includes c convolutional kernels of dimension c×1×1. The original input feature map n is processed by these c c×1×1 convolutional kernels. b3 Convolution is performed along the dimension of channel number to obtain c 1×h×w output features. Then, the c 1×h×w output features are stacked to obtain, as shown below. Figure 5 The third output feature map 506 is shown in (c). Where, f b3 The dimensions are c×h×w, and the region connected by the dashed lines in the figure is the third output feature map f. b3 The receptive field, that is, the third output feature map f b3 The range of information received by each feature point on the original input feature map.

[0103] It should be noted that the third output feature map f after convolution processing b3 The feature points on the surface possess information in the dimension of channel number; for details, please refer to... Figure 5 The third output feature map f shown in (c) b3 The area connected by dashed lines.

[0104] Because of the first output feature map f b1 Second output feature map f b2 and the third output feature map f b3 The feature points on the map receive information in the "one-dimensional" dimension on the corresponding input feature map. Therefore, in order to obtain sufficient contextual information, it is necessary to concatenate the information in each "one-dimensional" dimension to obtain the first target output feature map 510 with global information. The specific process of obtaining the first target output feature map 510 by concatenation includes, but is not limited to, the following methods:

[0105] S1, respectively, for the first output feature map f b1 Second output feature map f b2 Transpose the features to obtain a first transposed feature map and a second transposed feature map, wherein the dimensions of both the first transposed feature map and the second transposed feature map are c×h×w.

[0106] S2, concatenate the first transposed feature map, the second transposed feature map, and the third output feature map to obtain the first concatenated feature map CF1 (e.g., ...). Figure 5 As shown in 508), the dimension of the first spliced ​​feature map is 3c×h×w;

[0107] S3, perform dimensionality reduction on the first spliced ​​feature map CF1 to obtain the first target output feature map 510.

[0108] It should be noted that, due to the first output feature map f b1 The dimensions are h×c×w, and the second output feature map is f. b2 The dimensions of the first output feature map are w×c×h, which differs from the dimensions of the third output feature map (c×h×w), making direct concatenation impossible. Therefore, it is necessary to combine the first output feature map f... b1 Second output feature map f b2 The dimensions are transposed to obtain the first transposed feature map and the second transposed feature map with dimensions c×h×w.

[0109] Through the above embodiments provided in this application, in the process of performing multi-directional convolution on the original input feature map in the dimensions of feature map height, feature map width, and number of channels, the output feature maps obtained by convolution operations in each dimension are spliced ​​together in parallel. By reducing the dimensionality of the first spliced ​​feature map, each feature point on the final first target output feature map receives sufficient feature information from the original input feature map, thereby improving the ability to obtain context information and reducing the network computation load.

[0110] To further illustrate how this embodiment obtains contextual information in various dimensions through "multi-directional" convolution, thereby achieving a first target output feature map with sufficient contextual information with less computation, the following explanation uses the example of performing convolution on the original input feature map in the dimension of the number of channels.

[0111] The following detailed description, in conjunction with the accompanying drawings, is provided. Figure 4 and Figure 5 A convolution process in one dimension. Figure 5 Taking convolution on the dimension of channel number as an example, such as... Figure 6 As shown, assume the original input feature map n b3 The dimension is c×h×w. The original input feature map is convolved along the dimension of the number of channels c using a third set of convolutional kernels x3. This third set of convolutional kernels x3 includes c convolutional kernels of dimension c×1×1. By convolving the original input feature map along the dimension of the number of channels c using c c×1×1 convolutional kernels, c 1×h×w output features are obtained. Then, these c 1×h×w output features are concatenated to obtain the following... Figure 6 The third output feature map f shown b3 Among them, the third output feature map f b3 The dimensions are c×h×w.

[0112] Taking c=3, h=5, w=5 as an example, for an original input feature map with 3 input channels, each channel needs to be convolved with a convolution kernel, such as... Figure 6As shown, each channel is convolved once with a 3×1×1 convolution kernel, generating three feature maps. Then, the values ​​of each feature point on the three feature maps are summed to obtain the third output feature map f. b3 The third output feature map f b3 Each feature point on the map obtains feature information from the number of channels c of the original input feature map.

[0113] In a similar manner, convolution can be performed on both the height and width dimensions of the feature map. This allows the feature points in the resulting feature map to acquire sufficient contextual information through 1×1 convolutions across multiple dimensions.

[0114] However, in related technologies, in order to obtain the same contextual information, methods such as... Figure 7 The convolution process shown includes: (1) performing a first convolution operation on the original input feature map using a 3×3 convolution kernel to obtain the output feature map in Layer 2, wherein each feature point on the output feature map in Layer 2 only obtains 3×3 feature information on channel c of the original input feature map; (2) performing a second convolution operation on the output feature map in Layer 2 using a 3×3 convolution kernel to obtain the output feature map in Layer 3, wherein each feature point on the output feature map in Layer 3 obtains 5×5 feature information on channel c of the original input feature map.

[0115] As described above, compared to related technologies that use 3×3 convolution kernels to perform two convolution operations on the channel dimension of the original input feature map in order to receive 5×5 feature information on each feature point in the output feature map in the channel dimension of the original input feature map, the method in this embodiment, which uses three 3×1×1 convolution kernels for a single convolution operation, reduces the network computation and obtains more contextual information.

[0116] As an optional example, the above-described dimensionality reduction processing of the first concatenated feature map to obtain the first target output feature map includes:

[0117] The first concatenated feature map is convolved using the fourth set of convolutional kernels in the dimension of the number of channels to obtain the first target output feature map. Each convolutional kernel in the fourth set of convolutional kernels has a dimension of 3c×1×1, and the number of convolutional kernels in the fourth set of convolutional kernels is c.

[0118] Specifically, such as Figure 8 As shown, the first spliced ​​feature map CF1 (as shown) Figure 8The first concatenated feature map 508 (shown in Figure 508) has a dimension of 3c×h×w. In order to maintain consistency with the dimension c×h×w of the original input feature map, the first concatenated feature map 508 needs to be dimensionality reduced. The detailed process is as follows:

[0119] (1) Perform convolution operation between each channel in the first spliced ​​feature map CF1 and one of the convolution kernels in the fourth group (dimension 3c×1×1) to generate c feature maps with dimension 1×h×w.

[0120] (2) By concatenating the feature values ​​of the feature points on the c 1×h×w feature maps, the first target output feature map 510 of c×h×w can be obtained.

[0121] As an optional implementation, the above-described execution of the target operation in the target convolutional neural network on the first target output feature map includes:

[0122] Perform a superposition operation on the original input feature map and the first target output feature map.

[0123] It is understandable that in some application scenarios, although more contextual information is obtained when performing convolution operations on the original input feature map in the above embodiments, some feature information is easily lost because convolution is a process of extracting image features.

[0124] To obtain more comprehensive feature information from the original input feature map, it is common practice to overlay the original input feature map with the first target output feature map. This is done by adding the values ​​of each feature point on the original input feature map and the first target output feature map, thereby obtaining more contextual information.

[0125] It should be noted that in this embodiment, the target operation in the target convolutional neural network is performed on the first target output feature map, including but not limited to performing a superposition operation on the original input feature map and the first target output feature map, or performing operations on the pooling layer and fully connected layer in the target convolutional neural network to obtain the final output result of the target convolutional neural network.

[0126] As another important aspect of this application, the above-mentioned multi-directional convolution method (such as...) Figure 4 or Figure 5 The method shown (performing 1×1 convolutions in multiple dimensions) constitutes, as follows: Figure 9-14The multi-directional convolutional module shown is used. Since multi-directional convolutional modules can achieve a larger receptive field in the convolved feature map with less computation, they can be understood as lightweight convolutional modules. As an optional example, performing the target operation in the target convolutional neural network on the first target output feature map may further include, or, after performing the target operation in the target convolutional neural network on the first target output feature map, constructing a backbone network using multiple multi-directional convolutional modules. The structure of the multiple multi-directional convolutional modules in the backbone network can be set according to different tasks (e.g., image recognition). Optionally, the number of the multiple multi-directional convolutional modules can be 50-100. Optionally, for the feature map output by the backbone network, operations related to the above task can be further performed in the target convolutional neural network, such as performing operations on pooling layers and fully connected layers in the target convolutional neural network to obtain the final output result of the target convolutional neural network.

[0127] As an optional example, the above-described overlay operation on the original input feature map and the first target output feature map includes:

[0128] The original input feature map is convolved in the dimension of the number of channels using the third set of convolutional kernels to obtain the second target output feature map. Each convolutional kernel in the third set of convolutional kernels has a dimension of c×1×1, the number of convolutional kernels in the third set of convolutional kernels is c, and the dimension of the second target output feature map is c×h×w.

[0129] The first target output feature map and the second target output feature map are concatenated to form a second concatenated feature map, wherein the dimension of the second concatenated feature map is 2c×h×w;

[0130] The second concatenated feature map is dimensionality reduced to obtain the third target output feature map, where the dimension of the third target output feature map is c×h×w.

[0131] Specifically, such as Figure 9 As shown, perform the following steps:

[0132] S902, multi-directional convolution is performed on the original input feature map in the dimensions of channel number, feature map height, and feature width to obtain the first target output feature map F1; as an optional method, it can be done through Figure 4 or Figure 5 The described convolution method is implemented Figure 9 The multi-directional convolution module in the middle;

[0133] S904, a set of convolutional kernels is used to convolve the original input feature map in the dimension of the number of channels to obtain the second target output feature map F2. Each convolutional kernel in the above set of kernels can have a dimension of c×1×1. As an optional approach, the above set of convolutional kernels can be, but is not limited to, as shown below. Figure 4 or Figure 5 The third set of convolutional kernels shown is x3;

[0134] S906, the first target output feature map F1 and the second target output feature map F2 are concatenated to obtain the second concatenated feature map CF2, wherein the dimension of the second concatenated feature map CF2 is 2c×h×w;

[0135] S908, in order to maintain the same dimension as the first target output feature map F1, the second spliced ​​feature map CF2 is dimensionality reduced to obtain the third target output feature map F3, where the dimension of the third target output feature map F3 is c×h×w.

[0136] It should be noted that the information received by each feature point in the first target output feature map F1 obtained through multi-directional convolution is the contextual information of the original input feature map. In other words, the features in the first target output feature map F1 are the contextual features of the original input feature map obtained through multi-directional convolution. Conversely, the information received by each feature point in the second target output feature map F2 is the local information of the original input feature map in the dimension of its channel number; that is, the features in the second target output feature map F2 are the local features of the original input feature map itself obtained through unidirectional convolution.

[0137] Adding the contextual features obtained through multi-directional convolution to the local features of the original input feature map obtained through unidirectional convolution can not only make up for the feature information lost during the convolution operation, but also make the feature information of the original input feature map more comprehensive.

[0138] To ensure that the concatenated feature map has the same dimensionality as the original input feature map, the second concatenated feature map needs to undergo dimensionality reduction processing, including:

[0139] The second concatenated feature map is convolved in the dimension of the number of channels using the fifth group of convolutional kernels to obtain the third target output feature map. Each convolutional kernel in the fifth group has a dimension of 2c×1×1, and the number of convolutional kernels in the fifth group is c.

[0140] Specifically, such as Figure 10 As shown, by executing steps S1002 to S1008, a second spliced ​​feature map CF2 is obtained by splicing the first target output feature map F1 and the second target output feature map F2.

[0141] The second concatenated feature map CF2 has a dimension of 2c×h×w. To maintain consistency with the dimension c×h×w of the original input feature map, the second concatenated feature map CF2 needs to be dimensionality reduced. The specific process is as follows:

[0142] (1) Perform convolution operation between each channel in the second spliced ​​feature map CF2 and one of the convolution kernels (dimension 2c×1×1) in the fifth group of convolution kernels x5 to generate c feature maps with dimension 1×h×w;

[0143] (2) By concatenating the values ​​of the feature points on the c 1×h×w feature maps, we can obtain the third target output feature map F3 of c×h×w.

[0144] In the above embodiment, a third set of convolution kernels is used to convolve the original input feature map in the dimension of the number of channels to obtain the second target output feature map. The first target output feature map and the second target output feature map are then concatenated to obtain the second concatenated feature map to obtain more feature information.

[0145] It is understood that the above-described superposition operation of the original input feature map and the first target output feature map is only an example. In this embodiment, other classification methods can also be used to superimpose the original input feature map and the first target output feature map. For example, it can include adding features between the third target output feature map and the original input feature map, or superimposing the original input feature map and the first target output feature map in a different way than the above superposition operation.

[0146] As an optional example, the above-described overlay operation between the original input feature map and the first target output feature map further includes:

[0147] The original input feature map is added to the third target output feature map to obtain the fourth target output feature map, where the dimension of the fourth target output feature map is c×h×w.

[0148] Specifically, such as Figure 11 As shown, it includes the following steps:

[0149] S1102, perform multi-directional convolution on the original input feature map F0 (dimension c×h×w) in the dimensions of channel number, feature map height, and feature width to obtain the first target output feature map F1 (dimension c×h×w); as an optional method, it can be done through Figure 4 or Figure 5 The described convolution method is implemented Figure 11 The multi-directional convolution module in the middle;

[0150] S1104, a set of convolutional kernels is used to convolve the original input feature map in the dimension of the number of channels to obtain the second target output feature map F2 (dimension c×h×w). Each convolutional kernel in the above set of kernels can have a dimension of c×1×1. As an optional approach, the above set of convolutional kernels can be, but is not limited to, as shown below. Figure 4 or Figure 5 The third set of convolutional kernels shown is x3;

[0151] S1106, concatenate the first target output feature map F1 and the second target output feature map F2 to obtain the second concatenated feature map CF2, wherein the dimension of the second concatenated feature map CF2 is 2c×h×w;

[0152] S1108, In order to maintain the same dimension as the first target output feature map F1, the second concatenated feature map CF2 is dimensionality reduced to obtain the third target output feature map F3, wherein the dimension of the third target output feature map F3 is c×h×w;

[0153] S1110, the original input feature map F0 and the third target output feature map F3 are added together to obtain the fourth target output feature map, where the dimension of the fourth target output feature map is c×h×w.

[0154] Based on the analysis in the above embodiments, it can be seen that the features on the third target output feature map are the sum of the context features obtained by multi-directional convolution and the local features of the original input feature map obtained by unidirectional convolution.

[0155] Based on this, the features of the original input feature map are added to the features of the third target output feature map (the context features obtained through multi-directional convolution and the local features obtained through unidirectional convolution), so that the feature information received by each feature point on the fourth target output feature map after addition is more comprehensive.

[0156] As another alternative example, after performing the superposition operation on the original input feature map and the first target output feature map, the method further includes:

[0157] A 1×1 convolution operation is performed on the third target output feature map in the dimensions of channel number, feature map height, and feature map width to obtain the fourth target output feature map, where the dimensions of the fourth target output feature map are c×h×w.

[0158] Specifically, such as Figure 12 As shown, it includes the following steps:

[0159] S1202, perform multi-directional convolution on the original input feature map in the dimensions of channel number, feature map height, and feature width to obtain the first target output feature map F1 (dimension c×h×w); as an optional method, it can be done through Figure 4 or Figure 5 The described convolution method is implemented Figure 12 The multi-directional convolution module in the middle;

[0160] S1204, a set of convolutional kernels x1 is used to convolve the original input feature map in the dimension of the number of channels to obtain the second target output feature map F2 (dimension c×h×w); wherein, the dimension of each convolutional kernel in the above set of convolutional kernels can be c×1×1. As an optional method, the above set of convolutional kernels can be, but is not limited to, as shown in the figure. Figure 4 or Figure 5 The third set of convolutional kernels shown is x3;

[0161] S1206, The first target output feature map F1 (dimension c×h×w) and the second target output feature map F2 (dimension c×h×w) are concatenated to obtain the second concatenated feature map CF2, wherein the dimension of the second concatenated feature map CF2 is 2c×h×w;

[0162] S1208, In order to maintain the same dimension as the first target output feature map F1, the second concatenated feature map CF2 is dimensionality reduced to obtain the third target output feature map F3, wherein the dimension of the third target output feature map F3 is c×h×w;

[0163] S1210, perform a 1×1 convolution operation on the third target output feature map in the dimensions of channel number, feature map height, and feature map width to obtain the fourth target output feature map F4 (dimensions c×h×w).

[0164] The specific process of performing multi-directional convolution on the third target output feature map is similar to the process of performing multi-directional convolution on the original input feature map in the dimensions of channel number, feature map height, and feature width in step S1202, and will not be repeated here.

[0165] However, it should be noted that since the output feature map of the third target contains convolutional features after multi-directional convolution of the original input feature map and unidirectional convolution, performing multi-directional convolution on the output feature map of the third target again is equivalent to repeating the multi-directional convolution operation on the original input feature map.

[0166] It is easy to understand that, compared with the first feature information obtained by a single multi-directional convolution operation, the target feature information in the second feature information obtained after performing a second multi-directional convolution operation on the original input feature map is more prominent, which is more conducive to the identification of the target object and improves the accuracy of image recognition.

[0167] As another alternative example, the above-mentioned feature map processing methods also include:

[0168] The third set of convolution kernels is used to convolve the third target output feature map in the dimension of the number of channels to obtain the fifth target output feature map, where the dimension of the fifth target output feature map is c×h×w;

[0169] The output feature map of the fourth target is concatenated with the output feature map of the fifth target to form the third concatenated feature map, wherein the dimension of the third concatenated feature map is 2c×h×w;

[0170] The third concatenated feature is dimensionality reduced to obtain the sixth target output feature map, where the dimension of the sixth target output feature map is c×h×w.

[0171] Specifically, such as Figure 13 As shown, it includes the following steps:

[0172] S1302, perform multi-directional convolution on the original input feature map F0 (dimension c×h×w) in the dimensions of channel number, feature map height, and feature width to obtain the first target output feature map F1 (dimension c×h×w); as an optional method, it can be done through Figure 4 or Figure 5 The described convolution method is implemented Figure 13 The multi-directional convolution module in the middle;

[0173] S1304, a set of convolutional kernels is used to convolve the original input feature map in the dimension of the number of channels to obtain the second target output feature map F2 (dimension c×h×w); wherein, the dimension of each convolutional kernel in the above set of convolutional kernels can be c×1×1. As an optional method, the above set of convolutional kernels can be, but is not limited to, as shown in the figure. Figure 4 or Figure 5 The third set of convolutional kernels shown is x3;

[0174] S1306, The first target output feature map F1 and the second target output feature map F2 are concatenated to obtain the second concatenated feature map CF2, wherein the dimension of the second concatenated feature map CF2 is 2c×h×w;

[0175] S1308, In order to maintain the same dimension as the first target output feature map F1, the second concatenated feature map CF2 is dimensionality reduced to obtain the third target output feature map F3, wherein the dimension of the third target output feature map F3 is c×h×w;

[0176] S1310, perform a 1×1 convolution operation on the third target output feature map in the dimensions of channel number, feature map height, and feature map width to obtain the fourth target output feature map F4 (dimensions c×h×w).

[0177] S1312, a set of convolutional kernels is used to convolve the third target output feature map in the dimension of the number of channels to obtain the fifth target output feature map F5 (dimension c×h×w); wherein, the dimension of each convolutional kernel in the above set of convolutional kernels can be c×1×1. As an optional method, the above set of convolutional kernels can be, but is not limited to, as shown in the figure. Figure 4 or Figure 5 The third set of convolutional kernels shown is x3;

[0178] S1314, concatenate the fourth target output feature map F4 and the fifth target output feature map F5 to obtain the third concatenated feature map CF3, wherein the dimension of the third concatenated feature map CF3 is 2c×h×w;

[0179] S1316, dimensionality reduction is performed on the third spliced ​​feature map CF3 to obtain the sixth target output feature map.

[0180] The similarity between steps S1302 to S1308 and steps S1310 to S1316 lies in the fact that the principles of the two execution processes are the same, both including: performing multi-directional convolution and unidirectional convolution on the input feature map; concatenating the two output feature maps after convolution; and performing dimensionality reduction processing on the concatenated feature map, etc. For specific processes, please refer to the implementation process of the above embodiments, which will not be repeated here.

[0181] The difference between steps S1302 to S1308 and steps S1310 to S1316 is that the inputs for the multi-directional convolution operation are different: the input for the multi-directional convolution operation in step S1302 is the original input feature map, while the input for the multi-directional convolution operation in step S1308 is the third target output feature map.

[0182] By performing a first round of multi-directional and unidirectional convolution operations on the original input feature map in the embodiments provided in this application, a third target output feature map containing the contextual features and local features of the original input feature map is obtained. Using the same principle, a second round of multi-directional and unidirectional convolution operations are performed on the third target output feature map, so that the information received by each feature point on the sixth target output feature map includes both the contextual information of the third target output feature map and its local information (such as target feature information). This improves the ability to acquire contextual information and enhances the accuracy of pattern recognition.

[0183] As a preferred example, the above-mentioned feature map processing method further includes:

[0184] The output feature map of the sixth target is added to the original input feature map to obtain the output feature map of the seventh target, where the dimension of the output feature map of the seventh target is c×h×w.

[0185] Specifically, such as Figure 14 As shown, it includes the following steps:

[0186] S1402, perform multi-directional convolution on the original input feature map F0 (dimension c×h×w) in the dimensions of channel number, feature map height, and feature width to obtain the first target output feature map F1 (dimension c×h×w); as an optional method, it can be done through Figure 4 or Figure 5 The described convolution method is implemented Figure 14 The multi-directional convolution module in the middle;

[0187] S1404, a set of convolutional kernels is used to convolve the original input feature map in the dimension of the number of channels to obtain the second target output feature map F2 (dimension c×h×w); wherein, the dimension of each convolutional kernel in the above set of convolutional kernels can be c×1×1. As an optional method, the above set of convolutional kernels can be, but is not limited to, as shown in the figure. Figure 4 or Figure 5 The third set of convolutional kernels shown is x3;

[0188] S1406, The first target output feature map F1 and the second target output feature map F2 are concatenated to obtain the second concatenated feature map CF2, wherein the dimension of the second concatenated feature map CF2 is 2c×h×w;

[0189] S1408, In order to maintain the same dimension as the first target output feature map F1, the second concatenated feature map CF2 is dimensionality reduced to obtain the third target output feature map F3, wherein the dimension of the third target output feature map F3 is c×h×w;

[0190] S1410, perform a 1×1 convolution operation on the third target output feature map in the dimensions of channel number, feature map height, and feature map width to obtain the fourth target output feature map F4 (dimensions c×h×w); as an optional method, it can be done through... Figure 4 or Figure 5 The described convolution method is implemented Figure 14 The multi-directional convolution module in the middle;

[0191] S1412, a set of convolutional kernels is used to convolve the third target output feature map in the dimension of the number of channels to obtain the fifth target output feature map F5 (dimension c×h×w); wherein, the dimension of each convolutional kernel in the above set of convolutional kernels can be c×1×1. As an optional method, the above set of convolutional kernels can be, but is not limited to, as shown in the figure. Figure 4 or Figure 5 The third set of convolutional kernels shown is x3;

[0192] S1414, concatenate the fourth target output feature map F4 and the fifth target output feature map F5 to obtain the third concatenated feature map CF3, wherein the dimension of the third concatenated feature map CF3 is 2c×h×w;

[0193] S1416, Dimensionality reduction is performed on the third spliced ​​feature map CF3 to obtain the sixth target output feature map;

[0194] S1418, the output feature map of the sixth target is added to the original input feature map to obtain the output feature map of the seventh target.

[0195] As can be seen from the above embodiments provided in this application, after performing a first round of multi-directional convolution and unidirectional convolution operations on the original input feature map and a second round of multi-directional convolution and unidirectional convolution operations on the third target output feature map, the information received by each feature point on the sixth target output feature map includes both the context information of the third target output feature map and the local information of the third target output feature map (such as target feature information).

[0196] Building upon this, the features of the original input feature map are added to the features of the sixth target output feature map (the context features obtained through two multi-directional convolutions and the local features obtained through two unidirectional convolutions). This results in each feature point in the seventh target output feature map receiving more comprehensive feature information. Furthermore, this enhances the ability to acquire contextual information with less computation.

[0197] As another important aspect of this application, it will be as follows Figure 9-14The network structure shown, which includes multi-directional convolutional modules, is considered a lightweight convolutional module. As an optional example, performing the target operation in the target convolutional neural network on the first target output feature map may further include, or, after performing the target operation in the target convolutional neural network on the first target output feature map, constructing a backbone network using multiple lightweight convolutional modules. The structure of the multiple lightweight convolutional modules in the backbone network can be set according to different tasks (e.g., image recognition). Optionally, the number of the multiple lightweight convolutional modules can be 50-100. Optionally, for the feature map output by the backbone network, operations related to the aforementioned task in the target convolutional neural network can be further performed, such as performing operations on pooling layers and fully connected layers in the target convolutional neural network to obtain the final output result of the target convolutional neural network.

[0198] As an optional example, the above feature processing methods also include:

[0199] When the original input feature map is the input feature map corresponding to the target image, and the target convolutional neural network is used to identify the target object in the target image, the first target output feature map is used to represent the context information associated with the target object in the target image.

[0200] To better understand the first target's output feature map and contextual information, the following section combines... Figure 3 Provide a detailed description.

[0201] Specifically, such as Figure 3 As shown, assuming that a target convolutional neural network 304 is used to identify the target image 302, and the target object being identified is 300, then the original input feature map is the input feature map corresponding to the target image 302, where the dimension of the input feature map is c×h×w, c is the number of channels in the input feature map 306, h is the feature map height of the input feature map 306, and w is the feature map width of the input feature map 306.

[0202] During the process of recognizing the target object 300 in the target image 302 through the target convolutional neural network 304, 1×1 convolution operations are sequentially performed on the input feature map 306 in the dimensions of the number of channels c, the bitmap of the feature map height h, and the feature map width w to obtain feature information in each dimension of the input feature map 306 (such as obtaining feature information related to the object 300, seawater, or seagulls). Then, the feature information in each dimension is fused into global information so that each feature point on the output feature map 308 receives sufficient contextual information.

[0203] It is understood that the output feature map 308 in this embodiment is used to represent the context information associated with the target object 300, wherein the context information associated with the target object 300 includes, but is not limited to, feature information of seawater, seagulls, etc.

[0204] By employing the technical solution in this application embodiment, based on the contextual information associated with the target object 300, it can be determined that the scene in which the target object 300 is located is a vehicle sailing on seawater. Then, based on the feature information of seagulls and sails, the identification result of the target object 300 can be output as a boat, not a car. In contrast, related technologies only rely on the acquired feature information of the target object 300 for identification. However, since the target object 300 is shaped like a car with wheels, the identification result might be a car based on these feature information. Therefore, it can be seen that the contextual information (global information) associated with the target object obtained through this application embodiment not only improves the accuracy of identification but also has the technical advantages of low computational load and low requirements for hardware computing power.

[0205] To ensure more comprehensive feature information, this application also provides a technical solution for superimposing the original input feature map with the first target output feature map. The fused information after the superposition operation includes:

[0206] When the original input feature map is the input feature map corresponding to the target image, the target convolutional neural network is used to identify the target object in the target image, and the first target output feature map is used to represent the context information associated with the target object in the target image, the feature map obtained by performing a superposition operation on the original input feature map and the first target output feature map is used to represent the information after fusing the target object's own information and context information.

[0207] For ease of understanding, this embodiment still uses... Figure 3 Let's take an example to explain. In the process of obtaining the context information associated with the target object 300 through the above embodiments, due to the multi-directional convolution operation performed on the input feature map 306, the feature information of the target object 300 may be lost.

[0208] To obtain more comprehensive information, a convolution operation can be performed on the original input feature map 306 along the channel dimension to obtain the target object 300's own feature information. Then, the target object's own information is fused with the context information, and the fused global information is used to identify the target object 300, resulting in higher accuracy.

[0209] To better understand the above implementation schemes, this application also provides an overall flowchart of a feature map processing method, specifically, as follows: Figure 15 As shown, it includes the following steps:

[0210] S1502~S1504 transforms the pixel values ​​corresponding to each pixel in the original image to obtain the input feature map. Then, a 1×1 convolution operation is performed on each dimension of the input feature map, including the dimension of the number of channels of the input feature map, the dimension of the height of the feature map, and the dimension of the width of the feature map.

[0211] S1506 aggregates convolutions from various dimensions to form a multi-directional convolution module;

[0212] The specific operations in the multi-directional convolution module can be found in the detailed process of obtaining the first target output feature map in the above embodiments.

[0213] S1508 uses a multi-directional convolution module to build a lightweight basic convolution module;

[0214] Specifically, the detailed steps for performing multi-directional and unidirectional convolutions on the original input feature map can be referred to in the above embodiments. By concatenating and reducing the dimensionality of the output feature map obtained from multi-directional convolution and the output feature map obtained from unidirectional convolution, a feature map with the same dimension as the original input feature map is output. Each feature point on the output feature map receives the contextual feature information of the original input feature map and its own feature information obtained from the unidirectional convolution.

[0215] It should be noted that during the execution of step S1508, the convolutional kernel size used is 1×1. Therefore, in the process of fusing contextual feature information with its own feature information to obtain global information, it also has the characteristics of low computational cost and lightweight design. Furthermore, the basic convolutional module constructed using the multi-directional convolutional module also has the characteristics of low computational cost and lightweight design, that is, a lightweight basic convolutional module is obtained.

[0216] S1510 allows lightweight convolutional modules to be applied to existing networks or used to build new networks;

[0217] The lightweight convolution module in this application embodiment is applied in at least one of the following ways:

[0218] 1) The lightweight convolutional module in the embodiments of this application can be directly used to replace the basic convolutional module of any convolutional neural network, that is, only the convolutional layer in the convolutional neural network is replaced.

[0219] 2) Construct new neural networks using the lightweight convolution module in the embodiments of this application;

[0220] It is understandable that new convolutional neural networks or other neural networks built using the two application methods described above possess the characteristics of low computational cost and lightweight design of the aforementioned lightweight basic convolutional modules. Therefore, existing networks or other new networks that apply lightweight basic convolutional modules can also achieve the technical effects of reducing computational cost and enhancing the ability to acquire contextual information.

[0221] S1512, End of processing flow.

[0222] The feature map processing method in this application embodiment can perform multi-directional convolution on the original input feature map in multiple dimensions using a 1×1 convolution kernel, so that each feature point on the target output feature map receives sufficient feature information from the original input feature map, thus solving the technical problem of large computational load in the convolution process of feature maps in related technologies.

[0223] In addition, the lightweight convolutional module constructed through the embodiments of this application has strong versatility, which improves the applicability of the technical solution of this application. For example, it can be applied to products such as mobile phones, drones, and robots, avoiding the problem of not being able to run smoothly due to the poor hardware computing power of the products themselves. This makes the lightweight convolutional neural network of great significance in the application of pattern recognition algorithms.

[0224] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0225] According to another aspect of the present invention, a feature map processing apparatus for implementing the above-described feature map processing method is also provided. For example... Figure 16 As shown, the device includes:

[0226] The acquisition unit 1602 is used to acquire the original input feature map in the target convolutional neural network, wherein the dimension of the original input feature map is c×h×w, c is the number of channels of the original input feature map, h is the feature map height of the original input feature map, w is the feature map width of the original input feature map, and c, h and w are positive integers greater than or equal to 1.

[0227] The first processing unit 1604 is used to perform a 1×1 convolution operation on the original input feature map in the dimensions of the number of channels, the height of the feature map, and the width of the feature map to obtain the first target output feature map, wherein the dimensions of the first target output feature map are c×h×w.

[0228] The second processing unit 1606 is used to perform target operations in the target convolutional neural network on the first target output feature map.

[0229] Optionally, the first processing unit further includes:

[0230] The first processing module is used to transpose the original input feature map into a first input feature map, wherein the dimension of the first input feature map is h×c×w; and to convolve the first input feature map using a first set of convolutional kernels in the high dimension of the feature map to obtain a first output feature map, wherein the dimension of each convolutional kernel in the first set of convolutional kernels is h×1×1, the number of convolutional kernels in the first set of convolutional kernels is h, and the dimension of the first output feature map is h×c×w.

[0231] The second processing module is used to transpose the first output feature map into a second input feature map, wherein the second input feature map has a dimension of w×c×h; and to convolve the second input feature map using a second set of convolutional kernels along the width dimension of the feature map to obtain a second output feature map, wherein each convolutional kernel in the second set of convolutional kernels has a dimension of w×1×1, the number of convolutional kernels in the second set of convolutional kernels is w, and the dimension of the second output feature map is w×c×h.

[0232] The third processing module is used to transpose the second output feature map into a third input feature map, wherein the dimension of the third input feature map is c×h×w; and to convolve the third input feature map in the dimension of the number of channels using a third set of convolution kernels to obtain the first target output feature map, wherein the dimension of each convolution kernel in the third set of convolution kernels is c×1×1, and the number of convolution kernels in the third set of convolution kernels is c.

[0233] Optionally, the above-described 1×1 convolution operation is performed on the original input feature map in the dimensions of channel number, feature map height, and feature map width to obtain the first target output feature map, including:

[0234] The fourth processing module is used to transpose the original input feature map into a first input feature map, wherein the first input feature map has a dimension of h×c×w; and to convolve the first input feature map using a first set of convolutional kernels in the high dimension of the feature map to obtain a first output feature map, wherein each convolutional kernel in the first set of convolutional kernels has a dimension of h×1×1, the number of convolutional kernels in the first set of convolutional kernels is h, and the dimension of the first output feature map is h×c×w.

[0235] The fifth processing module is used to transpose the original input feature map into a second input feature map, wherein the second input feature map has a dimension of w×c×h; and to convolve the second input feature map using a second set of convolutional kernels along the width dimension of the feature map to obtain a second output feature map, wherein each convolutional kernel in the second set of convolutional kernels has a dimension of w×1×1, the number of convolutional kernels in the second set of convolutional kernels is w, and the dimension of the second output feature map is w×c×h.

[0236] The first convolution module is used to convolve the original input feature map in the dimension of the number of channels using the third set of convolution kernels to obtain the third output feature map. Each convolution kernel in the third set of convolution kernels has a dimension of c×1×1, the number of convolution kernels in the third set of convolution kernels is c, and the dimension of the third output feature map is c×h×w.

[0237] The sixth processing module is used to transpose the first output feature map into a first transposed feature map with dimensions c×h×w, transpose the second output feature map into a second transposed feature map with dimensions c×h×w, and concatenate the first transposed feature map, the second transposed feature map and the third output feature map into a first concatenated feature map, wherein the dimension of the first concatenated feature map is 3c×h×w.

[0238] The seventh processing module is used to perform dimensionality reduction processing on the first spliced ​​feature map to obtain the first target output feature map.

[0239] Optionally, the above-described dimensionality reduction processing of the first concatenated feature map to obtain the first target output feature map includes:

[0240] The second convolution module is used to convolve the first concatenated feature map in the dimension of the number of channels using the fourth group of convolution kernels to obtain the first target output feature map. The dimension of each convolution kernel in the fourth group of convolution kernels is 3c×1×1, and the number of convolution kernels in the fourth group of convolution kernels is c.

[0241] Optionally, the above-described target operation in the target convolutional neural network performed on the first target output feature map includes:

[0242] The first overlay module is used to perform an overlay operation on the original input feature map and the first target output feature map.

[0243] Optionally, the above-mentioned superposition operation on the original input feature map and the first target output feature map includes:

[0244] The third convolution module is used to convolve the original input feature map in the dimension of the number of channels using the third set of convolution kernels to obtain the second target output feature map. Each convolution kernel in the third set of convolution kernels has a dimension of c×1×1, the number of convolution kernels in the third set of convolution kernels is c, and the dimension of the second target output feature map is c×h×w.

[0245] The first splicing module is used to splice the first target output feature map and the second target output feature map into a second spliced ​​feature map, wherein the dimension of the second spliced ​​feature map is 2c×h×w;

[0246] The eighth processing module is used to perform dimensionality reduction processing on the second spliced ​​feature map to obtain the third target output feature map, wherein the dimension of the third target output feature map is c×h×w.

[0247] Optionally, the above-described dimensionality reduction processing of the second concatenated feature map yields the third target output feature map, including:

[0248] The ninth processing module is used to convolve the second concatenated feature map in the dimension of the number of channels using the fifth group of convolutional kernels to obtain the third target output feature map. The dimension of each convolutional kernel in the fifth group of convolutional kernels is 2c×1×1, and the number of convolutional kernels in the fifth group of convolutional kernels is c.

[0249] Optionally, the above-mentioned superposition operation between the original input feature map and the first target output feature map further includes:

[0250] The tenth processing module is used to add the features of the original input feature map and the third target output feature map to obtain the fourth target output feature map, wherein the dimension of the fourth target output feature map is c×h×w.

[0251] Optionally, after performing the superposition operation on the original input feature map and the first target output feature map, the apparatus further includes:

[0252] The eleventh processing module is used to perform a 1×1 convolution operation on the third target output feature map in the dimensions of the number of channels, the height of the feature map, and the width of the feature map to obtain the fourth target output feature map, wherein the dimensions of the fourth target output feature map are c×h×w.

[0253] Optionally, the above-mentioned feature map processing apparatus further includes:

[0254] The twelfth processing module is used to convolve the third target output feature map with the third set of convolution kernels in the dimension of the number of channels to obtain the fifth target output feature map, wherein the dimension of the fifth target output feature map is c×h×w;

[0255] The second stitching module is used to stitch the output feature map of the fourth target and the output feature map of the fifth target into a third stitched feature map, wherein the dimension of the third stitched feature map is 2c×h×w;

[0256] The thirteenth processing module is used to perform dimensionality reduction on the third concatenated feature to obtain the sixth target output feature map, where the dimension of the sixth target output feature map is c×h×w.

[0257] Optionally, the above-mentioned feature map processing apparatus further includes:

[0258] The fourteenth processing module is used to add the features of the sixth target output feature map to the original input feature map to obtain the seventh target output feature map, wherein the dimension of the seventh target output feature map is c×h×w.

[0259] Optionally, the aforementioned feature map processing device is further configured to, when the original input feature map is the input feature map corresponding to the target image and the target convolutional neural network is used to identify the target object in the target image, use the first target output feature map to represent the context information associated with the target object in the target image.

[0260] Optionally, the aforementioned feature map processing device is further configured to, when the original input feature map is the input feature map corresponding to the target image, the target convolutional neural network is used to identify the target object in the target image, and the first target output feature map is used to represent the context information associated with the target object in the target image, perform a superposition operation on the original input feature map and the first target output feature map to represent the information after the fusion of the target object's own information and context information.

[0261] It should be noted that the embodiments of the feature map processing device here can refer to the embodiments of the feature map processing method described above, and will not be repeated here.

[0262] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described feature map processing method is also provided. This electronic device may be... Figure 17 The terminal device or the server shown. This embodiment uses the electronic device as a server as an example for illustration. Figure 17 As shown, the electronic device includes a memory 1702 and a processor 1704. The memory 1702 stores a computer program, and the processor 1704 is configured to execute the steps of any of the above method embodiments via the computer program.

[0263] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0264] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0265] S1, obtain the original input feature map in the target convolutional neural network, where the dimension of the original input feature map is c×h×w, c is the number of channels of the original input feature map, h is the feature map height of the original input feature map, w is the feature map width of the original input feature map, and c, h and w are positive integers greater than or equal to 1;

[0266] S2, perform a 1×1 convolution operation on the original input feature map in the dimensions of the number of channels, the height of the feature map, and the width of the feature map to obtain the first target output feature map, wherein the dimensions of the first target output feature map are c×h×w;

[0267] S3, perform the target operation in the target convolutional neural network on the first target output feature map.

[0268] Alternatively, as those skilled in the art will understand, Figure 17 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 17 This does not limit the structure of the aforementioned electronic devices or electronic equipment. For example, electronic devices or electronic equipment may also include components that are more... Figure 17 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 17 The different configurations shown.

[0269] The memory 1702 can be used to store software programs and modules, such as the program instructions / modules corresponding to the resource preloading method and apparatus in this embodiment. The processor 1704 executes various functional applications and data processing by running the software programs and modules stored in the memory 1702, thereby implementing the aforementioned resource preloading method. The memory 1702 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1702 may further include memory remotely located relative to the processor 1704, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 1702 may be used, but is not limited to, for information such as training samples and convolutional features required for image recognition. As an example, such as... Figure 17As shown, the memory 1702 may include, but is not limited to, the acquisition unit 1602, the first processing unit 1604, and the second processing unit 1606 in the feature map processing device. Furthermore, it may include, but is not limited to, other module units in the feature map processing device, which will not be elaborated upon in this example.

[0270] Optionally, the transmission device 1706 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 1706 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 1706 is a radio frequency (RF) module, used for wireless communication with the Internet.

[0271] In addition, the above-mentioned electronic device also includes: a display 1708 for displaying the recognition result of the above-mentioned image recognition; and a connection bus 1710 for connecting the various module components in the above-mentioned electronic device.

[0272] In other embodiments, the aforementioned terminal device or server can be a node in a distributed system, wherein the distributed system can be a blockchain system, which is a distributed system formed by connecting multiple nodes through network communication. The nodes can form a peer-to-peer (P2P) network, and any form of computing device, such as a server, terminal, or other electronic device, can become a node in the blockchain system by joining this peer-to-peer network.

[0273] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the feature map processing methods provided in various alternative implementations of the server convolution operation described above, wherein the computer program is configured to execute the steps in any of the method embodiments described above at runtime.

[0274] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0275] S1, obtain the original input feature map in the target convolutional neural network, where the dimension of the original input feature map is c×h×w, c is the number of channels of the original input feature map, h is the feature map height of the original input feature map, w is the feature map width of the original input feature map, and c, h and w are positive integers greater than or equal to 1;

[0276] S2, perform a 1×1 convolution operation on the original input feature map in the dimensions of the number of channels, the height of the feature map, and the width of the feature map to obtain the first target output feature map, wherein the dimensions of the first target output feature map are c×h×w;

[0277] S3, perform the target operation in the target convolutional neural network on the first target output feature map.

[0278] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0279] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0280] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0281] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0282] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0283] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0284] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0285] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for processing feature maps, characterized in that, include: Obtain the original input feature map in the target convolutional neural network. Where the original input feature map is the input feature map corresponding to the target image, the target convolutional neural network is used to identify the target object in the target image, and the first target output feature map is used to represent the context information associated with the target object in the target image, the feature map obtained by performing a superposition operation on the original input feature map and the first target output feature map is used to represent the information after fusing the target object's own information with the context information. The dimension of the original input feature map is c×h×w, where c is the number of channels in the original input feature map, h is the feature map height of the original input feature map, w is the feature map width of the original input feature map, and c, h, and w are positive integers greater than or equal to 1. The first target output feature map is obtained by transposing the original input feature map, including one of the following: transposing the original input feature map into a first input feature map of dimension h×c×w; convolving the first input feature map in the height dimension of the feature map using a first convolution kernel of dimension h×1×1 to obtain a first output feature map; transposing the first output feature map into a second input feature map of dimension w×c×h; convolving the second input feature map in the width dimension of the feature map using a second convolution kernel of dimension w×1×1 to obtain a second output feature map; transposing the second output feature map into a third input feature map of dimension c×h×w; convolving the third input feature map in the number of channels dimension using a third convolution kernel of dimension c×1×1 to obtain the first target output feature map. Alternatively, the original input feature map is transposed in both the height and width dimensions to obtain the first input feature map and the second input feature map; the first input feature map is convolved in the height dimension using a first convolution kernel of dimension h×1×1 to obtain the first output feature map; the second input feature map is convolved in the width dimension using a second convolution kernel of dimension w×1×1 to obtain the second output feature map; the original input feature map is convolved in the number of channels dimension using a third convolution kernel of dimension c×1×1 to obtain the third output feature map; the first output feature map is transposed into a first transposed feature map of dimension c×h×w, the second output feature map is transposed into a second transposed feature map of dimension c×h×w, and the first concatenated feature map obtained by concatenating the first transposed feature map, the second transposed feature map, and the third output feature map is subjected to dimensionality reduction processing to obtain the first target output feature map; The target operation in the target convolutional neural network is performed on the first target output feature map.

2. The method according to claim 1, characterized in that, The step of convolving the first input feature map along the height dimension of the feature map to obtain the first output feature map includes: using a first set of convolution kernels to convolve the first input feature map along the height dimension of the feature map to obtain the first output feature map, wherein each convolution kernel in the first set of convolution kernels has a dimension of h×1×1, the number of convolution kernels in the first set of convolution kernels is h, and the dimension of the first output feature map is h×c×w; the step of convolving the second input feature map along the width dimension of the feature map to obtain the second output feature map includes: using a second set of convolution kernels to convolve the second input feature map along the width dimension of the feature map. The second output feature map is obtained by convolving the third input feature map on the dimension of the number of channels. Each convolutional kernel in the second set of convolutional kernels has a dimension of w×1×1, the number of convolutional kernels in the second set of convolutional kernels is w, and the dimension of the second output feature map is w×c×h. The step of convolving the third input feature map on the dimension of the number of channels to obtain the first target output feature map includes: using a third set of convolutional kernels to convolve the third input feature map on the dimension of the number of channels to obtain the first target output feature map. Each convolutional kernel in the third set of convolutional kernels has a dimension of c×1×1, and the number of convolutional kernels in the third set of convolutional kernels is c.

3. The method according to claim 1, characterized in that, The step of performing the target operation in the target convolutional neural network on the first target output feature map includes: An overlay operation is performed on the original input feature map and the first target output feature map.

4. The method according to claim 3, characterized in that, The step of performing a superposition operation on the original input feature map and the first target output feature map includes: The original input feature map is convolved in the dimension of the number of channels using a third set of convolutional kernels to obtain a second target output feature map. Each convolutional kernel in the third set of convolutional kernels has a dimension of c×1×1, the number of convolutional kernels in the third set of convolutional kernels is c, and the dimension of the second target output feature map is c×h×w. The first target output feature map and the second target output feature map are concatenated to form a second concatenated feature map, wherein the dimension of the second concatenated feature map is 2c×h×w; The second spliced ​​feature map is subjected to dimensionality reduction processing to obtain the third target output feature map, wherein the dimension of the third target output feature map is c×h×w.

5. The method according to claim 4, characterized in that, The step of performing dimensionality reduction processing on the second concatenated feature map to obtain the third target output feature map includes: The second concatenated feature map is convolved in the dimension of the number of channels using the fifth group of convolutional kernels to obtain the third target output feature map. The dimension of each convolutional kernel in the fifth group of convolutional kernels is 2c×1×1, and the number of convolutional kernels in the fifth group of convolutional kernels is c.

6. The method according to claim 4, characterized in that, The step of performing the superposition operation on the original input feature map and the first target output feature map further includes: The original input feature map is added to the third target output feature map to obtain the fourth target output feature map, wherein the dimension of the fourth target output feature map is c×h×w.

7. The method according to claim 4, characterized in that, After performing the overlay operation on the original input feature map and the first target output feature map, the method further includes: The third target output feature map is subjected to a 1×1 convolution operation in the dimensions of the number of channels, the height of the feature map, and the width of the feature map to obtain the fourth target output feature map, wherein the dimensions of the fourth target output feature map are c×h×w.

8. The method according to claim 7, characterized in that, The method further includes: The third set of convolutional kernels is used to convolve the third target output feature map in the dimension of the number of channels to obtain the fifth target output feature map, wherein the dimension of the fifth target output feature map is c×h×w; The fourth target output feature map and the fifth target output feature map are concatenated to form a third concatenated feature map, wherein the dimension of the third concatenated feature map is 2c×h×w; The third spliced ​​feature map is subjected to dimensionality reduction processing to obtain the sixth target output feature map, wherein the dimension of the sixth target output feature map is c×h×w.

9. The method according to claim 8, characterized in that, The method further includes: The sixth target output feature map is added to the original input feature map to obtain the seventh target output feature map, wherein the dimension of the seventh target output feature map is c×h×w.

10. The method according to any one of claims 1 to 9, characterized in that, When the original input feature map is the input feature map corresponding to the target image, and the target convolutional neural network is used to identify the target object in the target image, the first target output feature map is used to represent the context information associated with the target object in the target image.

11. A feature map processing apparatus, characterized in that, include: An acquisition unit is used to acquire the original input feature map in the target convolutional neural network. Where the original input feature map is the input feature map corresponding to the target image, the target convolutional neural network is used to identify the target object in the target image, and the first target output feature map is used to represent the context information associated with the target object in the target image, the feature map obtained by performing a superposition operation on the original input feature map and the first target output feature map is used to represent the information after fusing the target object's own information with the context information. The dimension of the original input feature map is c×h×w, where c is the number of channels in the original input feature map, h is the feature map height of the original input feature map, w is the feature map width of the original input feature map, and c, h, and w are positive integers greater than or equal to 1. A first processing unit is configured to obtain a first target output feature map by transposing the original input feature map, comprising one of the following: transposing the original input feature map into a first input feature map of dimension h×c×w; convolving the first input feature map with a first convolution kernel of dimension h×1×1 on the height dimension of the feature map to obtain a first output feature map; transposing the first output feature map into a second input feature map of dimension w×c×h; convolving the second input feature map with a second convolution kernel of dimension w×1×1 on the width dimension of the feature map to obtain a second output feature map; transposing the second output feature map into a third input feature map of dimension c×h×w; and convolving the third input feature map with a third convolution kernel of dimension c×1×1 on the number of channels dimension to obtain the first target output feature map. Alternatively, the original input feature map is transposed in both the height and width dimensions to obtain the first input feature map and the second input feature map; the first input feature map is convolved in the height dimension using a first convolution kernel of dimension h×1×1 to obtain the first output feature map; the second input feature map is convolved in the width dimension using a second convolution kernel of dimension w×1×1 to obtain the second output feature map; the original input feature map is convolved in the number of channels dimension using a third convolution kernel of dimension c×1×1 to obtain the third output feature map; the first output feature map is transposed into a first transposed feature map of dimension c×h×w, the second output feature map is transposed into a second transposed feature map of dimension c×h×w, and the first concatenated feature map obtained by concatenating the first transposed feature map, the second transposed feature map, and the third output feature map is subjected to dimensionality reduction processing to obtain the first target output feature map; The second processing unit is used to perform the target operation in the target convolutional neural network on the first target output feature map.

12. A computer-readable storage medium comprising a stored program, wherein, When the program is executed, it performs the method described in any one of claims 1 to 10.

13. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 10 through the computer program.

Images