Image processing method, electronic device, and storage medium

Abstract

The application discloses an image processing method, an electronic device and a storage medium. The image processing method comprises the following steps: inputting an acquired image to be processed into an information accumulation model; performing channel reorganization processing on an initial feature map of the acquired image to be processed through the information accumulation model, so as to obtain initial sub-feature maps corresponding to each channel group; performing information accumulation processing on the initial sub-feature maps corresponding to each channel group in a row direction and / or a column direction, so as to obtain target sub-feature maps corresponding to each channel group; and performing fusion processing on the target sub-feature maps corresponding to each channel group, so as to obtain a target feature map. Through the information accumulation processing in different directions, the utilization of image information can be improved on the basis of low overhead, so that the accumulation of global information in the target feature map is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an image processing method, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, in the field of image processing, the visual Transformer model, based on multi-head self-attention, has significantly improved the performance of many visual tasks by establishing long-range dependencies between pixels, becoming an important direction for current technological development. However, in application, the computational complexity of this mechanism is proportional to the square of the number of pixels, resulting in huge computational and memory overhead when processing high-resolution images, which severely limits real-time applications and hardware deployment.

[0003] Therefore, there is an urgent need for an image processing method that can utilize global image information while maintaining low overhead. Summary of the Invention

[0004] This application provides at least one image processing method, electronic device, and storage medium that can improve the utilization of image information with low overhead.

[0005] This application provides an image processing method, comprising: inputting an acquired image to be processed into an information accumulation model; performing channel recombination processing on the initial feature map of the acquired image to be processed through the information accumulation model to obtain initial sub-feature maps corresponding to each channel group; performing information accumulation processing on the initial sub-feature maps corresponding to each channel group in the row direction and / or column direction to obtain target sub-feature maps corresponding to each channel group; and performing fusion processing on the target sub-feature maps corresponding to each channel group to obtain target feature maps.

[0006] This application provides an image processing apparatus, including: a channel reconstruction module, an information accumulation module, and a fusion processing module; the channel reconstruction module is used to input the acquired image to be processed into an information accumulation model, and to perform channel reconstruction processing on the initial feature map of the acquired image to be processed through the information accumulation model to obtain the initial sub-feature map corresponding to each channel group; the information accumulation module is used to perform information accumulation processing on the initial sub-feature map corresponding to each channel group in the row direction and / or column direction to obtain the target sub-feature map corresponding to each channel group; the fusion processing module is used to perform fusion processing on the target sub-feature map corresponding to each channel group to obtain the target feature map.

[0007] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described image processing method.

[0008] This application provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the above-described image processing method.

[0009] The above scheme involves inputting the acquired image to be processed into an information accumulation model. The information accumulation model performs channel recombination processing on the initial feature maps of the acquired image to obtain initial sub-feature maps corresponding to each channel group. Information accumulation processing is then performed on the initial sub-feature maps corresponding to each channel group in the row and / or column directions to obtain target sub-feature maps corresponding to each channel group. Finally, the target sub-feature maps corresponding to each channel group are fused to obtain the target feature map. This approach reduces computational complexity by performing pre-channel recombination processing so that information accumulation processing can be performed on individual channel groups. Information accumulation processing in different directions on a single channel group yields the target sub-feature map of that single channel group, improving the utilization rate of global information in the image to be processed. Fusion processing is then performed on the target sub-feature maps of each channel group to ensure the integrity of image information in the image to be processed, thereby achieving the accumulation of global information in the target feature map and improving the efficiency of image processing.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0012] Figure 1 This is a schematic flowchart of an exemplary embodiment of the image processing method of this application; Figure 2 yes Figure 1 A schematic diagram of the sub-process of step S12; Figure 3 This is a schematic diagram of the weight generation module in an exemplary embodiment of the image processing method of this application; Figure 4 yes Figure 2 A schematic diagram of the sub-process of step S22; Figure 5 yes Figure 4 A schematic diagram of the sub-process of step S41; Figure 6 This is a schematic diagram illustrating the effect of the row and column information update module in an exemplary embodiment of the image processing method of this application; Figure 7 This is a schematic diagram illustrating the effect of the current vector to be updated and the candidate vectors to be updated in an exemplary embodiment of the image processing method of this application; Figure 8This is yet another schematic flowchart of an exemplary embodiment of the image processing method of this application; Figure 9 This is a schematic diagram of the structure of a preset image processing model in an exemplary embodiment of the image processing method of this application; Figure 10 This is a schematic diagram of the structure of an embodiment of the image processing apparatus of this application; Figure 11 This is a schematic diagram of the structure of an embodiment of the electronic device of this application; Figure 12 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0014] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The term "and / or" is merely a description of the association of related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, "many" in this document means two or more. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements, for example, including at least one of A, B, and C, which can mean including any one or more elements selected from the set consisting of A, B, and C. Additionally, the term "several" in this document means one or more.

[0016] This application provides several image processing methods and apparatuses. The application scenarios of these image processing methods include, but are not limited to, the generation of feature maps to be classified before an image classification task, the generation of feature maps to be segmented before an image segmentation task, or image classification scenarios, or image segmentation scenarios. The image processing methods of this application can involve various business domains when performing image classification or image segmentation tasks. For example, these business domains may include object recognition (e.g., vehicle and obstacle recognition), animal recognition (e.g., cat and dog recognition), etc. The executing entity of the image processing method can be an image processing apparatus, such as an information accumulation model, an image classification model, or an image segmentation model. The image classification model may include an information accumulation model, or the image segmentation model may include an information accumulation model. For example, the image processing apparatus can be located in a terminal device, server, or other processing device. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, etc. In some possible implementations, the image processing method can be implemented by a processor calling computer-readable instructions stored in memory.

[0017] Please see Figure 1 , Figure 1 This is a schematic flowchart of an exemplary embodiment of the image processing method of this application. Specifically, the image processing method is applied to an image acquisition device, and the image processing method may include the following steps: Step S11: Input the acquired image to be processed into the information accumulation model. The information accumulation model performs channel recombination processing on the initial feature map of the acquired image to be processed to obtain the initial sub-feature map corresponding to each channel group.

[0018] The image to be processed can be any image that requires image processing. The type of image to be processed varies depending on the business domain to which the image processing method described above is applied. The image to be processed represents an image acquired from a target object within that business domain. For example, if the business domain is target object recognition, and the target object is a vehicle, the image to be processed would be an image acquired from a vehicle. If the business domain is animal recognition, and the target object is an animal, the image to be processed would be an image acquired from an animal. The image to be processed can be acquired by acquiring a real-time image of the target object, or by retrieving an image from a pre-set database.

[0019] The initial feature map of the image to be processed represents the feature map obtained by performing preset feature extraction on the image to be processed. The initial feature map of the image to be processed can be obtained by first acquiring the image to be processed; then, the initial feature map of the image to be processed is obtained by performing preset feature extraction on the image to be processed, including: inputting the image to be processed into an image feature extraction module, and obtaining the initial feature map of the image to be processed output by the image feature extraction module. The image feature extraction module may be equipped with a preset feature extraction network. Specifically, the preset feature extraction network may be a convolutional neural network, a recurrent neural network, a long short-term memory network, a gated recurrent unit, a BiGRU neural network, or other feature extraction networks. In some application scenarios, the preset feature extraction network may be a preset perceptron network. The preset perceptron network may be obtained from at least one perceptron module. Each perceptron module in the at least one perceptron module may be a multilayer perceptron (MLP).

[0020] The information accumulation model represents the image information accumulation processing performed on the initial feature map of the image to be processed in the row and / or column directions to obtain the target feature map. The information accumulation model includes multiple cascaded functional modules, each of which performs channel reconstruction processing, information accumulation processing, and fusion processing on the initial feature map of the image to be processed to obtain the target feature map.

[0021] Channel reconfiguration processing represents the channel feature maps corresponding to at least two channel groups after pre-defined channel processing in the channel dimension of the initial feature map. The initial feature map of the image to be processed includes multiple channel sub-feature maps superimposed in the channel dimension, each channel sub-feature map having the same width and height. The initial sub-feature map includes at least one channel sub-feature map. Specifically, channel reconfiguration processing represents the initial sub-feature maps in the initial feature map undergoing channel grouping processing and / or channel compression processing to obtain initial sub-feature maps corresponding to at least two channel groups. The initial sub-feature map corresponding to each channel group represents the processing result of each channel sub-feature map in the initial feature map after channel reconfiguration processing.

[0022] In some application scenarios, channel reorganization of the initial feature map to obtain the initial sub-feature maps corresponding to each channel group can be achieved by grouping the channel sub-feature maps in the initial feature map into at least two channel groups, with each channel group containing at least one channel sub-feature map. The channel sub-feature maps of each channel group are then superimposed along the channel dimension to obtain the initial sub-feature map for the corresponding channel group. The total number of channel sub-feature maps in each channel group is the same as the total number of channel sub-feature maps in the initial feature map.

[0023] In other application scenarios, channel reorganization of the initial feature map to obtain the initial sub-feature maps corresponding to each channel group can be performed by channel compression of each channel sub-feature map in the initial feature map to obtain compressed channel sub-feature maps of at least two channel groups, with each channel group containing at least one compressed channel sub-feature map. The compressed channel sub-feature maps of each channel group are then superimposed along the channel dimension to obtain the initial sub-feature map of the corresponding channel group. The total number of compressed channel sub-feature maps for each channel group is less than the total number of channel sub-feature maps in the initial feature map.

[0024] In other application scenarios, channel reorganization of the initial feature map to obtain the initial sub-feature maps corresponding to each channel group can be achieved by grouping the channel sub-feature maps in the initial feature map into at least two channel groups, with each channel group containing at least two channel sub-feature maps. For each channel group, channel compression is performed on the corresponding channel sub-feature maps to obtain compressed channel sub-feature maps for the channel group. These compressed channel sub-feature maps are then superimposed along the channel dimension to obtain the initial sub-feature map for the corresponding channel group. The total number of compressed channel sub-feature maps for each channel group is less than the total number of channel sub-feature maps in the initial feature map.

[0025] Each initial sub-feature map corresponding to a channel group includes several vectors. Each vector in each initial sub-feature map can be a vector arranged along two preset dimensions. Specifically, the vectors arranged along the two preset dimensions can be vectors arranged along the channel dimension and width dimension, vectors arranged along the channel dimension and height dimension, or vectors arranged along the width dimension and height dimension. This application uses the channel dimension and width dimension as an example of the two preset dimensions.

[0026] Step S12: Perform information accumulation processing on the initial sub-feature maps corresponding to each channel group in the row direction and / or column direction to obtain the target sub-feature maps corresponding to each channel group.

[0027] The row and column directions correspond to the arrangement direction of each vector in the initial sub-feature map. The vectors in the initial sub-feature map are arranged according to two preset dimensions, and the row and column directions correspond to these two preset dimensions. If the vectors in the initial sub-feature map are arranged according to the channel dimension and the width dimension, the row direction can be the direction of the channel dimension, and the column direction can be the direction of the width dimension; or, the row direction can be the direction of the width dimension, and the column direction can be the direction of the channel dimension.

[0028] Information accumulation processing can be performed by fusing vectors in the row and / or column directions of each vector in the initial sub-feature map to obtain vectors after information accumulation, or by fusing vectors in the row and / or column directions of each vector in the initial sub-feature map and each vector in the initial sub-feature map that has undergone information accumulation to obtain vectors after information accumulation.

[0029] The target sub-feature map corresponding to the channel group includes several vectors. The total number of vectors in the target sub-feature map corresponding to the channel group is the same as the total number of vectors in the initial sub-feature map corresponding to the channel group. For vectors at corresponding positions between the initial sub-feature map and the target sub-feature map, the vectors in the target sub-feature map are the information-accumulated vectors obtained by processing the information accumulation from the vectors in the initial sub-feature map.

[0030] Specifically, step S12 above can be performed on the initial sub-feature map corresponding to each channel group by performing the following steps: performing information accumulation processing in at least one target direction on the initial sub-feature map corresponding to the channel group to obtain the target sub-feature map corresponding to the channel group. Each target direction can be a row direction or a column direction of the initial sub-feature map. Row directions include forward processing directions and / or reverse processing directions in the row direction. Specifically, a forward processing direction in the row direction can be a processing direction from left to right in the row direction; a reverse processing direction in the row direction can be a processing direction from right to left in the row direction. Column directions include forward processing directions and / or reverse processing directions in the column direction. A forward processing direction in the column direction can be a processing direction from top to bottom in the column direction; a reverse processing direction in the column direction can be a processing direction from bottom to top in the column direction. Wherein, the target direction represents the direction of the preset dimension of the vector arrangement distribution in the initial feature map, and the row direction and column direction represent the direction of the preset dimension of the vector arrangement distribution in the initial feature map. For example, in the initial feature map, the vectors are arranged in the dimensional and height directions. The target direction can be either the dimensional or height direction; for example, the row direction is the height direction and the column direction is the dimensional direction. Alternatively, in the initial feature map, the vectors are arranged in the dimensional and width directions. The target direction can be either the dimensional or width direction; for example, the row direction is the width direction and the column direction is the dimensional direction.

[0031] In some application scenarios, step S12 above involves performing information accumulation processing in the row direction on the initial sub-feature map corresponding to the channel group to obtain the target sub-feature map corresponding to the channel group; or performing information accumulation processing in the column direction on the initial sub-feature map corresponding to the channel group to obtain the target sub-feature map corresponding to the channel group; or performing information accumulation processing in both the row and column directions on the initial sub-feature map corresponding to the channel group to obtain the target sub-feature map corresponding to the channel group. It is understood that at least one target direction can be one target direction, two target directions, four target directions, or more than four target directions. The target directions can be the same direction or different directions.

[0032] For example, this application takes step S12 as an example of performing information accumulation processing on the initial sub-feature map corresponding to the channel group in four target directions to obtain the target sub-feature map corresponding to the channel group. Each target direction represents a different processing direction, specifically including forward and reverse processing directions in the row direction, and forward and reverse processing directions in the column direction. In step S12, this application does not limit the order of processing in each target direction. In some application scenarios, step S12 can be performed in parallel on the initial sub-feature map corresponding to the channel group in four target directions. Specifically, step S12 can be performed on the initial sub-feature map corresponding to the channel group in each of the four target directions to obtain the sub-feature map corresponding to the channel group after information accumulation in each target direction, and then the sub-feature map corresponding to the channel group after information accumulation in each target direction is fused to obtain the target sub-feature map corresponding to the channel group. In other application scenarios, step S12 above can be performing serial information accumulation processing on the initial sub-feature map corresponding to the channel group in four target directions. Specifically, step S12 can be performing information accumulation processing on the initial sub-feature map corresponding to each channel group in four target directions according to a preset processing order to obtain the target sub-feature map corresponding to each channel group. When performing serial information accumulation processing in four target directions, the four target directions include a first direction, a second direction, a third direction, and a fourth direction. The first direction can be a forward processing direction in the row direction, the second direction can be a reverse processing direction in the row direction, the third direction can be a forward processing direction in the column direction, and the fourth direction can be a reverse processing direction in the column direction. For example, the preset processing order can be performing information accumulation processing in the first direction, the second direction, the third direction, and the fourth direction sequentially on the initial sub-feature map corresponding to each channel group to obtain the target sub-feature map corresponding to the channel group, or it can be performing information accumulation processing in the first direction, the third direction, the second direction, and the fourth direction sequentially on the initial sub-feature map corresponding to each channel group to obtain the target sub-feature map corresponding to the channel group. It is understood that this application takes the execution of serial information accumulation processing on the initial sub-feature map corresponding to the channel group in four target directions as an example, and this application does not limit the execution order of serial information accumulation processing in each target direction, which will not be elaborated here.

[0033] Step S13: Perform fusion processing on the target sub-feature maps corresponding to each channel group to obtain the target feature map.

[0034] Fusion processing can include splicing, overlaying according to channel grouping order, or feature fusion. The target feature map represents the information accumulation result of the initial feature map of the image to be processed, which is the input information accumulation model to obtain the feature map output by the information accumulation model.

[0035] In some application scenarios, step S13 above can be to concatenate the target sub-feature maps corresponding to each channel group to obtain the target feature map, or to superimpose the target sub-feature maps corresponding to each channel group according to the channel grouping order to obtain the target feature map, or to input the target sub-feature maps corresponding to each channel group into the fusion module for feature fusion processing to obtain the target feature map. The fusion module is equipped with a preset feature fusion network, which may include, but is not limited to, a feature concatenation network, an attention-based fusion network, or a bilinear pooling-based fusion network. The preset attention network may be an attention-based feature extraction network and / or an attention-based fusion network. This application uses the preset feature fusion network as an example of a feature concatenation network.

[0036] The above scheme involves inputting the acquired image to be processed into an information accumulation model. The information accumulation model performs channel recombination processing on the initial feature maps of the acquired image to obtain initial sub-feature maps corresponding to each channel group. Information accumulation processing is then performed on the initial sub-feature maps corresponding to each channel group in the row and / or column directions to obtain target sub-feature maps corresponding to each channel group. Finally, the target sub-feature maps corresponding to each channel group are fused to obtain the target feature map. This approach reduces computational complexity by performing pre-channel recombination processing so that information accumulation processing can be performed on individual channel groups. Information accumulation processing in different directions on a single channel group yields the target sub-feature map of that single channel group, improving the utilization rate of global information in the image to be processed. Fusion processing is then performed on the target sub-feature maps of each channel group to ensure the integrity of image information in the image to be processed, thereby achieving the accumulation of global information in the target feature map and improving the efficiency of image processing.

[0037] In some embodiments, after step S13, this application also provides an image processing method, which includes: acquiring a target feature map corresponding to an image to be processed, the target feature map being obtained from steps S11 to S13; performing preset classification processing or preset segmentation processing on the target feature map to obtain an image processing result of the image to be processed. The target feature map is input into a preset processing module to obtain the image processing result of the image to be processed output by the preset processing module. Depending on the type of downstream task, the preset processing module may be equipped with a preset processing algorithm, which is used for preset segmentation processing or preset classification processing. The image processing result represents the classification result or segmentation result of the target object in the image to be processed.

[0038] It can be assumed that after obtaining the target feature map, the target feature map can be used to perform downstream tasks (image classification tasks or image segmentation tasks). This can improve the accuracy of the processing results of downstream tasks by using the target feature map with global information accumulation, while saving computing resources in the image processing process.

[0039] In some embodiments, the information accumulation model includes a channel grouping module and a channel compression module. The step of performing channel recombination processing on the initial feature map of the acquired image to be processed using the information accumulation model to obtain initial sub-feature maps corresponding to each channel group may include the following steps: performing channel grouping processing on the initial feature map of the image to be processed using the channel grouping module to obtain grouped sub-feature maps corresponding to each channel group; and performing channel compression processing on the grouped sub-feature maps corresponding to each channel group using the channel compression module to obtain the initial sub-feature maps corresponding to each channel group.

[0040] The initial feature map of the image to be processed includes multiple channel sub-feature maps superimposed along the channel dimension. The channel grouping module is used to group the channel sub-feature maps in the initial feature map to obtain at least two channel groups, each channel group including at least one grouped sub-feature map. The grouped sub-feature maps are simply the channel sub-feature maps after channel grouping. The grouped sub-feature map corresponding to a channel group represents the channel sub-feature map in the initial feature map after channel grouping. The channel grouping process can be performed by dividing the channel sub-feature maps in the initial feature map into a fixed number of channel groups according to a preset order along the channel dimension, resulting in at least two groups of grouped sub-feature maps. The preset order can be a polling order, a random order, etc. For example, the channel dimension of the grouped sub-feature maps in the initial sub-feature map corresponding to each channel group can be sorted according to the original order of the channel sub-feature maps in the initial feature map, a random order, or a preset order. The channel compression module is used to perform channel compression processing on the input feature map to reduce the number of channels in the input feature map. For each group of channels, the sub-feature map after grouping is compressed to obtain at least one compressed sub-feature map, and each compressed sub-feature map is used as the channel sub-feature map in the initial sub-feature map.

[0041] For example, in order to improve the representational capability of the target feature map output by the information accumulation model, this application pre-groups the data input to the information accumulation model by channels. For instance, the size of the initial feature map corresponding to the image to be processed after preset feature extraction processing is... C is the channel size, and H and W represent the height and width, respectively. Dividing the initial feature map into K groups along the channel direction, the number of channels in the sub-feature maps within each channel group becomes... Meanwhile, with the height and width remaining unchanged, the channels of the sub-feature maps corresponding to the channel groups can be named... ,Right now equal Then the size of the sub-feature map corresponding to each channel group becomes There are K channel groups in total. In the field of deep learning, the number of channels C is typically 4, 8, 16, 32, etc., so K typically takes values ​​of 2, 4, 8, etc. For example, the total number of channel groups is determined based on the total number of channels in the initial feature map, and it must be ensured that the total number of channels C in the initial feature map is divisible by the total number of channel groups K. The grouping method can be to first group the first... Divide the channels into groups, and then... Each channel is grouped into a set, and so on, to obtain the sub-feature maps corresponding to each channel group.

[0042] For example, to reduce computational load, channel compression is performed on the grouped sub-feature maps corresponding to each channel group. For instance, the channel compression module includes a convolution module, a normalization module (i.e., batch normalization), and a mapping module. The convolution module in the channel compression module can be equipped with a preset convolutional neural network. For example, the preset convolutional neural network is... The convolution, such as the mapping module in the channel compression module, can be used for ReLU activation. The data for each channel group is processed by the convolution module in the channel compression module. Become ,in Then, using BN and ReLU activation, the convolutional data in each channel group is processed to obtain the initial sub-feature map corresponding to each channel group.

[0043] It can be argued that channel grouping and channel compression can improve the feature representation capability of the initial sub-feature maps corresponding to each channel group while reducing the computational load.

[0044] Please see Figure 2 , Figure 2 yes Figure 1 A schematic diagram of the sub-process of step S12.

[0045] In some embodiments, the initial sub-feature map includes several initial vectors, and the information accumulation model includes a weight generation module and a row and column information update module. Step S12 above may include the following steps: performing the following steps on the initial sub-feature map corresponding to each channel group: Figure 2 The steps shown are as follows: Step S21: Obtain the initial update weights of each initial vector in the initial sub-feature map corresponding to the channel group through the weight generation module. Step S22: Input the initial sub-feature map and the initial update weights of each initial vector into the row and column information update module to obtain the target sub-feature map corresponding to the channel group output by the row and column information update module.

[0046] Each vector in the initial sub-feature map is used as an initial vector. The initial sub-feature map includes several initial vectors. The weight generation module generates initial update weights for each vector in the input feature map. The row and column information update module performs information accumulation processing on the input feature map in the row and / or column directions based on each vector in the input feature map and its corresponding initial update weights. The initial update weights characterize the importance of the corresponding initial vectors during row and column information accumulation; the higher the initial update weights, the higher the importance of the corresponding initial vectors during row and column information accumulation.

[0047] In some application scenarios, the weight generation module includes a mapping module. Step S21 above can involve directly inputting each initial vector from the initial sub-feature map into the mapping module within the weight generation module to obtain the initial update weights of each initial vector output by the mapping module. The mapping module in the weight generation module is equipped with a preset activation function or a fully connected layer. The preset activation function can be ReLU, Sigmoid, Softmax, etc. Essentially, the mapping result obtained by mapping each initial vector in the initial sub-feature map corresponding to the channel group is directly used as the initial update weight of each initial vector.

[0048] In other application scenarios, the weight generation module includes a convolution module and a mapping module. Step S21 can involve inputting each initial vector from the initial sub-feature map into the convolution module of the weight generation module to obtain the convolution result of each initial vector output by the convolution module; inputting the convolution result of each initial vector into the mapping module of the weight generation module to obtain the mapping result of each initial vector output by the mapping module; and then directly using the mapping result of each initial vector as the initial update weight of each initial vector.

[0049] In other application scenarios, the weight generation module includes a convolution module, a normalization module, and a mapping module. Step S21 can involve inputting each initial vector from the initial sub-feature map into the convolution module of the weight generation module to obtain the convolution result of each initial vector output by the convolution module; inputting the convolution result of each initial vector into the normalization module of the weight generation module to obtain the normalized result of each initial vector output by the normalization module; and inputting the normalized result of each initial vector into the mapping module of the weight generation module to obtain the mapping result of each initial vector output by the mapping module. The mapping result of each initial vector is then directly used as the initial update weight for each initial vector.

[0050] In some embodiments, the weight generation module includes several cascaded weight generation layers, each weight generation layer including a cascaded convolution module, a normalization module, and a mapping module. Step S21 above may include the following steps: inputting each vector in the current feature map to be processed into the convolution module in the current weight generation layer to obtain the convolution result of each vector in the current feature map output by the convolution module in the current weight generation layer. Inputting the convolution result of each vector in the current feature map to be processed into the normalization module in the current weight generation layer to obtain the normalized result of each vector in the current feature map output by the normalization module in the current weight generation layer. Inputting the normalized result of each vector in the current feature map to be processed into the mapping module in the current weight generation layer to obtain the mapping result of each vector in the current feature map output by the mapping module in the current weight generation layer. Using the mapping result of each vector output by the mapping module in the last weight generation layer as the initial update weight of each initial vector in the initial sub-feature map corresponding to the channel group.

[0051] The convolution module in the weight generation layer performs convolution processing on the input feature map to obtain the convolution results of each vector in the input feature map. A preset convolutional neural network can be set on the convolution module in the weight generation layer; the specific convolutional structure of the neural network is not limited here. The normalization module in the weight generation layer normalizes the input feature map to obtain the normalized results of each vector in the input feature map. The specific settings for the normalization module in the weight generation layer can be found in the relevant content above for the normalization module, and will not be repeated here. The mapping module in the weight generation layer maps the input feature map to obtain the mapped results of each vector in the input feature map. The specific settings for the mapping module in the weight generation layer can be found in the relevant content above for the mapping module, and will not be repeated here.

[0052] Each vector in the current feature map to be processed is a mapping result of each initial vector in the initial sub-feature map or the vector output by the mapping module in the previous weight generation layer. There can be one or more of these vectors, and the total number of weight generation layers in the weight generation module is not limited. This application uses a weight generation module with three weight generation layers as an example.

[0053] In some application scenarios, when the current weight generation layer is the first weight generation layer in the weight generation module, the current feature map to be processed is the initial sub-feature map. Each initial vector in the initial sub-feature map is input into the convolution module of the current weight generation layer to obtain the convolution result of each initial vector in the initial sub-feature map output by the convolution module of the current weight generation layer. The convolution result of each initial vector in the initial sub-feature map is input into the normalization module of the current weight generation layer to obtain the normalized result of each initial vector in the initial sub-feature map output by the normalization module of the current weight generation layer. The normalized result of each initial vector in the initial sub-feature map is input into the mapping module of the current weight generation layer to obtain the mapping result of each initial vector in the initial sub-feature map output by the mapping module of the current weight generation layer. The mapping result of each initial vector in the initial sub-feature map is used as the mapping result of each vector output by the first weight generation layer.

[0054] In other application scenarios, when the current weight generation layer is not the first weight generation layer in the weight generation module, each vector in the current feature map to be processed is the mapping result of each vector output by the mapping module in the previous weight generation layer; that is, the current feature map to be processed is the feature map output by the previous weight generation layer. The feature map output by the previous weight generation layer is input into the convolution module in the current weight generation layer to obtain the convolution result of each vector in the feature map output by the previous weight generation layer. The convolution result of each vector in the feature map output by the previous weight generation layer is input into the normalization module in the current weight generation layer to obtain the normalized result of each vector in the feature map output by the previous weight generation layer. The normalized result of each vector in the feature map output by the previous weight generation layer is input into the mapping module in the current weight generation layer to obtain the mapping result of each vector in the feature map output by the previous weight generation layer. The mapping result of each vector in the feature map output by the previous weight generation layer is used as the mapping result of each vector output by this non-first weight generation layer.

[0055] In response to the fact that the current weight generation layer is the last weight generation layer in the weight generation module, the mapping results of each vector output by the last weight generation layer are used as the initial update weights of each initial vector output by the weight generation module.

[0056] like Figure 3 As shown, the initial sub-features corresponding to each channel group serve as the input data for the weight generation module. This input data is... In the weight generation module, each weight generation layer is used to convolve the initial sub-feature map of each channel group. Taking a weight generation module with three cascaded weight generation layers as an example, the initial sub-feature map is sequentially input into the first and second weight generation layers for the first and second convolutions, respectively. The data size remains the same after the first and second convolutions. The mapping results of the vectors output by the second weight generation layer are input into the third weight generation layer to obtain the output of the third weight generation layer. After the third convolution, the number of data channels of the output of the third weight generation layer becomes 1, and the data size becomes... The output of the third weight generation layer is used as the initial update weights for each initial vector. The initial update weights for any initial vector can be represented as adaptive weights. .

[0057] In some application scenarios, step S22 above can be a weighted fusion of two adjacent initial vectors and their initial update weights in the initial sub-feature map along at least one target direction to obtain the updated vectors corresponding to the two adjacent initial vectors. Each target direction can be a row direction or a column direction corresponding to the initial sub-feature map. Specifically, at least one target direction can be any one, any two, any three, or all four of the first, second, third, and fourth directions mentioned above.

[0058] In some application scenarios, the row and column information update module includes multiple cascaded information update layers. Each information update layer is used to update the vectors in the feature map to be updated in the input information update layer in the target direction corresponding to the information update layer. The target direction includes the row direction or the column direction. The above step S22 may include the following steps: Based on the position of each current vector to be updated in the current feature map to be updated in the current update direction corresponding to the current information update layer, determine the updated vector corresponding to each current vector to be updated, including: In response to the current vector to be updated being the first vector to be updated in the current feature map to be updated in the current update direction, retain the current vector to be updated as the updated vector corresponding to the current vector to be updated. In response to the current vector to be updated being not the first vector to be updated in the current feature map to be updated in the current update direction, adjust the initial update weight corresponding to the current vector to be updated through the current information update layer to obtain the target update weight corresponding to the current vector to be updated; specifically, based on the updated vectors of the current vector to be updated and the candidate vectors to be updated, as well as the target update weight of the current vector to be updated, obtain the updated vector corresponding to the current vector to be updated. Among them, the candidate vector to be updated is the previous vector to be updated in the current update direction in the current feature map to be updated.

[0059] Please see Figure 4 , Figure 4 yes Figure 2 A schematic diagram of the sub-process of step S22.

[0060] In some embodiments, the row and column information update module includes multiple cascaded information update layers. Each information update layer is used to update the vectors in the feature map to be updated in the input information update layer in the target direction corresponding to the information update layer. The target direction includes the row direction or the column direction. Step S22 above may include the following steps: Step S41: Adjust the initial update weights corresponding to each current vector to be updated in the current feature map through the current information update layer to obtain the target update weights corresponding to each current vector to be updated.

[0061] The target direction corresponding to the information update layer represents the information accumulation direction of the input feature map of the information update layer. When the target direction corresponding to the information accumulation layer is the row direction, information is accumulated for each vector in the input feature map of the information update layer in the row direction. When the target direction corresponding to the information accumulation layer is the column direction, information is accumulated for each vector in the input feature map of the information update layer in the column direction. For the specific settings of the row and column directions in relation to the dimension, width, and height directions of the feature map, please refer to the above content, which will not be repeated here.

[0062] The information update layers in the row and column information update module have the same structure, but their parameters can be the same or different. The current feature map to be updated is either the initial sub-feature map or the updated feature map output by the previous information update layer. The current information update layer can be any one of the information update layers corresponding to the row and column information update module. The target update weight represents the importance of the corresponding vector when accumulating information in the target direction corresponding to each information update layer. The higher the target update weight, the higher the importance of the corresponding vector when accumulating row and column information in the target direction corresponding to the information update layer. The feature map to be updated is the feature map input to the current information update layer. The vector to be updated is any vector in the feature map to be updated. When the feature map to be updated is an initial sub-feature map, the vector to be updated is any initial vector in the initial sub-feature map. When the current information update layer is not the first information update layer in the row and column information update module, the initial update weight corresponding to the vector to be updated is the initial update weight of the initial vector at the same vector position among the initial vectors.

[0063] In some application scenarios, step S41 above can be used to adjust the initial update weight of the vector to be updated based on the preset adjustment value of the current information update layer and / or the position of the vector to be updated in the current feature map to be updated, thereby obtaining the target update weight of the vector to be updated. Specifically, step S41 above can be used to adjust the initial update weight of the vector to be updated based on the preset adjustment value of the current information update layer, thereby obtaining the target update weight of the vector to be updated, including: using the product or difference between the initial update weight of the vector to be updated and the preset adjustment value of the current information update layer as the target update weight of the vector to be updated. The preset adjustment value of the current information update layer is preset based on the number of layers in the current information update layer or the target direction corresponding to the current information update layer. Specifically, step S41 above can be used to adjust the initial update weight of the vector to be updated based on the position of the vector to be updated in the current feature map to be updated, thereby obtaining the target update weight of the vector to be updated. The adjustment process for the initial update weight of the vector to be updated differs depending on the position of the vector to be updated in the target direction in the current feature map to be updated. In other application scenarios, if the feature map to be updated includes multiple vector distributions with the same target direction (i.e., vector distributions with multiple rows or columns), the initial update weights corresponding to the current vectors to be updated at the same position in each target direction are adjusted in the same way. For example, the initial update weights corresponding to the vectors to be updated with the same index in the first or second row are adjusted in the same way.

[0064] In some application scenarios, when the current information update layer is the first information update layer in the row and column information update module, the feature map to be updated is the initial sub-feature map, and the vector to be updated is any initial vector in the initial sub-feature map. Step S41 above can be achieved by adjusting the initial update weights corresponding to each initial vector in the initial sub-feature map through the first information update layer to obtain the target update weights corresponding to each initial vector.

[0065] In some application scenarios, when the current information update layer is not the first information update layer in the row and column information update module, the feature map to be updated is the initial sub-feature map after information accumulation processing, and the vector to be updated is any vector in the initial sub-feature map after information accumulation processing. Step S41 can be performed by adjusting the initial update weights corresponding to each vector in the initial sub-feature map after information accumulation processing through a non-first information update layer to obtain the target update weights corresponding to each vector in the initial sub-feature map after information accumulation processing.

[0066] Please see Figure 5 , Figure 5 yes Figure 4A schematic diagram of the sub-process of step S41.

[0067] In some embodiments, the target direction corresponding to the current information update layer is taken as the current update direction; step S41 above may include the following steps: Step S51: In response to the fact that the current vector to be updated is the first vector to be updated in the current update direction in the current feature map, adjust the initial update weight corresponding to the current vector to be updated to a preset value to obtain the target update weight corresponding to the current vector to be updated.

[0068] The current update direction is the target direction corresponding to the current information update layer, representing the information accumulation direction of the input feature map of the current information update layer. When the current vector to be updated is the first vector to be updated in the current update direction within the current feature map, the initial update weights corresponding to the current vector to be updated are adjusted to a preset value, and this preset value is used as the target update weights corresponding to the current vector to be updated. For example, the preset value can be set to 0 or 1. This application uses 0 as an example.

[0069] Understandably, when the feature map to be updated includes multiple rows and columns of vectors to be updated, if the target direction or the current update direction is a row direction (e.g., a forward processing direction or a reverse processing direction in the row direction), step S51 is executed row by row in the feature map to be updated. For the current vector to be updated in each row of the feature map to be updated, if the current vector to be updated is the first vector to be updated in that row, a preset value is used as the target update weight of the current vector to be updated. If the target direction or the current update direction is a column direction (e.g., a forward processing direction or a reverse processing direction in the column direction), step S51 is executed column by column in the feature map to be updated. For the current vector to be updated in each column of the feature map to be updated, if the current vector to be updated is the first vector to be updated in that column, a preset value is used as the target update weight of the current vector to be updated.

[0070] Step S52: In response to the fact that the current vector to be updated is not the first vector to be updated in the current update direction in the current feature map, the target update weight corresponding to the current vector to be updated is obtained according to the current vector to be updated, the candidate vectors to be updated, and the initial update weight corresponding to the current vector to be updated.

[0071] Here, the candidate vector to be updated is the previous vector to be updated in the current update direction within the current feature map to be updated. It can be understood that when the feature map to be updated includes multiple rows and columns of vectors to be updated, if the target direction or the current update direction is a row direction (e.g., a forward processing direction or a reverse processing direction within a row direction), step S51 is executed row by row in the feature map to be updated. For the current vector to be updated in each row of the feature map to be updated, if the current vector to be updated is not the first vector to be updated in that row, the candidate vector to be updated is the previous vector to be updated in that row within the current update direction. For example, if the current update direction is a forward processing direction within a row, the candidate vector to be updated is the vector to the left of the current vector to be updated in that row. For example, if the current update direction is a reverse processing direction within a row, the candidate vector to be updated is the vector to the right of the current vector to be updated in that row. If the target direction or the current update direction is a column direction (e.g., a forward processing direction or a reverse processing direction in the column direction), step S51 is performed column by column in the feature map to be updated. For the current vector to be updated in each column of the feature map to be updated, if the current vector to be updated is not the first vector to be updated in that column, the candidate vector to be updated is the vector to be updated preceding the current vector to be updated in that row in the current update direction. For example, if the current update direction is a forward processing direction in the column direction, the candidate vector to be updated is the vector to be updated above the current vector to be updated in that column. For example, if the current update direction is a reverse processing direction in the column direction, the candidate vector to be updated is the vector to be updated below the current vector to be updated in that column.

[0072] In some application scenarios, the target update weight of the current vector to be updated is obtained based on the current vector to be updated, the candidate vectors to be updated, and the initial update weights corresponding to the current vector to be updated. This includes: weighted fusion of the current vector to be updated, the candidate vectors to be updated, and the initial update weights corresponding to the current vector to be updated to obtain the target update weights corresponding to the current vector to be updated; or, performing convolution and first mapping processing on the current vector to be updated and the candidate vectors to be updated respectively to obtain the mapping results corresponding to the current vector to be updated and the mapping results corresponding to the candidate vectors to be updated; using the product of the mapping results corresponding to the current vector to be updated and the mapping results corresponding to the candidate vectors to be updated as the initial mapping result corresponding to the current vector to be updated; using the difference between the initial mapping result corresponding to the current vector to be updated and the initial update weights corresponding to the current vector to be updated as the advanced mapping result corresponding to the current vector to be updated; and using the advanced mapping result corresponding to the current vector to be updated as the target update weights corresponding to the current vector to be updated; or, performing a second mapping processing on the advanced mapping result corresponding to the current vector to be updated to obtain the target mapping result corresponding to the current vector to be updated, and using the target mapping result corresponding to the current vector to be updated as the target update weights corresponding to the current vector to be updated. The first mapping process and the second mapping process can be mapping processes corresponding to the same or different activation functions. For example, the first mapping process can be a ReLU activation process, and the second mapping process can be a sigmoid activation process.

[0073] It can be argued that by determining different adjustment methods based on the position of the current vector to be updated in the feature map to be updated, the target update weights corresponding to the vector to be updated with higher accuracy can be adaptively generated, thereby improving the information accumulation ability of the updated vectors corresponding to the current vectors to be updated and improving the expressive ability of the updated vectors corresponding to the current vectors to be updated.

[0074] Step S42: Based on each current vector to be updated and the target update weight corresponding to each current vector to be updated, determine the updated vector corresponding to each current vector to be updated.

[0075] The updated vector corresponding to the vector to be updated represents the information accumulation result of the vector to be updated in the current information update layer.

[0076] In some application scenarios, for each current vector to be updated, the product of the current vector to be updated and the current vector to be updated is used as the updated vector corresponding to the current vector to be updated.

[0077] Step S43: Use the updated vectors corresponding to each current vector to be updated as vectors in the updated feature map output by the current information update layer.

[0078] For the current information update layer, the updated feature map obtained by taking the updated vectors corresponding to each vector to be updated in the feature map to be updated is used as the updated feature map, and the updated vectors corresponding to each vector to be updated are used as the vectors at the corresponding vector positions in the updated feature map.

[0079] Specifically, the updated feature map output by the last information update layer in the row and column information update module is used as the target sub-feature map corresponding to the channel group.

[0080] In some embodiments, the target direction corresponding to the current information update layer is taken as the current update direction. Step S42 above may include the following steps: In response to the current vector to be updated being the first vector to be updated in the current update direction in the current feature map to be updated, an updated vector corresponding to the current vector to be updated is obtained based on the current vector to be updated and the target update weight corresponding to the current vector to be updated. In response to the current vector to be updated being a non-first vector to be updated in the current update direction in the current feature map to be updated, an updated vector corresponding to the current vector to be updated is obtained based on the updated vectors of the current vector to be updated and the candidate vectors to be updated, as well as the target update weight of the current vector to be updated.

[0081] The candidate vector to be updated is the previous vector to be updated in the current update direction in the current feature map to be updated.

[0082] In some application scenarios, based on the current vector to be updated and the target update weight corresponding to the current vector to be updated, the updated vector corresponding to the current vector to be updated is obtained, including: taking the product between the current vector to be updated and the current vector to be updated as the updated vector corresponding to the current vector to be updated.

[0083] In some application scenarios, the updated vector corresponding to the current vector to be updated is obtained based on the updated vectors of the current vector to be updated, the candidate vectors to be updated, and the target update weight of the current vector to be updated. This includes: weighting and fusing the updated vectors of the current vector to be updated and the candidate vectors to be updated based on the target update weight of the current vector to be updated to obtain the updated vector corresponding to the current vector to be updated; or weighting and fusing the updated vectors of the current vector to be updated and the candidate vectors to be updated based on the target update weight of the current vector to be updated and the initial update weight of the candidate vectors to be updated to obtain the updated vector corresponding to the current vector to be updated; or using the difference between the preset value and the target update weight of the current vector to be updated as the target update weight of the candidate vector to be updated, and weighting and fusing the updated vectors of the current vector to be updated and the candidate vectors to be updated based on the target update weight of the current vector to be updated and the target update weight of the candidate vector to be updated to obtain the updated vector corresponding to the current vector to be updated.

[0084] For example, the row and column information update module includes four information update layers, and the row and column information update module can be executed serially as follows: Figure 6 The four steps shown involve each information update layer accumulating information in different directions on the input feature map to be updated. For example, information is accumulated from left to right in the row direction, from right to left in the row direction, from top to bottom in the column direction, and from bottom to top in the column direction, until the target sub-feature map is obtained.

[0085] The current update direction of the current information update layer can be at least one of the first, second, third, and fourth directions mentioned above. The execution logic of each information update layer can include the following: performing convolution and first mapping processes on the current vector to be updated and the candidate vector to be updated, respectively, to obtain the mapping result corresponding to the current vector to be updated and the mapping result corresponding to the candidate vector to be updated; taking the product between the mapping result corresponding to the current vector to be updated and the mapping result corresponding to the candidate vector to be updated as the initial mapping result corresponding to the current vector to be updated; taking the difference between the initial mapping result corresponding to the current vector to be updated and the initial update weight corresponding to the current vector to be updated as the advanced mapping result corresponding to the current vector to be updated; performing a second mapping process on the advanced mapping result corresponding to the current vector to be updated to obtain the target mapping result corresponding to the current vector to be updated, and taking the target mapping result corresponding to the current vector to be updated as the target update weight corresponding to the current vector to be updated.

[0086] To clearly illustrate this application, this application uses the current update direction of the current information update layer as the aforementioned first direction as an example, but it is not limited to the first direction as the current update direction, and will not be elaborated further thereafter. Specifically, the calculation of information accumulation from left to right in the row direction can include the following: First, convolution and first mapping processes are performed on the current vector to be updated and the candidate vector to be updated, respectively, to obtain the mapping results corresponding to the current vector to be updated and the candidate vector to be updated. The product of the mapping results corresponding to the current vector to be updated and the candidate vector to be updated is taken as the initial mapping result corresponding to the current vector to be updated. Specifically, the process of determining the initial mapping result corresponding to the current vector to be updated can refer to the following formula (1): Formula (1); Among them, such as Figure 7As shown, n and n-1 are the row direction indices. If the width of the feature map to be updated is W, then the range of the indices n is from 0 to W-1. n is the index of the current vector to be updated in the row direction, and n-1 is the index of the candidate vector to be updated in the row direction. This represents the current vector to be updated, with index n, extracted from the feature map to be updated. This represents the candidate vector with index n-1 extracted from the feature map to be updated. conV represents the above convolutional processing, which can be implemented by the convolutional module in the current information update layer, which has a 1x1 depthwise convolution. The first mapping process mentioned above can be implemented by the first mapping module in the current information update layer. Specifically, the first mapping process can be ReLU activation. This represents the initial mapping result corresponding to the current vector to be updated, used to indicate the similarity between the current vector to be updated and its neighboring vectors in the row direction. The calculated similarity is... The similarity between vectors with index n and n-1 in the same row. The larger the value, the more similar the numbers.

[0087] Then, the difference between the initial mapping result corresponding to the current vector to be updated and the initial update weight corresponding to the current vector to be updated is taken as the advanced mapping result corresponding to the current vector to be updated; the advanced mapping result corresponding to the current vector to be updated is subjected to a second mapping process to obtain the target mapping result corresponding to the current vector to be updated. Specifically, the process of determining the target mapping result corresponding to the current vector to be updated can refer to the following formulas (2) and (3): Formula (2); Formula (3); in, This represents the target mapping result corresponding to the vector to be updated. The adaptive weights are the initial update weights corresponding to the vector to be updated, and these values ​​are generated by the weight generation module. This represents the initial mapping result corresponding to the vector to be updated. The sigmoid function can be calculated using formula (3) above. The value of is the advanced mapping result corresponding to the current vector to be updated, that is, the difference between the initial mapping result corresponding to the current vector to be updated and the initial update weight corresponding to the current vector to be updated. It can be considered that using sigmoid will... Map the target to update the weights.

[0088] In some application scenarios, the difference between the preset value and the target update weight of the current vector to be updated is used as the target update weight of the candidate vector to be updated. The process of weighted fusion of the updated vectors of the current vector to be updated and the candidate vectors to be updated based on the target update weight of the current vector to be updated and the target update weight of the candidate vectors to be updated to obtain the updated vector corresponding to the current vector to be updated can be referred to the following formula (4): Formula (4); in, This represents the vector that needs to be updated. This represents the target update weight corresponding to the vector to be updated. This represents the updated vector of the candidate vector to be updated. This represents the updated vector of the current vector to be updated. The specific formula for information accumulation in the row direction from left to right can be found in formula (4) above. for This means that the first vector to be updated in each row is directly retained as the updated vector corresponding to the vector to be updated. It can be considered that the above formula (4) means that adjacent vectors in the feature map to be updated are weighted and combined; the more similar they are, the stronger the fusion, thus improving the expressive power of the vectors in the updated feature map. It can be understood that through the above calculation, each vector in a row of the feature map to be updated can be weighted and combined. This transforms the vectors to be updated into the updated vectors corresponding to each vector to be updated. For all rows in the feature map to be updated, process them once using the execution logic corresponding to the above formula (4), and the information accumulation from left to right in the row direction will be completed. The calculation process for the remaining information accumulation from right to left in the row direction, information accumulation from top to bottom in the column direction, and information accumulation from bottom to top in the column direction is the same as that for information accumulation from left to right in the row direction, and will not be repeated here.

[0089] It can be considered that by combining the target update weights of the vector and the updated vectors of the candidate vectors to be updated to obtain the updated vector of the current vector to be updated, the fusion of adjacent information in the feature map to be updated can be realized, thereby achieving the fusion of global information and improving the expressive power of the updated feature map output by each information update layer.

[0090] Please see Figure 8 , Figure 8 This is another schematic flowchart of an exemplary embodiment of the image processing method of this application.

[0091] In some embodiments, the image processing method further includes a training step for an information accumulation model, the training step comprising: Step S81: acquiring sample images and preset edge extraction results and / or sample image annotation results of the sample images. Step S82: performing channel recombination processing on the initial feature maps of the sample images using the information accumulation model to obtain initial sub-feature maps corresponding to each sample channel group. Step S83: obtaining a first loss based on the difference between the initial update weights of the sample initial sub-feature maps corresponding to each sample channel group and the preset edge extraction results of the sample images. And / or, Step S84: performing preset image processing on the target feature maps obtained based on the sample initial sub-feature maps corresponding to each sample channel group to obtain sample image processing results of the sample images. Step S85: obtaining a second loss based on the difference between the sample image processing results of the sample images and the sample image annotation results of the sample images. Step S86: adjusting the parameters in the information accumulation model based on the first loss and / or the second loss to obtain the trained information accumulation model.

[0092] The sample image can be an image of the same type as the image to be processed mentioned above, representing the training image data of the information accumulation model. The sample image annotation result represents the image annotation result of the sample image, specifically the annotation classification result or annotation segmentation result obtained by performing annotation processing on the sample image for downstream tasks. The preset edge extraction result of the sample image represents the extraction result obtained by performing edge extraction processing on the sample image. Edge extraction processing can be implemented based on an image edge extraction operator, specifically by inputting the sample image into the image edge extraction operator to obtain the preset edge extraction result of the sample image. For example, the image edge extraction operator can be the Canny image edge extraction operator.

[0093] In some application scenarios, step S82 above may involve performing channel grouping and / or channel compression on the initial feature maps of each channel in the sample image to obtain initial sub-feature maps corresponding to at least two sample channel groups. In other application scenarios, the information accumulation model includes a channel grouping module and a channel compression module. Step S82 above may involve performing channel grouping on the initial feature maps of the sample image using the channel grouping module to obtain grouped sub-feature maps corresponding to each sample channel group. Then, the channel compression module may perform channel compression on the grouped sub-feature maps corresponding to each sample channel group to obtain the initial sub-feature maps corresponding to each channel group. It is understood that step S82 above can refer to the specific process in step S11 above where the information accumulation model is used to perform channel recombination on the initial feature maps of the acquired image to be processed to obtain the initial sub-feature maps corresponding to each channel group; this will not be elaborated further here.

[0094] The initial update weights of the initial sub-feature maps corresponding to each sample channel group are at the same size as the preset edge extraction results of the sample image. The loss determined by the difference between the initial update weights of the initial sub-feature maps corresponding to each sample channel group and the preset edge extraction results of the sample image is used as the first loss.

[0095] The method of obtaining the target feature map based on the initial sub-feature map of each sample channel group is similar to step S12 above, and will not be repeated here.

[0096] Step S84 above may include: performing information accumulation processing on the initial sub-feature maps of each sample channel group in the row direction and / or column direction to obtain the target sub-feature map corresponding to each sample channel group. Then, performing fusion processing on the target sub-feature maps corresponding to each sample channel group to obtain the sample target feature map. This step is similar to step S13 above and will not be repeated here. Finally, performing preset classification processing or preset segmentation processing on the sample target feature map to obtain the sample image processing result of the sample image.

[0097] Specifically, information accumulation processing is performed on the initial sub-feature maps corresponding to each sample channel group in the row and / or column directions to obtain the target sub-feature maps corresponding to each sample channel group. The initial sub-feature maps include several initial sample vectors, and the information accumulation model includes a weight generation module and a row / column information update module. Specifically, the above-mentioned information accumulation processing on the initial sub-feature maps corresponding to each sample channel group in the row and / or column directions to obtain the target sub-feature maps corresponding to each sample channel group includes: performing the following steps for each initial sub-feature map corresponding to a sample channel group: obtaining the initial update weights of each initial sample vector in the initial sub-feature map corresponding to the sample channel group through the weight generation module; inputting the initial sub-feature map and the initial update weights of each initial sample vector into the row / column information update module to obtain the target sub-feature map corresponding to the sample channel group output by the row / column information update module.

[0098] The loss generated by the difference between the sample image processing result and the sample image annotation result is used as the second loss.

[0099] In some application scenarios, step S86 above can be used to adjust the parameters in the information accumulation model based on the target loss, resulting in a trained information accumulation model. The target loss includes a first loss and / or a second loss. For example, the target loss can be the first loss, or the second loss, or a weighted fusion result of the first and second losses.

[0100] For example, a pre-defined image processing model can be used to train the information accumulation model, such as... Figure 9The preset image processing model shown includes an information accumulation model and a preset processing module. During the application phase, the input to the preset image processing model is the image to be processed, and the output data is either the target feature map of the image to be processed or the image processing result of the image to be processed. During the training phase, the input to the preset image processing model is a sample image, and the output data is either the sample target feature map of the sample image or the sample image processing result of the sample image. In some application scenarios, such as... Figure 9 In the preset image processing model shown, each convolutional module can be set to 1×1 convolution.

[0101] During the training phase, adaptive weights are generated in the weight generation module; these are the initial updated weights of the initial sub-feature maps of each sample in the sample image. Considering that the adaptive weights are similar to indicators of flat edge strength, two loss functions can be designed for the preset image processing model. The design positions are based on, for example... Figure 9 The output data of the weight generation module shown (i.e., the initial updated weights of the initial sub-feature maps corresponding to each sample channel group, denoted as output1) is used to construct an adaptive weight loss, i.e., to construct the first loss, denoted as loss1, between the output data of the weight generation module and the preset edge extraction results. Figure 9 The output data of the preset processing module shown (i.e. the sample image processing result of the sample image, denoted as output2) and the sample image annotation result are used to construct the total loss of the preset image processing model, i.e., to construct the second loss, denoted as loss2.

[0102] During the training phase, the preset edge extraction results for constructing sample images can be obtained by using the Canny image edge extraction operator to extract edges from the training data. The horizontal and vertical edge extraction templates of the Canny operator can be represented as follows: and The process of using the two templates mentioned above to extract image edges, i.e., obtaining the preset edge extraction results of the sample image, can be referred to the following formula (5): Formula (5); Here, Ldata represents the training data, which is typically RGB images such as JPG and PNG as sample images. It can convert sample images to grayscale, and Gray performs the RGB-to-grayscale conversion operation. This is a convolution operation. `max` is a maximum value operation. and The above information can be used as a reference, and will not be repeated here. The preset edge extraction result represents the sample image.

[0103] Specifically, the process of constructing the first loss can be referred to the following formula (6): Formula (6); in, The 1 norm is defined as the sum of the absolute values ​​of all its elements. This indicates an image downsampling operation. Downsampling is a necessary operation because depth typically downsamples data during feature extraction, and the downsampling factor varies depending on the feature extraction model. Therefore, the downsampling operation ensures that the size of the preset edge extraction result (ledge) of the sample image is consistent with the initial update weights (output1) of the initial sub-feature map corresponding to the sample channel group. `number` represents the number of elements / vectors contained in `output1`. This represents the first loss, namely the first loss loss1 mentioned above.

[0104] The second loss is designed based on the actual usage needs in the preset image processing model (i.e., the specific type of downstream task). For example, if the preset processing module is recognition and detection, then loss2 is the cross-entropy loss. If the preset processing module is segmentation, then L1 loss is used.

[0105] For example, in step S86 above, this application performs training in three steps, specifically including the following: Please refer to Figure 9First, in the first training phase: the image feature extraction module, the row and column information update module, and the preset processing module are frozen. The channel grouping module, the channel compression module, and the weight generation module are trained using loss1. The image feature extraction module uses a publicly available pre-trained image classification network such as VGG16 or ResNet. Training continues until the preset image processing model converges. Then, in the second training phase: the image feature extraction module, the channel grouping module, the channel compression module, and the weight generation module are frozen. The row and column information update module and the preset processing module are trained using loss2. This continues until convergence. Finally, in the third training phase: only loss2 is used, the image feature extraction module, the channel grouping module, the channel compression module, and the preset processing module are frozen, and the weight generation module and the row and column information update module are trained. The learning rate for the first and second training phases is set to 0.0001. For the third training phase, the learning rate for both the weight generation module and the row and column information update module is set to 0.0001. That is, the learning rate for the weight generation module is one-tenth that of the row and column information update module. The main purpose of setting different learning rates in the three stages is to ensure the weight generation module converges quickly. To prevent overfitting from affecting the overall model performance, a smaller learning rate is set. Since the initial updated weights resemble an indicator of flat edge strength, a loss 1 is designed and trained using loss 1 in the first stage. Its main purpose is to initialize the parameters of the weight generation module, effectively accelerating the model's convergence in the subsequent two training stages. Through the learning in the second and third training stages, especially the third stage, the parameters of the weight generation module are gradually adjusted. After the model fully converges, the initial updated weights output by the weight generation module can resemble, but not exactly resemble, image edges, improving the effectiveness of the weight generation module's parameters.

[0106] It is understandable that after the preset image processing model is trained, that is, after the information accumulation model in the preset image processing model is trained, the trained information accumulation model is used to execute the above steps S11 to S13. The preset image processing model can be used directly to output the image processing result from the image to be processed, or only the information accumulation model in the preset image processing model can be used to output the target feature map from the image to be processed.

[0107] It is understandable that the initial update weights in this application... Used to adjust the target update weight The basic idea behind this approach is that when the image to be processed is in a flat region, adjacent vectors are relatively similar, and the value should be increased accordingly. The value of should be reduced when the image to be processed is in an edge region, as the difference between adjacent vectors is relatively large. The value of . Therefore, the initial update weight can be similar to an indicator of the strength of a flat edge.

[0108] The above scheme involves inputting the acquired image to be processed into an information accumulation model. The information accumulation model performs channel recombination processing on the initial feature maps of the acquired image to obtain initial sub-feature maps corresponding to each channel group. Information accumulation processing is then performed on the initial sub-feature maps corresponding to each channel group in the row and / or column directions to obtain target sub-feature maps corresponding to each channel group. Finally, the target sub-feature maps corresponding to each channel group are fused to obtain the target feature map. This approach reduces computational complexity by performing pre-channel recombination processing so that information accumulation processing can be performed on individual channel groups. Information accumulation processing in different directions on a single channel group yields the target sub-feature map of that single channel group, improving the utilization rate of global information in the image to be processed. Fusion processing is then performed on the target sub-feature maps of each channel group to ensure the integrity of image information in the image to be processed, thereby achieving the accumulation of global information in the target feature map and improving the efficiency of image processing.

[0109] Please see Figure 10 , Figure 10 This is a schematic diagram of an embodiment of the image processing apparatus of this application. The image processing apparatus 100 includes a channel reconstruction module 101, an information accumulation module 102, and a fusion processing module 103. The channel reconstruction module 101 is used to input the acquired image to be processed into an information accumulation model, and to perform channel reconstruction processing on the initial feature map of the acquired image to be processed through the information accumulation model to obtain the initial sub-feature map corresponding to each channel group. The information accumulation module 102 is used to perform information accumulation processing on the initial sub-feature map corresponding to each channel group in the row direction and / or column direction to obtain the target sub-feature map corresponding to each channel group. The fusion processing module 103 is used to perform fusion processing on the target sub-feature map corresponding to each channel group to obtain the target feature map.

[0110] Please refer to the image processing method for the functions performed by each module; they will not be repeated here.

[0111] The above scheme involves inputting the acquired image to be processed into an information accumulation model. The information accumulation model performs channel recombination processing on the initial feature maps of the acquired image to obtain initial sub-feature maps corresponding to each channel group. Information accumulation processing is then performed on the initial sub-feature maps corresponding to each channel group in the row and / or column directions to obtain target sub-feature maps corresponding to each channel group. Finally, the target sub-feature maps corresponding to each channel group are fused to obtain the target feature map. This approach reduces computational complexity by performing pre-channel recombination processing so that information accumulation processing can be performed on individual channel groups. Information accumulation processing in different directions on a single channel group yields the target sub-feature map of that single channel group, improving the utilization rate of global information in the image to be processed. Fusion processing is then performed on the target sub-feature maps of each channel group to ensure the integrity of image information in the image to be processed, thereby achieving the accumulation of global information in the target feature map and improving the efficiency of image processing.

[0112] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 110 includes a memory 111 and a processor 112. The processor 112 is used to execute program instructions stored in the memory 111 to implement the steps in the above-described image processing method embodiment. In a specific implementation scenario, the electronic device 110 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 110 may also include mobile devices such as laptops and tablets, which are not limited here.

[0113] Specifically, processor 112 controls itself and memory 111 to implement the steps in the above-described image processing method embodiments. Processor 112 can also be referred to as a CPU (Central Processing Unit). Processor 112 may be an integrated circuit chip with signal processing capabilities. Processor 112 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 112 can be implemented using integrated circuit chips.

[0114] The above scheme involves inputting the acquired image to be processed into an information accumulation model. The information accumulation model performs channel recombination processing on the initial feature maps of the acquired image to obtain initial sub-feature maps corresponding to each channel group. Information accumulation processing is then performed on the initial sub-feature maps corresponding to each channel group in the row and / or column directions to obtain target sub-feature maps corresponding to each channel group. Finally, the target sub-feature maps corresponding to each channel group are fused to obtain the target feature map. This approach reduces computational complexity by performing pre-channel recombination processing so that information accumulation processing can be performed on individual channel groups. Information accumulation processing in different directions on a single channel group yields the target sub-feature map of that single channel group, improving the utilization rate of global information in the image to be processed. Fusion processing is then performed on the target sub-feature maps of each channel group to ensure the integrity of image information in the image to be processed, thereby achieving the accumulation of global information in the target feature map and improving the efficiency of image processing.

[0115] Please see Figure 12 , Figure 12 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 120 stores program instructions 1201 thereon, which, when executed by a processor, implement the steps in any of the above-described image processing method embodiments.

[0116] The above scheme involves inputting the acquired image to be processed into an information accumulation model. The information accumulation model performs channel recombination processing on the initial feature maps of the acquired image to obtain initial sub-feature maps corresponding to each channel group. Information accumulation processing is then performed on the initial sub-feature maps corresponding to each channel group in the row and / or column directions to obtain target sub-feature maps corresponding to each channel group. Finally, the target sub-feature maps corresponding to each channel group are fused to obtain the target feature map. This approach reduces computational complexity by performing pre-channel recombination processing so that information accumulation processing can be performed on individual channel groups. Information accumulation processing in different directions on a single channel group yields the target sub-feature map of that single channel group, improving the utilization rate of global information in the image to be processed. Fusion processing is then performed on the target sub-feature maps of each channel group to ensure the integrity of image information in the image to be processed, thereby achieving the accumulation of global information in the target feature map and improving the efficiency of image processing.

[0117] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0118] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0120] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0121] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An image processing method, characterized in that, The method includes: The acquired image to be processed is input into the information accumulation model. The information accumulation model performs channel recombination processing on the initial feature map of the acquired image to be processed to obtain the initial sub-feature map corresponding to each channel group. The initial sub-feature maps corresponding to each channel group are processed by accumulating information in the row direction and / or column direction to obtain the target sub-feature maps corresponding to each channel group. The target sub-feature maps corresponding to each channel group are fused to obtain the target feature map.

2. The method according to claim 1, characterized in that, The initial sub-feature map includes several initial vectors, and the information accumulation model includes a weight generation module and a row and column information update module; the step of performing information accumulation processing on the initial sub-feature map corresponding to each channel group in the row direction and / or column direction to obtain the target sub-feature map corresponding to each channel group includes: For each of the channel groups, perform the following steps for the initial sub-feature map: The initial update weights of each initial vector in the initial sub-feature map corresponding to the channel group are obtained through the weight generation module. The initial sub-feature map and the initial update weights of each initial vector are input into the row and column information update module to obtain the target sub-feature map corresponding to the channel group output by the row and column information update module.

3. The method according to claim 2, characterized in that, The row and column information update module includes multiple cascaded information update layers. Each information update layer is used to perform information update processing on each vector in the feature map to be updated input to the information update layer in the target direction corresponding to the information update layer. The target direction includes the row direction or the column direction. The step of inputting the initial sub-feature map and the initial update weights of each initial vector into the row and column information update module to obtain the target sub-feature map corresponding to the channel group output by the row and column information update module includes: The initial update weights corresponding to each current vector to be updated in the current feature map to be updated are adjusted by the current information update layer to obtain the target update weights corresponding to each current vector to be updated. The current feature map to be updated is the initial sub-feature map or the updated feature map output by the previous information update layer. Based on each current vector to be updated and the target update weight corresponding to each current vector to be updated, determine the updated vector corresponding to each current vector to be updated. The updated vectors corresponding to each current vector to be updated are used as vectors in the updated feature map output by the current information update layer; Among them, the updated feature map output by the last information update layer in the row and column information update module is the target sub-feature map corresponding to the channel group.

4. The method according to claim 3, characterized in that, The current update direction is the target direction corresponding to the current information update layer; the step of determining the updated vector corresponding to each current vector to be updated based on each current vector to be updated and the target update weight corresponding to each current vector to be updated includes: In response to the fact that the current vector to be updated is the first vector to be updated in the current update direction in the current feature map, the updated vector corresponding to the current vector to be updated is obtained based on the current vector to be updated and the target update weight corresponding to the current vector to be updated; In response to the fact that the current vector to be updated is not the first vector to be updated in the current update direction in the current feature map to be updated, the updated vector corresponding to the current vector to be updated is obtained according to the updated vectors of the current vector to be updated and the candidate vectors to be updated, as well as the target update weight of the current vector to be updated. The candidate vector to be updated is the previous vector to be updated in the current feature map to be updated in the current update direction.

5. The method according to claim 3, characterized in that, The current update direction is the target direction corresponding to the current information update layer; the step of adjusting the initial update weights corresponding to each current vector to be updated in the current feature map through the current information update layer to obtain the target update weights corresponding to each current vector to be updated includes: In response to the fact that the current vector to be updated is the first vector to be updated in the current update direction in the current feature map, the initial update weight corresponding to the current vector to be updated is adjusted to a preset value to obtain the target update weight corresponding to the current vector to be updated; In response to the fact that the current vector to be updated is not the first vector to be updated in the current update direction in the current feature map to be updated, the target update weight corresponding to the current vector to be updated is obtained according to the current vector to be updated, the candidate vector to be updated, and the initial update weight corresponding to the current vector to be updated, wherein the candidate vector to be updated is the previous vector to be updated in the current feature map to be updated in the current update direction.

6. The method according to claim 2, characterized in that, The weight generation module includes several cascaded weight generation layers, each of which includes a cascaded convolution module, a normalization module, and a mapping module; the step of obtaining the initial update weights of each initial vector in the initial sub-feature map corresponding to the channel group through the weight generation module includes: Each vector in the current feature map to be processed is input into the convolution module in the current weight generation layer to obtain the convolution result of each vector in the current feature map to be processed output by the convolution module in the current weight generation layer. The current feature map to be processed is each initial vector in the initial sub-feature map or the mapping result of each vector output by the mapping module in the previous weight generation layer. The convolution result of each vector in the current feature map to be processed is input into the normalization module in the current weight generation layer to obtain the normalization result of each vector in the current feature map to be processed output by the normalization module in the current weight generation layer. The normalized result of each vector in the current feature map to be processed is input into the mapping module in the current weight generation layer to obtain the mapping result of each vector in the current feature map to be processed output by the mapping module in the current weight generation layer. The mapping results of each vector output by the mapping module in the last weight generation layer are used as the initial update weights of each initial vector in the initial sub-feature map corresponding to the channel group.

7. The method according to claim 1, characterized in that, The information accumulation model includes a channel grouping module and a channel compression module. The step of performing channel recombination processing on the initial feature map of the acquired image to be processed through the information accumulation model to obtain the initial sub-feature map corresponding to each channel group includes: The initial feature map of the image to be processed is processed by channel grouping through the channel grouping module to obtain the grouped sub-feature map corresponding to each channel group; The channel compression module performs channel compression processing on the grouped sub-feature maps corresponding to each channel group to obtain the initial sub-feature maps corresponding to each channel group.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes a training step for the information accumulation model, the training step comprising: Obtain sample images and the preset edge extraction results of the sample images and / or the sample image annotation results of the sample images; The initial feature map of the sample image is reorganized using the information accumulation model to obtain the initial sub-feature map of each sample channel group. A first loss is obtained based on the difference between the initial update weights of the initial sub-feature maps corresponding to each sample channel group and the preset edge extraction results of the sample images; and / or, The target feature map obtained from the initial sub-feature map of each sample channel group is processed by a preset image to obtain the sample image processing result of the sample image; the second loss is obtained based on the difference between the sample image processing result of the sample image and the sample image annotation result of the sample image. The parameters in the information accumulation model are adjusted based on the first loss and / or the second loss to obtain the trained information accumulation model.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to perform the method as claimed in any one of claims 1-8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they are used to implement the method as described in any one of claims 1-8.

Images