Image processing method, deep learning model training method, device and equipment

Abstract

The present disclosure provides an image processing method related to the field of artificial intelligence, particularly deep learning. A specific implementation scheme comprises: determining a shift offset value based on a first feature map of an image to be processed; shifting a data block in the first feature map based on the shift offset value to obtain a shifted first feature map; and obtaining a second feature map of the image to be processed based on the shifted first feature map. The present disclosure also provides a training method, apparatus, electronic device, and storage medium for a deep learning model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of deep learning technology. More specifically, the present disclosure provides an image processing method, a training method for a deep learning model, an apparatus, an electronic device, and a storage medium. Background Art

[0002] The image can be processed based on the attention mechanism to obtain the image features. For example, the image can be processed using the Transformer model based on the coarse-grained global attention mechanism or the fine-grained local attention mechanism. Summary of the Invention

[0003] The present disclosure provides an image processing method, a training method, an apparatus, a device, and a storage medium for a deep learning model.

[0004] According to one aspect of the present disclosure, an image processing method is provided, the method comprising: determining a shift offset value based on a first feature map of an image to be processed; shifting a data block in the first feature map based on the shift offset value to obtain a shifted first feature map; and obtaining a second feature map of the image to be processed based on the shifted first feature map.

[0005] According to another aspect of the present disclosure, a training method for a deep learning model is provided, wherein the deep learning model includes a first subnetwork and a second subnetwork, and the method includes: inputting a sample image into the first subnetwork to obtain a first feature map; inputting the first feature map into the second subnetwork to obtain a second feature map of the image to be processed; obtaining a detection value of the sample image based on the second feature map; and adjusting parameters of the first subnetwork and parameters of the second subnetwork based on the difference between the detection value of the sample image and the label of the sample image to obtain a trained deep learning model; wherein the second feature map is obtained by the following operations: shifting a data block in the first feature map to obtain a shifted first feature map; and obtaining the second feature map of the image to be processed based on the shifted first feature map.

[0006] According to another aspect of the present disclosure, an image processing device is provided, the device comprising: a determination module for determining a shift offset value based on a first feature map of an image to be processed; a shift module for shifting a data block in the first feature map according to the shift offset value to obtain a shifted first feature map; and a first obtaining module for obtaining a second feature map of the image to be processed from the shifted first feature map.

[0007] According to another aspect of the present disclosure, a training device for a deep learning model is provided, wherein the deep learning model includes a first subnetwork and a second subnetwork, and the device includes: a second acquisition module for inputting a sample image into the first subnetwork to obtain a first feature map; a third acquisition module for inputting the first feature map into the second subnetwork to obtain a second feature map of the image to be processed; a fourth acquisition module for obtaining a detection value of the sample image based on the second feature map; and an adjustment module for adjusting the parameters of the first subnetwork and the parameters of the second subnetwork based on the difference between the detection value of the sample image and the label of the sample image to obtain a trained deep learning model; wherein the second feature map is obtained by performing relevant operations by the following submodules: a second shift submodule for shifting the data block in the first feature map to obtain the shifted first feature map; and a second acquisition submodule for obtaining the second feature map of the image to be processed based on the shifted first feature map.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided according to the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method provided according to the present disclosure when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of an image processing method according to another embodiment of the present disclosure;

[0015] Figure 3Ais a schematic diagram of a first characteristic diagram according to an embodiment of the present disclosure;

[0016] Figure 3B is a schematic diagram of multiple data blocks of a first feature map according to an embodiment of the present disclosure;

[0017] Figure 3C is a schematic diagram of first to fourth data blocks of a first feature map according to an embodiment of the present disclosure;

[0018] Figure 3D is a schematic diagram of a shifted first feature map according to one embodiment of the present disclosure;

[0019] Figure 4A is a schematic diagram of a first characteristic diagram according to another embodiment of the present disclosure;

[0020] Figure 4B is a schematic diagram of a first data block and a second data block of a first feature map according to another embodiment of the present disclosure;

[0021] Figure 4C is a schematic diagram of a shifted first characteristic map according to another embodiment of the present disclosure;

[0022] Figure 5 is a schematic diagram of multiple feature tiles according to one embodiment of the present disclosure;

[0023] Figures 6A to 6B is a schematic diagram of an image processing method according to an embodiment of the present disclosure;

[0024] Figure 6C is a schematic diagram of a deep learning model according to one embodiment of the present disclosure;

[0025] Figure 7 is a flowchart of a method for training a deep learning model according to one embodiment of the present disclosure;

[0026] Figure 8 is a schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure;

[0027] Figure 9 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure;

[0028] Figure 10 is a block diagram of a training apparatus for a deep learning model according to another embodiment of the present disclosure; and

[0029] Figure 11 1 is a block diagram of an electronic device to which an image processing method and / or a training method for a deep learning model can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] For example, the Swin (Shifted Windows) Transformer model can process images based on a fine-grained local self-attention mechanism.

[0032] The Swin Transformer model can set a local window on the image to restrict the model to computations within the window. The Swin Transformer model can enhance feature interactions between windows by shifting window splits or downsampling, thereby obtaining global information outside the window.

[0033] For example, when processing an image, the Swin Transformer model can shift the image's feature map according to preset parameters to improve the model's performance. However, these preset parameters limit the model's learning ability and prevent it from fully realizing its performance.

[0034] Figure 1 is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0035] like Figure 1 As shown, the method 100 may include operations S110 to S130.

[0036] In operation S110 , a shift offset value is determined according to a first feature map of an image to be processed.

[0037] For example, various feature extraction models may be used to extract features from the image to be processed to obtain a first feature map.

[0038] In one example, the feature extraction model may include at least one module or sub-network built based on a Swin Transformer model.

[0039] For example, the number of channels of the first feature map may be N, where N is an integer greater than 1. Convolution may be performed on the first feature map to compress the number of channels of the first feature map, thereby obtaining a convolved first feature map. The number of channels of the convolved first feature map may be, for example, M. M feature subgraphs may be determined based on the convolved first feature map. Each feature subgraph corresponds to a channel of the convolved first feature map.

[0040] For another example, a pooling operation can be performed on each feature subgraph to obtain a pooled feature subgraph. A normalization operation can be performed on each pooled feature subgraph to obtain each ratio. A total of M ratios can be obtained, and the M ratios are used as M shift offset values. In one example, M = 2.

[0041] In operation S120, the data block in the first feature map is shifted according to the shift offset value to obtain a shifted first feature map.

[0042] For example, the first feature map includes at least one data block. The size of each data block can be determined. The M ratios described above can be applied to one or more of the at least one data block to adjust the size of the data block accordingly. In one example, the M ratios described above can be applied to one data block to adjust the size of the data block, thereby obtaining a first data block.

[0043] For another example, the position of the data block may be adjusted to obtain a shifted first feature map. In one example, the position of the first data block mentioned above in the first feature map may be adjusted to obtain a shifted first feature map.

[0044] In operation S130 , a second feature map of the image to be processed is obtained according to the shifted first feature map.

[0045] In the embodiment of the present disclosure, the shifted first feature map may be divided into a plurality of feature map blocks.

[0046] For example, the size of each feature map block may be the same. In one example, the shifted first feature map may be divided into four feature map blocks.

[0047] In the embodiment of the present disclosure, multiple feature map blocks may be processed to obtain a second feature map of the image to be processed.

[0048] For example, each feature block can be processed in various ways. In one example, multi-head self-attention feature detection can be performed on each feature block.

[0049] By determining the shift offset value based on the first feature map, the embodiment of the present disclosure can more flexibly enhance the feature interaction between windows, more comprehensively obtain the global features of the input image, and further improve the performance of the model.

[0050] In some embodiments, different from method 100 , convolution may be performed on each feature map block to obtain a second feature map.

[0051] Figure 2 is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0052] like Figure 2 As shown, method 210 can determine the shift offset value based on the first feature map. Detailed description will be given below in conjunction with operations S211 to S213.

[0053] In operation S211 , a first feature subgraph and a second feature subgraph of a first feature graph are determined.

[0054] For example, a 1*1 convolution kernel can be used to convolve the first feature map to reduce the number of channels in the first feature map from N to 2, thereby obtaining the convolved first feature map. Based on the convolved first feature map, a first feature sub-map and a second feature sub-map are determined. Each feature sub-map corresponds to a channel in the convolved first feature map.

[0055] In operation S212 , a first mean value of the first characteristic subgraph and a second mean value of the second characteristic subgraph are determined respectively.

[0056] For example, the first mean and the second mean may be determined by performing global average pooling on the first feature sub-graph and the second feature sub-graph respectively.

[0057] In operation S213 , a shift offset value is determined based on the first mean value and the second mean value.

[0058] For example, the first mean and the second mean are normalized to obtain a first ratio and a second ratio as the shift offset values. In one example, the first mean and the second mean can be normalized using a Sigmoid function.

[0059] In some embodiments, shifting the data block in the first feature map according to the shift offset value to obtain the shifted first feature map includes: applying the first ratio and the second ratio to the first feature map respectively to obtain the first data block in the first feature map; and shifting the first data block to obtain the shifted first feature map. Figures 3A to 3D Provide detailed explanation.

[0060] Figure 3A is a schematic diagram of a first characteristic diagram according to an embodiment of the present disclosure.

[0061] like Figure 3A As shown, the first feature map 301 is a 12*12 feature map.

[0062] Figure 3B is a schematic diagram of multiple data blocks of a first feature map according to an embodiment of the present disclosure.

[0063] like Figure 3B As shown, four first windows can be used to divide the first feature map 301 into four data blocks. The four first windows can be first window 311, first window 312, first window 313, and first window 314. Each first window corresponds to a data block.

[0064] Figure 3C 2 is a schematic diagram of first to fourth data blocks of a first feature graph according to an embodiment of the present disclosure.

[0065] like Figure 3C As shown, the first ratio and the second ratio described above can be applied to the first feature map 301. For example, the first ratio described above is applied to adjust the height of the first window 311 of the first feature map 301, and the second ratio described above is applied to adjust the width of the first window 311 of the first feature map 301, thereby obtaining an adjusted first window 311'. In this embodiment, the first ratio is 0.5, and the second ratio is 2 / 3.

[0066] Based on the adjusted first window 311', other adjusted first windows can be determined using various methods. For example, the width of the adjusted first window 312' can be determined based on the width of the adjusted first window 311' and the width of the first feature map. The height of the adjusted first window 312' can then be determined based on the height of the adjusted first window 311'. Similarly, the height of the adjusted first window 313' can be determined based on the height of the adjusted first window 311' and the height of the first feature map. The width of the adjusted first window 313' can then be determined based on the width of the adjusted first window 311'. Next, the adjusted first windows 314' corresponding to other areas of the first feature map 301 are determined.

[0067] A first data block 321 may be determined based on the adjusted first window 311'. Similarly, a second data block 322 may be determined based on the adjusted first window 312'. A third data block 323 may be determined based on the adjusted first window 313'. A fourth data block 324 may be determined based on the adjusted second window 314'.

[0068] Figure 3D is a schematic diagram of a shifted first feature map according to an embodiment of the present disclosure.

[0069] like Figure 3D As shown, you can Figure 3CShift the first data block 321 in , for example, shift it two positions counterclockwise, from the upper left corner to the lower right corner of the first feature map. Figure 3C Shift the second data block 322 to the fourth data block 323 in the , and obtain Figure 3D The first feature map 302 is shown after shifting.

[0070] In some other embodiments, shifting the data block in the first feature map according to the shift offset value to obtain the shifted first feature map includes: applying the first ratio and the second ratio to the first feature map respectively to obtain the first data block and the second data block in the first feature map; and shifting the first data block and the second data block to obtain the shifted first feature map. Figures 4A to 4C Provide detailed explanation.

[0071] Figure 4A is a schematic diagram of a first characteristic diagram according to another embodiment of the present disclosure.

[0072] like Figure 4A As shown, similar to the first feature map 301 described above, the first feature map 401 is also a 12*12 feature map.

[0073] Figure 4B 2 is a schematic diagram of a first data block and a second data block of a first feature graph according to another embodiment of the present disclosure.

[0074] The second window may be determined based on the first feature map 401. The size of the second window may at least cover the first feature map 401. In this embodiment, the first feature map 401 is regarded as a data block.

[0075] Next, the first ratio and the second ratio can be applied to first feature map 401. For example, the first ratio is applied to adjust the width of a second window of first feature map 401 to obtain adjusted second window 411. For another example, the second ratio is applied to adjust the height of another second window of first feature map 401. Furthermore, based on the adjusted width of second window 411, the width of the second window is adaptively determined to obtain adjusted second window 412. In this embodiment, the first ratio is 1 / 3, and the second ratio is 0.25.

[0076] A first data block 421 may be determined based on the adjusted second window 411. A second data block 422 may be determined based on the adjusted second window 412.

[0077] Another second window is then adjusted based on the adjusted second window 411 and the adjusted second window 412 to obtain an adjusted second window 413. The adjusted second window 413 can cover other areas on the first feature map 401. The third data block 423 can be determined based on the adjusted second window 413.

[0078] Figure 4C is a schematic diagram of a shifted first feature map according to another embodiment of the present disclosure.

[0079] like Figure 4C As shown, the first data block 421 may be shifted.

[0080] It is understood that the shifting can be performed in various ways. For example, the first data block 421 can be moved to the other side of the first feature map. The positions of the second data block 422 and the third data block 423 can then be swapped to obtain the shifted first feature map 402.

[0081] Figure 5 is a schematic diagram of multiple feature blocks according to an embodiment of the present disclosure.

[0082] like Figure 5 As shown, in this embodiment, the four feature blocks are feature block 531, feature block 532, feature block 533 and feature block 534. The four feature blocks can be obtained by, for example Figure 3D The shifted first feature map 302 shown is obtained by dividing.

[0083] In some embodiments, processing multiple feature blocks to obtain a second feature map of the image to be processed includes: performing multi-head self-attention feature detection on each feature block in the multiple feature blocks to obtain the second feature map of the image to be processed.

[0084] Figures 6A to 6B is a schematic diagram of an image processing method according to an embodiment of the present disclosure.

[0085] Next, we will combine Figures 3A to 3D 、 Figure 5 、 Figure 6A and Figure 6B The principle of the image processing method provided by the present disclosure is explained in detail.

[0086] like Figure 6A As shown, a deep learning model can be used to process the image to be processed to obtain a second feature map of the image to be processed.

[0087] The deep learning model may include, for example Figure 6AThe feature extraction stage Stage_600 is shown. The feature extraction stage Stage_600 includes at least a first sub-network 610 and a second sub-network 620. The feature extraction stage Stage_600 can take the image to be processed as input and output a second feature map.

[0088] For example, at least one first sub-network 610 may extract a first feature map of the image to be processed. For example, the first feature map may be the first feature map 301 described above.

[0089] The second sub-network 620 may process the first feature map to obtain a second feature map.

[0090] like Figure 6B As shown, the second sub-network 620 may include a multi-head self-attention detection branch 621 and a shift offset value determination branch 622.

[0091] For example, the multi-head self-attention detection branch 621 may include an LN layer (Layer Normalization) 6211, a DW-MSA (Deformable Windows Multi-head Self-Attention) layer 6212, an LN layer 6213 and a multi-layer perceptron layer 6214.

[0092] For another example, the shift offset value determination branch 622 may include a convolution layer 6221 , a global average pooling layer 6222 , and a mapping layer 6223 .

[0093] The input of the LN layer 6211 may be the first feature map. The LN layer 6211 may perform a normalization operation on the first feature map to obtain a normalized first feature map.

[0094] The convolution layer 6221 can use a 1*1 convolution kernel to convolve the normalized first feature map to determine the first feature sub-map and the second feature sub-map. The global average pooling layer 6222 can determine the first mean of the first feature sub-map, and can also determine the second mean of the second feature sub-map. The mapping layer 6223 can map the first mean and the second mean to values ​​within a predetermined range, respectively. In an example, the mapping layer 6223 can use a Sigmoid function to map the first mean to a first ratio, or can map the second mean to a second ratio. The first ratio and the second ratio can serve as the shift offset values ​​described above. For example, in this embodiment, the first ratio can be 0.5 as described above, and the second ratio can be 2 / 3 as described above.

[0095] The DW-MSA layer can apply the first ratio and the second ratio to the normalized first feature map to obtain a first data block of the first feature map. In one example, the first data block 321 to the fourth data block 324 described above can be obtained.

[0096] The DW-MSA layer may shift the first data block to obtain a shifted first feature map. In an example, the shifted first feature map may be the shifted first feature map 302 described above.

[0097] The DW-MSA layer may also divide the shifted first feature map into a plurality of feature map blocks. In one example, the plurality of feature map blocks are the feature map blocks 531, 532, 533, and 534 described above.

[0098] The DW-MSA layer can also perform multi-head self-attention feature detection on multiple feature blocks and output intermediate feature maps.

[0099] Next, the intermediate feature map and the normalized first feature map can be fused to generate a fused feature map. LN layer 6213 takes the fused feature map as input and outputs a normalized fused feature map. Multilayer perceptron layer 6214 takes the normalized fused feature map as input and outputs a target feature map. Based on the target feature map and the fused feature map, a second feature map can be generated.

[0100] In some embodiments, the product of the shift offset value output by the mapping layer 6223 described above and the window may not match the size of the first feature map (for example, the product is not an integer). The shift offset value can be interpolated so that the interpolated shift offset value can be used to shift the data block in the feature map.

[0101] Figure 6C is a schematic diagram of a deep learning model according to an embodiment of the present disclosure.

[0102] like Figure 6C As shown, the deep learning model includes four feature extraction stages, namely feature extraction stage Stage_600_1, feature extraction stage Stage_600_2, feature extraction stage Stage_600_3 and feature extraction stage Stage_600_4.

[0103] For example, the deep learning model further includes a partitioning layer 650. The partitioning layer 650 can perform patch partitioning on the image to be processed to obtain an input image of the feature extraction stage Stage_600_1.

[0104] Feature extraction stage Stage_600_1 includes an embedding layer 630_1, a first sub-network 610_1, and a second sub-network 620_1. Embedding layer 630_1 performs linear embedding on the input image to produce an embedded image. First sub-network 610_1 outputs a first feature map based on the embedded image. Second sub-network 620_1 processes the first feature map and outputs a second feature map.

[0105] Feature extraction stage Stage_600_2 includes a fusion layer 640_2, a first sub-network 610_2, and a second sub-network 620_2. Fusion layer 640_2 performs patch merging on the first second feature map to produce a first fused image. First sub-network 610_2 outputs a second first feature map based on the first fused image. Second sub-network 620_2 processes the second first feature map to produce a second second feature map.

[0106] Feature extraction stage Stage_600_3 includes a fusion layer 640_3, a first sub-network 610_3, and a second sub-network 620_3. Feature extraction stage Stage_600_4 includes a fusion layer 640_4, a first sub-network 610_4, and a second sub-network 620_4. It is understood that feature extraction stages Stage_600_3 and Stage_600_4 are identical or similar to feature extraction stage Stage_600_2 and are not further described in this disclosure. The output of feature extraction stage Stage_600_4 may be the fourth second feature map.

[0107] In some embodiments, the method provided by the present disclosure further includes: detecting a target object in the image to be processed based on a second feature map of the image to be processed.

[0108] Figure 7 4 is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure.

[0109] The deep learning model includes a first sub-network and a second sub-network.

[0110] like Figure 7 As shown, the method 700 may include operations S710 to S740.

[0111] In operation S710, a sample image is input into a first sub-network to obtain a first feature map.

[0112] For example, the first sub-network can perform feature extraction on the sample image to obtain a first feature map.

[0113] It can be understood that the way the first sub-network processes the sample image is the same as or similar to the way the first sub-network 610 processes the image to be processed described above, and this disclosure will not elaborate on it here.

[0114] In operation S720, the first feature map is input into the second sub-network to obtain a second feature map of the image to be processed.

[0115] In an embodiment of the present disclosure, the second feature map is obtained by the following operations: shifting the data block in the first feature map to obtain the shifted first feature map; and obtaining the second feature map of the image to be processed based on the shifted first feature map.

[0116] For example, the shifted first feature map may be divided into a plurality of feature map blocks; and the plurality of feature map blocks may be processed to obtain a second feature map of the image to be processed.

[0117] It can be understood that the way in which the second sub-network processes the first feature map of the sample image is the same as or similar to the way in which the second sub-network 620 processes the first feature map of the image to be processed described above, and this disclosure will not go into details here.

[0118] In operation S730 , a detection value of the sample image is obtained according to the second feature map.

[0119] In operation S740 , parameters of the first sub-network and parameters of the second sub-network are adjusted according to the difference between the detection value of the sample image and the label of the sample image to obtain a trained deep learning model.

[0120] For example, a cross entropy loss function can be used to determine the difference between the detection value and the label, and the parameters of the first sub-network and the second sub-network can be adjusted based on the difference.

[0121] Through the embodiments of the present disclosure, the parameters of the second sub-network can be adjusted, and the various parameters in the process of shifting the data blocks of the first feature map can be adjusted, thereby improving the learning ability of the model and further improving the performance of the model.

[0122] In some embodiments, the deep learning model includes multiple feature extraction stages, each feature extraction stage includes a first sub-network and a second sub-network; the second sub-network includes a multi-head self-attention detection branch and a shift offset value determination branch.

[0123] For example, a feature extraction stage may be the feature extraction stage Stage_600 described above. The first sub-network may be the first sub-network 610 described above. The second sub-network may be the second sub-network 620 described above. The self-attention detection branch may be the self-attention detection branch 621 described above. The shift offset value determination branch may be the shift offset value determination branch 622 described above.

[0124] In some embodiments, the shift offset value determination branch includes: a convolution layer for determining a first feature sub-map and a second feature sub-map of a first feature map; a global average pooling layer for respectively determining a first mean of the first feature sub-map and a second mean of the second feature sub-map; and a mapping layer for respectively mapping the first mean and the second mean to values ​​within a predetermined range.

[0125] For example, the convolution layer may be the convolution layer 6221 described above. The global average pooling layer may be the global average pooling layer 6222 described above. The mapping layer may be the mapping layer 6223 described above.

[0126] Figure 8 It is a schematic diagram of a deep learning model training method according to an embodiment of the present disclosure.

[0127] like Figure 8 As shown, the deep learning model Model_800 may include feature extraction stages Stage_800_1, ..., feature extraction stages Stage_800_K, and detection stages Stage_Dect. K is an integer greater than or equal to 1. In one example, K=2. In one example, the structure of each feature extraction stage may be similar to, for example, Figure 6A The feature extraction stage Stage_600 shown is the same or similar.

[0128] like Figure 8 As shown, deep learning model Model_800 can take sample image 801 as input and output detection value 802 for sample image 801. For example, after sample image 801 is processed by K feature extraction stages, it is converted into the Kth second feature map. In one example, assuming K = 2, the input of feature extraction stage Stage_800_K is the first second feature map output by feature extraction stage Stage_800_1. The output of feature extraction stage Stage_800_K is the Kth second feature map.

[0129] The detection stage Stage_Dect takes the Kth second feature map as input and outputs a detection value 802.

[0130] Based on the detection value 802 and the label 803 of the sample image 801, a difference 804 between the two can be determined. Parameters of the deep learning model Model_800 can be adjusted based on the difference 804. For example, the parameters of the first sub-network and the second sub-network in each feature extraction stage can be adjusted.

[0131] In some embodiments, unlike the deep learning model Model_800, in this embodiment, K=4.

[0132] Figure 9 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0133] like Figure 9 As shown, the apparatus 900 may include a determination module 910 , a shift module 920 and a first obtaining module 930 .

[0134] The determination module 910 is configured to determine a shift offset value according to the first feature map.

[0135] The shift module 920 is configured to shift the data block in the first feature map according to the shift offset value to obtain a shifted first feature map.

[0136] The first obtaining module 930 is configured to obtain a second feature map of the image to be processed according to the shifted first feature map.

[0137] In some embodiments, the determination module includes: a first determination submodule, used to determine the first feature subgraph and the second feature subgraph of the first feature graph; a second determination submodule, used to respectively determine the first mean of the first feature subgraph and the second mean of the second feature subgraph; and a third determination submodule, used to determine the shift offset value based on the first mean and the second mean.

[0138] In some embodiments, the second determining submodule includes: a global average pooling unit, configured to determine the first mean and the second mean by performing global average pooling on the first feature subgraph and the second feature subgraph, respectively.

[0139] In some embodiments, the third determining submodule includes: a normalizing unit, configured to normalize the first mean and the second mean respectively to obtain a first ratio and a second ratio as the shift offset value.

[0140] In some embodiments, the shift module includes: a first acquisition submodule, used to apply the first ratio and the second ratio to the first feature map respectively to obtain the first data block and the second data block in the first feature map; and a first shift submodule, used to shift the first data block and the second data block to obtain the shifted first feature map.

[0141] In some embodiments, the first acquisition module includes: a division submodule for dividing the shifted first feature map into multiple feature map blocks; and a processing submodule for processing the multiple feature map blocks to obtain a second feature map of the image to be processed.

[0142] In some embodiments, the processing submodule includes: a multi-head self-attention feature detection unit, which is used to perform multi-head self-attention feature detection on each feature block in the multiple feature blocks to obtain a second feature map of the image to be processed.

[0143] In some embodiments, the apparatus 900 further includes: a detection module configured to detect a target object in the image to be processed based on a second feature map of the image to be processed.

[0144] Figure 10 1 is a block diagram of a training apparatus for a deep learning module according to another embodiment of the present disclosure. The deep learning model includes a first sub-network and a second sub-network.

[0145] like Figure 10 As shown, the apparatus 1000 may include a second obtaining module 1010 , a third obtaining module 1020 , a fourth obtaining module 1030 and an adjusting module 1040 .

[0146] The second obtaining module 1010 is used to input the sample image into the first sub-network to obtain a first feature map.

[0147] The third obtaining module 1020 is configured to input the first feature map into a second sub-network to obtain a second feature map of the image to be processed.

[0148] The fourth obtaining module 1030 is configured to obtain a detection value of the sample image according to the second feature map.

[0149] The adjustment module 1040 is used to adjust the parameters of the first sub-network and the parameters of the second sub-network according to the difference between the detection value of the sample image and the label of the sample image to obtain a trained deep learning model.

[0150] For example, the second feature map is obtained by performing relevant operations by the following sub-modules: a second shift sub-module, used to shift the data block in the first feature map to obtain the shifted first feature map; and a second acquisition sub-module, used to obtain the second feature map of the image to be processed based on the shifted first feature map.

[0151] In some embodiments, the deep learning model includes multiple feature extraction stages, each feature extraction stage includes the first sub-network and the second sub-network; the second sub-network includes a multi-head self-attention detection branch and a shift offset value determination branch.

[0152] In some embodiments, the shift offset value determination branch includes: a convolution layer for determining a first feature sub-map and a second feature sub-map of the first feature map; a global average pooling layer for respectively determining a first mean of the first feature sub-map and a second mean of the second feature sub-map; and a mapping layer for respectively mapping the first mean and the second mean to values ​​within a predetermined range.

[0153] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0154] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0155] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0156] like Figure 11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0157] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0158] The computing unit 1101 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as image processing methods and / or deep learning model training methods. For example, in some embodiments, the image processing methods and / or deep learning model training methods can be implemented as computer software programs that are tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the image processing method and / or deep learning model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the image processing method and / or the deep learning model training method in any other appropriate manner (e.g., by means of firmware).

[0159] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0161] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0163] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0164] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0165] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0166] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An image processing method, comprising: Determining a shift offset value according to a first feature map of the image to be processed; Shifting the data block in the first feature map according to the shift offset value to obtain a shifted first feature map; as well as Obtaining a second feature map of the image to be processed according to the shifted first feature map; The determining of the shift offset value according to the first feature map includes: Performing convolution on the number of channels of the first feature map to compress the number of channels of the first feature map; Determining a first feature subgraph and a second feature subgraph of the first feature graph according to the convolved first feature graph, wherein each feature subgraph corresponds to a channel of the convolved first feature graph; Determining a first mean of the first characteristic subgraph and a second mean of the second characteristic subgraph respectively; and The shift offset value is determined according to the first mean value and the second mean value.

2. The method according to claim 1, wherein The determining of the first mean of the first characteristic subgraph and the second mean of the second characteristic subgraph respectively comprises: The first mean and the second mean are determined by performing global average pooling on the first feature subgraph and the second feature subgraph respectively.

3. The method according to claim 1 or 2, wherein: Determining the shift offset value according to the first mean value and the second mean value includes: The first mean and the second mean are normalized respectively to obtain a first ratio and a second ratio as the shift offset value.

4. The method according to claim 3, wherein: The step of shifting the data block in the first feature map according to the shift offset value to obtain the shifted first feature map includes: Applying the first ratio and the second ratio to the first feature map respectively to obtain a first data block and a second data block in the first feature map; and The first data block and the second data block are shifted to obtain the shifted first feature map.

5. The method according to any one of claims 1 to 4, wherein: Obtaining a second feature map of the image to be processed according to the shifted first feature map includes: Dividing the shifted first feature map into a plurality of feature map blocks; and The plurality of feature map blocks are processed to obtain a second feature map of the image to be processed.

6. The method according to claim 5, wherein: The processing of the plurality of feature image blocks to obtain a second feature image of the image to be processed includes: Multi-head self-attention feature detection is performed on each feature block in the multiple feature blocks to obtain a second feature map of the image to be processed.

7. The method according to claim 1, further comprising: Detecting a target object in the image to be processed according to the second feature map of the image to be processed.

8. A method for training a deep learning model, the deep learning model comprising a first sub-network and a second sub-network, the method comprising: Input the sample image into the first sub-network to obtain the first feature map; Inputting the first feature map into the second sub-network to obtain a second feature map of the image to be processed; Obtaining a detection value of the sample image according to the second feature map; as well as Adjusting parameters of the first sub-network and parameters of the second sub-network according to a difference between the detection value of the sample image and the label of the sample image to obtain a trained deep learning model; The second feature map is obtained by the following operation: Shifting the data block in the first feature map according to the shift offset value to obtain a shifted first feature map; and Obtaining a second feature map of the image to be processed according to the shifted first feature map; The second sub-network includes a shift offset value determination branch, and the shift offset value determination branch includes: A convolution layer, configured to convolve the number of channels of the first feature map to compress the number of channels of the first feature map, and determine a first feature subgraph and a second feature subgraph of the first feature map based on the convolved first feature map, wherein each feature subgraph corresponds to a channel of the convolved first feature map; a global average pooling layer, configured to respectively determine a first mean of the first feature sub-graph and a second mean of the second feature sub-graph; and The mapping layer is used to map the first mean value and the second mean value to values ​​within a predetermined range respectively to obtain the shift offset value.

9. The method according to claim 8, wherein The deep learning model includes a plurality of feature extraction stages, each feature extraction stage includes the first sub-network and the second sub-network; The second sub-network also includes a multi-head self-attention detection branch.

10. An image processing device, comprising: A determination module, configured to determine a shift offset value based on a first feature map of the image to be processed; A shift module, configured to shift the data block in the first feature map according to the shift offset value to obtain a shifted first feature map; as well as A first obtaining module, configured to obtain a second feature map of the image to be processed according to the shifted first feature map; Wherein, the determination module includes: A convolution submodule, configured to perform convolution on the number of channels of the first feature map to compress the number of channels of the first feature map; A first determining submodule, configured to determine a first feature subgraph and a second feature subgraph of the first feature graph according to the convolved first feature graph, wherein each feature subgraph corresponds to a channel of the convolved first feature graph; a second determining submodule, configured to respectively determine a first mean of the first characteristic subgraph and a second mean of the second characteristic subgraph; and A third determining submodule is configured to determine the shift offset value according to the first mean value and the second mean value.

11. The device according to claim 10, wherein The second determining submodule includes: A global average pooling unit is configured to determine the first mean and the second mean by performing global average pooling on the first feature sub-image and the second feature sub-image respectively.

12. The device according to claim 10 or 11, wherein The third determining submodule includes: A normalization unit is configured to normalize the first mean and the second mean respectively to obtain a first ratio and a second ratio as the shift offset value.

13. The device according to claim 12, wherein The shift module includes: A first obtaining submodule, configured to apply the first ratio and the second ratio to the first feature map, respectively, to obtain a first data block and a second data block in the first feature map; and The first shift submodule is configured to shift the first data block and the second data block to obtain the shifted first feature map.

14. The device according to any one of claims 10 to 13, wherein: The first obtaining module includes: a division submodule, configured to divide the shifted first feature map into a plurality of feature map blocks; and The processing submodule is used to process the multiple feature image blocks to obtain a second feature image of the image to be processed.

15. The device according to claim 14, wherein The processing submodule includes: The multi-head self-attention feature detection unit is used to perform multi-head self-attention feature detection on each feature block in the multiple feature blocks to obtain a second feature map of the image to be processed.

16. The apparatus according to claim 10, further comprising: A detection module is used to detect a target object in the image to be processed based on the second feature map of the image to be processed.

17. A training device for a deep learning model, the deep learning model comprising a first sub-network and a second sub-network, the device comprising: A second acquisition module is used to input the sample image into the first sub-network to obtain a first feature map; A third acquisition module is used to input the first feature map into the second sub-network to obtain a second feature map of the image to be processed; a fourth obtaining module, configured to obtain a detection value of the sample image according to the second feature map; as well as an adjustment module, configured to adjust parameters of the first sub-network and parameters of the second sub-network according to a difference between the detection value of the sample image and the label of the sample image, to obtain a trained deep learning model; The second feature map is obtained by performing related operations by the following submodules: a second shift submodule, configured to shift the data block in the first feature map according to a shift offset value to obtain a shifted first feature map; and A second obtaining submodule is configured to obtain a second feature map of the image to be processed according to the shifted first feature map; The second sub-network includes a shift offset value determination branch, and the shift offset value determination branch includes: A convolution layer, configured to convolve the number of channels of the first feature map to compress the number of channels of the first feature map, and determine a first feature subgraph and a second feature subgraph of the first feature map based on the convolved first feature map, wherein each feature subgraph corresponds to a channel of the convolved first feature map; a global average pooling layer, configured to respectively determine a first mean of the first feature sub-graph and a second mean of the second feature sub-graph; and The mapping layer is used to map the first mean value and the second mean value to values ​​within a predetermined range respectively to obtain the shift offset value.

18. The device according to claim 17, wherein The deep learning model includes a plurality of feature extraction stages, each feature extraction stage includes the first sub-network and the second sub-network; The second sub-network also includes a multi-head self-attention detection branch.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images