Image label adding method and device, electronic equipment and storage medium

Abstract

The present disclosure provides an image label adding method and device, electronic equipment and storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of computer vision and image recognition. The specific implementation scheme comprises: performing feature extraction on an image to obtain a first feature map; determining an attention parameter of the first feature map, adjusting the feature values in each channel of the first feature map based on the attention parameter to obtain a target reinforced feature map; performing feature extraction on the target reinforced feature map to obtain a second feature map; determining a weight coefficient of each channel of the second feature map, adjusting the feature values in the channel based on the weight coefficient of each channel of the second feature map to obtain a final feature map; and determining a label corresponding to the image based on the final feature map. The added steps can improve the data processing efficiency and the accuracy of the label recognition result, thereby reducing the error rate of the prompt picture and reducing the cost of human review.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of computer vision and image recognition. Background Technology

[0002] In related technologies, in order to label sensitive scenes in images, existing technologies usually use graph neural network models or network models containing LSTM modules to add multiple labels to the image to indicate the corresponding sensitive scenes. However, the above models take a long time to process images and are not fast enough in high-task scenarios, failing to meet the real-time requirements of labeling. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for adding image tags.

[0004] According to a first aspect of this disclosure, a method for adding image tags is provided, the method comprising:

[0005] Feature extraction is performed on the image to obtain the first feature map;

[0006] Determine the attention parameters of the first feature map, and adjust the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhancement feature map;

[0007] Feature extraction is performed on the target enhanced feature map to obtain a second feature map;

[0008] Determine the weight coefficient of each channel of the second feature map, and adjust the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain the final feature map;

[0009] The label corresponding to the image is determined based on the final feature map.

[0010] In some embodiments of this disclosure, an attention parameter of a first feature map is determined, and the feature values ​​in each channel of the first feature map are adjusted based on the attention parameter to obtain a target enhancement feature map, including:

[0011] Determine the channel attention parameters for each channel of the first feature map, and adjust the feature values ​​in that channel based on the channel attention parameters of each channel of the first feature map to obtain the initial enhanced feature map;

[0012] Determine the spatial attention parameters of the initial enhanced feature map, and adjust the feature values ​​in each channel of the initial enhanced feature map based on the spatial attention parameters to obtain the target enhanced feature map. Each channel of the initial enhanced feature map corresponds to the same spatial attention parameter.

[0013] In some embodiments of this disclosure, determining the weight coefficient of each channel of the second feature map, and adjusting the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain the final feature map, includes:

[0014] Determine the weight coefficient of each channel of the second feature map, and adjust the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain the intermediate feature map;

[0015] Feature extraction is performed on the intermediate feature map to obtain the third feature map;

[0016] Determine the weight coefficient of each channel in the third feature map, and adjust the feature value in that channel based on the weight coefficient of each channel in the third feature map to obtain the final feature map.

[0017] In some embodiments of this disclosure, determining the label corresponding to the image based on the final feature map includes:

[0018] Based on the feature value of each channel in the final feature map, calculate the label confidence score corresponding to that channel;

[0019] The label of the image is determined based on the label confidence of each channel of the final feature map.

[0020] In some embodiments of this disclosure, the target enhancement feature map is obtained from a convolutional block attention layer in a modified convolutional neural network, the convolutional block attention layer being located between two convolutional layers in the convolutional neural network;

[0021] The convolutional layer before the convolutional block attention layer is used to obtain the first feature map, and the convolutional layer after the convolutional block attention layer is used to obtain the second feature map.

[0022] In some embodiments of this disclosure, the convolutional block attention layer is located between the first and second convolutional layers of the convolutional neural network;

[0023] The first convolutional layer is used to obtain the first feature map, and the second to Nth convolutional layers are used to obtain the second feature map, where N is greater than or equal to 2.

[0024] In some embodiments of this disclosure, the final feature map is obtained by a compressed excitation layer in a convolutional neural network, which is located after the Nth convolutional layer of the convolutional neural network. The Nth convolutional layer is located after the convolutional block attention layer, and N is greater than or equal to 2.

[0025] In some embodiments of this disclosure, the intermediate feature map is obtained from the first compressed activation layer in the convolutional neural network, which is located after the Nth convolutional layer of the convolutional neural network;

[0026] The third feature map is obtained from the (N+1)th convolutional layer after the first compressed excitation layer, where N is greater than or equal to 2;

[0027] The final feature map is obtained from the second compressed activation layer in the convolutional neural network, which is located after the (N+1)th convolutional layer.

[0028] According to a second aspect of this disclosure, an image tag adding device is provided, the image tag adding device comprising a first feature extraction module, a feature enhancement module, a second feature extraction module, a weight determination module, and a tag determination module;

[0029] The first feature extraction module is used to extract features from the image to obtain a first feature map;

[0030] The feature enhancement module is used to determine the attention parameters of the first feature map, and adjust the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhanced feature map;

[0031] The second feature extraction module is used to extract features from the target enhancement feature map to obtain the second feature map;

[0032] The weight determination module is used to determine the weight coefficient of each channel of the second feature map, and adjust the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain the final feature map;

[0033] The label determination module is used to determine the label corresponding to the image based on the final feature map.

[0034] In some embodiments of this disclosure, when the feature enhancement module is used to determine the attention parameters of the first feature map and adjust the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhanced feature map, it is specifically used for:

[0035] Determine the channel attention parameters for each channel of the first feature map, and adjust the feature values ​​in that channel based on the channel attention parameters of each channel of the first feature map to obtain the initial enhanced feature map;

[0036] Determine the spatial attention parameters of the initial enhanced feature map, and adjust the feature values ​​in each channel of the initial enhanced feature map based on the spatial attention parameters to obtain the target enhanced feature map. Each channel of the initial enhanced feature map corresponds to the same spatial attention parameter.

[0037] In some embodiments of this disclosure, when the weight determination module determines the weight coefficient of each channel of the second feature map and adjusts the feature values ​​in that channel based on the weight coefficient of each channel of the second feature map to obtain the final feature map, it is specifically used for:

[0038] Determine the weight coefficient of each channel of the second feature map, and adjust the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain the intermediate feature map;

[0039] Feature extraction is performed on the intermediate feature map to obtain the third feature map;

[0040] Determine the weight coefficient of each channel in the third feature map, and adjust the feature value in that channel based on the weight coefficient of each channel in the third feature map to obtain the final feature map.

[0041] In some embodiments of this disclosure, when the label determination module is used to determine the label corresponding to an image based on the final feature map, it is specifically used for:

[0042] Based on the feature value of each channel in the final feature map, calculate the label confidence score corresponding to that channel;

[0043] The label of the image is determined based on the label confidence of each channel of the final feature map.

[0044] In some embodiments of this disclosure, the target enhancement feature map is obtained from a convolutional block attention layer in a modified convolutional neural network, the convolutional block attention layer being located between two convolutional layers in the convolutional neural network;

[0045] The convolutional layer before the convolutional block attention layer is used to obtain the first feature map, and the convolutional layer after the convolutional block attention layer is used to obtain the second feature map.

[0046] In some embodiments of this disclosure, the convolutional block attention layer is located between the first and second convolutional layers of the convolutional neural network;

[0047] The first convolutional layer is used to obtain the first feature map, and the second to Nth convolutional layers are used to obtain the second feature map, where N is greater than or equal to 2.

[0048] In some embodiments of this disclosure, the final feature map is obtained by a compressed excitation layer in a convolutional neural network, which is located after the Nth convolutional layer of the convolutional neural network. The Nth convolutional layer is located after the convolutional block attention layer, and N is greater than or equal to 2.

[0049] In some embodiments of this disclosure, the intermediate feature map is obtained from the first compressed activation layer in the convolutional neural network, which is located after the Nth convolutional layer of the convolutional neural network;

[0050] The third feature map is obtained from the (N+1)th convolutional layer after the first compressed excitation layer, where N is greater than or equal to 2;

[0051] The final feature map is obtained from the second compressed activation layer in the convolutional neural network, which is located after the (N+1)th convolutional layer.

[0052] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0053] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the method described in the first aspect.

[0054] According to a fifth aspect of this disclosure, a computer program item is provided, comprising a computer program that, when executed by a processor, implements the method according to the first aspect.

[0055] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0056] The beneficial effects of the technical solution provided in this disclosure are:

[0057] The image tag addition method provided in this embodiment adds a step to adjust the feature values ​​in the feature map through attention parameters to highlight the effective region in the image, and a step to adjust the feature values ​​in the channel based on the weight coefficient of each channel of the feature map to model the relationship between the tags. The above-mentioned added steps can improve the data processing efficiency and the accuracy of the tag recognition results, thereby reducing the error rate of the prompt image and reducing the cost of human review. Attached Figure Description

[0058] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0059] Figure 1 A schematic diagram of a modified convolutional neural network provided in this disclosure is shown;

[0060] Figure 2 A schematic diagram of another modified convolutional neural network provided in this disclosure is shown;

[0061] Figure 3 A flowchart illustrating a method for adding image tags provided in this disclosure is shown.

[0062] Figure 4 A flowchart illustrating the method for adding image labels using a modified convolutional neural network, as provided in this disclosure, is shown.

[0063] Figure 5 A schematic diagram of an image tag adding device provided in this disclosure is shown;

[0064] Figure 6 A schematic block diagram of an example electronic device that can be used to implement this disclosure is shown. Detailed Implementation

[0065] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0066] It should be understood that in the embodiments of this disclosure, the character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0067] In related technologies, in order to label sensitive scenes in images, existing technologies usually use graph neural network models or network models containing LSTM modules to add multiple labels to the image to indicate the corresponding sensitive scenes. However, the above models take a long time to process images and are not fast enough in high-task scenarios, failing to meet the real-time requirements of labeling.

[0068] The image tag addition method provided in this embodiment adds a step to adjust the feature values ​​in the feature map through attention parameters to highlight the effective region in the image, and a step to adjust the feature values ​​in the channel based on the weight coefficient of each channel of the feature map to model the relationship between the tags. The above-mentioned added steps can improve the data processing efficiency and the accuracy of the tag recognition results, thereby reducing the error rate of the prompt image and reducing the cost of human review.

[0069] The execution subject of this method can be a terminal device, a computer, a server, or other devices with data processing capabilities. No limitation is placed on the execution subject of this method. In some embodiments, the execution subject of the image tag adding method provided in this disclosure can be a terminal device on the vehicle (such as an onboard computer).

[0070] Optionally, the terminal device can be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. The specific type of terminal device is not limited in the embodiments of this disclosure.

[0071] In some embodiments, the server can be a single server, or it can be a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. This disclosure does not limit the specific implementation of the server.

[0072] The following is an illustrative example of the method for adding image tags provided in this disclosure.

[0073] The image labeling method disclosed herein can be implemented based on a modified convolutional neural network, such as VGGNet, Xception, MobileNet, and ResNet101. Since various GPUs offer greater optimization for convolutional neural networks, their inference time is also faster. Therefore, the classification network in the form of a convolutional neural network can be modified by adding an adaptation module at an appropriate location within the classification network, enabling the modified classification network to add multiple labels to the image to indicate the corresponding scene.

[0074] Figure 1 A schematic diagram of a modified convolutional neural network provided in this disclosure is shown, such as... Figure 1 As shown, a Convolutional Block Attention Module (CBAM) can be placed between two convolutional layers in a convolutional neural network. The convolutional layers before the CBAM can be used to obtain the first feature map of the image. The CBAM can then obtain a target enhancement feature map based on the first feature map. The convolutional layers after the CBAM can extract features from the target enhancement feature map to obtain a second feature map. For example, the CBAM can be located between the first and second convolutional layers of the convolutional neural network. The first convolutional layer is used to obtain the first feature map, and the second to Nth convolutional layers are used to obtain the second feature map, where N is greater than or equal to 2. Placing the CBAM after the first convolutional layer can highlight the effective regions in the image early, allowing subsequent steps to process the data in the effective regions more specifically, improving data processing efficiency and saving computational resources.

[0075] Optionally, such as Figure 1 As shown, a squeezed-and-excitation network (SE-NET) can be set after the Nth convolutional layer in a convolutional neural network, where N is greater than or equal to 2. The squeezed-and-excitation layer can obtain the final feature map based on the second feature map. After the squeezed-and-excitation layer, an average pooling layer, a fully connected layer, and a sigmoid layer can be set. The final feature map can then be used to determine the label corresponding to the image.

[0076] Figure 2 A schematic diagram of another modified convolutional neural network provided in this disclosure is shown, such as... Figure 2 As shown, two compressed activation layers can be set in a convolutional neural network, namely the first compressed activation layer and the second compressed activation layer. Specifically, the first compressed activation layer is located after the Nth convolutional layer of the convolutional neural network, followed by the (N+1)th convolutional layer, and then the second compressed activation layer is set after the (N+1)th convolutional layer. Here, the first compressed activation layer can obtain an intermediate feature map based on the second feature map, the (N+1)th convolutional layer can extract features from the intermediate feature map to obtain a third feature map, and the second compressed activation layer can obtain the final feature map based on the third feature map. After the second compressed activation layer, an average pooling layer, a fully connected layer, and a sigmoid layer can be set. The average pooling layer, the fully connected layer, and the sigmoid layer can be used to determine the label corresponding to the image.

[0077] Figure 3 A flowchart illustrating a method for adding image tags provided in this disclosure is shown, such as... Figure 3 As shown, the method mainly includes the following steps:

[0078] S310: Perform feature extraction on the image to obtain the first feature map.

[0079] In S310, an image can be input into a convolutional layer of a convolutional neural network, and feature extraction can be performed on the image based on the convolutional layer to obtain a first feature map. Alternatively, an image can be input into the first convolutional layer of the convolutional neural network, and feature extraction can be performed on the image based solely on the first convolutional layer to obtain the first feature map.

[0080] S320: Determine the attention parameters of the first feature map, and adjust the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhancement feature map.

[0081] In S320, the first feature map can be input into the convolutional block attention layer, which can determine the attention parameters of the first feature map and adjust the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhanced feature map.

[0082] Optionally, the attention parameters of the first feature map may include channel attention parameters and spatial attention parameters, wherein each channel of the initial enhanced feature map corresponds to the same spatial attention parameter. It can be understood that the channel attention parameters and spatial attention parameters are two complementary attention parameters. The channel attention parameters focus on "what" is meaningful in a given input image, while spatial attention focuses on "where" information in a given input image. Adjusting the feature values ​​in the first feature map based on these two complementary attention parameters can better highlight the effective regions in the image, facilitating targeted processing of the data in the effective regions in subsequent steps, thereby improving data processing efficiency and the accuracy of label recognition results. Specifically, the convolutional block attention layer can determine the channel attention parameters for each channel of the first feature map, adjust the feature values ​​in that channel based on the channel attention parameters of the first feature map to obtain the initial enhanced feature map; then, it determines the spatial attention parameters of the initial enhanced feature map, adjusts the feature values ​​in each channel of the initial enhanced feature map based on the spatial attention parameters, and obtains the target enhanced feature map.

[0083] S330: Extract features from the target enhanced feature map to obtain a second feature map.

[0084] In S330, the target enhancement feature map can be input into the convolutional layer after the attention layer of the convolutional block in the convolutional neural network. The convolutional layer here is used to extract features from the target enhancement feature map, and then the second feature map is output.

[0085] Optionally, the target enhancement feature map can be input into the second convolutional layer of the convolutional neural network, and the target enhancement feature map can be extracted (N-1) times using the second to Nth convolutional layers. Finally, the second feature map is output by the Nth convolutional layer.

[0086] S340: Determine the weight coefficient of each channel of the second feature map, and adjust the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain the final feature map.

[0087] In S340, the second feature map can be input into the compressed activation layer of the convolutional neural network. The compressed activation layer determines the weight coefficient of each channel of the second feature map, adjusts the feature value in that channel based on the weight coefficient, and then outputs the final feature map. By adjusting the feature value in each channel of the second feature map through the weight coefficient, the relationship between the labels is modeled, making it easier to more accurately identify the label corresponding to the image.

[0088] Optionally, embodiments of this disclosure may determine the weight coefficient of each channel of the second feature map, adjust the feature value in the channel based on the weight coefficient of each channel of the second feature map to obtain an intermediate feature map; then perform feature extraction on the intermediate feature map to obtain a third feature map; finally determine the weight coefficient of each channel of the third feature map, adjust the feature value in the channel based on the weight coefficient of each channel of the third feature map to obtain a final feature map.

[0089] Specifically, the convolutional neural network (CNN) includes a first and a second compression activation layer. The first compression activation layer follows the Nth convolutional layer, followed by the (N+1)th convolutional layer, and then the second compression activation layer. Here, the second feature map is input to the first compression activation layer, which determines the weight coefficients for each channel of the second feature map. Based on these weight coefficients, the feature values ​​in each channel are adjusted to output an intermediate feature map. The (N+1)th convolutional layer extracts the intermediate feature map and outputs a third feature map. The second compression activation layer determines the weight coefficients for each channel of the third feature map and adjusts the feature values ​​in each channel to output the final feature map.

[0090] S350: Determine the label corresponding to the image based on the final feature map.

[0091] In S350, the final feature map can be input into the mean pooling layer, and then the label corresponding to the image can be determined based on the mean pooling layer, the fully connected layer, and the sigmoid layer.

[0092] Optionally, embodiments of this disclosure may calculate the label confidence level corresponding to each channel based on the feature value of each channel in the final feature map; then, based on the label confidence level corresponding to each channel in the final feature map, determine the label corresponding to the image.

[0093] Specifically, the final feature map can be defined as (C, H, W), where C is the number of channels in the final feature map, and H and W represent different dimensions within the channels. The mean pooling layer normalizes the final feature map (C, H, W) to (C, 1, 1), averaging the feature values ​​across the H and W dimensions of each channel. Then, (C, 1, 1) is input to a fully connected layer, which maps (C, 1, 1) to M dimensions to obtain M scores. Here, each channel of the final feature map corresponds to one score. The sigmoid layer normalizes each score to between 0 and 1, obtaining the label confidence score for each channel of the final feature map.

[0094] Understandably, if the confidence level of a label is greater than the corresponding confidence threshold, then the corresponding label can be assigned to the image; if the confidence level of each label is less than the corresponding confidence threshold, then it is determined that the image does not need to be labeled.

[0095] Figure 4 The flowchart illustrating the method for adding image labels using a modified convolutional neural network, as provided in this disclosure, is shown. Figure 4 The convolutional neural network used is ResNet101, such as Figure 4 As shown, the convolutional block attention layer is placed after the first convolutional layer of ResNet101. After the convolutional block attention layer, there are the first to the 99th convolutional layers of ResNet101. The first compressed activation layer is placed after the 99th convolutional layer. The first compressed activation layer is placed after the first to the 100th convolutional layers of ResNet101. The second compressed activation layer is placed after the 100th convolutional layer. After the second compressed activation layer, there are mean pooling layers, fully connected layers, and sigmoid layers.

[0096] exist Figure 4In this process, the image is input into the first convolutional layer of ResNet101, where feature extraction is performed to obtain the first feature map. This first feature map is then input into a convolutional block attention layer, which determines the attention parameters of the first feature map and adjusts the feature values ​​in each channel of the first feature map based on these parameters to obtain the target enhancement feature map. The convolutional block attention layer then inputs the target enhancement feature map into the second convolutional layer of the convolutional neural network. Features are extracted 98 times from the second to the 99th convolutional layer, and finally, the 99th convolutional layer outputs the second feature map. The second feature map is input into the first compressed activation layer. The first compressed activation layer determines the weight coefficient of each channel of the second feature map. Based on the weight coefficient of each channel of the second feature map, the feature value in that channel is adjusted to output an intermediate feature map. The 100th convolutional layer can extract the intermediate feature map and output a third feature map. The second compressed activation layer determines the weight coefficient of each channel of the third feature map. Based on the weight coefficient of each channel of the third feature map, the feature value in that channel is adjusted to output a final feature map. The final feature map is defined as (C, H, W), where C is the number of channels in the final feature map, and H and W represent different dimensions in the channels, respectively. The mean pooling layer normalizes the final feature map (C, H, W) to (C, 1, 1), averaging the feature values ​​across the H and W dimensions for each channel. Then, (C, 1, 1) is input to a fully connected layer, which maps (C, 1, 1) to M dimensions, yielding M scores. Here, each channel of the final feature map corresponds to one score. The sigmoid layer normalizes each score to a value between 0 and 1, obtaining the label confidence score for each channel of the final feature map. Essentially, if a label confidence score is greater than a corresponding confidence threshold, the corresponding label can be assigned to the image.

[0097] Based on the same principle as the image tag adding method described above, this disclosure provides an image tag adding device. Figure 5 A schematic diagram of an image tag adding device provided in this disclosure is shown, such as... Figure 5 As shown, the image labeling device 500 includes a first feature extraction module 510, a feature enhancement module 520, a second feature extraction module 530, a weight determination module 540, and a label determination module 550.

[0098] The first feature extraction module 510 is used to extract features from the image to obtain a first feature map;

[0099] The feature enhancement module 520 is used to determine the attention parameters of the first feature map, and adjust the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhanced feature map;

[0100] The second feature extraction module 530 is used to extract features from the target enhanced feature map to obtain the second feature map;

[0101] The weight determination module 540 is used to determine the weight coefficient of each channel of the second feature map, and adjust the feature value in the channel based on the weight coefficient of each channel of the second feature map to obtain the final feature map;

[0102] The label determination module 550 is used to determine the label corresponding to the image based on the final feature map.

[0103] The image tag adding device provided in this embodiment adds a step to adjust the feature values ​​in the feature map through attention parameters to highlight the effective region in the image, and a step to adjust the feature values ​​in the channel based on the weight coefficient of each channel of the feature map to model the relationship between the tags. The above-mentioned added steps can improve the data processing efficiency and the accuracy of the tag recognition results, thereby reducing the error rate of the prompt image and reducing the cost of human review.

[0104] In some embodiments of this disclosure, when the feature enhancement module 520 determines the attention parameters of the first feature map and adjusts the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhanced feature map, it is specifically used for:

[0105] Determine the channel attention parameters for each channel of the first feature map, and adjust the feature values ​​in that channel based on the channel attention parameters of each channel of the first feature map to obtain the initial enhanced feature map;

[0106] Determine the spatial attention parameters of the initial enhanced feature map, and adjust the feature values ​​in each channel of the initial enhanced feature map based on the spatial attention parameters to obtain the target enhanced feature map. Each channel of the initial enhanced feature map corresponds to the same spatial attention parameter.

[0107] In some embodiments of this disclosure, when the weight determination module 540 determines the weight coefficient of each channel of the second feature map and adjusts the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain the final feature map, it is specifically used for:

[0108] Determine the weight coefficient of each channel of the second feature map, and adjust the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain the intermediate feature map;

[0109] Feature extraction is performed on the intermediate feature map to obtain the third feature map;

[0110] Determine the weight coefficient of each channel in the third feature map, and adjust the feature value in that channel based on the weight coefficient of each channel in the third feature map to obtain the final feature map.

[0111] In some embodiments of this disclosure, when the label determination module 550 is used to determine the label corresponding to an image based on the final feature map, it is specifically used for:

[0112] Based on the feature value of each channel in the final feature map, calculate the label confidence score corresponding to that channel;

[0113] The label of the image is determined based on the label confidence of each channel of the final feature map.

[0114] In some embodiments of this disclosure, the target enhancement feature map is obtained from a convolutional block attention layer in a modified convolutional neural network, the convolutional block attention layer being located between two convolutional layers in the convolutional neural network;

[0115] The convolutional layer before the convolutional block attention layer is used to obtain the first feature map, and the convolutional layer after the convolutional block attention layer is used to obtain the second feature map.

[0116] In some embodiments of this disclosure, the convolutional block attention layer is located between the first and second convolutional layers of the convolutional neural network;

[0117] The first convolutional layer is used to obtain the first feature map, and the second to Nth convolutional layers are used to obtain the second feature map, where N is greater than or equal to 2.

[0118] In some embodiments of this disclosure, the final feature map is obtained by a compressed excitation layer in a convolutional neural network, which is located after the Nth convolutional layer of the convolutional neural network. The Nth convolutional layer is located after the convolutional block attention layer, and N is greater than or equal to 2.

[0119] In some embodiments of this disclosure, the intermediate feature map is obtained from the first compressed activation layer in the convolutional neural network, which is located after the Nth convolutional layer of the convolutional neural network;

[0120] The third feature map is obtained from the (N+1)th convolutional layer after the first compressed excitation layer, where N is greater than or equal to 2;

[0121] The final feature map is obtained from the second compressed activation layer in the convolutional neural network, which is located after the (N+1)th convolutional layer.

[0122] It is understood that the modules of the image tag adding device in this embodiment have the function of implementing the corresponding steps of the image tag adding method described above. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The modules can be software and / or hardware; each module can be implemented individually or multiple modules can be integrated. For a detailed description of the functions of each module of the image tag adding device, please refer to the corresponding description of the image tag adding method described above, which will not be repeated here.

[0123] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0124] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program project.

[0125] In an exemplary embodiment, the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in the above embodiments. The electronic device may be the computer or server described above.

[0126] In an exemplary embodiment, the readable storage medium may be a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the above embodiments.

[0127] In an exemplary embodiment, the computer program item includes a computer program that, when executed by a processor, implements the method described in the above embodiments.

[0128] Figure 6 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0129] like Figure 6As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0130] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0131] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the image tagging method. For example, in some embodiments, the image tagging method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the image tagging method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the image tagging method by any other suitable means (e.g., by means of firmware).

[0132] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard projects (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0136] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0137] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0138] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0139] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for adding image tags, the method comprising: The first feature map is obtained by extracting features from the image using the first convolutional layer of the convolutional neural network. The attention parameters of the first feature map are determined by the convolutional block attention layer in the convolutional neural network, and the feature values ​​in each channel of the first feature map are adjusted based on the attention parameters to obtain the target enhanced feature map; The second feature map is obtained by extracting features from the second to the Nth convolutional layers in the convolutional neural network, where N is greater than or equal to 2. The weight coefficients of each channel of the second feature map are determined by the compressed activation layer in the convolutional neural network. Based on the weight coefficients of each channel of the second feature map, the feature values ​​in that channel are adjusted to obtain the final feature map. The intermediate feature map is obtained by the first compressed activation layer in the convolutional neural network, which is located after the Nth convolutional layer. The third feature map is obtained by the (N+1)th convolutional layer after the first compressed activation layer, where N is greater than or equal to 2. The final feature map is obtained by the second compressed activation layer in the convolutional neural network, which is located after the (N+1)th convolutional layer. The label corresponding to the image is determined based on the final feature map; The step of determining the label corresponding to the image based on the final feature map includes: Based on the feature values ​​of each channel of the final feature map, the label confidence corresponding to that channel is calculated; this includes: normalizing the final feature map (C,H,W) to (C,1,1), and averaging the feature values ​​in the H and W dimensions of each channel; inputting (C,1,1) into a fully connected layer, which maps (C,1,1) to M dimensions to obtain M scores, with each channel of the final feature map corresponding to one score; and normalizing each score to between 0 and 1 using a Sigmoid layer to obtain the label confidence corresponding to each channel of the final feature map. Based on the label confidence score of each channel of the final feature map, the label corresponding to the image is determined; including: if the label confidence score is greater than the corresponding confidence score threshold, the corresponding label is assigned to the image; if the label confidence score is less than the corresponding confidence score threshold, the image is determined not to be labeled. In the convolutional neural network, the convolutional block attention layer is located between the first convolutional layer and the second convolutional layer, and the compressed activation layer is located after the Nth convolutional layer.

2. The method according to claim 1, wherein, The step of determining the attention parameters of the first feature map and adjusting the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhancement feature map includes: Determine the channel attention parameter for each channel of the first feature map, and adjust the feature value in that channel based on the channel attention parameter for each channel of the first feature map to obtain an initial enhanced feature map; The spatial attention parameters of the initial enhanced feature map are determined, and the feature values ​​in each channel of the initial enhanced feature map are adjusted based on the spatial attention parameters to obtain the target enhanced feature map, wherein each channel of the initial enhanced feature map corresponds to the same spatial attention parameter.

3. The method according to claim 1, wherein, The step of determining the weight coefficient of each channel of the second feature map, and adjusting the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain the final feature map includes: Determine the weight coefficient of each channel of the second feature map, and adjust the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain an intermediate feature map; The intermediate feature map is used to extract features to obtain the third feature map; The weight coefficient of each channel of the third feature map is determined, and the feature value in that channel is adjusted based on the weight coefficient of each channel of the third feature map to obtain the final feature map.

4. An image tag adding device, the device comprising: The first feature extraction module is used to extract features from the image using the first convolutional layer of the convolutional neural network to obtain a first feature map; The feature enhancement module is used to determine the attention parameters of the first feature map by the convolutional block attention layer in the convolutional neural network, and adjust the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhanced feature map; The second feature extraction module is used to extract features from the target enhancement feature map from the second to the Nth convolutional layers in the convolutional neural network to obtain a second feature map, where N is greater than or equal to 2; wherein, the intermediate feature map is obtained from the first compressed activation layer in the convolutional neural network, which is located after the Nth convolutional layer; the third feature map is obtained from the (N+1)th convolutional layer after the first compressed activation layer, where N is greater than or equal to 2; and the final feature map is obtained from the second compressed activation layer in the convolutional neural network, which is located after the (N+1)th convolutional layer; The weight determination module is used to determine the weight coefficient of each channel of the second feature map by the compressed activation layer in the convolutional neural network, and adjust the feature value in the channel based on the weight coefficient of each channel of the second feature map to obtain the final feature map; A label determination module is used to determine the label corresponding to the image based on the final feature map; Specifically, when the label determination module determines the label corresponding to the image based on the final feature map, it is used for: Based on the feature values ​​of each channel of the final feature map, the label confidence corresponding to that channel is calculated. Specifically, this involves: normalizing the final feature map (C,H,W) to (C,1,1), and averaging the feature values ​​in the H and W dimensions of each channel; inputting (C,1,1) into a fully connected layer, which maps (C,1,1) to M dimensions to obtain M scores, with each channel of the final feature map corresponding to one score; and normalizing each score to between 0 and 1 using a Sigmoid layer to obtain the label confidence corresponding to each channel of the final feature map. Based on the label confidence score of each channel of the final feature map, the label corresponding to the image is determined; specifically, if the label confidence score is greater than the corresponding confidence score threshold, the corresponding label is assigned to the image; if the label confidence score is less than the corresponding confidence score threshold, it is determined that the image does not need to be labeled. In the convolutional neural network, the convolutional block attention layer is located between the first convolutional layer and the second convolutional layer, and the compressed activation layer is located after the Nth convolutional layer.

5. The apparatus according to claim 4, wherein, When the feature enhancement module determines the attention parameters of the first feature map and adjusts the feature values ​​in each channel of the first feature map based on the attention parameters to obtain the target enhanced feature map, it is specifically used for: Determine the channel attention parameter for each channel of the first feature map, and adjust the feature value in that channel based on the channel attention parameter for each channel of the first feature map to obtain an initial enhanced feature map; The spatial attention parameters of the initial enhanced feature map are determined, and the feature values ​​in each channel of the initial enhanced feature map are adjusted based on the spatial attention parameters to obtain the target enhanced feature map, wherein each channel of the initial enhanced feature map corresponds to the same spatial attention parameter.

6. The apparatus according to claim 4, wherein, The weight determination module, when determining the weight coefficient of each channel of the second feature map and adjusting the feature value in that channel based on the weight coefficient to obtain the final feature map, is specifically used for: Determine the weight coefficient of each channel of the second feature map, and adjust the feature value in that channel based on the weight coefficient of each channel of the second feature map to obtain an intermediate feature map; The intermediate feature map is used to extract features to obtain the third feature map; The weight coefficient of each channel of the third feature map is determined, and the feature value in that channel is adjusted based on the weight coefficient of each channel of the third feature map to obtain the final feature map.

7. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1-3.

8. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method according to any one of claims 1-3.

9. A computer program project comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-3.

Images