Training method of image multi-label classification model and image classification identification method

Abstract

The invention discloses a training method of an image multi-label classification model and an image classification identification method, and the method comprises the steps: carrying out the label labeling of each original sample image in an original sample image set, obtaining a first sample image set, and enabling each first sample image in the first sample image set to be associated with at least one positive label; configuring position information for each positive label associated with each first sample image; positioning a target object corresponding to each positive label in each first sample image according to the position information of the positive labels, removing the positioned and determined target object from the first sample image, and updating the corresponding positive label to a negative label to obtain a second sample image set; and training the network model according to the first sample image set and the second sample image set to obtain an image multi-label classification model, and carrying out image classification identification according to the image multi-label classification model. According to the invention, the recognition accuracy of the model on the small target object in the image can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a training method for an image multi-label classification model, an image classification and recognition method, an apparatus, a computer-readable storage medium, a computer program product, and a terminal device. Background Art

[0002] The media resource library stores a large amount of audio and video, book covers and illustrations, pictures and short video materials uploaded by users, etc. These pictures and videos may contain inappropriate content, so security audits are required. Traditional media resource security audits usually rely on manual audits, which are inefficient and prone to errors. Especially when small-scale bad content appears in pictures or videos, it is easy for the human eye to ignore it. With the development of machine vision, image classification models based on deep learning can realize image classification and recognition, and filter out pictures and video frames that do not meet the requirements, thereby solving the problems existing in traditional media resource security audits.

[0003] Current deep learning-based image classification models generally train features and patterns from raw image data, and use these features and patterns to classify or identify image data. In the training process, the image classification model mainly identifies the corresponding target in the image based on the label. Among them, the Convolutional Neural Network (CNN) is the most commonly used deep learning model. It can effectively extract image features through a combination of convolutional layers, pooling layers, and fully connected layers, and achieve multi-label classification through end-to-end training. Before training, for supervised learning, the training sample set is usually labeled, and for the image multi-label classification task, since each image in the training sample set may be associated with one or more labels, it is necessary to assign a set (at least one) of corresponding labels to each image. The label can be a descriptive text (such as "cat", "dog", etc.) or a unique digital identifier. In addition, after preprocessing operations such as scaling, cropping, and normalization of each image, the training sample set can be sent as input data to the convolutional neural network for training.

[0004] However, the sizes of targets corresponding to different labels may vary significantly. The labels of large targets in images occupy more pixel areas in the image and are therefore more easily noticed by the model during feature extraction and classification. In contrast, for small targets in images, even if the image already has a label corresponding to a small target, the model may not be able to recognize the small target corresponding to the label. The labels of small targets are ignored due to insufficient feature information, which ultimately leads to missed detection of small targets or misidentification of the background as being related to the label of the small target, thereby affecting the accuracy of small target recognition. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a training method for an image multi-label classification model, an image classification and recognition method, an apparatus, a computer-readable storage medium, a computer program product and a terminal device, which can solve the problem that small targets in images are easily missed and avoid interference from the background, thereby improving the accuracy of the model in recognizing small targets.

[0006] In order to achieve the above object, a first aspect of the present invention provides a method for training an image multi-label classification model, comprising:

[0007] Labeling the target object in each original sample image in the original sample image set according to a preset label set to obtain a first sample image set; wherein each first sample image in the first sample image set is associated with at least one positive label, and the positive label indicates that the target object corresponding to the label in the label set exists in the first sample image;

[0008] According to the bounding box information of the target object corresponding to the positive label in the first sample image, configuring the position information for each positive label associated with each first sample image;

[0009] According to the position information of the positive label, locate the target object corresponding to each positive label in each first sample image, remove the located target object from the first sample image, and update the corresponding positive label to a negative label to obtain a second sample image set;

[0010] A preset network model is trained according to the first sample image set and the second sample image set to obtain an image multi-label classification model, so as to classify and identify images according to the image multi-label classification model.

[0011] Furthermore, labeling the target object in each original sample image in the original sample image set according to the preset label set specifically includes:

[0012] Compare each original sample image in the original sample image set with each label in the preset label set to determine whether there is a target object corresponding to the label in the label set in the original sample image;

[0013] If yes, the target object in the original sample image is marked as a positive label;

[0014] If not, the target object that does not exist in the original sample image is marked as a negative label.

[0015] Further, according to the position information of the positive label, the target object corresponding to each positive label is located in each first sample image, and the located target object is removed from the first sample image, and the corresponding positive label is updated to a negative label to obtain a second sample image set, which specifically includes:

[0016] According to the position information of the positive label, locate the target object corresponding to each positive label in each first sample image;

[0017] For each first sample image, the target area corresponding to each target object determined by positioning in the first sample image is updated to the background area, and the corresponding positive label is updated to the negative label, so as to obtain a processed first sample image; wherein the method of updating the target area to the background area includes: updating the color channel values ​​of all pixels in the target area to the same preset value, or updating the color channel value of each pixel in the target area to a random value within a preset color range;

[0018] The processed first sample images are used as second sample images, and a second sample image set is obtained according to all the second sample images.

[0019] Furthermore, the training of a preset network model according to the first sample image set and the second sample image set to obtain an image multi-label classification model specifically includes:

[0020] For each iteration in the model training process, according to the relationship between the currently generated random value and the preset probability threshold, selecting the first sample image in the first sample image set or the second sample image in the second sample image set as the training sample image;

[0021] Input the training sample image into a preset network model for classification prediction to obtain a label classification prediction result;

[0022] Input the label classification prediction result and the real label data of the training sample image into a preset loss function to obtain a total loss value of the current label classification;

[0023] Back propagation is performed according to the total loss value to update the model parameters of the network model until the loss function converges to obtain an image multi-label classification model.

[0024] In order to achieve the above object, a second embodiment of the present invention provides an image classification and recognition method, comprising:

[0025] Obtain an image to be recognized;

[0026] The target object in the image to be identified is classified and identified according to a preset image multi-label classification model to obtain a recognition result; wherein the image multi-label classification model is trained using the image multi-label classification model training method described in any one of the first aspects above.

[0027] In order to achieve the above-mentioned purpose, the third aspect of the present invention also provides a training device for an image multi-label classification model, comprising:

[0028] A labeling module is used to label the target object in each original sample image in the original sample image set according to a preset label set to obtain a first sample image set; wherein each first sample image in the first sample image set is associated with at least one positive label, and the positive label indicates that the target object corresponding to the label in the label set exists in the first sample image;

[0029] A position information configuration module, configured to configure position information for each positive label associated with each first sample image according to the bounding box information of the target object corresponding to the positive label in the first sample image;

[0030] A sample image processing module is used to locate the target object corresponding to each positive label in each first sample image according to the position information of the positive label, remove the located target object from the first sample image, and update the corresponding positive label to a negative label to obtain a second sample image set;

[0031] A model training module is used to train a preset network model according to the first sample image set and the second sample image set to obtain an image multi-label classification model so as to classify and identify images according to the image multi-label classification model.

[0032] In order to achieve the above-mentioned purpose, the fourth aspect of the present invention further provides an image classification and recognition device, comprising:

[0033] An image acquisition module to be identified, used to acquire an image to be identified;

[0034] The image classification module to be identified is used to classify and identify the target object in the image to be identified according to a preset image multi-label classification model to obtain a recognition result; wherein the image multi-label classification model is trained using the image multi-label classification model training method described in any one of the first aspects above.

[0035] The fifth aspect embodiment of the present invention also provides a computer-readable storage medium, including a stored computer program, which, when running, controls the device where the computer-readable storage medium is located to execute the training method of the image multi-label classification model described in any one of the first aspect above, or to execute the image classification and recognition method described in any one of the second aspect above.

[0036] The sixth aspect of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the training method of the image multi-label classification model described in any one of the first aspect above, or implements the image classification and recognition method described in any one of the second aspect above.

[0037] The seventh aspect embodiment of the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when executing the computer program, the processor implements the training method of the image multi-label classification model described in any one of the first aspect above, or implements the image classification and recognition method described in any one of the second aspect above.

[0038] Compared with the prior art, the embodiments of the present invention provide a training method for an image multi-label classification model, an image classification and recognition method, an apparatus, a computer-readable storage medium, a computer program product and a terminal device, wherein a target object in each original sample image in the original sample image set is labeled according to a preset label set to obtain a first sample image set, wherein each first sample image in the first sample image set is associated with at least one positive label, and the positive label indicates a target object corresponding to a label in the label set existing in the first sample image; according to the bounding box information of the target object corresponding to the positive label in the first sample image, position information is configured for each positive label associated with each first sample image; according to the position information of the positive label, the target object corresponding to each positive label is located in each first sample image, and the located target object is removed from the first sample image, and the corresponding positive label is updated to a negative label to obtain a second sample image set; according to the first sample image set and the second sample image set, a preset network model is trained to obtain an image multi-label classification model, so as to classify and recognize images according to the image multi-label classification model. The embodiment of the present invention uses the first sample image and the second sample image (containing only background information) for model training, which can not only promote the model to learn the association between the classification label and the target object in the image, but also enable the model to focus on learning the background features in the image, enhance the model's ability to distinguish between the target object and the background, and strengthen the model's learning relationship between the label and small targets, thereby solving the problem of small targets in the image being easily missed, and avoiding interference caused by the background, thereby improving the accuracy of the model in identifying small targets in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a preferred embodiment of a training method for an image multi-label classification model provided by the first aspect of the present invention;

[0040] FIG. 2A to FIG. 2D is an example diagram of updating a target area to a background area provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic structural diagram of a residual module provided by an embodiment of the present invention;

[0042] Figure 4 is a flowchart of a preferred embodiment of an image classification and recognition method provided by the second aspect of the present invention;

[0043] Figure 5 It is a structural block diagram of a preferred embodiment of a terminal device provided in the seventh aspect of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this technical field without creative work are within the scope of protection of the present invention.

[0045] The first embodiment of the present invention provides a method for training an image multi-label classification model. Figure 1 FIG. 1 is a flowchart of a preferred embodiment of a method for training an image multi-label classification model provided by the first aspect of the present invention, wherein the method comprises steps S11 to S14:

[0046] Step S11, labeling the target object in each original sample image in the original sample image set according to a preset label set to obtain a first sample image set; wherein each first sample image in the first sample image set is associated with at least one positive label, and the positive label indicates that the target object corresponding to the label in the label set exists in the first sample image;

[0047] Step S12: configuring position information for each positive label associated with each first sample image according to the bounding box information of the target object corresponding to the positive label in the first sample image;

[0048] Step S13: locate the target object corresponding to each positive label in each first sample image according to the position information of the positive label, remove the located target object from the first sample image, and update the corresponding positive label to a negative label to obtain a second sample image set;

[0049] Step S14: training a preset network model according to the first sample image set and the second sample image set to obtain an image multi-label classification model, so as to classify and recognize images according to the image multi-label classification model.

[0050] Specifically, first, a label set is pre-set, and the label set includes all possible labels in the original sample image set. Then, for each original sample image in the original sample image set, a corresponding label can be selected from the pre-set label set according to the target object contained in each original sample image, and the target object in each original sample image is labeled respectively. Correspondingly, the original sample image with added label information (that is, the labeled original image sample) is used as the first sample image, and all the first sample images are collected together to obtain the first sample image set, wherein each first sample image is labeled with a positive label, and the positive label indicates that there is a target object corresponding to the label in the label set in the first sample image. It can be understood that each original sample image contains at least one target object, and each first sample image obtained after labeling is associated with at least one positive label; then, for each positive label associated with each first sample image, there will be a corresponding target object in the first sample image to which it is associated, then, according to the edge of the target object corresponding to each positive label in the first sample image, Bounding box information, respectively configure corresponding position information (i.e., bounding box information) for each positive label associated with each first sample image; then, for each first sample image in the first sample image set, according to the position information of each positive label associated with the first sample image, locate the target object corresponding to each positive label in each first sample image, and remove each target object determined by positioning from the first sample image, so that the target area corresponding to the target object in the first sample image (i.e., the area indicated by the bounding box of the target object) is updated to a pure background area, and the positive label corresponding to each target object determined by positioning is updated to a negative label, and accordingly, the updated first sample image is used as the second sample image, and all the second sample images are collected together to obtain the second sample image set; finally, according to the obtained first sample image set and the second sample image set, the pre-set network model is trained to obtain the trained network model accordingly, and the trained network model can be used as an image multi-label classification model to classify and recognize the image according to the image multi-label classification model, thereby identifying each target object in the image.

[0051] It should be noted that, in general, the bounding box of the target object in the image is a rectangular box. At this time, the position information of the positive label can be defined by the coordinate positions of the lower left corner and the upper right corner of the corresponding rectangular bounding box. Assuming that the coordinates of the pixel point of the lower left corner (leftmost and bottommost) of a rectangular bounding box in an original sample image are determined to be (Xmin, Ymin), and the coordinates of the pixel point of the upper right corner (rightmost and topmost) are (Xmax, Ymax), then, the position information (bounding box information) of the corresponding positive label can be expressed as (Xmin, Ymin, Xmax, Ymax), and accordingly, the position information (bounding box information) of all positive labels associated with the first sample image corresponding to the original sample image can be expressed as [(X1min, Y1min, X1max, Y1max), (X2min, Y2min, X2max, Y2max), ...].

[0052] It should be noted that the first sample image set actually only contains the original sample images and their corresponding positive labels, and does not contain the location information of the positive labels. The purpose of configuring the location information for the positive labels is to remove the target objects from the first sample images to obtain the second sample image set. Therefore, in the model training stage, when using the first sample image and the second sample image (containing only background information) for model training, the first sample image does not directly tell the model the location information of the target object (including large targets and small targets) in the image. The corresponding bounding box, but through the second sample image with the target object removed, the model is allowed to perform comparative training and learning, thereby indirectly telling the model the location information of the target object, so that the model pays more attention to the area of ​​interest.

[0053] It can be understood that after the image multi-label classification model is obtained by training based on the first sample image set and the second sample image set, pre-prepared test sample images (usually sample images that have not been used by the model in the training phase) can be input into the trained model for testing to evaluate the generalization ability of the image multi-label classification model to new image data.

[0054] A training method for an image multi-label classification model provided by an embodiment of the first aspect of the present invention is mainly used in the training process of the image multi-label classification model. In the model training stage, a first sample image and a second sample image (containing only background information) are used for model training. When the first sample image is used for model training, the position information contained in the bounding box corresponding to the target object (including large targets and small targets) in the image can be used to make the model pay more attention to the area of ​​interest, and promote the model to learn the association between the classification label and the target object in the image. When the second sample image is used for model training, the model can be focused on learning the background features in the image, enhancing the model's ability to distinguish between the target object and the background, and strengthening the association between the model learning label and the small target object, so that when the image is classified and recognized according to the image multi-label classification model, even if the target object in the image is very small, the model can accurately identify the target object with different background features from the background, thereby solving the problem of easy omission of small targets in the image, and avoiding interference caused by the background, thereby improving the accuracy of the model in recognizing small targets in the image, and at the same time improving the accuracy of the model in recognizing multi-label images.

[0055] In one of the optional embodiments, labeling the target object in each original sample image in the original sample image set according to the preset label set specifically includes:

[0056] Compare each original sample image in the original sample image set with each label in the preset label set to determine whether there is a target object corresponding to the label in the label set in the original sample image;

[0057] If yes, the target object in the original sample image is marked as a positive label;

[0058] If not, the target object that does not exist in the original sample image is marked as a negative label.

[0059] Specifically, in combination with the above embodiments, when labeling the target object in each original sample image in the original sample image set, each original sample image in the original sample image set can be compared with each label in the preset label set to determine whether there is a target object corresponding to the label in the label set in the original sample image; when it is determined that there is a target object corresponding to the label in the label set in the original sample image, the corresponding label can be selected from the label set to label the target object, and the target object is marked as a positive label; when it is determined that there is no target object corresponding to the label in the label set in the original sample image, the non-existent target object is directly marked as a negative label.

[0060] It should be noted that when labeling is actually performed, the label information of each original sample image can be represented in the form of a vector. The number of elements in the vector is the same as the number of labels contained in the label set. The element value in the vector is 1 or 0, where 1 represents a positive label, that is, it indicates that there is a target object corresponding to a label in the label set in the original sample image, and 0 represents a negative label, that is, it indicates that there is no target object corresponding to a label in the label set in the original sample image.

[0061] For example, suppose there is a multi-label classification task about animal images, and the preset label set includes "cat", "dog", "bird" and "rabbit"; for an image containing a cat and a rabbit, when labeling, it can be labeled as two positive labels "cat" and "rabbit", and the positive labels "cat" and "rabbit" can be converted into a vector [1, 0, 0, 1] to represent the label information of the image.

[0062] In one of the optional embodiments, locating the target object corresponding to each positive label in each first sample image according to the position information of the positive label, removing the located target object from the first sample image, and updating the corresponding positive label to a negative label to obtain the second sample image set specifically includes:

[0063] According to the position information of the positive label, locate the target object corresponding to each positive label in each first sample image;

[0064] For each first sample image, the target area corresponding to each target object determined by positioning in the first sample image is updated to the background area, and the corresponding positive label is updated to the negative label, so as to obtain a processed first sample image; wherein the method of updating the target area to the background area includes: updating the color channel values ​​of all pixels in the target area to the same preset value, or updating the color channel value of each pixel in the target area to a random value within a preset color range;

[0065] The processed first sample images are used as second sample images, and a second sample image set is obtained according to all the second sample images.

[0066] Specifically, in combination with the above embodiment, when obtaining the second sample image set according to the first sample image set, since the position information of the positive label is the bounding box information of the corresponding target object in the first sample image associated with it, for each first sample image in the first sample image set, the target object corresponding to each positive label can be located in each first sample image according to the position information of each positive label associated with the first sample image. For each target object determined by positioning, the color channel values ​​of all pixels contained in the target area corresponding to the target object in the first sample image (that is, the area indicated by the bounding box of the target object) can be updated to the same preset value, or the color channel value of each pixel contained in the target area corresponding to the target object in the first sample image can be updated to a random value within a preset color range, so that the target area corresponding to the target object in the first sample image is updated to a pure background area, and the positive label corresponding to each target object determined by positioning is updated to a negative label. Accordingly, the first sample image after the update processing is used as the second sample image, and all the second sample images are grouped together to obtain the second sample image set.

[0067] For example, suppose there is a multi-label classification task for animal images, and the pre-set label set includes "cat", "dog", "bird" and "rabbit"; for an image containing "cat" and "rabbit", when labeling, it can be labeled as the two positive labels "cat" and "rabbit", and the positive labels "cat" and "rabbit" are converted into a vector [1, 0, 0, 1] to represent the label information of the image; further, after the target area corresponding to "cat" and "rabbit" in the image is updated to the background area, and the two positive labels "cat" and "rabbit" are updated to negative labels, the corresponding label information is expressed as [0, 0, 0, 0].

[0068] It should be noted that there are three main processing methods for updating the target area corresponding to the target object in the first sample image to a pure background area: the first method is to set the color channel values ​​of all pixels in the target area to 0 (that is, update to the same preset value, which is 0), so that the target image in the target area becomes a black background image; the second method is to set the color channel values ​​of all pixels in the target area to the same randomly generated value in the range of 0 to 255 (that is, update to the same preset value, which is a randomly generated value in the range of 0 to 255), so that the target image in the target area becomes a color background image with the same color; the third method is to set the color channel value of each pixel in the target area to a randomly generated value in the range of 0 to 255 (that is, update to a random value in the preset color range, and each pixel corresponds to a randomly generated value in the range of 0 to 255), and each pixel is randomly set to a color, so that the target image in the target area becomes a color background image with different colors.

[0069] Combination FIG. 2A to FIG. 2D FIG. 1 is an example diagram of updating a target area to a background area provided by an embodiment of the present invention. Figure 2A is a first sample image, in which “person” and “dog” are positive labels of the first sample image, and the position information of each positive label (that is, the bounding box information corresponding to “person” and “dog”) has been determined; Figure 2B The second sample image is obtained by updating each target area (i.e., the target area corresponding to "person" and "dog") in the first sample image to the background area using the first method, such as Figure 2B As shown, the target area becomes a pure black background area; Figure 2C The second sample image is a second sample image in which each target area (i.e., the target area corresponding to "person" and "dog") in the first sample image is updated to the background area by using the second method described above, such as Figure 2C As shown, the target area becomes a colored background area with the same color ( Figure 2C The images shown are grayscale and should be in color); Figure 2D The second sample image is obtained by updating each target area (i.e., the target area corresponding to "person" and "dog") in the first sample image to the background area using the third method, such as Figure 2D As shown, the target area becomes a colored background area with different colors ( Figure 2D The images shown are in grayscale and should be in color.)

[0070] In one optional embodiment, the training of a preset network model according to the first sample image set and the second sample image set to obtain an image multi-label classification model specifically includes:

[0071] For each iteration in the model training process, according to the relationship between the currently generated random value and the preset probability threshold, selecting the first sample image in the first sample image set or the second sample image in the second sample image set as the training sample image;

[0072] Input the training sample image into a preset network model for classification prediction to obtain a label classification prediction result;

[0073] Input the label classification prediction result and the real label data of the training sample image into a preset loss function to obtain a total loss value of the current label classification;

[0074] Back propagation is performed according to the total loss value to update the model parameters of the network model until the loss function converges to obtain an image multi-label classification model.

[0075] Specifically, in combination with the above embodiment, when the preset network model is trained according to the first sample image set and the second sample image set, multiple iterations are generally required, and in each iteration, the training sample image to be input into the network model for training and learning can be the first sample image in the first sample image set or the second sample image in the second sample image set. The specific sample image to be selected can be determined according to the size relationship between the random value generated in the current iteration and the preset probability threshold, and the first sample image in the first sample image set is determined to be selected as the training sample image, or the second sample image in the second sample image set is determined to be selected as the training sample image. Then, the selected training sample image is input into the network model for classification and pre-training. The label classification prediction result corresponding to the training sample image is measured, and the label classification prediction result corresponding to the training sample image is obtained; then, the label classification prediction result corresponding to the training sample image and the real label data of the training sample image are input into a preset loss function for loss calculation, and the total loss value of the label classification corresponding to the current iteration is obtained; finally, a back propagation algorithm is used to perform back propagation according to the total loss value of the label classification obtained in the current iteration to update the model parameters (such as weight parameters) of the network model, and through multiple iterations until the loss function converges, a trained network model is obtained (the network model can learn the association between the classification label and the image target by comparing the first sample image and the second sample image), and the trained network model can be used as an image multi-label classification model.

[0076] It should be noted that when selecting training sample images from the first sample image set and the second sample image set, a probability threshold in the range of 0 to 1 can be set in advance. When the random value generated in this iteration (also in the range of 0 to 1) is greater than the above probability threshold, the first sample image in the first sample image set can be selected as the training sample image. When the random value generated in this iteration is less than or equal to the above probability threshold, the second sample image in the second sample image set can be selected as the training sample image.

[0077] It should be noted that before inputting the training sample images into the network model for model training, the training sample images can also be preprocessed by scaling, cropping, normalization, etc. to remove irrelevant information in the training sample images, and the preprocessed training sample images are obtained accordingly. The preprocessed training sample images are then input into the network model for model training.

[0078] It should be noted that the pre-set network model can adopt the current mainstream convolutional neural network model. In the embodiment of the present invention, the network model can specifically use the ResNet (deep residual network) model. The deep residual network is a deep convolutional neural network, and its core component is the residual module. The residual module allows direct connection between the input and output layers, includes multiple convolutional layers and a skip connection, and usually also includes operations such as batch normalization and activation functions; Figure 3 , which is a schematic diagram of the structure of a residual module provided by an embodiment of the present invention. The input of the residual module is a, and it includes two convolutional layers ( Figure 3 weight layer shown) and a skip connection, and also includes an activation function ( Figure 3 The skip connection adds the input a directly to the output F(a) of the second convolutional layer to form a residual connection. The final output F(a)+a will continue to be passed as a new input to the next residual module. This connection method of the residual module enables the network to learn the residual part, that is, the difference between the input and the output, so that the network can still be trained and optimized when the depth increases, and effectively alleviate the problem of gradient disappearance.

[0079] Furthermore, the deep residual network adds a global average pooling layer after the last convolutional layer, which can convert the feature map into a feature vector. The purpose of this layer is to convert each feature map into a single value, that is, the average value of the feature map. For example, assuming that the size of the feature map output by the last convolutional layer is (H, W, C), where H and W represent the height and width of the feature map, respectively, and C is the number of channels of the feature map, then global average pooling can compress each H*W feature map into a value, and finally output a feature map of size (1, 1, C), which can be regarded as a feature vector with a length of C.

[0080] Furthermore, the feature vector obtained after the global average pooling layer can be used as the numerical input of the fully connected layer. An independent fully connected layer is used for each classification label. Each fully connected layer obtains different weight matrices and bias items through model training. After passing through the fully connected layer, a numerical value is output, which represents the prediction score of the corresponding classification label, and the Sigmoid function is used to convert it into a probability value between 0 and 1, that is, the prediction probability of each classification label is obtained (equivalent to the label classification prediction result of the image). The expression of the Sigmoid function is: x represents the output of the fully connected layer, and Sigmoid(x) represents the probability value obtained after conversion by the Sigmoid function.

[0081] Furthermore, the loss function is calculated based on the predicted probability values ​​of each classification label obtained above and the real labels corresponding to each classification label (equivalent to the real label data of the image). In the image multi-label classification task, the binary cross entropy function can be used to calculate the loss value of each label, and the loss values ​​of all labels are added to obtain the total loss value; assuming that the number of training samples is N, the i-th training sample corresponds to M classification labels, L j It represents the loss value of the j-th classification label of the i-th training sample, and its expression is: y i Indicates the true label data (0 or 1) of the jth classification label corresponding to the i-th training sample, p i represents the predicted probability value of the jth classification label; L total It represents the total loss value of the i-th training sample, which is the sum of the loss values ​​of all M classification labels. Its expression is:

[0082] Furthermore, according to the total loss value calculated by the loss function, the back-propagation algorithm can be used to back-propagate the error information between the model's prediction results and the true labels from the output layer to the input layer layer by layer, and at the same time, the weight parameters in the model are updated according to the error information. Through the above continuous iterations, the loss function is gradually converged, thereby minimizing the total loss value and optimizing the network model, and finally a trained network model is obtained.

[0083] The second embodiment of the present invention provides an image classification and recognition method, see Figure 4 FIG. 1 is a flowchart of a preferred embodiment of an image classification and recognition method provided by the second aspect of the present invention, wherein the method comprises steps S21 to S22:

[0084] Step S21, obtaining an image to be identified;

[0085] Step S22, classify and identify the target object in the image to be identified according to a preset image multi-label classification model to obtain a recognition result; wherein the image multi-label classification model is trained using the image multi-label classification model training method described in any embodiment of the first aspect above.

[0086] Specifically, after obtaining an image multi-label classification model by training the image multi-label classification model training method described in any embodiment of the first aspect above, the image to be identified can be obtained first, and then the image to be identified can be classified and identified using the trained image multi-label classification model to obtain a corresponding recognition result.

[0087] It should be noted that images containing inappropriate information will be strictly reviewed in the media library. By adopting the embodiment of the present invention, images to be reviewed can be obtained from the media library, and the trained image multi-label classification model can be used to classify and identify the images to be reviewed, so as to screen out images containing preset types of inappropriate content.

[0088] Exemplarily, an embodiment of the present invention can obtain a training model by pre-training and learning the model for a large number of pictures containing guns and corresponding pure background pictures without guns. In actual image classification and recognition, the image to be reviewed is input into the training model. For each image to be reviewed, the model will output a set of probability values ​​for the classification label, among which the predicted probability of the label corresponding to the image containing guns is higher than a preset threshold. At this time, the model will determine that there is a gun target in the image.

[0089] It should be noted that the embodiments of the present invention can help the system model to conduct a more comprehensive review and analysis of the scenes of key frames in images and videos, help to quickly identify and process images or videos containing illegal, sensitive or bad information, and can greatly improve the review efficiency and reduce the burden of manual review. At the same time, it can also reduce the risks that media content may bring during the dissemination process and ensure the security and compliance of platform content.

[0090] The third aspect of the present invention further provides a training device for an image multi-label classification model, which is used to implement the training method for an image multi-label classification model described in any embodiment of the first aspect, and the training device includes:

[0091] A labeling module is used to label the target object in each original sample image in the original sample image set according to a preset label set to obtain a first sample image set; wherein each first sample image in the first sample image set is associated with at least one positive label, and the positive label indicates that the target object corresponding to the label in the label set exists in the first sample image;

[0092] A position information configuration module, configured to configure position information for each positive label associated with each first sample image according to the bounding box information of the target object corresponding to the positive label in the first sample image;

[0093] A sample image processing module is used to locate the target object corresponding to each positive label in each first sample image according to the position information of the positive label, remove the located target object from the first sample image, and update the corresponding positive label to a negative label to obtain a second sample image set;

[0094] A model training module is used to train a preset network model according to the first sample image set and the second sample image set to obtain an image multi-label classification model so as to classify and identify images according to the image multi-label classification model.

[0095] Preferably, the label marking module specifically includes:

[0096] A label determination unit, used to compare each original sample image in the original sample image set with each label in the preset label set, so as to determine whether there is a target object corresponding to the label in the label set in the original sample image;

[0097] A positive label marking unit, used for marking the target object existing in the original sample image as a positive label if yes;

[0098] The negative label marking unit is used to mark the target object that does not exist in the original sample image as a negative label if no.

[0099] Preferably, the sample image processing module specifically includes:

[0100] An object positioning unit, used to locate an object corresponding to each positive label in each first sample image according to the position information of the positive label;

[0101] The sample image processing unit is used to update the target area corresponding to each target object determined by positioning in the first sample image to the background area, and update the corresponding positive label to the negative label, so as to obtain a processed first sample image; wherein the method of updating the target area to the background area includes: updating the color channel values ​​of all pixels in the target area to the same preset value, or updating the color channel value of each pixel in the target area to a random value within a preset color range;

[0102] The second sample image set acquisition unit is used to use the processed first sample images as second sample images, and obtain a second sample image set according to all the second sample images.

[0103] Preferably, the model training module specifically includes:

[0104] a training sample image selection unit, configured to select, for each iteration in the model training process, a first sample image in the first sample image set or a second sample image in the second sample image set as a training sample image according to a magnitude relationship between a currently generated random value and a preset probability threshold;

[0105] A label classification prediction unit, used to input the training sample image into a preset network model for classification prediction to obtain a label classification prediction result;

[0106] A loss calculation unit, used to input the label classification prediction result and the real label data of the training sample image into a preset loss function to obtain a total loss value of the current label classification;

[0107] The model optimization unit is used to perform back propagation according to the total loss value to update the model parameters of the network model until the loss function converges to obtain an image multi-label classification model.

[0108] It should be noted that the training device for an image multi-label classification model provided in the embodiment of the third aspect of the present invention can implement all the processes of the training method for an image multi-label classification model described in any embodiment of the first aspect above. The functions of each module and unit in the training device and the technical effects achieved are respectively the same as the functions and technical effects achieved by the training method for the image multi-label classification model described in the embodiment of the first aspect above, and will not be repeated here.

[0109] The fourth aspect of the present invention further provides an image classification and recognition device, which is used to implement the image classification and recognition method described in any one of the embodiments of the second aspect, and the recognition device includes:

[0110] An image acquisition module to be identified, used to acquire an image to be identified;

[0111] The image classification module to be identified is used to classify and identify the target object in the image to be identified according to a preset image multi-label classification model to obtain a recognition result; wherein the image multi-label classification model is trained using the image multi-label classification model training method described in any embodiment of the first aspect above.

[0112] It should be noted that the image classification and recognition device provided in the embodiment of the fourth aspect of the present invention can implement all the processes of the image classification and recognition method described in any embodiment of the second aspect above. The functions of each module in the recognition device and the technical effects achieved are respectively the same as the functions and technical effects achieved by the image classification and recognition method described in the embodiment of the second aspect above, and will not be repeated here.

[0113] The fifth aspect of the present invention also provides a computer-readable storage medium, including a stored computer program, which, when running, controls the device where the computer-readable storage medium is located to execute the training method of the image multi-label classification model described in any embodiment of the first aspect above, or to execute the image classification and recognition method described in any embodiment of the second aspect above.

[0114] The sixth aspect of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the training method of the image multi-label classification model described in any embodiment of the first aspect above, or implements the image classification and recognition method described in any embodiment of the second aspect above.

[0115] The seventh aspect of the present invention further provides a terminal device, see Figure 5 As shown, it is a structural block diagram of a preferred embodiment of a terminal device provided in the seventh aspect of the present invention, wherein the terminal device includes a processor 10, a memory 20, and a computer program stored in the memory 20 and configured to be executed by the processor 10, and when executing the computer program, the processor 10 implements the training method of the image multi-label classification model described in any embodiment of the first aspect above, or implements the image classification and recognition method described in any embodiment of the second aspect above.

[0116] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 20 and executed by the processor 10 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0117] The processor 10 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 10 may be any conventional processor. The processor 10 is the control center of the terminal device, and various parts of the terminal device are connected using various interfaces and lines.

[0118] The memory 20 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 20 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 20 can also be other volatile solid-state storage devices.

[0119] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 5 The structural block diagram shown is only a structural example of the above-mentioned terminal device and does not constitute a structural limitation of the above-mentioned terminal device. The above-mentioned terminal device may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0120] In summary, the training method, image classification and recognition method, device, computer-readable storage medium, computer program product and terminal device of an image multi-label classification model provided by the embodiments of the present invention can promote the model to learn the association between classification labels and image targets by introducing a first sample image (equivalent to the original image with a label) and a second sample image (equivalent to the pure background image) as training sample images for model training. During the training process, the model calculates the loss function based on the input image and the corresponding true label, and updates the weight parameters of the model through back propagation. Furthermore, when the model learns the original image with the target object and the pure background image respectively, which features will be learned? This contrast learning method enhances the model's ability to distinguish between targets and backgrounds, by comparing which features are related to the target and which features are related to the background. Even for small targets in the image, although they account for a small proportion of the image, by introducing pure background images, the model can focus on learning the features of the background. This helps the model to more accurately identify those features that appear on small targets and are different from the background when processing the original image later, thereby giving higher weights to those features that are different from the background, thereby improving the model's accuracy in identifying small targets in the image, significantly improving the pain point of small targets in the image being difficult to identify, and at the same time improving the model's accuracy in multi-label recognition of images.

[0121] In addition, the embodiments of the present invention can enhance the robustness of the model to different background changes by additionally introducing pure background images for model training. In practical applications, the background may change due to factors such as lighting, occlusion, and noise. If the model only learns the original image with the target object during the training process, it may be too sensitive to the background changes and cause misjudgment. However, if the model also learns the pure background image, it can better avoid the interference caused by these irrelevant background changes and pay more attention to the features related to the target object, thereby improving the generalization ability of the model.

[0122] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A training method for an image multi-label classification model, characterized in that: include: Labeling the target object in each original sample image in the original sample image set according to a preset label set to obtain a first sample image set; wherein each first sample image in the first sample image set is associated with at least one positive label, and the positive label indicates that the target object corresponding to the label in the label set exists in the first sample image; According to the bounding box information of the target object corresponding to the positive label in the first sample image, configuring the position information for each positive label associated with each first sample image; According to the position information of the positive label, locate the target object corresponding to each positive label in each first sample image, remove the located target object from the first sample image, and update the corresponding positive label to a negative label to obtain a second sample image set; A preset network model is trained according to the first sample image set and the second sample image set to obtain an image multi-label classification model, so as to classify and identify images according to the image multi-label classification model.

2. The method for training a multi-label image classification model according to claim 1, wherein: The step of labeling the target object in each original sample image in the original sample image set according to the preset label set specifically includes: Compare each original sample image in the original sample image set with each label in the preset label set to determine whether there is a target object corresponding to the label in the label set in the original sample image; If yes, the target object in the original sample image is marked as a positive label; If not, the target object that does not exist in the original sample image is marked as a negative label.

3. The method for training an image multi-label classification model according to claim 1, wherein: The method locates the target object corresponding to each positive label in each first sample image according to the position information of the positive label, removes the located target object from the first sample image, and updates the corresponding positive label to a negative label to obtain a second sample image set, specifically including: According to the position information of the positive label, locate the target object corresponding to each positive label in each first sample image; For each first sample image, the target area corresponding to each target object determined by positioning in the first sample image is updated to the background area, and the corresponding positive label is updated to the negative label, so as to obtain the processed first sample image; wherein the method of updating the target area to the background area includes: updating the color channel values ​​of all pixels in the target area to the same preset value, or updating the color channel value of each pixel in the target area to a random value within a preset color range; The processed first sample images are used as second sample images, and a second sample image set is obtained according to all the second sample images.

4. The method for training an image multi-label classification model according to claim 1, wherein: The training of a preset network model according to the first sample image set and the second sample image set to obtain an image multi-label classification model specifically includes: For each iteration in the model training process, according to the magnitude relationship between the currently generated random value and the preset probability threshold, selecting the first sample image in the first sample image set or the second sample image in the second sample image set as the training sample image; Input the training sample image into a preset network model for classification prediction to obtain a label classification prediction result; Input the label classification prediction result and the real label data of the training sample image into a preset loss function to obtain a total loss value of the current label classification; Back propagation is performed according to the total loss value to update the model parameters of the network model until the loss function converges to obtain an image multi-label classification model.

5. An image classification and recognition method, characterized in that: include: Obtain an image to be recognized; The target object in the image to be identified is classified and identified according to a preset image multi-label classification model to obtain a recognition result; wherein the image multi-label classification model is trained using the training method of the image multi-label classification model as described in any one of claims 1 to 4.

6. A training device for an image multi-label classification model, characterized in that: include: A labeling module is used to label the target object in each original sample image in the original sample image set according to a preset label set to obtain a first sample image set; wherein each first sample image in the first sample image set is associated with at least one positive label, and the positive label indicates that the target object corresponding to the label in the label set exists in the first sample image; A position information configuration module, configured to configure position information for each positive label associated with each first sample image according to the bounding box information of the target object corresponding to the positive label in the first sample image; A sample image processing module is used to locate the target object corresponding to each positive label in each first sample image according to the position information of the positive label, remove the located target object from the first sample image, and update the corresponding positive label to a negative label to obtain a second sample image set; A model training module is used to train a preset network model according to the first sample image set and the second sample image set to obtain an image multi-label classification model so as to classify and identify images according to the image multi-label classification model.

7. An image classification and recognition device, characterized in that: include: An image acquisition module to be identified, used to acquire an image to be identified; The image classification module to be identified is used to classify and identify the target object in the image to be identified according to a preset image multi-label classification model to obtain a recognition result; wherein the image multi-label classification model is trained using the training method of the image multi-label classification model described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that: It includes a stored computer program, which, when running, controls the device where the computer-readable storage medium is located to execute the training method of the image multi-label classification model as described in any one of claims 1 to 4, or executes the image classification and recognition method as described in claim 5.

9. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the training method of the image multi-label classification model as described in any one of claims 1 to 4, or implements the image classification and recognition method as described in claim 5.

10. A terminal device, characterized in that: The invention comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When executing the computer program, the processor implements the training method of the image multi-label classification model as described in any one of claims 1 to 4, or implements the image classification and recognition method as described in claim 5.

Images