Image classification method, apparatus and device

Abstract

This application proposes an image classification method, apparatus, and device. The method includes: identifying whether an image to be processed is a beautified image using a pre-trained face detection model and a beautification recognition model; classifying the image to be processed using a pre-trained multi-class classification model to obtain the probability of the image belonging to each image category; and identifying whether the image to be processed contains an image of a target person using a pre-trained person classification model. This application integrates a beautification recognition model for identifying beautified images, a multi-class classification model for classifying multiple image categories, a person classification model for identifying whether an image contains a target person, and so on. By using these multiple models to automatically classify the image to be processed, the method offers strong real-time performance, enabling automatic classification of a large number of images and improving the efficiency of image data structured processing.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image classification method, apparatus, and device. Background Technology

[0002] Currently, a massive number of images are generated online every day. Based on this vast amount of images, services such as text-based image search and image-by-image search can be provided to users. Before providing these services, it is necessary to classify the images and structure the image data.

[0003] Related technologies typically rely on manual classification of user-uploaded images, which is time-consuming and labor-intensive. Alternatively, these technologies may explicitly define a one-to-one correspondence between uploaded images and relevant category tags to guide users in uploading images according to the tags. However, in reality, many users do not upload images according to the regulations, resulting in very low accuracy in image structuring. Summary of the Invention

[0004] This application proposes an image classification method, apparatus, and device, integrating a beauty recognition model for identifying beautified images, a multi-class classification model for classifying various image categories, and a person classification model for identifying whether an image contains a target person. These multiple models automatically classify the images to be processed, offering strong real-time performance and enabling automatic classification of large numbers of images, thus improving the efficiency of image data structure processing.

[0005] The first aspect of this application proposes an image classification method, including:

[0006] The system uses a pre-trained face detection model and a beautification model to identify whether the image to be processed is a beautified image.

[0007] The image to be processed is classified by a pre-trained multi-class classification model to obtain the probability of the image to be processed belonging to each image category.

[0008] The image to be processed is identified by a pre-trained person classification model to determine whether it contains the image of the target person.

[0009] In some embodiments of this application, the step of identifying whether an image to be processed is a beautified image using a pre-trained face detection model and a beautification recognition model includes:

[0010] The pre-trained face detection model determines whether there are face regions in the image to be processed that meet the preset face classification conditions.

[0011] If so, the image of a face region that meets the preset face classification conditions is identified by a pre-trained beautification recognition model to determine whether it is a beautified image.

[0012] In some embodiments of this application, determining whether a face region satisfying preset face classification conditions exists in the image to be processed using a pre-trained face detection model includes:

[0013] The pre-trained face detection model is used to detect whether the image to be processed contains at least one face region.

[0014] If so, determine whether the at least one face region contains a target face region whose face area ratio is greater than a preset threshold;

[0015] If the target face region exists, then it is determined that there is a face region in the image to be processed that meets the preset face classification conditions.

[0016] In some embodiments of this application, determining whether the at least one face region contains a target face region whose face area ratio is greater than a preset threshold includes:

[0017] Detect the facial landmarks corresponding to each facial region;

[0018] Based on the facial key points corresponding to each facial region, the area of ​​each facial region is determined.

[0019] Based on the area of ​​each face region and the area of ​​the image to be processed, calculate the proportion of the face area corresponding to each face region;

[0020] Based on the proportion of face area corresponding to each face region, determine whether the at least one face region contains a target face region whose face area proportion is greater than a preset threshold.

[0021] In some embodiments of this application, the step of identifying whether an image of a face region that meets the preset face classification conditions is a beautified image using a pre-trained beautification recognition model includes:

[0022] Extract the face regions that meet the preset face classification conditions from the image to be processed to obtain the corresponding face image;

[0023] Generate a face-aligned image corresponding to the face image;

[0024] The face alignment image is identified as a beautified image by a pre-trained beautification recognition model.

[0025] In some embodiments of this application, generating the face-aligned image corresponding to the face image includes:

[0026] The face image is scaled up to a first image of a preset size;

[0027] Obtain multiple facial key points from the face image;

[0028] Based on preset standard key points and multiple facial key points of the face image, an alignment operation is performed on the face image to obtain a second image;

[0029] The first image and the second image are merged based on color channels to obtain the face-aligned image corresponding to the face image.

[0030] In some embodiments of this application, before identifying whether the image to be processed is a beautified image using a pre-trained face detection model and beautification recognition model, the method further includes:

[0031] Construct the structure of a beauty recognition model for recognizing beautified images;

[0032] Obtain the first training set;

[0033] The beauty recognition model is trained based on the first training set.

[0034] In some embodiments of this application, the structure for constructing a beauty recognition model for recognizing beautified images includes:

[0035] A preset number of convolutional modules are connected sequentially, where the preset number is an integer greater than or equal to 3;

[0036] The last convolutional module among the predetermined number of sequentially connected convolutional modules is connected to a predetermined high-efficiency network to obtain the structure of a beauty recognition model for recognizing beautified images.

[0037] In some embodiments of this application, obtaining the first training set includes:

[0038] Acquire multiple beautified facial images and multiple unbeautified facial images;

[0039] Generate a face alignment image corresponding to each acquired face image;

[0040] The first training set is obtained by labeling each face-aligned image with a classification label.

[0041] In some embodiments of this application, the step of labeling each face-aligned image with a classification tag includes:

[0042] Label each beautified face-aligned image with a beautification tag, and label each unbeautified face-aligned image with an unbeautified tag; and / or,

[0043] In the beautified face alignment image, beautification labels are marked on the beautified face parts.

[0044] In some embodiments of this application, training the beauty recognition model based on the first training set includes:

[0045] Obtain face-aligned images from the first training set;

[0046] The acquired face-aligned image is input into the beauty recognition model to obtain the classification result of the face image corresponding to the face-aligned image;

[0047] Based on the classification results of the face image, the loss value for the current training period is calculated using a preset classification-center loss function.

[0048] In some embodiments of this application, the classification result includes the probability that a face image belongs to each image category, and each image category includes at least beautified images;

[0049] The step of calculating the loss value for the current training period based on the classification result of the face image using a preset classification-center loss function includes:

[0050] Based on the number of image categories and the probability that a face image belongs to each image category, the first classification loss value corresponding to the face image is calculated using the first classification loss function included in the preset classification-center loss function;

[0051] Based on the feature vectors and feature center points of each image category corresponding to the face image, the center loss value corresponding to the face image is calculated using the center loss function included in the classification-center loss function;

[0052] Calculate the loss value for the current training period based on the first classification loss value and the center loss value.

[0053] In some embodiments of this application, the classification result also includes the probability that an image of at least one facial region in a face image belongs to each image category;

[0054] The step of calculating the loss value for the current training period based on the classification result of the face image using a preset classification-center loss function includes:

[0055] Based on the number of the at least one face part, the number of image categories, and the probability that each face part belongs to each image category, the second classification loss value corresponding to the face image is calculated using the second classification loss function included in the classification-center loss function;

[0056] The loss value for the current training period is calculated based on the first classification loss value, the second classification loss value, and the center loss value.

[0057] In some embodiments of this application, the classification result includes the probability that an image of at least one facial feature in a face image belongs to each image category;

[0058] The step of calculating the loss value for the current training period based on the classification result of the face image using a preset classification-center loss function includes:

[0059] Based on the number of the at least one face part, the number of image categories, and the probability that each face part belongs to each image category, the second classification loss value corresponding to the face image is calculated using the second classification loss function included in the classification-center loss function;

[0060] Based on the feature vectors and feature center points of each image category corresponding to the face image, the center loss value corresponding to the face image is calculated using the center loss function included in the classification-center loss function;

[0061] The loss value for the current training period is calculated based on the second classification loss value and the center loss value.

[0062] In some embodiments of this application, before classifying the image to be processed using a pre-trained multi-class classification model, the method further includes:

[0063] Obtain a second training set, which includes multiple images, each labeled with a category label corresponding to its respective category.

[0064] Construct the structure of the first neural network model for multi-label classification;

[0065] The first neural network model is trained using the second training set to obtain a multi-class classification model.

[0066] In some embodiments of this application, the structure for constructing the first neural network model for multi-label classification includes:

[0067] By connecting the output of the pre-defined high-efficiency network to the input of the bidirectional long short-term memory network, a first neural network model for multi-label classification is obtained.

[0068] In some embodiments of this application, training the constructed first neural network model based on the second training set to obtain a multi-class classification model includes:

[0069] Multiple images are obtained from the second training set;

[0070] The acquired images are input into the first neural network model to obtain the classification results for each image.

[0071] Based on the classification result of each image, the loss value for the current training cycle is calculated using the cross-entropy loss function and the smoothing loss function.

[0072] In some embodiments of this application, after training the constructed first neural network model based on the second training set to obtain a multi-class classification model, the method further includes:

[0073] The accuracy of the multi-class classification model in classifying each image category is determined respectively;

[0074] If there is a first image category with an accuracy lower than a preset threshold, then train the branch model corresponding to the first image category;

[0075] The multi-class classification model is modified using the branching model to obtain the modified multi-class classification model.

[0076] In some embodiments of this application, training the branch model corresponding to the first image category includes:

[0077] Obtain a third training set corresponding to the first image category. The third training set includes multiple images, each of which is labeled with a category label corresponding to the first image category.

[0078] A structure is constructed for a second neural network model to classify the first image category separately, the second neural network model including a branch model corresponding to the first image category;

[0079] The second neural network model is trained based on the third training set to obtain the trained branch model.

[0080] In some embodiments of this application, the structure for constructing a second neural network model for separately classifying the first image category includes:

[0081] By sequentially connecting a predetermined number of fully connected layers, a branch model corresponding to the first image category is obtained;

[0082] Connect the output of the preset efficient network in the multi-class classification model to the input of the branch model to obtain a second neural network model for separately classifying the first image category.

[0083] In some embodiments of this application, the step of modifying the multi-class classification model using the branching model to obtain a modified multi-class classification model includes:

[0084] Connect the input of the trained branch model to the output of the preset high-efficiency network in the multi-class classification model;

[0085] The output of the branch model is connected to the output of the bidirectional long short-term memory network in the multi-class classification model through the fusion module to obtain the modified multi-class classification model.

[0086] The fusion module is used to fuse the first classification result corresponding to the first image category output by the branch model and the second classification result corresponding to the first image category output by the bidirectional long short-term memory network.

[0087] In some embodiments of this application, the step of classifying the image to be processed using a pre-trained multi-class classification model to obtain the probability that the image to be processed belongs to each image category includes:

[0088] The feature vector of the image to be processed is extracted by the preset high-efficiency network in the modified multi-class classification model.

[0089] The feature vector is classified by the branch model in the modified multi-class classification model to obtain the first probability that the image to be processed belongs to the first image category.

[0090] The feature vector is classified by the bidirectional long short-term memory network in the modified multi-class classification model to obtain the probability that the image to be processed belongs to each image category, including the second probability that the image to be processed belongs to the first image category.

[0091] The first probability and the second probability are fused to obtain the final probability that the image to be processed belongs to the first image category.

[0092] In some embodiments of this application, after obtaining the probability that the image to be processed belongs to each image category, the method further includes:

[0093] The probability that the image to be processed belongs to the second image category is greater than the preset probability threshold corresponding to the second image category, where the second image category is an image category that includes body parts;

[0094] The face detection model is used to detect whether the image to be processed contains a face region that meets the preset face classification conditions.

[0095] If the image to be processed contains a face region that meets the preset face classification conditions, then the beautification recognition model is used to identify whether the image of the face region in the image to be processed is a beautified image;

[0096] If it is determined that the image of the face region in the image to be processed is a beautified image, then it is determined that the image to be processed does not belong to the second image category.

[0097] In some embodiments of this application, after obtaining the probability that the image to be processed belongs to each image category, the method further includes:

[0098] The probability that the image to be processed belongs to a third image category is greater than a preset probability threshold corresponding to the third image category, wherein the third image category includes stitched images;

[0099] The pre-trained recognition model identifies whether the image to be processed is a stitched image containing the target content.

[0100] In some embodiments of this application, identifying whether the image to be processed is a spliced ​​image containing target content using a pre-trained recognition model includes:

[0101] The image to be processed is input into a trained recognition model, and the feature extraction network in the recognition model is used to extract features from the image to obtain a feature map.

[0102] Based on the feature map, the splicing line position in the image to be processed is predicted by the splicing line prediction network in the recognition model;

[0103] Based on the feature map and the splicing line position, the classification network in the recognition model determines whether the image to be processed is a spliced ​​image containing the target content.

[0104] In some embodiments of this application, feature extraction is performed on the image to be processed through a feature extraction network in the recognition model to obtain a feature map, including:

[0105] The image to be processed is then processed by convolutional layers in the feature extraction network of the recognition model and then input into the downsampling residual network of the feature extraction network.

[0106] The image after convolution is downsampled multiple times using the downsampling residual network to obtain a feature map.

[0107] In some embodiments of this application, the step of performing multiple downsampling processes on the convolutionally processed image through the downsampling residual network includes:

[0108] The convolutional image is downsampled by the first downsampled residual block in the downsampled residual network to obtain a first intermediate feature, and the first intermediate feature is input into the second downsampled residual block in the downsampled residual network.

[0109] The first intermediate feature is downsampled by the second downsampled residual block to obtain the second intermediate feature, and the second intermediate feature is input into the third downsampled residual block in the downsampled residual network;

[0110] The second intermediate feature is downsampled by the third downsampled residual block to obtain the third intermediate feature, and the third intermediate feature is input into the fourth downsampled residual block in the downsampled residual network;

[0111] The third intermediate feature is downsampled using the fourth downsampled residual block to obtain a feature map.

[0112] In some embodiments of this application, predicting the position of the stitching line in the image to be processed based on the feature map using the stitching line prediction network in the recognition model includes:

[0113] The splicing line prediction network obtains the first intermediate feature, the second intermediate feature, and the third intermediate feature generated by the feature extraction network during the feature extraction process, and predicts the splicing line position in the image to be processed based on the feature map, the first intermediate feature, the second intermediate feature, and the third intermediate feature.

[0114] In some embodiments of this application, the stitching line prediction network predicts the stitching line position in the image based on the feature map, the first intermediate feature, the second intermediate feature, and the third intermediate feature, including:

[0115] The first feature fusion module in the splicing line prediction network performs feature fusion on the feature map, the first intermediate feature, the second intermediate feature, and the third intermediate feature, and outputs the result.

[0116] The horizontal stitching line prediction module uses the fused features output by the first feature fusion module to predict the position of the horizontal stitching line in the image to be processed.

[0117] The vertical stitching line prediction module uses the fused features output by the first feature fusion module to predict the position of the vertical stitching line in the image to be processed.

[0118] In some embodiments of this application, feature fusion is performed on the feature map, the first intermediate feature, the second intermediate feature, and the third intermediate feature by the first feature fusion module, including:

[0119] The feature map is upsampled by the first upsampling layer in the first feature fusion module and then input into the first fusion module in the first feature fusion module.

[0120] The first fusion module fuses the upsampled feature map with the third intermediate feature and then performs convolution processing to obtain the first fused feature, which is then input into the second upsampled layer in the first feature fusion module.

[0121] The first fused feature is upsampled by the second upsampling layer and then input into the second fusion module in the first feature fusion module.

[0122] The second fusion module fuses the upsampled first fusion feature with the second intermediate feature and then performs convolution processing to obtain the second fusion feature, which is then input into the third upsampling layer in the first feature fusion module.

[0123] The second fused feature is upsampled by the third upsampling layer and then input into the third fusion module in the first feature fusion module.

[0124] The third fusion module fuses the upsampled second fusion feature with the first intermediate feature and then performs convolution processing to obtain the fused feature.

[0125] In some embodiments of this application, the horizontal stitching line prediction module uses the fused features output by the first feature fusion module to predict the position of the horizontal stitching line in the image to be processed, including:

[0126] The first separation module in the horizontal splicing line prediction module separates the fused features by column, and then fuses the multiple separated column features according to the channel dimension to obtain a single column feature, which is then input into the first processing module in the horizontal splicing line prediction module.

[0127] After the first processing module performs multiple channel dimensionality reduction processes on a column feature, the horizontal splicing line feature is obtained and input into the first output layer of the horizontal splicing line prediction module.

[0128] The first output layer determines the position of the horizontal stitching line in the image based on the horizontal stitching line feature.

[0129] In some embodiments of this application, determining the position of the horizontal stitching line in the image based on the horizontal stitching line feature through the first output layer includes:

[0130] Using a first window with a first preset interval, traverse the horizontal splicing line feature starting from the first feature value with a preset step size;

[0131] Each time the first window moves, the first position of the horizontal splicing line in the horizontal splicing line feature is extracted and recorded based on the feature value contained in the first window;

[0132] When the traversal is complete, for each recorded first position, the position of the horizontal stitching line in the image is determined based on the size of the image and the first position.

[0133] In some embodiments of this application, the step of extracting and recording the first position of the horizontal splicing line in the horizontal splicing line feature based on the feature values ​​contained in the first window includes:

[0134] Obtain feature values ​​greater than a threshold within the first window, and select the largest feature value from the obtained feature values;

[0135] Determine if there is a recorded first position;

[0136] If not, the position of the maximum feature value is recorded as the first position of a horizontal splicing line in the horizontal splicing line feature;

[0137] If so, the interval between the position of the maximum feature value and the first position of the most recent record is compared with the first preset interval;

[0138] If the value is greater than the first preset interval, the position of the maximum feature value is recorded as the first position of another horizontal splicing line in the feature of the horizontal splicing line.

[0139] If it is less than the first preset interval, then the first position of the most recent record is updated according to the position of the maximum feature value.

[0140] In some embodiments of this application, updating the first position of the most recently recorded record based on the position of the maximum eigenvalue includes:

[0141] If the maximum eigenvalue is greater than the eigenvalue corresponding to the first position of the most recent record, then the first position of the most recent record is updated using the position of the maximum eigenvalue.

[0142] In some embodiments of this application, the vertical stitching line prediction module uses the fused features output by the first feature fusion module to predict the position of the vertical stitching line in the image, including:

[0143] The second separation module in the vertical splicing line prediction module separates the fused features by row, and then fuses the multiple separated row features according to the channel dimension to obtain a single row feature, which is then input into the second processing module in the vertical splicing line prediction module.

[0144] After performing multiple channel dimensionality reduction processes on a row feature through the second processing module, the vertical splicing line feature is obtained and input into the second output layer of the vertical splicing line prediction module.

[0145] The second output layer determines the position of the vertical stitching line in the image based on the vertical stitching line feature.

[0146] In some embodiments of this application, determining the position of the vertical stitching line in the image based on the vertical stitching line feature through the second output layer includes:

[0147] Using a second window with a second preset interval, traverse the longitudinal splicing line feature starting from the first feature value with a preset step size;

[0148] Each time the second window moves, the second position of the vertical splicing line in the vertical splicing line feature is extracted and recorded based on the feature value contained in the second window;

[0149] When the traversal is complete, for each recorded second position, the position of the vertical stitching line in the image is determined based on the size of the image and the second position.

[0150] In some embodiments of this application, extracting and recording the second position of the longitudinal splicing line in the longitudinal splicing line feature based on the feature values ​​contained in the second window includes:

[0151] Obtain feature values ​​greater than the threshold within the second window, and select the largest feature value from the obtained feature values;

[0152] Determine if there is a recorded second position;

[0153] If not, the position of the maximum feature value is recorded as the second position of a vertical splicing line in the vertical splicing line feature;

[0154] If so, the interval between the position of the maximum feature value and the second position of the most recently recorded value is compared with the second preset interval;

[0155] If it is greater than the second preset interval, the position of the maximum feature value is recorded as the second position of another longitudinal splicing line in the longitudinal splicing line feature;

[0156] If it is less than the second preset interval, then the second position of the most recent record is updated according to the position of the maximum feature value.

[0157] In some embodiments of this application, updating the second position of the most recently recorded data based on the position of the maximum eigenvalue includes:

[0158] If the maximum feature value is greater than the feature value corresponding to the second position of the most recent record, then the second position of the most recent record is updated using the position of the maximum feature value.

[0159] In some embodiments of this application, determining whether the image to be processed is a spliced ​​image containing target content based on the feature map and the splicing line position using a classification network in the recognition model includes:

[0160] The feature map and the splicing line feature are fused by the second feature fusion module in the classification network, and the fused feature is input into the third processing module of the classification network.

[0161] The third processing module performs multiple channel dimensionality reduction processes on the fused features to obtain features with a preset channel dimension, which are then input into the third output layer of the classification network.

[0162] The third output layer determines whether the image is a stitched image containing the target content based on features of a preset channel dimension.

[0163] In some embodiments of this application, the method further includes a training process for the recognition model:

[0164] Obtain a first stitched image containing the target content as a positive sample, and mark the horizontal stitching line position and / or vertical stitching line position in the first stitched image, as well as the first category label corresponding to the third image category;

[0165] Obtain a second stitched image and / or a non-stitched image that does not contain the target content as a negative sample, and mark the horizontal stitching line position and / or the vertical stitching line position in the second stitched image, and mark the second stitched image and the non-stitched image with the second category label corresponding to the third image category;

[0166] The constructed recognition model is trained using the positive and negative samples.

[0167] In some embodiments of this application, after obtaining the probability that the image to be processed belongs to each image category, the method further includes:

[0168] The probability that the image to be processed belongs to the fourth image category is greater than the preset probability threshold corresponding to the fourth image category, wherein the fourth image category is an image category that includes text;

[0169] The text information in the image to be processed is identified using an optical character recognition model;

[0170] The text information is segmented into words to obtain one or more keywords;

[0171] Determine whether the preset dictionary contains at least one of the aforementioned keywords;

[0172] If so, the image to be processed is determined to belong to the fourth image category containing the target content.

[0173] In some embodiments of this application, before identifying whether the image to be processed includes the target person using a pre-trained person classification model, the method further includes:

[0174] Obtain a fourth training set, which includes multiple images labeled with the target person and multiple images that do not contain the target person;

[0175] The character classification model is trained based on the fourth training set.

[0176] An embodiment of the second aspect of this application provides an image classification apparatus, comprising:

[0177] The beauty recognition module is used to identify whether the image to be processed is a beauty-enhanced image by using a pre-trained face detection model and a beauty recognition model.

[0178] The multi-class classification module is used to classify the image to be processed using a pre-trained multi-class classification model, and obtain the probability that the image to be processed belongs to each image category.

[0179] The target person recognition module is used to identify whether the image to be processed contains the target person through a pre-trained person classification model.

[0180] An embodiment of the third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0181] An embodiment of the fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method described in the first aspect above.

[0182] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0183] In this embodiment, a beauty recognition model for identifying beautified images, a multi-class classification model for classifying various image categories, a person classification model for identifying whether an image contains a target person, and so on, are integrated. These multiple models automatically classify the images to be processed, offering strong real-time performance and enabling automatic classification of large numbers of images, thus improving the efficiency of image data structure processing.

[0184] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0185] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0186] In the attached diagram:

[0187] Figure 1 A flowchart of an image classification method provided in an embodiment of this application is shown;

[0188] Figure 2 This illustration shows a process for determining the proportion of a human face area according to an embodiment of this application;

[0189] Figure 3 This illustration shows a structural diagram of a beauty recognition model for recognizing beautified images according to an embodiment of this application;

[0190] Figure 4 This illustration shows a process for generating a face-aligned image according to an embodiment of this application;

[0191] Figure 5 This illustration shows a first schematic diagram of a beauty recognition model provided in an embodiment of this application classifying face-aligned images;

[0192] Figure 6 This illustration shows a second schematic diagram of a beauty recognition model provided in an embodiment of this application classifying face-aligned images;

[0193] Figure 7 A third schematic diagram is shown illustrating how the beauty recognition model provided in an embodiment of this application classifies face-aligned images;

[0194] Figure 8 This invention provides a schematic diagram showing a comparison of feature classification before and after the introduction of a center loss function according to an embodiment of this application.

[0195] Figure 9 A schematic diagram of the structure of a first neural network model provided in an embodiment of this application is shown;

[0196] Figure 10 A schematic diagram of the structure of a second neural network model provided in an embodiment of this application is shown;

[0197] Figure 11 A schematic diagram of the structure of a modified multi-class classification model provided in an embodiment of this application is shown;

[0198] Figure 12A schematic diagram illustrating the image classification effect of a multi-class classification model provided in an embodiment of this application is shown;

[0199] Figure 13 A schematic diagram of the network structure of a recognition model for recognizing spliced ​​images containing target content, provided in an embodiment of this application, is shown.

[0200] Figure 14 This application provides schematic diagrams illustrating different splicing styles according to an embodiment.

[0201] Figure 15 This illustration shows a schematic diagram of the structure of a feature extraction network provided in an embodiment of this application;

[0202] Figure 16 This illustration shows a schematic diagram of the structure of a splicing line prediction network provided in an embodiment of this application;

[0203] Figure 17 This illustration shows a schematic diagram of the structure of a horizontal splicing line prediction module provided in one embodiment of this application;

[0204] Figure 18 This illustration shows a comparison diagram of an input image and an output image marked with splicing line positions and classification results, provided in an embodiment of this application.

[0205] Figure 19 This illustration shows a structural schematic diagram of a longitudinal splicing line prediction module provided in an embodiment of this application;

[0206] Figure 20 This illustration shows a schematic diagram of the structure of a classification network provided in an embodiment of this application;

[0207] Figure 21 A schematic diagram of the overall structure of an identification model provided in an embodiment of this application is shown;

[0208] Figure 22 This illustration shows a schematic diagram of the process of recognizing an image to be processed using an optical character recognition model according to an embodiment of this application;

[0209] Figure 23 This paper illustrates another flowchart of an image classification method provided in one embodiment of the present application;

[0210] Figure 24 This invention provides a schematic diagram of the structure of an image classification device according to an embodiment of the present application.

[0211] Figure 25 This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0212] Figure 26A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0213] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0214] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0215] The following description, in conjunction with the accompanying drawings, describes an image classification method, apparatus, and device according to embodiments of this application.

[0216] Currently, a massive amount of images are generated online every day. Based on this vast image database, services such as text-based image search and image-by-image search can be provided to users. Before providing these services, the images must first be categorized and structured. Related technologies typically rely on manual categorization of user-uploaded images, which is time-consuming and labor-intensive. Alternatively, some technologies explicitly define a one-to-one correspondence between uploaded images and relevant category tags to guide users in uploading images according to the tags. However, in reality, many users do not upload images according to these rules, resulting in low accuracy in image structuring.

[0217] Based on this, embodiments of this application provide an image classification method that integrates multiple neural network models for image classification, such as a face detection model for detecting faces, a beauty recognition model for recognizing beautified images, a multi-class classification model for classifying multiple image categories, a person classification model for identifying whether an image contains a target person, and so on. By using these multiple models to automatically classify the images to be processed, the method offers strong real-time performance, enabling automatic classification of large numbers of images and improving the efficiency of image data structured processing.

[0218] See Figure 1 The method specifically includes the following steps 101-103. The implementation of this application does not limit the execution order of steps 101, 102, and 103. In practical applications, these three steps can be executed simultaneously or in any order.

[0219] Step 101: Identify whether the image to be processed is a beautified image using a pre-trained face detection model and a beautification recognition model.

[0220] A pre-trained face detection model is used to determine whether there are face regions in the image to be processed that meet the preset face classification conditions.

[0221] Specifically, a pre-trained face detection model is used to detect whether the image to be processed contains at least one face region. If so, it is determined whether the at least one face region contains a target face region whose face area occupies a pre-defined threshold. If a target face region exists, it is determined that the image to be processed contains a face region that meets the pre-defined face classification criteria.

[0222] like Figure 2 As shown, the image to be processed, 'a', is input into the face detection model. The face detection model detects whether the image contains a face region and marks the face region with a rectangular bounding box, as shown. Figure 2 As shown in Figure b. Then, a facial landmark detection model is used to detect facial landmarks corresponding to each face region. Based on the facial landmarks corresponding to each face region, the area of ​​each face region is determined. Specifically, based on the facial contour points and eyebrow landmarks among the facial landmarks, the face region is drawn, as shown below. Figure 2 The white face region is shown in image c. Then, based on the area of ​​each face region and the area of ​​the image to be processed, the proportion of the face area corresponding to each face region is calculated. That is, for each face region, the following calculation is performed. Figure 2 The white area shown in Figure c represents the proportion of the entire image's area to the face region.

[0223] The proportion of the face area corresponding to each face region is compared with a preset threshold. If a target face region exists whose face area proportion is greater than the preset threshold, then that target face region is determined to meet the preset face classification conditions. If the proportion of the face area corresponding to all face regions is less than or equal to the preset threshold, then it is determined that there are no face regions in the image to be processed that meet the preset face classification conditions.

[0224] The aforementioned preset threshold can be 3% or 4%, etc. This application embodiment does not limit the value of the preset threshold, and it can be set according to the needs in actual application.

[0225] After determining the face regions in the image to be processed that meet the preset face classification conditions using the above method, the pre-trained beautification recognition model identifies whether the image containing the face regions that meet the preset face classification conditions is a beautified image.

[0226] Images that do not include a complete face region, or images that include a complete face region but whose face area is less than or equal to a preset threshold, will no longer be processed using a beauty recognition model.

[0227] Before processing with the beauty recognition model, the beauty recognition model is trained through steps A1-A3, including:

[0228] A1: Construct the structure of a beauty recognition model for recognizing beautified images.

[0229] A preset number of convolutional modules are sequentially connected. The preset number can be an integer greater than or equal to 3. For example, the preset number can be 3 or 4. This application embodiment does not limit the specific value of the preset number; it can be set according to requirements in practical applications.

[0230] The aforementioned convolutional module may include a convolutional layer (CONV), a batch normalization layer (BN), and an activation layer (RELU) connected in sequence.

[0231] The last convolutional module in a predetermined number of sequentially connected convolutional modules is connected to a predetermined high-efficiency network to obtain the structure of a beauty recognition model for recognizing beautified images. The predetermined high-efficiency network can be any of the eight high-efficiency networks B0 to B7 (EfficientNet-B0 to EfficientNet-B7), or any other neural network capable of image classification.

[0232] like Figure 3 As shown, a schematic example of the structure of this beauty recognition model is illustrated. Figure 3 The diagram only shows four convolutional modules; in actual applications, there may be other numbers of convolutional modules. Figure 3 The middle arrow indicates the data flow direction; the output of the previous convolutional module serves as the input of the next convolutional module, and the output of the last convolutional module serves as the input of the preset high-efficiency network. Figure 3 The default high-efficiency network is EfficientNet-B0, but other networks can also be used in practical applications.

[0233] A2: Obtain the first training set.

[0234] Acquire a large number of face images, each of which may include one or more face regions. For a face image containing multiple face regions, extract images of each face region separately to obtain multiple face images containing only one face region. When extracting face regions, you can extract face images that include the complete face region above the neck and shoulders.

[0235] The aforementioned large number of facial images, each representing only a single facial region, includes multiple beautified facial images and multiple unbeautiful facial images. The beautified facial images include both those obtained through image enhancement applications and those captured by a camera showing a face after actual makeup application. The beautification processes performed by the image enhancement applications include one or more combinations of distortion techniques such as face slimming, eye enlargement, and smile enhancement, as well as non-distortion techniques such as skin smoothing, whitening, sharpening, and filters. The actual makeup application includes one or more combinations of foundation makeup, eye makeup, eyebrow shaping, blush application, and lipstick application.

[0236] After obtaining multiple face images using the above method, a face-aligned image corresponding to each face image is generated. For each face image, it is first scaled to a preset size, such as 224*224 or 226*226. This embodiment does not limit the specific value of the preset size; it can be set according to requirements in practical applications.

[0237] Then, multiple facial landmarks in the face image are detected using a preset facial landmark detection model. The number of detected facial landmarks can be 106 or 117, etc. Based on the preset standard landmarks and the multiple facial landmarks in the face image, an alignment operation is performed on the face image to obtain the second image.

[0238] The preset standard facial landmarks may include preset standard center points for the left eye, right eye, nose tip, left corner of mouth, and right corner of mouth. The center points for the left eye, right eye, nose tip, left corner of mouth, and right corner of mouth from the detected facial landmarks in the face image are aligned one-to-one with the preset standard facial landmarks. For example, the center point of the left eye in the face image is aligned with the standard center point of the left eye, and the center point of the right eye in the face image is aligned with the standard center point of the right eye, and so on.

[0239] Before performing the alignment operation described above, a facial region image can be extracted from the facial image based on multiple facial key points. This facial region image includes only the complete face region, excluding areas such as the shoulders and neck. Then, the corresponding coordinate points of this facial region image are aligned using the methods described above. Alternatively, after performing the alignment operation, the multiple facial key points of the aligned facial image are redefined, and a facial region image including only the complete face region is extracted from the aligned facial image based on these aligned key points.

[0240] After obtaining an image containing only the complete face region through the above operations, this face region image is scaled to the preset size to obtain a second image. This second image is equivalent to enlarging the facial region in the original face image. The first and second images are then merged based on color channels to obtain a face-aligned image corresponding to the original face image. The process of generating the face-aligned image is as follows: Figure 4 As shown.

[0241] Assuming the preset size is 224*224, the first and second images are both 224*224*3, and the face-aligned image obtained by merging based on color channels is 224*224*6.

[0242] After generating face-aligned images for each face image using the methods described above, each face-aligned image is labeled with a tag indicating its image category, which can include beautified images, unedited images, etc. Specifically, each beautified face-aligned image is labeled with a "beautified" tag, and each unedited face-aligned image is labeled with a "unedited" tag.

[0243] In other embodiments of this application, beautification tags can also be added to the beautified facial areas in the beautified face-aligned image. For example, if foundation makeup is applied in the beautified face-aligned image, a beautification tag corresponding to the foundation makeup on the facial skin is added. If eye makeup is applied, a beautification tag corresponding to the eye makeup is added. If eyeliner is applied, a beautification tag corresponding to the eyeliner is added. If eyebrows are applied, a beautification tag corresponding to the eyebrows is added. If blush is applied, a beautification tag corresponding to the cheek area is added. If lipstick is applied, a beautification tag corresponding to the lips is added.

[0244] The first training set is composed of each face-aligned image labeled above. Each image in the first training set may contain only one label indicating whether the image is beautified. Alternatively, each image may include at least one label indicating whether the facial features are beautified. Alternatively, each image may include both a label indicating whether the image is beautified and at least one label indicating whether the facial features are beautified.

[0245] A3: Based on the first training set mentioned above, train the constructed beauty recognition model.

[0246] Face-aligned images are acquired from the first training set. The number of face-aligned images acquired from the first training set in each training cycle can be multiple. The acquired face-aligned images are input into the constructed beauty recognition model. A predetermined number of convolutional modules connected sequentially in the beauty recognition model are used to extract features from the face-aligned images. The finally extracted feature images are input into a predetermined high-efficiency network.

[0247] For example, refer to Figure 5-6 The first convolutional module takes a 224*224*6 face-aligned image and inputs it into the first convolutional module. This module expands the image's channels to 64, outputting a 224*224*64 feature image. The second convolutional module expands the image's channels to 128, outputting a 224*224*128 feature image. The third convolutional module reduces the image's channels to 3, outputting a 224*224*3 feature image. Finally, the fourth convolutional module maintains the same number of channels and outputs a 224*224*3 feature image to the EfficientNet-B0 network.

[0248] A pre-defined high-efficiency network classifies the input feature image to obtain the classification result of the face image corresponding to the face alignment image. Then, based on the classification result of the face image, the loss value of the current training cycle is calculated using a pre-defined classification-center loss function. The classification result includes the probability that the face image belongs to each image category, and each image category includes at least beautified images.

[0249] In one implementation, the face-aligned image is labeled only with a tag indicating whether it is a beautified image. Correspondingly, the classification result only includes the probability that the face image belongs to a beautified image and the probability that it belongs to a plain image.

[0250] For example, refer to Figure 5 The EfficientNet-B0 network classifies the face image based on this feature image and outputs that the probability of the face image corresponding to the face alignment image is 0.98 for a beautified image and 0.02 for a plain image.

[0251] The loss value for the current period is calculated based on the classification results. Specifically, based on the number of image categories and the probability that the face image belongs to each image category, the first classification loss value corresponding to the face image is calculated using the first classification loss function shown in formula (1), which is part of the preset classification-center loss function. Based on the feature vectors and feature center points of each image category corresponding to the face image, the center loss value corresponding to the face image is calculated using the center loss function shown in formula (2), which is part of the classification-center loss function. Based on the first classification loss value and the center loss value, the loss value for the current training period is calculated using the classification-center loss function shown in formula (3).

[0252]

[0253]

[0254] L=Lsoftmax1+λ1Lcenterloss…(3)

[0255] In formulas (1), (2), and (3) above, L is the loss value of the current training cycle, Lsoftmax1 is the first classification loss value, Lcenterloss is the center loss value, and λ1 is the balance factor controlling the two loss functions. m is the number of image categories, p i x represents the probability that a face image belongs to the i-th image category. i Let c be the feature vector of the i-th image category corresponding to the face image. yi It is the feature center point of the i-th image category corresponding to the face image.

[0256] Where, x i and c yi Both are high-dimensional feature vectors. The L2 distance between them is taken as the optimization index of the center loss. The smaller the center loss value Lcenterloss, the closer the feature is to the center point, and the better the training effect.

[0257] In another implementation, the face-aligned image is labeled with only at least one tag indicating whether a facial feature has been beautified. Correspondingly, the classification result only includes the probability that an image with at least one facial feature belongs to each image category.

[0258] For example, refer to Figure 6 The EfficientNet-B0 network classifies based on this feature image and outputs the probability of beautifying multiple facial features, such as... Figure 6 The probability of applying foundation is 0.04, the probability of applying eye makeup is 0.12, the probability of applying eyeliner is 0.04, the probability of applying eyebrows is 0.02, the probability of applying blush is 0.67, and the probability of applying lipstick is 0.08.

[0259] The loss value for the current period is calculated based on the classification results. Specifically, based on the number of at least one face part, the number of image categories, and the probability that each face part belongs to each image category, the second classification loss value corresponding to the face image is calculated using the second classification loss function shown in formula (4), which includes the classification-center loss function. Based on the feature vectors and feature center points of each image category corresponding to the face image, the center loss value corresponding to the face image is calculated using the center loss function shown in formula (2). Based on the second classification loss value and the center loss value, the loss value for the current training period is calculated using the classification-center loss function shown in formula (5).

[0260]

[0261] L=Lsoftmax2+λ2Lcenterloss…(5)

[0262] In formulas (4) and (5) above, L is the loss value of the current training cycle, Lsoftmax2 is the second classification loss value, Lcenterloss is the center loss value, and λ2 is the balance factor controlling the two loss functions. m is the number of image categories, n is the number of at least one face region mentioned above, and p ij Let be the probability that the image of the j-th face region in a face image belongs to the i-th image category.

[0263] In another implementation, the face-aligned image is labeled with a tag indicating whether the image is beautified and at least one tag indicating whether a facial feature is beautified. Correspondingly, the classification result includes the probability that the face image belongs to a beautified image, the probability that it belongs to a plain image, and the probability that at least one facial feature in the face image belongs to each image category.

[0264] For example, refer to Figure 7 The EfficientNet-B0 network classifies the face image based on this feature image, outputting a probability of 0.98 for a beautified image and a probability of 0.02 for a natural face image. It also outputs the probabilities of beautifying multiple facial features, such as... Figure 7 The probability of applying foundation is 0.04, the probability of applying eye makeup is 0.12, the probability of applying eyeliner is 0.04, the probability of applying eyebrows is 0.02, the probability of applying blush is 0.67, and the probability of applying lipstick is 0.08.

[0265] The loss value for the current period is calculated based on the classification results. Specifically, the first classification loss value is calculated using formula (1) above, the center loss value is calculated using formula (2) above, and the second classification loss value is calculated using formula (4) above. Then, based on the first classification loss value, the second classification loss value, and the center loss value, the loss value for the current training period is calculated using the classification-center loss function shown in formula (6).

[0266] L=Lsoftmax1+λ3Lcenterloss+Lsoftmax2…(6)

[0267] In formula (6), L is the loss value of the current training cycle, Lsoftmax1 is the first classification loss value, Lsoftmax2 is the second classification loss value, Lcenterloss is the center loss value, and λ3 is the balance factor that controls the two loss functions.

[0268] In any of the above implementations, after calculating the loss value of the current training period, it is determined whether the current number of training periods has reached the preset number of training periods. If so, training is stopped, and the model parameters of the training period with the smallest loss value are obtained from the trained periods. The structure of the beauty recognition model constructed above and the model parameters corresponding to the training period with the smallest loss value are determined as the trained beauty recognition model. If the current number of training periods has not reached the preset number of training periods, the learning and training process continues in the above manner until a trained beauty recognition model is obtained.

[0269] In this embodiment, considering that classifying beautified and bare-faced images is a relatively difficult metric learning problem, a center loss function is introduced for constraint learning during the training of the beautification recognition model. The center loss function can pull all points of the same category towards the center point of that category. Ultimately, this ensures that the features of all beautified images are relatively close together, and the features of all bare-faced images are also relatively close together. This results in the features of beautified and bare-faced images being relatively far apart, achieving a better classification effect for beautified and bare-faced images.

[0270] To validate the classification results, the final features are dimensionality reduced and output to a two-dimensional plane for visualization. For example... Figure 8 As shown in Figure a, Figure a presents the result of training using only the classification loss function softmax-crossentrop, followed by 2D dimensionality reduction and visualization of the output features. Figure b presents the result of using both classification loss softmax-crossentrop and center loss, followed by 2D dimensionality reduction and visualization of the output features. The comparison in the figures clearly shows that using center loss effectively separates the beauty and natural beauty features, significantly reducing the mixing of the two features.

[0271] After training the beauty recognition model using the above method, the model is used to identify face regions in the image to be processed that meet the preset face classification conditions. Specifically, face regions that meet the preset face classification conditions are extracted from the image to be processed to obtain the corresponding face images. The extracted face images can include the entire face region above the shoulders and neck. A face-aligned image corresponding to the face image is then generated. The specific generation process is the same as the operation of generating the face-aligned image during step A2 of the above-mentioned process of obtaining the training set, and will not be repeated here.

[0272] Then, the trained beautification recognition model identifies whether the face-aligned image is a beautified image. For each face image extracted from the image to be processed, a corresponding face-aligned image is generated. Each face-aligned image is then input into the beautification recognition model trained through steps A1-A3 above. The beautification recognition model extracts features from each input face-aligned image and classifies them based on the extracted features. It outputs the probability that the face image corresponding to each face-aligned image belongs to each image category, including beautified images.

[0273] In one implementation, the trained beauty recognition model only outputs the probability that a face image belongs to each image category, including beautified images. If the probability of belonging to a beautified image is greater than 0.5, the face image is considered beautified; if the probability is less than or equal to 0.5, the face image is not beautified. For example, assuming the image categories only include beautified and plain images, and the probability of belonging to a beautified image is 0.98 while the probability of belonging to a plain image is 0.02, then the face image is determined to be a beautified image.

[0274] In another implementation, the trained beautification recognition model only outputs the probability of beautifying at least one facial feature in a face image. If the sum of the probabilities of beautifying at least one facial feature is greater than 0.5, or if there is a probability greater than 0.5 among the probabilities of beautifying at least one facial feature, then the face image is considered beautified. For example, suppose at least one feature includes facial skin, eyes, eyebrows, cheeks, and lips. Suppose the beautification recognition model outputs a probability of 0.04 for applying foundation to facial skin, 0.12 for applying eye makeup to the eyes, 0.04 for applying eyeliner to the eyes, 0.02 for applying eyebrows to the eyebrows, 0.67 for applying blush to the cheeks, and 0.08 for applying lipstick to the lips. The sum of these probabilities is 0.97, or there is a probability greater than 0.5 (0.67), therefore the face image is determined to be beautified.

[0275] In another implementation, the trained beautification recognition model outputs the probability that a face image belongs to each image category and the probability that at least one facial feature has been beautified. Each image category includes beautified images. If the probability of belonging to a beautified image is greater than 0.5, the face image is considered beautified; if the probability is less than or equal to 0.5, the face image is not beautified. Furthermore, the probability of at least one facial feature being beautified clearly identifies which facial feature in the face image has undergone beautification processing.

[0276] Each face image in the image to be processed is identified individually using the above method to determine whether it is a beautified image. The recognition results for each face image can then be labeled on the image to be processed, allowing users to visually determine whether each face image in the image to be processed is a beautified image after the image is displayed.

[0277] This application's embodiments can also be applied to scenarios where the effects of cosmetic surgery are evaluated. When users view evaluations of cosmetic surgery procedures, they typically prefer to see natural, unedited facial images in the reviews to more accurately assess the results. Therefore, upon receiving user evaluations of cosmetic surgery procedures, if the evaluation includes a facial image, the method provided in this application's embodiments can first detect whether the facial image is filtered or not. If it is filtered, the user can be prompted to provide a natural, unedited image. Alternatively, evaluations containing filtered images can be placed at the end of the cosmetic surgery procedure's evaluation list, allowing other users to see evaluations containing natural, unedited images first.

[0278] In this embodiment, a beautification recognition model is used to classify the face-aligned image corresponding to the face image, obtaining the probability that the face image belongs to each image category, including beautified images. This accurately determines whether a face image is beautified. The face-aligned image includes global and local features of the face region, and processing the face-aligned image yields more accurate classification results. Furthermore, the beautification recognition model is trained using a center loss function, enabling the model to learn more discriminative features, allowing it to further separate the image categories and classify them more accurately. This beautification recognition model can also identify whether facial features are beautified, is simple and efficient, and can simultaneously output the probability that a face image belongs to each image category and the probability that facial features are beautified. For images to be processed, including beautified and unbeautified face images, it can identify whether each face image is beautified, with high accuracy, avoiding ambiguity in the classification of images containing multiple faces.

[0279] Step 102: Classify the image to be processed using a pre-trained multi-class classification model to obtain the probability of the image to be processed belonging to each image category.

[0280] Before classifying the image to be processed using a pre-trained multi-class classification model, the multi-class classification model is first trained through steps B1-B3, including:

[0281] B1: Obtain the second training set, which includes multiple images, each labeled with the category label corresponding to each image category.

[0282] First, determine all image categories that need to be classified. Image categories can be determined based on business needs. For example, in fields such as medical and cosmetic surgery, image categories may include surgical sites, puzzles, surgical procedures, instruments and medicines, indoor environments, outdoor environments, people, or others. Other categories may include other types of images uploaded by users, such as food photos, landscape photos, cartoon images, emoticons, advertising images, game screenshots, icons, WeChat chat screenshots, and any other non-medical / surgical related images. This application does not limit the specific number of image categories or the method of classification; in practical applications, image categories can be determined based on business needs.

[0283] A large number of images are acquired. For each acquired image, a category label is assigned to determine the image category. The category label can be a binary label. For example, an image showing a surgical site would have a category label of 1 for the surgical site, while the category labels for other image categories such as jigsaw puzzles, surgical procedures, instruments and medications, indoor environments, outdoor environments, people, or others would all be 0.

[0284] A second training set is formed by combining multiple images labeled with category tags corresponding to each image category.

[0285] B2: The structure for constructing the first neural network model used for multi-label classification.

[0286] By connecting the output of the pre-defined high-efficiency network to the input of a Bi-directional Long Short-Term Memory (BiLSTM) network, a first neural network model for multi-label classification is obtained.

[0287] The preset high-efficiency network can be any one of the eight high-efficiency networks from B0 to B7, namely EfficientNet-B0 to EfficientNet-B7, or it can be any other neural network capable of image classification.

[0288] like Figure 9 As shown, taking the EfficientNet-B0 network as a preset high-efficiency network as an example, the structure of the first neural network model is illustrated. Figure 9The arrows indicate the direction of data transmission. The EfficientNet-B0 network consists of a first convolutional layer (conv), seven MBconv layers, and a second convolutional layer (conv) connected sequentially. The first convolutional layer (conv) has a 3x3 kernel size, the second convolutional layer (conv) has a 1x1 kernel size, and the six MBconv layers have kernel sizes of 3x3, 3x3, 5x5, 3x3, 5x5, 5x5, and 3x3, respectively. The output of the second convolutional layer (conv) is connected to the input of the bidirectional long short-term memory network. The second convolutional layer (conv) can also be replaced with a fully connected layer.

[0289] The embodiments of this application do not limit the order in which steps B1 and B2 are executed. Step B1 can be executed first, followed by step B2. Alternatively, step B2 can be executed first, followed by step B1. Or, steps B1 and B2 can be executed simultaneously.

[0290] B3: Train the constructed first neural network model based on the second training set to obtain a multi-class classification model.

[0291] Multiple images are acquired from the second training set. The number of images acquired can be the number of images that the first neural network model can process in parallel; this number can be called the batch size. Each acquired image is input into the constructed first neural network model. In this embodiment, each image input into the first neural network model is scaled to a preset size, such as 224*224 or 226*226. The image of the preset size is input into the first neural network model, where a high-efficiency network extracts the feature vector of the image. The extracted feature vector is then input into a bidirectional long short-term memory network. The bidirectional long short-term memory network classifies the image based on the feature vector and outputs the classification result, i.e., the probability that the image belongs to each image category.

[0292] like Figure 9As shown, a 224*224 image is input into EfficientNet-B0. The first convolutional layer (conv) processes the image and outputs a 112*112 feature map. This 112*112 feature map is input into the first MBconv layer, which outputs a 112*112 feature map to the second MBconv layer. The second MBconv layer outputs a 56*56 feature map to the third MBconv layer. The third MBconv layer outputs a 28*28 feature map to the fourth MBconv layer. The fourth MBconv layer outputs a 14*14 feature map to the fifth MBconv layer. The fifth MBconv layer outputs a 14*14 feature map to the sixth MBconv layer. The sixth MBconv layer outputs a 7*7 feature map to the seventh MBconv layer, which in turn outputs a 7*7 feature map to the second convolutional layer (conv). The second convolutional layer (conv) outputs a 1*1280 feature vector to the bidirectional long short-term memory network. Assuming the image categories include eight types: surgical site, puzzle, surgical procedure, instruments and medicines, indoor environment, outdoor environment, people, or others, the final bidirectional long short-term memory network outputs a 1*8 classification result, which represents the probability that the image belongs to one of these eight categories: surgical site, puzzle, surgical procedure, instruments and medicines, indoor environment, outdoor environment, people, or others.

[0293] In this embodiment, a bidirectional long short-term memory network is introduced into the first neural network model, enabling the first neural network model to learn the preceding and following relationships and the preceding and following relationships between features of different categories. This allows the model to learn the correlation between features of different categories, enhances the robustness of the model, and improves the accuracy of the model in multi-class classification.

[0294] After obtaining the classification results of each image in the current training period using the above method, the loss value of the current training period is calculated based on the classification results of each image using the cross-entropy loss function and the smoothing loss function.

[0295] The cross-entropy loss function can be sigmoidcrossentrop, and the smoothing loss function can be smoothL1. The cross-entropy loss function is shown in Equation (7), and the smoothing loss function is shown in Equation (8). After calculating the cross-entropy loss value using Equation (7) and the smoothing loss value using Equation (8), the loss value for the current training period is calculated using Equation (9).

[0296]

[0297]

[0298] Loss_all = Loss * 0.5 + smooth L1*0.5…(9)

[0299] In formulas (7), (8), and (9), Loss_all is the loss value for the current training cycle, and LOSS is the cross-entropy loss value. This represents the smoothing loss value. M is the number of image categories, and N is the number of categories with different labels. For example, for binary classification, the number of categories is 2. in This represents the category label corresponding to the i-th image category, such as the label for surgical site, the label for puzzle, etc. x represents the probability of the i-th image category predicted by the first neural network model. i This represents the absolute value of the difference between the predicted probability corresponding to the i-th image category and the value of the category label corresponding to the i-th image category in the image.

[0300] After calculating the loss value for the current training cycle using the above method, determine whether the number of training cycles has reached the preset number. If so, stop training, determine the training cycle with the smallest loss value from the trained cycles, and combine the model parameters trained in the training cycle with the smallest loss value with the structure of the first neural network model to obtain a trained multi-class classification model. If not, continue training using the above method until the preset number of training cycles is reached to obtain a trained multi-class classification model.

[0301] After obtaining the multi-class classification model using the above method, the model can be used to simultaneously classify multiple image categories. Specifically, the image to be processed is acquired, scaled to a preset size, and then input into the trained multi-class classification model. The multi-class classification model classifies the image to be processed and outputs the probability that the image belongs to each image category.

[0302] A multi-class classification model is obtained by training on multi-label images. This model can classify multiple image categories simultaneously using only one model, improving the efficiency of multi-class classification. Furthermore, the bidirectional long short-term memory network included in the trained multi-class classification model allows it to learn the correlations between features of different image categories, resulting in more accurate classification and better robustness.

[0303] A multi-class classification model, which classifies multiple image categories simultaneously, inevitably leads to mutual influence between features of different image categories. Therefore, the classification accuracy for a few image categories may be low. To address this issue, after training the multi-class classification model as described above, the accuracy of the model for each image category can be determined separately.

[0304] Specifically, a test set can be obtained, which includes multiple images belonging to different image categories. The images in the test set are input into a trained multi-class classification model, and the probability of each image belonging to each image category is obtained. The image category with the highest probability is the predicted image category. For each image category, based on the predicted image category and the actual image category of each image, the number of images whose predicted and actual categories match is counted. The ratio between the number of matching images and the total number of images in the test set that actually belong to that image category is calculated, and this ratio is determined as the accuracy corresponding to that image category.

[0305] The accuracy of each image category is compared with a preset threshold. If no image category has an accuracy lower than the preset threshold, it indicates that the trained multi-class classification model has high accuracy for each image category and no adjustment is needed. If a first image category has an accuracy lower than the preset threshold, a branch model corresponding to the first image category is trained. Then, this branch model is used to correct the multi-class classification model, resulting in a corrected multi-class classification model.

[0306] The aforementioned preset threshold can be 70% or 50%, etc. This application embodiment does not limit the specific value of the preset threshold; it can be determined according to requirements in practical applications.

[0307] Specifically, the branch model corresponding to the first image category is trained through the following steps C1-C3:

[0308] C1: Obtain the third training set corresponding to the first image category. The third training set includes multiple images, each labeled with the category label corresponding to the first image category.

[0309] The category labels for all image categories other than the first image category in each image in the second training set can be deleted, leaving only the category label for the first image category, to obtain the third training set.

[0310] Alternatively, a large number of images can be acquired again, and the category label corresponding to the first image category can be labeled in each acquired image to obtain a third training set.

[0311] C2: Construct the structure of a second neural network model for separately classifying the first image category. The second neural network model includes a branch model corresponding to the first image category.

[0312] A predetermined number of fully connected layers are sequentially connected in series to obtain the branch model corresponding to the first image category. The predetermined number can be 3 or 4, etc. The embodiments of this application do not limit the specific value of the predetermined number, and it can be set according to the needs in actual applications.

[0313] Connect the output of the preset efficient network in the multi-class classification model trained above to the input of the branch model above to obtain a second neural network model for separately classifying the first image category.

[0314] C3: Train the second neural network model based on the third training set to obtain the trained branch model.

[0315] Multiple images are acquired from the third training set, the number of which can be the number of images that the second neural network model can process in parallel. Each acquired image is input into the constructed second neural network model. In this embodiment, each image input into the second neural network model is also scaled to a preset size. The image of the preset size is input into the first neural network model, where a preset high-efficiency network extracts the feature vector of the image. The extracted feature vector is then input into the branch model corresponding to the first image category. The branch model classifies the image based on the feature vector and outputs the classification result, i.e., the probability that the image belongs to the first image category.

[0316] like Figure 10 As shown, assuming the branching model consists of three sequentially connected fully connected layers, the output of a pre-defined efficient network, EfficientNet-B0, is connected to the first fully connected layer of the branching model. A 224*224 image is input into EfficientNet-B0, which outputs a 1*1280 feature vector, which is then fed into the first fully connected layer. The first fully connected layer outputs a 1*320 feature vector, which is fed into the second fully connected layer. The second fully connected layer outputs a 1*80 feature vector, which is fed into the third fully connected layer. The third fully connected layer outputs a 1*1 classification result. Assuming the first image belongs to the surgical procedure category, this classification result represents the probability that the image belongs to the surgical procedure category.

[0317] After obtaining the classification results of each image in the current training cycle through the above method, the loss value of the current training cycle is calculated based on the classification results of each image using the cross-entropy loss function of formula (7), the smoothing loss function of formula (8), and the total loss function of formula (9).

[0318] After calculating the loss value of the current training period using the above method, determine whether the number of training periods has reached the preset number of times. If so, stop training, determine the training period with the smallest loss value from the trained periods, and combine the model parameters and structure of the branch model trained in the training period with the smallest loss value to obtain the trained branch model. If not, continue training using the above method until the preset number of training periods is reached to obtain the trained branch model.

[0319] After training the branch model corresponding to the first image category using the above method, this branch model is used to correct the multi-class classification model trained in steps B1-B3. Specifically, the input of the trained branch model is connected to the output of a pre-set efficient network in the multi-class classification model. The output of the branch model is then connected to the output of the bidirectional long short-term memory network in the multi-class classification model via a fusion module, resulting in the corrected multi-class classification model. The fusion module is used to fuse the first classification result corresponding to the first image category output by the branch model with the second classification result corresponding to the first image category output by the bidirectional long short-term memory network.

[0320] like Figure 11 As shown, the input of the first fully connected layer of the branch model is connected to the output of the EfficientNet-B0 network, and the output of the last fully connected layer of the branch model is connected to the fusion module. The output of the bidirectional long short-term memory network is also connected to the fusion module. The data output by the fusion module is the final probability corresponding to the first image category. Figure 11 Taking eight image categories—surgical site, puzzle, surgical procedure, instruments and drugs, indoor environment, outdoor environment, people, and others—as an example, a branching model for the surgical procedure was trained. The probability output by the branching model for the surgical procedure is fused with the probability of the surgical procedure category output by the bidirectional long short-term memory network to obtain the final probability corresponding to the surgical procedure category.

[0321] The fusion module can perform a weighted fusion operation to weight and fuse the first probability corresponding to the first image category output by the branch model with the second probability corresponding to the first image category output by the bidirectional long short-term memory network. Specifically, the weighted fusion can be performed using the following formula (10).

[0322] feature=feature1*p+feature2*(1-p)…(10)

[0323] Where feature is the final probability corresponding to the first image category after fusion, feature1 is the first probability corresponding to the first image category output by the branch model, feature2 is the second probability corresponding to the first image category output by the bidirectional long short-term memory network, and P is the fusion factor.

[0324] To determine the fusion factor P, in this embodiment, 11 numbers—0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, and 1.0—can be substituted into formula (10) above to record the final probability of the predicted first image category when P takes different values. In the test set, the final probability of the predicted first image is compared with the actual situation, and the accuracy of the prediction is calculated for each value of P. The value of P with the highest accuracy is then selected. This value can be 0.7 or 0.8, etc.

[0325] In this embodiment, the number of first image categories that need to be corrected by the branch model can be one or more. For cases where there are multiple first image categories, a branch model corresponding to each first image category can be trained separately as described above. Then, the branch model corresponding to each first image category is connected in parallel to the multi-class classification model trained in steps B1-B3. The input of each branch model corresponding to a first image category is connected to the output of a preset high-efficiency network, and the output of each branch model corresponding to a first image category is connected to the output of a bidirectional long short-term memory network through a fusion module.

[0326] After obtaining the modified multi-class classification model through the above method, the modified model can be used to classify images into multiple image categories. Specifically, the image to be processed is obtained, and the modified multi-class classification model is used to classify the image to be processed, obtaining the probability of the image to be processed belonging to each image category.

[0327] The image to be processed is scaled to a preset size and then input into a preset efficient network in a modified multi-class classification model. The preset efficient network extracts feature vectors from the image. These feature vectors are then input into a branch model and a bidirectional long short-term memory network corresponding to each first image category. Each branch model classifies the feature vectors to obtain the first probability that the image belongs to each first image category. The bidirectional long short-term memory network further classifies the feature vectors to obtain the probability that the image belongs to each image category, including a second probability. For each first image category, a fusion module weighted and fused the first and second probabilities to obtain the final probability that the image belongs to each first image category.

[0328] After obtaining the final probability of the image to be processed belonging to each image category, the image category with the highest probability is determined as the image category to which the image to be processed belongs, thus realizing the classification of multiple image categories through a single model and identifying the image category to which the image to be processed belongs.

[0329] Taking an image category comprising eight categories—surgical site, mosaic, surgical procedure, instruments and medications, indoor environment, outdoor environment, people, and others—as an example, the classification method provided in this application embodiment achieves the following final image classification effect: Figure 12 As shown.

[0330] In this embodiment, a preset probability threshold can be set for each image category. For example, in the medical field, each image category can include eight categories: surgical site, puzzle, surgical procedure, instruments and medicines, indoor environment, outdoor environment, people, and others. The preset probability threshold for surgical site can be 0.6, for puzzle can be 0.5, for instruments and medicines can be 0.6, for surgical procedure can be 0.5, for indoor environment can be 0.9, for outdoor environment can be 0.5, and for people can be 0.6. Since the "other" category contains too much uncertainty, including everything unrelated to medical surgery such as food, scenery, emojis, and game screenshots, the features that the multi-class classification model can learn are quite complex. To ensure accuracy, the preset probability threshold for the "other" category can be 0.9.

[0331] Multi-class image classification outputs the probability that the image to be processed belongs to each image category. For each image category, if the probability corresponding to that category is greater than a preset probability threshold, then the image to be processed is determined to belong to that category. Through multi-class image classification, an image to be processed can belong to one or more image categories. For example, an image to be processed can simultaneously belong to both the surgical site and human categories.

[0332] In this embodiment, a multi-class classification model is trained to simultaneously classify multiple image categories using a single neural network model. This multi-class classification model includes a bidirectional long short-term memory (LSTM) network, which learns the correlations between features of different categories, resulting in more robust features. During the classification process for each image category, if the classification accuracy for a particular image category is low, a branch model corresponding to that image category is trained. This branch model is then used to correct the multi-class classification model, addressing the issue of low classification accuracy for that image category. The model training process employs a combination of cross-entropy loss and smoothing loss functions, resulting in smoother training and better training performance.

[0333] In some other embodiments of this application, the above-mentioned image categories also include a second image category, which is an image category that includes body parts. For an image category that includes body parts, the body part included in the image may be a face. In fields such as beauty and plastic surgery, users prefer to browse images that include a face without makeup, while images that include a face with filters are of low reference value to users. Therefore, after obtaining the probability of the image to be processed belonging to each image category in the above manner, the probability of the image to be processed belonging to the second image category is compared with a preset probability threshold corresponding to the second image category. This preset probability threshold can be 0.6 or 0.7, etc. If the probability of the image to be processed belonging to the second image category is less than or equal to the preset probability threshold corresponding to the second image category, then it is determined that the image to be processed does not belong to the second image category. If it is determined that the probability of the image to be processed belonging to the second image category is greater than the preset probability threshold corresponding to the second image category, then it is determined that the image to be processed belongs to the second image category. Then, the face detection model in step 101 is used to detect whether the image to be processed contains a face region that meets the preset face classification conditions. If a face region that meets the preset face classification conditions exists, then the beauty recognition model in step 101 is used to further identify whether the image of the face region is a beauty image.

[0334] If the beautification recognition model outputs a probability greater than 0.5 that the image of the face region belongs to the beautified image category, then the image of the face region is determined to be a beautified image, and the image to be processed is determined not to belong to the second image category. If the beautification recognition model outputs a probability greater than 0.5 that the image of the face region belongs to the unbeautified image category, then the image of the face region is determined to be an unbeautified image, and the image to be processed is determined to belong to the second image category.

[0335] In essence, when a multi-class classification model determines that the image to be processed belongs to the second image category, and the image contains a face region that meets the preset face classification conditions, a face detection model and a beautification recognition model are used to verify whether the face region is a beautified image. If it is determined to be a beautified image, the image to be processed is considered not to belong to the second image category. In other words, the face detection model and the beautification recognition model correct the classification results of the multi-class classification model regarding the second image category, thereby improving the accuracy and reliability of the classification results for the second image category.

[0336] In other embodiments of this application, the aforementioned image categories also include a third image category, which includes a stitched image, which is an image composed of multiple sub-images. After determining that the image to be processed belongs to the third image category through a multi-class classification model, a pre-trained recognition model is further used to identify whether the image to be processed is a stitched image containing the target content.

[0337] Before using a recognition model to determine whether an image to be processed is a stitched image containing the target content, the recognition model needs to be trained first. Specifically, first, a recognition model is built. Figure 13 The model structure is shown. Then, a first stitched image containing the target content is obtained as a positive sample, and the horizontal stitching line position and / or vertical stitching line position are marked in the first stitched image, as well as the first category label corresponding to the third image category. A second stitched image and / or non-stitched image that does not contain the target content are obtained as negative samples, and the horizontal stitching line position and / or vertical stitching line position are marked in the second stitched image, as well as the second category label corresponding to the third image category is marked in the second stitched image and the non-stitched image. Finally, the constructed recognition model is trained using positive and negative samples until the stitching line prediction network and classification network in the recognition model converge.

[0338] In this designation, the first stitched image containing the target content refers to a stitched image in which every sub-image contains the target content. If any sub-image does not contain the target content, it is considered the second stitched image, i.e., a negative sample. The first category label is used to characterize images that are stitched images containing the target content, while the second category label is used for non-stitched images or stitched images that do not contain the target content.

[0339] During the annotation process, only three pieces of information need to be annotated: the horizontal and / or vertical splicing lines, and whether it belongs to the puzzle category containing the target content. Therefore, the annotation requirements are few and simple.

[0340] like Figure 14 As shown, 10 different splicing styles are provided as examples. It can be seen from these splicing styles that the splicing line features in the figure are either horizontal lines or vertical lines.

[0341] Therefore, in the actual annotation process, in order to avoid the problem that the splicing lines of the annotation are not completely horizontal or vertical due to human jitter, a pixel coordinate system is first established in the image. For the horizontal splicing lines of the annotation, the average value of the vertical coordinate of the entire horizontal splicing line is taken as the final annotation result. For the vertical splicing lines of the annotation, the average value of the horizontal coordinate of the entire vertical splicing line is taken as the final annotation result.

[0342] The specific calculation formula is as follows:

[0343]

[0344]

[0345] Where m represents the number of elements in the horizontal splicing line of the label, and n represents the number of elements in the vertical splicing line of the label; horizontal_position[y i] indicates retrieving the y-coordinate of the element at the horizontal splicing line, vertical_position[x i [] indicates the x-coordinate of the element at the vertical splicing line; mean_horizontal_position indicates the final result of the horizontal annotation, and mean_vertical_position indicates the final result of the vertical annotation.

[0346] It should be noted that during training, the loss function for the splicing line prediction network can use a regression-based L1 loss function, the formula of which is as follows:

[0347]

[0348]

[0349] Among them, smooth(x i ) represents the loss value of the i-th feature among the splicing line features predicted by the splicing line prediction network, x i Let w represent the predicted value of the i-th feature in the splicing line feature, w·h represent the size of the splicing line feature, and loss represent the predicted value of the i-th feature in the splicing line feature. L1 This represents the loss value of the splicing line prediction network.

[0350] For the loss function of classification networks, the cross-entropy loss function can be used. The formula for calculating the function is as follows:

[0351]

[0352] Where Loss represents the loss value of the classification network, N represents the total number of classification categories, and in this embodiment, N = 2, p n This represents the label value of the nth category in the training sample annotations. This represents the prediction value of the nth classification category of the output by the classification network based on the input training samples.

[0353] In one optional implementation, to avoid interference from black edges in the original image on the stitching line prediction, the process of acquiring the model input image can identify whether there are black edges in the original image. If there are, the original image is cropped to remove the black edges, and then the cropped image is scaled to obtain a uniform size image that meets the model input requirements. If there are no black edges, the original image can be scaled directly.

[0354] Optionally, the original image can be converted to a grayscale image, and then the grayscale values ​​of each row or column edge pixel in the grayscale image can be accumulated. If the accumulated sum is zero, then the row or column edge pixel is determined to be a black edge.

[0355] After training the recognition model using the above method, the image to be processed is input into the trained recognition model. The feature extraction network in the recognition model extracts features from the image to obtain a feature map. First, the image to be processed is convolved by the convolutional layer in the feature extraction network and then input into the downsampling residual network in the feature extraction network. For example, assuming the input image is 224*224*3, after convolution with a 3*3 kernel and a stride of 2, it can output a feature map of 112*112*32. The image feature resolution is halved after processing by the convolutional layer, but the number of channels increases.

[0356] Then, a downsampling residual network is used to perform multiple downsampling processes on the convolutional image to obtain feature maps. See also Figure 15 As shown, the downsampling residual network consists of four cascaded downsampling residual blocks. Each downsampling residual block is composed of a series of bottleneck residual layers stacked together. The feature map acquisition process includes: downsampling the convolutional image through the first downsampling residual block to obtain a first intermediate feature, and inputting the first intermediate feature into the second downsampling residual block; downsampling the first intermediate feature through the second downsampling residual block to obtain a second intermediate feature, and inputting the second intermediate feature into the third downsampling residual block; downsampling the second intermediate feature through the third downsampling residual block to obtain a third intermediate feature, and inputting the third intermediate feature into the fourth downsampling residual block; and downsampling the third intermediate feature through the fourth downsampling residual block to obtain the feature map.

[0357] It should be noted that these four downsampling residual blocks perform a series of downsampling operations on the input image, gradually reducing its resolution. Furthermore, during this process, the first intermediate feature generated by the first downsampling residual block, the second intermediate feature generated by the second downsampling residual block, and the third intermediate feature generated by the third downsampling residual block are all used as intermediate features output from the stitching prediction network.

[0358] Since the four downsampling residual blocks mentioned above are all composed of a series of residual convolutional layers, they all belong to residual networks. By using residual networks for feature extraction, problems such as gradient explosion and gradient vanishing that occur as the network depth increases can be avoided, making network training easier.

[0359] Those skilled in the art will understand that the number of downsampled residual blocks in the downsampled residual network can be set according to actual needs, as long as it ensures that the final output feature map is of a certain size. Figure 15 The four downsampling residual blocks given are merely illustrative examples.

[0360] This completes the feature extraction process. By using a residual network to extract the content features of the image, the network depth can be increased to extract deeper content features, thereby improving the prediction accuracy of the stitching prediction network and the classification network.

[0361] After obtaining the feature map in the above manner, the splicing line position in the image to be processed is predicted based on the feature map by the splicing line prediction network in the recognition model.

[0362] In an optional specific embodiment, the first intermediate feature, the second intermediate feature, and the third intermediate feature generated by the feature extraction network during the feature extraction process can be obtained by the splicing line prediction network, and the splicing line position in the image can be predicted based on the feature map, the first intermediate feature, the second intermediate feature, and the third intermediate feature.

[0363] Since the feature extraction network is the module closest to the model input, the intermediate features (including the first to third intermediate features) generated during the feature extraction process can be used as shallow features, and the features (i.e. feature maps) output by the feature extraction network to the splicing line prediction network can be used as deep features. By fusing the shallow features generated in the middle of the feature extraction network and the output deep features, and predicting the splicing line position based on the fused features, the model can have better output accuracy.

[0364] Specifically, the first feature fusion module in the splicing line prediction network performs feature fusion on the feature map, the first intermediate feature, the second intermediate feature, and the third intermediate feature, and outputs the result. In an optional embodiment, such as... Figure 16 The splicing line prediction network structure shown has intermediate features output by the feature extraction network, including a first intermediate feature, a second intermediate feature, and a third intermediate feature. These three intermediate features are respectively input into the three fusion modules set in the first feature fusion module for fusion.

[0365] The specific fusion process of the first feature fusion module includes: upsampling the feature map through the first upsampling layer and then inputting it into the first fusion module; fusing the upsampled feature map with the third intermediate feature through the first fusion module and then performing convolution processing to obtain the first fused feature, which is then input into the second upsampling layer; upsampling the first fused feature through the second upsampling layer and then inputting it into the second fusion module; fusing the upsampled first fused feature with the second intermediate feature through the second fusion module and then performing convolution processing to obtain the second fused feature, which is then input into the third upsampling layer; upsampling the second fused feature through the third upsampling layer and then inputting it into the third fusion module; and fusing the upsampled second fused feature with the first intermediate feature through the third fusion module and then performing convolution processing to obtain the fused feature.

[0366] The purpose of upsampling in each of the above upsampling layers is to obtain features of the same size as the intermediate features to be fused. Specifically, the size of the upsampled feature map output by the first upsampling layer is the same as the size of the third intermediate feature; the size of the upsampled first fused feature output by the second upsampling layer is the same as the size of the second intermediate feature; and the size of the upsampled second fused feature output by the third upsampling layer is the same as the size of the first intermediate feature. Since each fusion module is based on the channel dimension, a convolutional process can be performed after each fusion before outputting the result.

[0367] In this embodiment of the invention, in order to reduce the computational complexity of the subsequent horizontal and vertical stitching prediction modules, the fused feature of the size of the first intermediate feature is finally taken as the prediction input, instead of the fused feature of the input image size.

[0368] Then, the horizontal stitching line prediction module uses the fused features output by the first feature fusion module to predict the position of the horizontal stitching line in the image. In an optional embodiment, such as... Figure 17 The horizontal stitching line prediction module structure shown includes the following process: the first separation module separates the fused features by column, and the multiple separated column features are fused according to the channel dimension to obtain a single column feature, which is then input into the first processing module. The first processing module performs multiple channel dimensionality reduction processes on the single column feature to obtain the horizontal stitching line feature, which is then input into the first output layer. The first output layer determines the position of the horizontal stitching line in the image based on the horizontal stitching line feature.

[0369] The purpose of the horizontal stitching line prediction module is to learn the position of the horizontal stitching line. The horizontal stitching line is a horizontal line in the input image. Therefore, the fused features are broken down by column and then fused into a column feature based on the channel dimension. After multiple dimensionality reduction processes by the first processing module, it becomes a column feature vector with 1 channel. For example, a 1*1 convolution kernel can be used to achieve the dimensionality reduction process each time.

[0370] In other words, the horizontal splicing line feature is a column feature vector with one channel, and the length of this column feature vector is consistent with the height of the fused features input to the prediction module. For example... Figure 17 The output of the first processing module shown in the figure Figure 18 The column feature vector corresponding to the input image shown is due to Figure 18 The horizontal stitching line of the input image is located in the middle of the image, so the eigenvalues ​​in the middle part of the column feature vector are relatively high.

[0371] Optionally, for the process of determining the position of the horizontal stitching line in the image based on the horizontal stitching line feature through the first output layer, a first window with a first preset interval can be used to traverse the horizontal stitching line feature starting from the first feature value of the horizontal stitching line feature with a preset step size. Each time the first window moves, the first position of the horizontal stitching line in the horizontal stitching line feature is extracted and recorded based on the feature value contained in the first window. When the traversal ends, for each recorded first position, the position of the horizontal stitching line in the image is determined based on the image size and the first position.

[0372] Here, the first preset interval refers to the minimum interval between two pre-set horizontal stitching lines. After the first window traversal is completed, several first positions are recorded, indicating the number of horizontal stitching lines in the image. Since the recorded first positions actually refer to their positions within the horizontal stitching line features, it is necessary to convert the recorded first positions to their positions within the input image based on the input image size. The specific conversion calculation formula is as follows:

[0373]

[0374] Among them, positiont_pred 输入 This indicates the normalized position of the horizontal stitching line in the input image; leave_index indicates the position of the horizontal stitching line extracted from the horizontal stitching line features, i.e., the first position. This indicates the ratio of the height of the input image to the size of the horizontal stitching line feature; ε represents the preset error correction value, which can be 1.5; H 输入 This indicates the height of the input image.

[0375] Furthermore, since there are scaling operations and / or black border cropping operations between the input image and the original image, if the stitching line position in the input image is to be converted back to the original image, the scaling factor and the size of the cropped black border also need to be considered.

[0376] The specific conversion formula is as follows:

[0377] positiont_pred 原始 =positiont_pred 输入 *H 原始 +start_position...(17)

[0378] Among them, H 原始 The height of the original image is represented by `start_position`, and the size of the black border that was cropped out is represented by `start_position`.

[0379] It should be noted that, regarding the process of extracting and recording the first position of a horizontal splicing line in the horizontal splicing line feature based on the feature values ​​contained in the first window, this can be achieved by obtaining feature values ​​greater than a threshold within the first window, selecting the largest feature value from the obtained feature values, and determining whether a first position has already been recorded. If not, it indicates that this is the position of the first extracted horizontal splicing line in the horizontal splicing line feature, and the position of the largest feature value is recorded as the first position of a horizontal splicing line in the horizontal splicing line feature. If so, the interval between the position of the largest feature value and the most recently recorded first position is compared with a first preset interval. If it is greater than the first preset interval, it indicates that there is another horizontal splicing line, and the position of the largest feature value is recorded as the first position of the other horizontal splicing line in the horizontal splicing line feature. If it is less than the first preset interval, it indicates that the largest feature value still belongs to the most recently recorded horizontal splicing line, and the first position of the most recently recorded line is updated according to the position of the largest feature value.

[0380] In the first window, the feature values ​​greater than the threshold refer to feature values ​​that may be horizontal splicing lines.

[0381] Optionally, for the process of updating the first position of the most recent record based on the position of the maximum feature value, the feature value corresponding to the first position of the most recent record can be compared with the maximum feature value. If it is less than the maximum feature value, the first position of the most recent record is updated using the position of the maximum feature value, so that the recorded first position is always the position indicated by the maximum feature value in the horizontal splicing line.

[0382] This application embodiment also uses the longitudinal stitching line prediction module to predict the position of the longitudinal stitching line in the fused feature image output by the first feature fusion module.

[0383] In an alternative embodiment, such as Figure 19 The vertical stitching line prediction module structure shown includes the following process: the fused features are separated by row by row by the second separation module, and the multiple row features obtained by separation are fused according to the channel dimension to obtain a row feature, which is then input into the second processing module. The second processing module performs multiple channel dimensionality reduction processing on the row feature to obtain the vertical stitching line feature, which is then input into the second output layer. The second output layer determines the position of the vertical stitching line in the image based on the vertical stitching line feature.

[0384] The purpose of the vertical stitching line prediction module is to learn the position of the vertical stitching line. The vertical stitching line is a vertical line in the input image. Therefore, the fused features are broken down by row and then fused into a row feature based on the channel dimension. After multiple dimensionality reduction processes by the second processing module, the vertical stitching line feature is obtained until it becomes a row feature vector with 1 channel. For example, a 1*1 convolution kernel can be used to achieve the dimensionality reduction process each time.

[0385] In other words, the vertical splicing line feature is a row feature vector with 1 channel, and the length of this row feature vector is consistent with the width of the fused feature input to the prediction module. For example... Figure 19 The output of the second processing module shown above Figure 18 The row feature vector corresponding to the input image shown is due to Figure 18 The vertical stitching line of the input image is located in the middle of the image, so the eigenvalues ​​of the middle part of the row feature vector are relatively high.

[0386] Optionally, for the process of determining the position of the vertical stitching line in the image based on the vertical stitching line feature through the second output layer, a second window with a second preset interval can be used to traverse from the first feature value of the vertical stitching line feature with a preset step size. Each time the second window moves, the second position of the vertical stitching line in the vertical stitching line feature is extracted and recorded based on the feature value contained in the second window. When the traversal ends, for each recorded second position, the position of the vertical stitching line in the image is determined based on the size of the image and the second position.

[0387] The second preset interval refers to the minimum interval between two pre-set vertical stitching lines. After traversing the second window, several second positions are recorded, indicating the number of vertical stitching lines in the image. Since the recorded second positions actually refer to their positions within the vertical stitching line features, it is necessary to convert the recorded second positions to their positions within the input image based on the input image size. The specific conversion calculation formula is as follows:

[0388]

[0389] Among them, positiont_pred 输入 This indicates the normalized position of the vertical stitching line in the input image; leave_index indicates the position of the vertical stitching line extracted from the vertical stitching line features, i.e., the second position. This represents the ratio of the width of the input image to the size of the vertical stitching line feature; ε represents the preset error correction value, which can be 1.5; W 输入 This indicates the width of the input image.

[0390] Furthermore, since there are scaling operations and / or black border cropping operations between the input image and the original image, if the stitching line position in the input image is to be converted back to the original image, the scaling factor and the size of the cropped black border also need to be considered.

[0391] The specific conversion formula is as follows:

[0392] positiont_pred 原始 =positiont_pred 输入 *W 原始 +start_position...(19)

[0393] Among them, W 原始 `start_position` represents the width of the original image, and `start_position` represents the size of the black border to be cropped.

[0394] It should be noted that, regarding the process of extracting and recording the second position of a vertical splicing line in the vertical splicing line feature based on the feature values ​​contained in the second window, feature values ​​greater than a threshold are obtained within the second window, and the largest feature value is selected from the obtained feature values. It is then determined whether there is a recorded second position. If not, it indicates that this is the position of the first extracted vertical splicing line in the vertical splicing line feature, and the position of the largest feature value is recorded as the second position of a vertical splicing line in the vertical splicing line feature. If there is, the interval between the position of the largest feature value and the most recently recorded second position is compared with the second preset interval. If it is greater than the second preset interval, it indicates that there is another vertical splicing line, and the position of the largest feature value is recorded as the second position of another vertical splicing line in the vertical splicing line feature. If it is less than the second preset interval, it indicates that the largest feature value still belongs to the most recently recorded vertical splicing line, and the second position of the most recently recorded line is updated according to the position of the largest feature value.

[0395] In the second window, the feature values ​​greater than the threshold refer to feature values ​​that may be vertical splicing lines.

[0396] Optionally, for the process of updating the second position of the most recently recorded record based on the position of the maximum feature value, if the maximum feature value is greater than the feature value corresponding to the second position of the most recently recorded record, then the second position of the most recently recorded record is updated using the position of the maximum feature value, so that the recorded second position is always the position indicated by the maximum feature value in the vertical splicing line.

[0397] As described above, the calculation principle of the vertical stitching line position in the image by the second output layer is the same as described above. Figure 17The calculation principle for the position of the horizontal stitching line in the image in the first output layer is similar, except that the height in Formula 16 and Formula 17 above is changed to the width.

[0398] This completes the stitching line prediction process. Before prediction, the shallow intermediate features and deep features are fused together by the first feature fusion module to improve prediction accuracy. Since the horizontal stitching line extends horizontally while the vertical stitching line extends vertically in the stitched image, their feature predictions differ. Therefore, different processing methods are used for the horizontal and vertical stitching line prediction modules.

[0399] The splicing prediction network given above uses only three simple processing methods: upsampling, convolution, and splicing. There are no complex computational processes involved, resulting in fast prediction speed.

[0400] After obtaining the feature map and stitching line positions of the image to be processed using the above method, the classification network in the recognition model determines whether the image to be processed is a stitched image containing the target content based on the feature map and stitching line positions. In an exemplary scenario, such as... Figure 18 As shown, the input image is composed of four images stitched together, and each image is related to the surgery. After the input image is input into the recognition model, the model processes it and outputs a classification result marked with the predicted horizontal stitching line, vertical stitching line, and stitched image containing surgical content.

[0401] First, the feature map and splicing line features are fused by the second feature fusion module in the classification network, and then the fused features are input into the third processing module in the classification network.

[0402] Among them, the feature map belongs to the content description feature, and the splicing line feature belongs to the splicing line description feature. The splicing line feature is composed of horizontal splicing line feature and vertical splicing line feature. Therefore, by fusing the content description feature and the splicing line description feature for classification, the classification accuracy can be improved.

[0403] In one optional implementation, to ensure a balanced proportion between content description features and splicing line description features, the size and channels of the feature map need to be converted to be the same as those of the splicing line features before fusion, such as... Figure 20The specific structure of the second feature fusion module is as follows: For feature map transformation, a global pooling layer converts the feature map into a 1*1 feature map, and then a convolutional layer converts the 1*1 feature map into content description features with a preset channel dimension; For the transformation of horizontal and vertical splicing line features, different deformation operation layers are used to transform the horizontal and vertical splicing line features into horizontal and vertical splicing line description features with the same size and dimension as the content features; A fusion layer fuses the content description features, horizontal splicing line description features, and vertical splicing line description features with the same size and dimension based on the channel dimension to obtain the fused features.

[0404] Then, the fused features are subjected to multiple channel dimensionality reduction processes by the third processing module to obtain features of a preset channel dimension, which are then input into the third output layer of the classification network. The purpose of these multiple channel dimensionality reduction processes is to obtain features of a preset channel dimension, which can be achieved using 1*1 convolutional kernels for each dimensionality reduction step. This preset channel dimension is the same as the final required number of classification categories; in this invention, the number of classification categories is 2, therefore the preset channel dimension is 2.

[0405] The third output layer determines whether an image is a stitched image containing target content based on features of a preset channel dimension. Specifically, if the classification result indicates that the input image is a stitched image containing target content, it means that each sub-image in the input image contains the target content. The target content varies depending on the training samples used in the actual application scenario.

[0406] The classification process is now complete. Before classification, the accuracy can be improved by fusing the content description features from the feature extraction network with the splicing line description features from the splicing line prediction network. The modules in the classification network described above are implemented using three simple processes: deformation, pooling, convolution, and splicing. There are no complex computational processes involved, resulting in fast classification speed.

[0407] Based on the description of the above embodiments, as Figure 21To identify the overall structure of the model, a 224*224*3 image is input. After feature extraction through the convolutional layers and the first to fourth downsampled residual blocks in the feature extraction network 10, a 7*7*1280 feature map is output. This feature map enters the stitching line prediction network 20. The first intermediate feature (56*56*24), the second intermediate feature (28*28*32), and the third intermediate feature (14*14*160) generated during feature extraction also enter the stitching line prediction network 20. The first feature fusion module in module 0 performs a series of fusion operations on the 7*7*1280 feature map, the 56*56*24 first intermediate feature, the 28*28*32 second intermediate feature, and the 14*14*160 third intermediate feature, and outputs a 56*56*32 fused feature. This 56*56*32 fused feature enters the horizontal splicing line prediction module and the vertical splicing line prediction module to predict the horizontal splicing line and the vertical splicing line respectively, so as to obtain the horizontal splicing line position 1 and the vertical splicing line position 2.

[0408] For the classification network 30, the 7*7*1280 feature map is first processed into 1*1*56 content description features through max pooling and convolutional layers. For the horizontal and vertical splicing line features output by the splicing line prediction network 20, the deformation operation layer is used to convert them into 1*1*56 horizontal splicing line description features and 1*1*56 vertical splicing line description features, respectively. These three features are fused by the fusion layer to output a fused feature of 1*1*168. Then, after dimensionality reduction processing by the third processing module, a 1*1*2 feature is output. Finally, the classification result is output by the discrimination of the third output layer.

[0409] This completes the process of recognizing stitched images containing the target content. By employing a deep learning network model, it simultaneously predicts the position of the stitching lines and the stitching classification result, providing filtering conditions for image selection. Furthermore, the stitching classification result specifically determines whether the stitched image contains the target content. For example, in the medical aesthetics field, it can predict whether a user-shared image is a stitched image containing surgical sites, solving the problem of recognizing stitched images before and after surgery.

[0410] Furthermore, the classification network and splicing line prediction network in this model are both lightweight in design, with fast prediction speed, which is conducive to application deployment on mobile phones or robots with relatively low processing power.

[0411] In some other embodiments of this application, the multi-class classification model can classify multiple image categories, including a fourth image category, which is an image category containing text, i.e., an image containing text regions. After obtaining the probability of the image to be processed belonging to each image category through the multi-class classification model, the probability of the image to be processed belonging to the fourth image category is compared with a preset probability threshold corresponding to the fourth image category. This preset probability threshold can be 0.9 or 0.95, etc. If the probability of the image to be processed belonging to the fourth image category is less than or equal to the preset probability threshold corresponding to the fourth image category, then it is determined that the image to be processed does not belong to the fourth image category. If the probability of the image to be processed belonging to the fourth image category is determined to be greater than the preset probability threshold corresponding to the fourth image category, then it is determined that the image to be processed belongs to the fourth image category.

[0412] After determining that the image to be processed belongs to the fourth image category, it can be further determined whether the image to be processed belongs to the fourth image category containing target content. Specifically, the text information in the image to be processed is identified using an Optical Character Recognition (OCR) model. The text information is segmented into words to obtain one or more keywords. The Jieba word segmentation algorithm can be used for word segmentation. For each keyword identified from the image to be processed, a preset dictionary is queried to see if the keyword is contained in the preset dictionary. If it is determined that all keywords identified from the image to be processed are not in the preset dictionary, then the image to be processed is determined not to belong to the fourth image category containing target content. If it is determined that the preset dictionary contains at least one keyword identified from the image to be processed, then the image to be processed is determined to belong to the fourth image category containing target content.

[0413] The fourth image category and target content can both be determined based on business needs. For example, the fourth image category could be an outdoor environment image containing text, and the target content could be related to cosmetic surgery or medical procedures. Therefore, the fourth image category containing the target content could be an outdoor medical environment. Each keyword in the text information of the image to be processed is identified using the above method. These keywords could be "hospital," "outpatient clinic," "emergency room," "cosmetic surgery," etc. A pre-defined dictionary contains a large number of medical-related words. Based on at least one keyword identified in the image to be processed from the pre-defined dictionary, the image to be processed is determined to belong to an outdoor medical environment.

[0414] like Figure 22 As shown, the image to be processed is input into the optical character recognition model to identify keywords. The model then searches a preset dictionary to see if the identified keywords are present, thus obtaining the recognition result. Specifically, if the preset dictionary contains the keywords corresponding to the image to be processed, the image is determined to belong to the fourth image category containing the target content.

[0415] Step 103: Identify whether the image to be processed contains the target person by using a pre-trained person classification model.

[0416] Before using a people classification model to identify whether an image contains the target person, the people classification model is first trained. Specifically, a fourth training set is obtained, which includes multiple images labeled with the target person and multiple images that do not contain the target person. The people classification model is then trained based on this fourth training set.

[0417] Multiple images are acquired from the fourth training set and input into the constructed person classification model to obtain the classification result for each image, which includes the probability that the image contains the target person. Based on the classification result for each image, the loss value for the current training epoch is calculated.

[0418] After calculating the loss value for the current training period, determine whether the current number of training periods has reached the preset number of training periods. If so, stop training, and obtain the model parameters of the training period with the smallest loss value from the trained periods. The constructed character classification model and the model parameters corresponding to the training period with the smallest loss value are then identified as the trained character classification model. If the current number of training periods has not reached the preset number of training periods, continue training in the above manner until a trained character classification model is obtained.

[0419] The image to be processed is then input into a trained person classification model to obtain the probability that the image contains the target person. The target person can be set according to business needs, such as a doctor, nurse, traffic police officer, firefighter, student, etc.

[0420] To facilitate understanding of the image classification direction in the embodiments of this application, the following description is provided in conjunction with the accompanying drawings. Figure 23 As shown, the image to be processed is input into a face detection model, a multi-class classification model, and a person classification model, respectively. The face detection model can detect the presence of a complete face region in the image, with the face region area exceeding a preset threshold; or it can detect the absence of a complete face region; or it can detect a complete face region, with the face region area exceeding a preset threshold. For images containing a complete face region with a face region area exceeding the preset threshold, a beauty recognition model is used to classify the image to determine whether it is a beautified image or a plain image.

[0421] For the branch of the multi-class classification model that inputs the image to be processed, the model can classify images into eight categories: surgical site, mosaic, surgical procedure, indoor environment, outdoor environment, people, and others. The multi-class classification model then outputs the probability that the image to be processed belongs to each of these eight categories. The surgical site can include images before surgery, images after surgery, and images showing the surgical outcome. If the image to be processed is identified as belonging to a surgical site and contains a face region, the identification result can be corrected using a beautification model. That is, the beautification model identifies whether the face region in the image to be processed is a beautified image; if it is, the image to be processed is determined not to belong to a surgical site; if the face region in the image to be processed is not a beautified image, the image to be processed is determined to belong to a surgical site.

[0422] For the above jigsaw puzzle categories, once it is determined that the image to be processed belongs to the jigsaw puzzle category, it can also be classified into the recognition model used to identify jigsaw puzzles containing target content, in order to determine whether the image to be processed is a jigsaw puzzle image containing target content.

[0423] For the above-mentioned outdoor environment categories, once it is determined that the image to be processed belongs to the outdoor environment category, the image to be processed can be input into the optical character recognition model to recognize the text information in the image to be processed, and then to determine whether the image to be processed belongs to the outdoor medical environment.

[0424] For the branch that inputs the image to be processed into the person classification model, the person classification model determines whether the image to be processed contains the target person, such as a doctor or nurse.

[0425] In this embodiment, a beauty recognition model for identifying beautified images, a multi-class classification model for classifying various image categories, a person classification model for identifying whether an image contains a target person, and so on, are integrated. These multiple models automatically classify the images to be processed, offering strong real-time performance and enabling automatic classification of large numbers of images, thus improving the efficiency of image data structure processing.

[0426] This application also provides an image classification apparatus for performing the image classification method provided in any of the above embodiments. Figure 24 As shown, the device includes:

[0427] The beauty recognition module 201 is used to identify whether the image to be processed is a beauty image by using a pre-trained face detection model and a beauty recognition model;

[0428] The multi-class classification module 202 is used to classify the image to be processed using a pre-trained multi-class classification model, and obtain the probability of the image to be processed belonging to each image category.

[0429] The target person recognition module 203 is used to identify whether the image to be processed contains the target person through a pre-trained person classification model.

[0430] The beautification recognition module 201 is used to determine whether there is a face region in the image to be processed that meets the preset face classification conditions through a pre-trained face detection model; if so, it uses the pre-trained beautification recognition model to identify whether the image of the face region that meets the preset face classification conditions is a beautified image.

[0431] The beauty recognition module 201 is used to detect whether the image to be processed contains at least one face region through a pre-trained face detection model; if so, it determines whether the at least one face region contains a target face region whose face area ratio is greater than a preset threshold; if a target face region exists, it determines that there is a face region in the image to be processed that meets the preset face classification conditions.

[0432] The beauty recognition module 201 is used to detect facial key points corresponding to each face region; determine the area of ​​each face region based on the facial key points corresponding to each face region; calculate the face area ratio corresponding to each face region based on the area of ​​each face region and the area of ​​the image to be processed; and determine whether at least one face region contains a target face region whose face area ratio is greater than a preset threshold based on the face area ratio corresponding to each face region.

[0433] The beautification recognition module 201 is used to extract the face region that meets the preset face classification conditions from the image to be processed to obtain the corresponding face image; generate the face alignment image corresponding to the face image; and identify whether the face alignment image is a beautified image through a pre-trained beautification recognition model.

[0434] The beauty recognition module 201 is used to scale a face image to a first image of a preset size; acquire multiple facial key points of the face image; perform an alignment operation on the face image based on preset standard key points and multiple facial key points of the face image to obtain a second image; and merge the first image and the second image based on color channels to obtain a face-aligned image corresponding to the face image.

[0435] The device also includes: a model training module for constructing the structure of a beauty recognition model for recognizing beautified images; obtaining a first training set; and training the beauty recognition model based on the first training set.

[0436] The model training module is used to sequentially connect a preset number of convolutional modules, where the preset number is an integer greater than or equal to 3; the last convolutional module in the sequentially connected preset number of convolutional modules is connected to a preset high-efficiency network to obtain the structure of a beauty recognition model for recognizing beautified images.

[0437] The model training module is used to acquire multiple beautified face images and multiple unbeautified face images; generate face alignment images corresponding to each acquired face image; and label each face alignment image with a classification label to obtain the first training set.

[0438] The model training module is used to annotate beautification labels in each beautified face-aligned image and to annotate bare-face labels in each bare-face-aligned image; and / or, to annotate beautified facial features in the beautified face-aligned images.

[0439] The model training module is used to acquire face-aligned images from the first training set; input the acquired face-aligned images into the beauty recognition model to obtain the classification results of the face images corresponding to the face-aligned images; and calculate the loss value of the current training cycle based on the classification results of the face images using a preset classification-center loss function.

[0440] The classification results include the probability that a face image belongs to each image category, and each image category includes at least beautified images; the model training module is used to calculate the first classification loss value corresponding to the face image based on the number of image categories and the probability that the face image belongs to each image category, using the first classification loss function included in the preset classification-center loss function; based on the feature vectors and feature center points of each image category corresponding to the face image, the center loss value corresponding to the face image is calculated using the center loss function included in the classification-center loss function; based on the first classification loss value and the center loss value, the loss value of the current training cycle is calculated.

[0441] The classification results also include the probability that an image of at least one facial feature in a face image belongs to each image category; the model training module is used to calculate the second classification loss value corresponding to the face image through the second classification loss function included in the classification-center loss function, based on the number of at least one facial feature, the number of image categories, and the probability that an image of each facial feature belongs to each image category; and the loss value of the current training cycle is calculated based on the first classification loss value, the second classification loss value, and the center loss value.

[0442] The classification results include the probability that an image with at least one facial feature belongs to each image category; the model training module is used to calculate the second classification loss value corresponding to the facial image using the second classification loss function included in the classification-center loss function, based on the number of at least one facial feature, the number of image categories, and the probability that each facial feature belongs to each image category; based on the feature vectors and feature centers of each image category corresponding to the facial image, the center loss value corresponding to the facial image is calculated using the center loss function included in the classification-center loss function; and based on the second classification loss value and the center loss value, the loss value for the current training cycle is calculated.

[0443] The multi-class classification module 202 is used to obtain a second training set, which includes multiple images, each of which is labeled with a category label corresponding to each image category; to construct the structure of a first neural network model for multi-label classification; and to train the constructed first neural network model based on the second training set to obtain a multi-class classification model.

[0444] The multi-class classification module 202 is used to connect the output of a preset high-efficiency network to the input of a bidirectional long short-term memory network to obtain a first neural network model for multi-label classification.

[0445] The multi-class classification module 202 is used to acquire multiple images from the second training set; input the acquired images into the constructed first neural network model to obtain the classification result of each image; and calculate the loss value of the current training cycle based on the classification result of each image using the cross-entropy loss function and the smoothing loss function.

[0446] The multi-class classification module 202 is used to determine the accuracy of the multi-class classification model for each image category; if there is a first image category with an accuracy lower than a preset threshold, then the branch model corresponding to the first image category is trained; the branch model is used to correct the multi-class classification model to obtain the corrected multi-class classification model.

[0447] The multi-class classification module 202 is used to obtain a third training set corresponding to the first image category. The third training set includes multiple images, each of which is labeled with a category label corresponding to the first image category. The module constructs a structure for a second neural network model to classify the first image category separately. The second neural network model includes a branch model corresponding to the first image category. The module trains the second neural network model based on the third training set to obtain the trained branch model.

[0448] The multi-class classification module 202 is used to sequentially connect a preset number of fully connected layers to obtain a branch model corresponding to the first image category; and to connect the output of the preset efficient network in the multi-class classification model with the input of the branch model to obtain a second neural network model for separately classifying the first image category.

[0449] The multi-class classification module 202 is used to connect the input of the trained branch model to the output of the preset efficient network in the multi-class classification model; and to connect the output of the branch model to the output of the bidirectional long short-term memory network in the multi-class classification model through the fusion module to obtain the corrected multi-class classification model; wherein, the fusion module is used to fuse the first classification result corresponding to the first image category output by the branch model and the second classification result corresponding to the first image category output by the bidirectional long short-term memory network.

[0450] The multi-class classification module 202 is used to extract feature vectors of the image to be processed through a preset efficient network in the modified multi-class classification model; classify the feature vectors through a branch model in the modified multi-class classification model to obtain a first probability that the image to be processed belongs to a first image category; classify the feature vectors through a bidirectional long short-term memory network in the modified multi-class classification model to obtain the probability that the image to be processed belongs to each image category, including a second probability that the image to be processed belongs to the first image category; and fuse the first probability and the second probability to obtain the final probability that the image to be processed belongs to the first image category.

[0451] The multi-category classification module 202 is used to determine whether the probability of the image to be processed belonging to a second image category is greater than a preset probability threshold corresponding to the second image category, wherein the second image category includes images containing body parts; to detect whether the image to be processed contains a face region that meets the preset face classification conditions through a face detection model; if the image to be processed contains a face region that meets the preset face classification conditions, then to identify whether the image of the face region in the image to be processed is a beautified image through a beautification recognition model; if it is determined that the image of the face region in the image to be processed is a beautified image, then it is determined that the image to be processed does not belong to the second image category.

[0452] The device also includes: a jigsaw puzzle module, used to determine whether the probability of the image to be processed belonging to a third image category is greater than a preset probability threshold corresponding to the third image category, the third image category including jigsaw puzzle images; and to identify whether the image to be processed is a jigsaw puzzle image containing target content through a pre-trained recognition model.

[0453] The image stitching module is used to input the image to be processed into the trained recognition model. The feature extraction network in the recognition model extracts features from the image to obtain a feature map. Based on the feature map, the stitching line prediction network in the recognition model predicts the position of the stitching line in the image to be processed. Based on the feature map and the stitching line position, the classification network in the recognition model determines whether the image to be processed is a stitched image containing the target content.

[0454] The image stitching module is used to convolve the image to be processed by the convolutional layer of the feature extraction network in the recognition model and then input it into the downsampling residual network in the feature extraction network. The downsampling residual network performs multiple downsampling processes on the convolutional image to obtain a feature map.

[0455] The image stitching module is used to downsample the convolutional image using a first downsampled residual block in the downsampled residual network to obtain a first intermediate feature, and then input the first intermediate feature into a second downsampled residual block in the downsampled residual network; the second downsampled residual block is used to downsample the first intermediate feature to obtain a second intermediate feature, and then input the second intermediate feature into a third downsampled residual block in the downsampled residual network; the third downsampled residual block is used to downsample the second intermediate feature to obtain a third intermediate feature, and then input the third intermediate feature into a fourth downsampled residual block in the downsampled residual network; finally, the fourth downsampled residual block is used to downsample the third intermediate feature to obtain a feature map.

[0456] The stitching module is used to obtain the first, second, and third intermediate features generated by the feature extraction network during the feature extraction process through the stitching line prediction network, and to predict the stitching line position in the image to be processed based on the feature map, the first, second, and third intermediate features.

[0457] The jigsaw puzzle module is used to perform feature fusion and output of the feature map, the first intermediate feature, the second intermediate feature, and the third intermediate feature through the first feature fusion module in the jigsaw puzzle prediction network; the horizontal jigsaw puzzle prediction module uses the fused features output by the first feature fusion module to predict the position of the horizontal jigsaw puzzle in the image to be processed; and the vertical jigsaw puzzle prediction module uses the fused features output by the first feature fusion module to predict the position of the vertical jigsaw puzzle in the image to be processed.

[0458] The jigsaw puzzle module is used to upsample the feature map through the first upsampling layer in the first feature fusion module and then input it into the first fusion module in the first feature fusion module; the first fusion module fuses the upsampled feature map with a third intermediate feature and then performs convolution processing to obtain a first fused feature, which is then input into the second upsampling layer in the first feature fusion module; the second upsampling layer upsamples the first fused feature and then inputs it into the second fusion module in the first feature fusion module; the second fusion module fuses the upsampled first fused feature with a second intermediate feature and then performs convolution processing to obtain a second fused feature, which is then input into the third upsampling layer in the first feature fusion module; the third upsampling layer upsamples the second fused feature and then inputs it into the third fusion module in the first feature fusion module; the third fusion module fuses the upsampled second fused feature with the first intermediate feature and then performs convolution processing to obtain the fused feature.

[0459] The jigsaw puzzle module is used to separate the fused features column by column by the first separation module in the horizontal stitching line prediction module, and then fuse the multiple separated column features according to the channel dimension to obtain a single column feature, which is then input into the first processing module in the horizontal stitching line prediction module. The first processing module performs multiple channel dimensionality reduction processes on the single column feature to obtain the horizontal stitching line feature, which is then input into the first output layer in the horizontal stitching line prediction module. The first output layer determines the position of the horizontal stitching line in the image based on the horizontal stitching line feature.

[0460] The jigsaw puzzle module is used to traverse the horizontal stitching line feature starting from the first feature value of the horizontal stitching line feature with a preset step size using a first window with a first preset interval. Each time the first window moves, the first position of the horizontal stitching line in the horizontal stitching line feature is extracted and recorded based on the feature value contained in the first window. When the traversal ends, for each recorded first position, the position of the horizontal stitching line in the image is determined based on the size of the image and the first position.

[0461] The jigsaw puzzle module is used to acquire feature values ​​greater than a threshold within a first window and select the largest feature value from the acquired feature values; determine if there is a recorded first position; if not, record the position of the largest feature value as the first position of a horizontal splicing line in the horizontal splicing line feature; if there is, compare the interval between the position of the largest feature value and the most recently recorded first position with a first preset interval; if it is greater than the first preset interval, record the position of the largest feature value as the first position of another horizontal splicing line in the horizontal splicing line feature; if it is less than the first preset interval, update the most recently recorded first position according to the position of the largest feature value.

[0462] The jigsaw puzzle module is used to update the first position of the most recent record by using the position of the largest feature value if the largest feature value is greater than the feature value corresponding to the first position of the most recent record.

[0463] The jigsaw puzzle module is used to separate the fused features row by row by the second separation module in the vertical stitching line prediction module, and then fuse the multiple separated row features according to the channel dimension to obtain a single row feature, which is then input into the second processing module in the vertical stitching line prediction module. The second processing module performs multiple channel dimensionality reduction processes on the single row feature to obtain the vertical stitching line feature, which is then input into the second output layer in the vertical stitching line prediction module. The second output layer determines the position of the vertical stitching line in the image based on the vertical stitching line feature.

[0464] The jigsaw puzzle module is used to traverse the vertical stitching line feature starting from the first feature value of the vertical stitching line feature with a preset step size using a second window with a second preset interval. Each time the second window moves, the second position of the vertical stitching line feature in the vertical stitching line feature is extracted and recorded based on the feature value contained in the second window. When the traversal ends, for each recorded second position, the position of the vertical stitching line in the image is determined based on the size of the image and the second position.

[0465] The jigsaw puzzle module is used to acquire feature values ​​greater than a threshold within the second window and select the largest feature value from the acquired feature values; determine if there is a recorded second position; if not, record the position of the largest feature value as the second position of a vertical splicing line in the vertical splicing line feature; if there is, compare the interval between the position of the largest feature value and the most recently recorded second position with a second preset interval; if it is greater than the second preset interval, record the position of the largest feature value as the second position of another vertical splicing line in the vertical splicing line feature; if it is less than the second preset interval, update the most recently recorded second position according to the position of the largest feature value.

[0466] The jigsaw puzzle module is used to update the second position of the most recent record by using the position of the largest feature value if the largest feature value is greater than the feature value corresponding to the second position of the most recent record.

[0467] The jigsaw puzzle module is used to fuse the feature map and the splicing line features through the second feature fusion module in the classification network, and input the fused features into the third processing module in the classification network. The third processing module performs multiple channel dimensionality reduction processes on the fused features to obtain features of a preset channel dimension and inputs them into the third output layer in the classification network. The third output layer determines whether the image is a spliced ​​image containing the target content based on the features of the preset channel dimension.

[0468] The model training module is further configured to acquire a first stitched image containing the target content as a positive sample, and to annotate the horizontal stitching line position and / or vertical stitching line position in the first stitched image, as well as the first category label corresponding to the third image category; acquire a second stitched image and / or an unstitched image that does not contain the target content as a negative sample, and to annotate the horizontal stitching line position and / or vertical stitching line position in the second stitched image, and to annotate the second stitched image and the unstitched image with the second category label corresponding to the third image category; and train the constructed recognition model using the positive and negative samples.

[0469] The device also includes: a text recognition module, used to determine that the probability of the image to be processed belonging to a fourth image category is greater than the preset probability threshold corresponding to the fourth image category, wherein the fourth image category is an image category that includes text; to recognize the text information in the image to be processed through an optical character recognition model; to segment the text information to obtain one or more keywords; to determine whether a preset dictionary contains at least one keyword; if so, to determine that the image to be processed belongs to the fourth image category containing the target content.

[0470] The model training module is also used to obtain a fourth training set, which includes multiple images labeled with the target person and multiple images that do not contain the target person; and to train a person classification model based on the fourth training set.

[0471] The image classification device and the image classification method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0472] This application also provides an electronic device for performing the image classification method described above. Please refer to... Figure 25 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 25 As shown, the electronic device 8 includes: a processor 800, a memory 801, a bus 802, and a communication interface 803. The processor 800, the communication interface 803, and the memory 801 are connected via the bus 802. The memory 801 stores a computer program that can run on the processor 800. When the processor 800 runs the computer program, it executes the image classification method provided in any of the foregoing embodiments of this application.

[0473] The memory 801 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 803 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0474] Bus 802 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs. After receiving an execution instruction, the processor 800 executes the program. The image classification method disclosed in any of the foregoing embodiments of this application can be applied to the processor 800, or implemented by the processor 800.

[0475] The processor 800 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 800 or by instructions in software form. The processor 800 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 801. Processor 800 reads the information in memory 801 and, in conjunction with its hardware, completes the steps of the above method.

[0476] The electronic device provided in this application embodiment and the image classification method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0477] This application also provides a computer-readable storage medium corresponding to the image classification method provided in the foregoing embodiments. Please refer to... Figure 26 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the image classification method provided in any of the foregoing embodiments.

[0478] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0479] The computer-readable storage medium provided in the above embodiments of this application and the image classification method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0480] It should be noted that:

[0481] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0482] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0483] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0484] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image classification method, characterized in that, include: The system uses a pre-trained face detection model and a beautification model to identify whether the image to be processed is a beautified image. The image to be processed is classified by a pre-trained multi-class classification model to obtain the probability of the image to be processed belonging to each image category. The image to be processed is identified by a pre-trained person classification model to determine whether it contains an image of the target person. After obtaining the probability that the image to be processed belongs to each image category, the method further includes: The probability that the image to be processed belongs to a third image category is greater than a preset probability threshold corresponding to the third image category, wherein the third image category includes stitched images; The image to be processed is input into a trained recognition model, and the feature extraction network in the recognition model is used to extract features from the image to obtain a feature map. Based on the feature map, the splicing line position in the image to be processed is predicted by the splicing line prediction network in the recognition model; Based on the feature map and the splicing line position, the classification network in the recognition model determines whether the image to be processed is a spliced ​​image containing the target content.

2. The method according to claim 1, characterized in that, The step of identifying whether an image to be processed is a beautified image using a pre-trained face detection model and a beautification recognition model includes: The pre-trained face detection model determines whether there are face regions in the image to be processed that meet the preset face classification conditions. If so, extract the face regions that meet the preset face classification conditions from the image to be processed to obtain the corresponding face image; Generate a face-aligned image corresponding to the face image; The face alignment image is identified as a beautified image by a pre-trained beautification recognition model.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Connect a preset number of convolutional modules sequentially; The last convolutional module in the predetermined number of sequentially connected convolutional modules is connected to a predetermined high-efficiency network to obtain the structure of a beauty recognition model for recognizing beautified images; Obtain the first training set; The beauty recognition model is trained based on the first training set.

4. The method according to claim 3, characterized in that, The acquisition of the first training set includes: Acquire multiple beautified facial images and multiple unbeautified facial images; Generate a face alignment image corresponding to each acquired face image; Label each beautified face-aligned image with a beautification label, and label each unbeautified face-aligned image with an unbeautified label to obtain the first training set; and / or, In the beautified face alignment image, beautification labels are marked on the beautified face parts to obtain the first training set.

5. The method according to claim 3, characterized in that, The step of training the beauty recognition model based on the first training set includes: Obtain face-aligned images from the first training set; The acquired face-aligned image is input into the beauty recognition model to obtain the classification result of the face image corresponding to the face-aligned image; Based on the classification results of the face image, the loss value for the current training period is calculated using a preset classification-center loss function.

6. The method according to claim 5, characterized in that, The classification results include the probability that a face image belongs to each image category, and each image category includes at least beautified images; The step of calculating the loss value for the current training period based on the classification result of the face image using a preset classification-center loss function includes: Based on the number of image categories and the probability that a face image belongs to each image category, the first classification loss value corresponding to the face image is calculated using the first classification loss function included in the preset classification-center loss function; Based on the feature vectors and feature center points of each image category corresponding to the face image, the center loss value corresponding to the face image is calculated using the center loss function included in the classification-center loss function; Calculate the loss value for the current training period based on the first classification loss value and the center loss value.

7. The method according to claim 6, characterized in that, The classification results also include the probability that an image with at least one facial feature belongs to each image category; The step of calculating the loss value for the current training period based on the classification result of the face image using a preset classification-center loss function includes: Based on the number of the at least one face part, the number of image categories, and the probability that each face part belongs to each image category, the second classification loss value corresponding to the face image is calculated using the second classification loss function included in the classification-center loss function; The loss value for the current training period is calculated based on the first classification loss value, the second classification loss value, and the center loss value.

8. The method according to claim 5, characterized in that, The classification results include the probability that an image containing at least one facial feature belongs to each image category; The step of calculating the loss value for the current training period based on the classification result of the face image using a preset classification-center loss function includes: Based on the number of the at least one face part, the number of image categories, and the probability that each face part belongs to each image category, the second classification loss value corresponding to the face image is calculated using the second classification loss function included in the classification-center loss function; Based on the feature vectors and feature center points of each image category corresponding to the face image, the center loss value corresponding to the face image is calculated using the center loss function included in the classification-center loss function; The loss value for the current training period is calculated based on the second classification loss value and the center loss value.

9. The method according to claim 1, characterized in that, Before classifying the image to be processed using a pre-trained multi-class classification model, the method further includes: Obtain a second training set, which includes multiple images, each labeled with a category label corresponding to its respective category. By connecting the output of the pre-defined high-efficiency network to the input of the bidirectional long short-term memory network, a first neural network model for multi-label classification is obtained. The first neural network model is trained using the second training set to obtain a multi-class classification model.

10. The method according to claim 9, characterized in that, After training the constructed first neural network model based on the second training set to obtain a multi-class classification model, the process further includes: The accuracy of the multi-class classification model in classifying each image category is determined respectively; If there is a first image category with an accuracy lower than a preset threshold, then train the branch model corresponding to the first image category; The multi-class classification model is modified using the branching model to obtain the modified multi-class classification model.

11. The method according to claim 10, characterized in that, The step of classifying the image to be processed using a pre-trained multi-class classification model to obtain the probability of the image belonging to each image category includes: The feature vector of the image to be processed is extracted by the preset high-efficiency network in the modified multi-class classification model. The feature vector is classified by the branch model in the modified multi-class classification model to obtain the first probability that the image to be processed belongs to the first image category. The feature vector is classified by the bidirectional long short-term memory network in the modified multi-class classification model to obtain the probability that the image to be processed belongs to each image category, including the second probability that the image to be processed belongs to the first image category. The first probability and the second probability are fused to obtain the final probability that the image to be processed belongs to the first image category.

12. The method according to any one of claims 1, 9-11, characterized in that, After obtaining the probability that the image to be processed belongs to each image category, the process further includes: The probability that the image to be processed belongs to the second image category is greater than the preset probability threshold corresponding to the second image category, where the second image category is an image category that includes body parts; The face detection model is used to detect whether the image to be processed contains a face region that meets the preset face classification conditions. If the image to be processed contains a face region that meets the preset face classification conditions, then the beautification recognition model is used to identify whether the image of the face region in the image to be processed is a beautified image; If it is determined that the image of the face region in the image to be processed is a beautified image, then it is determined that the image to be processed does not belong to the second image category.

13. The method according to claim 1, characterized in that, The step of predicting the position of the stitching line in the image to be processed based on the feature map using the stitching line prediction network in the recognition model includes: The first intermediate feature, the second intermediate feature, and the third intermediate feature generated by the feature extraction network during the feature extraction process are obtained through the splicing line prediction network. The first feature fusion module in the splicing line prediction network performs feature fusion on the feature map, the first intermediate feature, the second intermediate feature, and the third intermediate feature, and outputs the result. The horizontal stitching line prediction module uses the fused features output by the first feature fusion module to predict the position of the horizontal stitching line in the image to be processed. The vertical stitching line prediction module uses the fused features output by the first feature fusion module to predict the position of the vertical stitching line in the image to be processed.

14. The method according to claim 13, characterized in that, The horizontal stitching line prediction module uses the fused features output by the first feature fusion module to predict the position of the horizontal stitching line in the image to be processed, including: The first separation module in the horizontal splicing line prediction module separates the fused features by column, and then fuses the multiple separated column features according to the channel dimension to obtain a single column feature, which is then input into the first processing module in the horizontal splicing line prediction module. After the first processing module performs multiple channel dimensionality reduction processes on a column feature, the horizontal splicing line feature is obtained and input into the first output layer of the horizontal splicing line prediction module. The first output layer determines the position of the horizontal stitching line in the image based on the horizontal stitching line feature.

15. The method according to claim 13, characterized in that, The vertical stitching line prediction module uses the fused features output by the first feature fusion module to predict the position of the vertical stitching line in the image, including: The second separation module in the vertical splicing line prediction module separates the fused features by row, and then fuses the multiple separated row features according to the channel dimension to obtain a single row feature, which is then input into the second processing module in the vertical splicing line prediction module. After performing multiple channel dimensionality reduction processes on a row feature through the second processing module, the vertical splicing line feature is obtained and input into the second output layer of the vertical splicing line prediction module. The second output layer determines the position of the vertical stitching line in the image based on the vertical stitching line feature.

16. The method according to claim 1, characterized in that, The step of determining whether the image to be processed is a spliced ​​image containing target content based on the feature map and the splicing line position, using the classification network in the recognition model, includes: The feature map and the splicing line feature are fused by the second feature fusion module in the classification network, and the fused feature is input into the third processing module of the classification network. The third processing module performs multiple channel dimensionality reduction processes on the fused features to obtain features with a preset channel dimension, which are then input into the third output layer of the classification network. The third output layer determines whether the image is a stitched image containing the target content based on features of a preset channel dimension.

17. The method according to any one of claims 1, 9-11, characterized in that, After obtaining the probability that the image to be processed belongs to each image category, the process further includes: The probability that the image to be processed belongs to the fourth image category is greater than the preset probability threshold corresponding to the fourth image category, wherein the fourth image category is an image category that includes text; The text information in the image to be processed is identified using an optical character recognition model; The text information is segmented into words to obtain one or more keywords; Determine whether the preset dictionary contains at least one of the aforementioned keywords; If so, the image to be processed is determined to belong to the fourth image category containing the target content.

18. An image classification device, characterized in that, include: The beauty recognition module is used to identify whether the image to be processed is a beauty-enhanced image by using a pre-trained face detection model and a beauty recognition model. The multi-class classification module is used to classify the image to be processed using a pre-trained multi-class classification model, and obtain the probability that the image to be processed belongs to each image category. The target person recognition module is used to identify whether the image to be processed contains the target person through a pre-trained person classification model; After obtaining the probability that the image to be processed belongs to each image category, the method further includes: The probability that the image to be processed belongs to a third image category is greater than a preset probability threshold corresponding to the third image category, wherein the third image category includes stitched images; The image to be processed is input into a trained recognition model, and the feature extraction network in the recognition model is used to extract features from the image to obtain a feature map. Based on the feature map, the splicing line position in the image to be processed is predicted by the splicing line prediction network in the recognition model; Based on the feature map and the splicing line position, the classification network in the recognition model determines whether the image to be processed is a spliced ​​image containing the target content.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-17.

Images