Content risk control processing method and device based on clothing identification and computing equipment

Abstract

The invention discloses a content risk control processing method and device based on costume identification and computing equipment. The method comprises the following steps: acquiring a costume image in to-be-processed content; inputting the costume image into a trained costume identification model, so that the costume identification model extracts target features of each dimension of the costume image through a feature extraction network of each dimension, and fusing the target features of each dimension through a feature derivation network and obtaining prediction classification information based on the fused features; and performing content risk control processing on the to-be-processed content according to the prediction classification information. Through the above mode, the feature information of the multiple dimensions of the costume image is extracted for fusion classification, so that more accurate costume identification can be realized, and the content risk control capability can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a content risk control processing method, apparatus, computing device, computer storage medium and computer program product based on clothing recognition. Background Art

[0002] At present, the forms of information dissemination on the Internet are diverse, covering a variety of media such as text, video, pictures, and a combination of text and pictures. Clothing images with negative connotations appearing on the Internet will cause the spread of bad values ​​and endanger network information security. Therefore, it is necessary to conduct content risk control from the clothing dimension to prevent clothing images with negative connotations from spreading on the Internet, thereby guiding the spread of correct values.

[0003] Due to the wide variety of clothing, different styles, complex structures, rich semantic attributes, and many changes in small details such as clothing accessories and patterns, it is difficult to distinguish different types of clothing. In addition, clothing of the same category also varies in color and style, resulting in large intra-class variance. A single feature is difficult to distinguish, resulting in low clothing recognition accuracy. Currently commonly used clothing recognition methods are difficult to achieve ideal recognition effects. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a content risk control processing method, apparatus, computing device, computer storage medium and computer program product based on clothing recognition that overcome the above problems or at least partially solve the above problems.

[0005] According to one aspect of the present application, a content risk control processing method for clothing recognition is provided, comprising:

[0006] Obtain clothing images in the content to be processed;

[0007] Input the clothing image into the trained clothing recognition model, so that the clothing recognition model can extract the target features of each dimension of the clothing image through the feature extraction network of each dimension, fuse the target features of each dimension through the feature derivation network, and obtain the prediction classification information based on the fused features;

[0008] Based on the predicted classification information, content risk control is performed on the content to be processed.

[0009] Optionally, the feature extraction network of each dimension includes: a pre-trained network and an intermediate processing network; the intermediate processing network includes multiple fine-tuning networks and a feature fusion network;

[0010] The multiple fine-tuning networks are respectively connected to the multiple encoding layers of the pre-trained network to perform fine-tuning processing on the initial features output by the multiple encoding layers to obtain multiple fine-tuning features;

[0011] The feature fusion network is connected to multiple fine-tuning networks to fuse multiple fine-tuning features to obtain the target features of this dimension.

[0012] Optionally, the initial features output by the multiple encoding layers include: bottom-level features, middle-level features, and high-level features.

[0013] Optionally, before inputting the clothing image into the trained clothing recognition model, the method further comprises:

[0014] In the process of training the feature extraction network of each dimension in the clothing recognition model, the network parameters of the pre-trained network are fixed, and the network parameters of the intermediate processing network of the dimension are fine-tuned according to the loss function of the dimension;

[0015] In the process of training the clothing recognition model as a whole, the network parameters of the pre-trained network in the feature extraction network of each dimension are fixed, and the network parameters of the intermediate processing network and the feature derivation network in the feature extraction network of each dimension are fine-tuned according to the joint loss function;

[0016] Among them, the joint loss function is determined according to the loss functions of each dimension.

[0017] Optionally, the multiple dimensions include: overall clothing dimension, head accessories dimension, upper body clothing dimension and lower body clothing dimension.

[0018] Optionally, performing content risk control processing on the content to be processed according to the predicted classification information further includes:

[0019] Determine whether the predicted classification information belongs to the control classification list;

[0020] If so, conduct risk control management on the content to be processed;

[0021] If not, release the pending content.

[0022] According to another aspect of the present application, a content risk control processing device based on clothing recognition is provided, comprising:

[0023] An acquisition module, adapted to acquire clothing images in the content to be processed;

[0024] The recognition module is suitable for inputting the clothing image into the trained clothing recognition model, so that the clothing recognition model can respectively extract the target features of each dimension of the clothing image through the feature extraction network of each dimension, fuse the target features of each dimension through the feature derivation network, and obtain the prediction classification information based on the fused features;

[0025] The processing module is suitable for performing content risk control processing on the content to be processed based on the predicted classification information.

[0026] Optionally, the feature extraction network of each dimension includes: a pre-trained network and an intermediate processing network; the intermediate processing network includes multiple fine-tuning networks and a feature fusion network;

[0027] The multiple fine-tuning networks are respectively connected to the multiple encoding layers of the pre-trained network to perform fine-tuning processing on the initial features output by the multiple encoding layers to obtain multiple fine-tuning features;

[0028] The feature fusion network is connected to multiple fine-tuning networks to fuse multiple fine-tuning features to obtain the target features of this dimension.

[0029] Optionally, the initial features output by the multiple encoding layers include: bottom-level features, middle-level features, and high-level features.

[0030] Optionally, the device further comprises:

[0031] The training module is suitable for fixing the network parameters of the pre-trained network and fine-tuning the network parameters of the intermediate processing network of the dimension according to the loss function of the dimension during the training of the feature extraction network of each dimension in the clothing recognition model; and fixing the network parameters of the pre-trained network in the feature extraction network of each dimension and fine-tuning the network parameters of the intermediate processing network and the feature derivation network in the feature extraction network of each dimension according to the joint loss function during the training of the clothing recognition model as a whole;

[0032] Among them, the joint loss function is determined according to the loss functions of each dimension.

[0033] Optionally, the multiple dimensions include: overall clothing dimension, head accessories dimension, upper body clothing dimension and lower body clothing dimension.

[0034] Optionally, the processing module is further adapted to:

[0035] Determine whether the predicted classification information belongs to the control classification list;

[0036] If so, conduct risk control management on the content to be processed;

[0037] If not, release the pending content.

[0038] According to another aspect of the present application, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0039] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned content risk control processing method based on clothing recognition.

[0040] According to another aspect of the present application, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned content risk control processing method based on clothing recognition.

[0041] According to another aspect of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the above-mentioned content risk control processing method based on clothing recognition.

[0042] According to the content risk control processing method, device, computing device, computer storage medium and computer program product based on clothing recognition provided by the embodiments of the present application, clothing images in the content to be processed are obtained; the clothing images are input into the trained clothing recognition model, so that the clothing recognition model can extract the target features of each dimension of the clothing image through the feature extraction network of each dimension, fuse the target features of each dimension through the feature derivation network and obtain the predicted classification information based on the fused features; according to the predicted classification information, the content to be processed is subjected to content risk control processing. In the above manner, the feature information of multiple dimensions of the clothing image is extracted and fused and classified, which can achieve more accurate clothing recognition, thereby improving the content risk control capability.

[0043] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0045] Figure 1 A flowchart of a content risk control processing method based on clothing recognition provided by an embodiment of the present application is shown;

[0046] Figure 2 A flowchart of a content risk control processing method based on clothing recognition provided by another embodiment of the present application is shown;

[0047] Figure 3 A schematic diagram of the structure of a clothing recognition model provided in an embodiment of the present application is shown;

[0048] Figure 4A functional structure diagram of a content risk control processing device based on clothing recognition provided by an embodiment of the present application is shown;

[0049] Figure 5 A schematic diagram of the structure of a computing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0051] Figure 1 FIG. 1 is a flowchart of a content risk control processing method based on clothing recognition provided by an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0052] Step S110, obtaining clothing images in the content to be processed.

[0053] The method of the embodiment of the present application can be used to perform risk control management on content in information platforms (such as video platforms, social media, news platforms, etc.). Risk control verification can be performed before the content is released to prevent negative content from being released to the information platform. Risk control verification can also be performed after the content is released to prevent negative content from being widely disseminated on the information platform.

[0054] The content to be processed is the content that needs to be verified for risk control. It can be in the form of videos, pictures, content combining pictures and text, content combining videos and text, etc. For the content to be processed, the clothing images contained therein are extracted, the categories of the clothing images are identified, and content risk control processing is performed based on the clothing identification results.

[0055] In step S120, the clothing image is input into the trained clothing recognition model, so that the clothing recognition model extracts target features of each dimension of the clothing image through feature extraction networks of each dimension, fuses the target features of each dimension through a feature inference network, and obtains predicted classification information based on the fused features.

[0056] The clothing recognition model is pre-trained and contains feature extraction networks and feature derivation networks in multiple dimensions. The feature extraction network in each dimension is used to extract the target features of the input image in that dimension, and the feature derivation network is used to fuse the target features output by the feature extraction networks in each dimension, and obtain the predicted classification information of the input image based on the fused features.

[0057] Among them, the multi-dimensional feature extraction network can extract features of multiple different dimensions of clothing, such as the features of the clothing as a whole, the features of special areas of clothing, the features of local areas of clothing, and so on.

[0058] Step S130: performing content risk control on the content to be processed according to the predicted classification information.

[0059] Specifically, it is determined whether the predicted classification information belongs to the control classification list, where the control classification list records each clothing category that needs to be controlled, and the control classification list can be set according to actual business needs. If so, risk control management is performed on the content to be processed; if not, the content to be processed is released.

[0060] If the predicted classification information belongs to the control classification list, it means that the clothing image is information with negative meaning, and the pending content will be subject to risk control management. For example, for pending content that has not been put online, it is prohibited to release the pending content and reject the online request for the pending content. For pending content that has been put online, the clothing images in the pending content whose predicted classification information belongs to the control classification list will be blocked or the entire pending content will be blocked. On the contrary, if the predicted classification information does not belong to the control classification list, it is determined that the pending content has passed the risk control verification and the pending content is released.

[0061] In summary, according to the content risk control processing method based on clothing recognition provided by this embodiment, clothing images in the content to be processed are obtained; the clothing images are input into the trained clothing recognition model, so that the clothing recognition model can extract the target features of each dimension of the clothing image through the feature extraction network of each dimension, and the target features of each dimension are fused through the feature derivation network and the predicted classification information is obtained based on the fused features; according to the predicted classification information, the content to be processed is subjected to content risk control processing. In the above manner, the feature information of multiple dimensions of the clothing image is extracted and fused and classified, which can achieve more accurate clothing recognition, thereby improving the content risk control capability.

[0062] Figure 2 A flowchart of a content risk control processing method based on clothing recognition provided by another embodiment of the present application is shown. The method identifies the clothing category of a clothing image through a clothing recognition model and performs content risk control processing based on the clothing category.

[0063] First, the network structure of the clothing recognition model is introduced. The clothing recognition model contains feature extraction networks and feature derivation networks in multiple dimensions. The feature extraction network in each dimension includes: the feature extraction network in each dimension includes: a pre-trained network and an intermediate processing network; the intermediate processing network includes multiple fine-tuning networks and a feature fusion network; the multiple fine-tuning networks are respectively connected to the multiple encoding layers of the pre-training network, and are used to fine-tune the initial features output by the multiple encoding layers to obtain multiple fine-tuning features; the feature fusion network is connected to the multiple fine-tuning networks, and is used to fuse the multiple fine-tuning features to obtain the target features of the dimension.

[0064] Figure 3 FIG. 2 shows a schematic diagram of the structure of the clothing recognition model provided in the embodiment of the present application. Figure 3 As shown, the clothing recognition model includes 4-dimensional feature extraction networks (i.e., feature extraction networks 1-4) and feature derivation network 5. It should be noted that Figure 3 Only the specific structure of the feature extraction network of one dimension is shown, and the structures of the feature extraction networks of other dimensions are consistent with it.

[0065] The feature extraction network 1 includes a pre-trained network 10 and an intermediate processing network 11.

[0066] The pre-trained network 10 is specifically an image encoder, which includes multiple encoding layers (i.e., encoding layer 101-encoding layer N). The intermediate processing network 11 includes a fine-tuning network 111, a fine-tuning network 112, and a fine-tuning network 113. The fine-tuning network 111, the fine-tuning network 112, and the fine-tuning network 113 are respectively used to fine-tune the initial features output by the encoding layer X, the encoding layer Y, and the encoding layer N. The fine-tuning features output by the fine-tuning network 111, the fine-tuning network 112, and the fine-tuning network 113 are then fused through the feature fusion network 114 to obtain the target features of the dimension.

[0067] Similarly, feature extraction network 2, feature extraction network 3 and feature extraction network 4 follow the same processing method and output a target feature respectively. Feature derivation network 5 fuses the target features output by the four feature extraction networks and then deduces the predicted classification information.

[0068] In an optional manner, the initial features output by the multiple coding layers include: bottom-level features, middle-level features, and high-level features. The bottom-level features are the early coding layers ( Figure 3 The initial features output by the encoding layer X in the pre-trained network retain more detailed information such as color, edge, texture, etc. of the clothing, which is used to distinguish similar clothing categories; the middle-level features are the intermediate encoding layers ( Figure 3The initial features of the encoding layer Y in the pre-trained network are usually high-level semantic information, which can represent the parts or accessories of clothing. The high-level features are the last encoding layer of the pre-trained network ( Figure 3 The initial features output by the encoding layer N in , the high-level features are more advanced abstract information, which can describe the overall information of the clothing. In this way, multiple levels of feature information of the image can be obtained, which can improve the recognition accuracy of the model.

[0069] In an optional manner, each fine-tuning network includes one or more MLP (multi-layer perceptron) networks, and the fine-tuning process can be expressed as:

[0070] x l+1 =F(x l ,W l )

[0071] Among them, W l represents the network parameters in the MLP network. The F(·) function represents the feature calculation process corresponding to the MLP network, including linear calculation and activation function calculation. l represents the initial feature, x l+1 Represents fine-tuning features.

[0072] The initial features of the three layers are processed by the fine-tuning network respectively, and then fused by the feature fusion network. The fusion processing method can be expressed as:

[0073]

[0074] Among them, x represents the fused feature, Concat() represents the fusion method, It represents the features of the underlying features after fine-tuning. It represents the features of the middle-level features after fine-tuning. It represents the features of high-level features after fine-tuning.

[0075] Among them, both the feature fusion network and the feature derivation network contain multiple MLP networks, and the network parameters of these MLP networks are optimized and updated during the model training process.

[0076] like Figure 2 As shown, the method comprises the following steps:

[0077] Step S210, in the process of training the feature extraction network of each dimension in the clothing recognition model, the network parameters of the pre-trained network are fixed, and the network parameters of the intermediate processing network of the dimension are fine-tuned according to the loss function of the dimension.

[0078] In the method of the embodiment of the present application, a multi-stage training method is used to train the clothing recognition model. In the first stage, each feature extraction network is trained separately. In the process of training the feature extraction network of each dimension, the network parameters of the pre-trained network are frozen, and other network structures (i.e., intermediate processing networks) in the feature extraction network except the pre-trained network participate in the training, and the network parameters of the network structures participating in the training are fine-tuned.

[0079] Specifically, the feature extraction network for each dimension is trained according to the training data set of that dimension. During the training process, the feature extraction network also outputs category labels and scores. The loss function value is calculated based on the output results and the training data, and the network parameters of the network structure involved in the training are fine-tuned according to the loss function value.

[0080] In an optional way, in the process of training the feature extraction network of each dimension in the clothing recognition model, focal loss is used as the loss function to optimize the network weight parameters, so that the model pays more attention to samples with clothing classification errors and difficult classification. The model already has good classification ability for samples with very accurate predictions. Reducing the attention to these samples will not affect the model's classification ability for these samples. Using focal loss can improve the prediction ability for these samples, thereby improving the overall effect. The calculation method of focal loss is as follows:

[0081] L focalloss =-(1-p t ) r log(p t )

[0082] Among them, L focalloss represents the loss function of dimension, p t It indicates how close the output result is to the training data label, r is an adjustable factor, and optionally, when the loss function value is minimized, the training of the feature extraction network is stopped.

[0083] In an optional manner, the multiple dimensions include the overall clothing dimension and the local clothing dimension, and the local clothing dimension includes: head accessories dimension, upper body clothing dimension, and lower body clothing dimension. That is, one feature extraction network is used to extract features of the overall clothing area, and the other three feature extraction networks are used to extract features of the local clothing area, and the four feature extraction networks focus on the overall features of the clothing, head accessories features, upper body features, and lower body features.

[0084] Among them, the training data corresponding to the overall clothing dimension is the overall clothing image, and the training data corresponding to the local clothing dimension is the local image of the clothing image.

[0085] Through the above method, the global regional features of clothing are obtained through a feature extraction network, and the more fine-grained local regional features of clothing are obtained through three feature extraction networks in parallel. The global features and local features obtained by the four feature extraction networks are fused and classified, which can achieve more accurate clothing recognition.

[0086] In an optional manner, a training data set for training a feature extraction network of a local clothing dimension is constructed in the following manner: based on the annotation information of the sample clothing images in the global clothing data set, images of the local clothing dimension are cropped from the sample clothing images, and the cropped local images are used to construct a training data set of the local clothing dimension (i.e., the local clothing data set), and the local clothing data set can be stored in a folder of the corresponding dimension.

[0087] Among them, the sample clothing images in the global clothing dataset are labeled by categories to obtain labels in four dimensions. Specifically: labeling is performed from the overall clothing dimension, head accessories dimension, upper body clothing dimension, and lower body clothing dimension to obtain clothing category labels, head accessories labels, upper body clothing labels, and lower body clothing labels of the sample clothing images.

[0088] For example, the label of a sample clothing image is {"classLabel": 1, labelHead:{"class": 10, "box": [x1, y1, h1, w1]}, labelBody:{"class": 11, "box": [x2, y2, h2, w2]}, labelTail:{"class": 12, "box": [x3, y3, h3, w3]}}. Among them, classLabel represents the clothing category label, and the value behind it is 1, which represents the specific clothing category. Different clothing categories correspond to different values. LabelHead, labelBody, and labelTail represent the head accessories label, upper body clothing label, and lower body clothing label. They divide the overall clothing into three more detailed labels, and the values ​​behind them are the corresponding specific clothing expressions. class represents the category of the corresponding label, box represents the position of the corresponding label in the sample clothing image, xi and yi represent the center point position of the label, and hi and wi represent the width and height of the corresponding label in the sample clothing image.

[0089] In an optional manner, initial sample clothing images are collected through an external public system and an internal system. For the collected initial sample clothing images, the image encoder in the CLIP model is used to extract features from each initial sample clothing image, and the similarity between each pair of initial sample clothing images is calculated. One of the initial sample clothing images is eliminated based on the similarity, specifically, the initial sample clothing images with poor quality between two initial sample clothing images whose similarity reaches a preset value are eliminated, thereby completing the cleaning of the sample data, and the remaining initial sample clothing images constitute a global clothing data set.

[0090] For the global clothing dataset, it is divided into a training dataset and a test dataset according to the ratio. The training dataset is used for training, and the test dataset is used for testing. Similarly, for the local clothing dataset, it is divided into a training dataset and a test dataset according to the ratio. The training dataset is used to train the feature extraction network, and the test dataset is used to test the feature extraction network.

[0091] In an optional method, the global clothing dataset and the local clothing dataset are respectively subjected to image enhancement processing to obtain an expanded global clothing dataset and an expanded local clothing dataset, and training is performed based on the expanded global clothing dataset and the expanded local clothing dataset. Specifically, by randomly changing the brightness, contrast, saturation, and hue in a mixed enhancement method, sample data with different lighting changes is increased. In this way, the problem of too few sample data due to data collection difficulties is solved, and the problem of overfitting and imbalance of positive and negative samples caused by too few sample data is avoided. In addition, the core of the clothing recognition algorithm is to extract features that can express the same category of clothing. By expanding the sample data, the model recognition effect can also be further improved.

[0092] Step S220, in the process of training the entire clothing recognition model, the network parameters of the pre-trained network in the feature extraction network of each dimension are fixed, and the network parameters of the intermediate processing network and the feature derivation network in the feature extraction network of each dimension are fine-tuned according to the joint loss function.

[0093] Among them, the joint loss function is determined according to the loss functions of each dimension.

[0094] In the first stage, the feature extraction network of each dimension is trained separately. After the first stage of training is completed, the second stage of training is entered to carry out the overall training of the clothing recognition model. In the second stage of training, the network parameters of the pre-trained network in the feature extraction network of each dimension are frozen. All network structures in the clothing recognition model except each pre-trained network (i.e., the feature derivation network and the intermediate processing network in the feature extraction network of each dimension) are involved in the training, and the network parameters of the network structures involved in the training are fine-tuned. In addition, the training data used are sample clothing images in the global clothing dataset.

[0095] Each feature extraction network is jointly trained through a joint loss function to optimize the entire network. The joint loss function is expressed as follows:

[0096] L final =aL all +bL head +cL body +dL tail

[0097] Among them, L final represents the joint loss function, L all , L head , L body , L tail They respectively represent the loss function of the overall clothing dimension, the loss function of the head accessories dimension, the loss function of the upper body clothing dimension, and the loss function of the lower body clothing dimension. a, b, c, and d represent the weight factors of the losses of each part, which can be set according to the clothing area concerned by each dimension. For example, the overall clothing dimension pays more attention to the global characteristics of clothing, so the weight factor of the loss function of this dimension can be set to a larger value than other weight factors. By jointly fine-tuning the joint loss function, each feature extraction network can be more focused on the clothing area it is responsible for.

[0098] In the method of the embodiment of the present application, the training of the clothing recognition model is achieved through multi-stage training combined with transfer learning. The network parameters of the pre-trained network always remain unchanged. In this way, the construction of the clothing recognition model can be simplified and the training of the clothing recognition model can be accelerated.

[0099] In an optional way, the pre-trained network is constructed based on the network parameters of the image encoder in the CLIP model. The CLIP model is a multimodal model based on large-scale training, which includes a text encoder and an image encoder. The image encoder in the CLIP model has a strong feature extraction capability through large-scale data training and can extract rich image features and knowledge information. The image encoder is split out, and the network parameters of the image encoder are extracted. The corresponding network structure is constructed based on the extracted network parameters to obtain the pre-trained network as the backbone structure of the entire clothing recognition model.

[0100] Step S230, obtaining clothing images in the content to be processed.

[0101] The content to be processed is the content that needs to be verified for risk control. It can be in the form of videos, pictures, content combining pictures and text, content combining videos and text, etc. For the content to be processed, the clothing images contained therein are extracted, the categories of the clothing images are identified, and content risk control processing is performed based on the clothing identification results.

[0102] In step S240, the clothing image is input into the trained clothing recognition model, so that the clothing recognition model extracts target features of each dimension of the clothing image through feature extraction networks of each dimension, fuses the target features of each dimension through a feature inference network, and obtains predicted classification information based on the fused features.

[0103] The clothing recognition model includes feature extraction networks and feature derivation networks in multiple dimensions. The feature extraction network in each dimension is used to extract the target features of the input image in that dimension, and the feature derivation network is used to fuse the target features output by the feature extraction networks in each dimension, and obtain the predicted classification information of the input image based on the fused features.

[0104] Step S250: performing content risk control on the content to be processed according to the predicted classification information.

[0105] Specifically, it is determined whether the predicted classification information belongs to the control classification list, where the control classification list records each clothing category that needs to be controlled, and the control classification list can be set according to actual business needs. If so, risk control management is performed on the content to be processed; if not, the content to be processed is released.

[0106] In summary, according to the content risk control processing method based on clothing recognition provided by this embodiment, a method for realizing content risk control recognition based on a multi-feature fine-grained classification method is provided, and a new clothing recognition model is constructed by using the image encoder of the multimodal large model, namely the CLIP model, as the base network. A feature extraction network with multiple branches is constructed to focus on the global features and local features of clothing, and the bottom-level features, middle-level features and high-level features extracted by the pre-trained network in each branch are extracted, fine-tuned and then fused, so as to obtain more stable features of different granularities and different levels, and the target features obtained by fusion of multiple branches are fused together to derive clothing categories, so as to fully obtain and utilize the global features and minute detail features of clothing, thereby improving the accuracy of clothing recognition; further, the clothing recognition model is trained by a multi-stage transfer learning method, and joint training is performed by a joint loss function, so as to accelerate the model training process, further improve the model recognition effect, and obtain better recognition effect with less sample data.

[0107] During specific implementation, a computer terminal can be used to build the network structure in the embodiment of the present application, and a clothing recognition model can be obtained through training. The clothing recognition model is then deployed on various intelligent terminals such as servers and FPGAs, and clothing images are recognized in real time through the clothing recognition model.

[0108] Figure 4 The functional structure diagram of the content risk control processing device based on clothing recognition provided by the embodiment of the present application is shown. Figure 4 As shown, the device comprises:

[0109] An acquisition module 41, adapted to acquire clothing images in the content to be processed;

[0110] The recognition module 42 is adapted to input the clothing image into the trained clothing recognition model, so that the clothing recognition model can extract target features of each dimension of the clothing image through feature extraction networks of each dimension, fuse the target features of each dimension through a feature derivation network, and obtain prediction classification information based on the fused features;

[0111] The processing module 43 is adapted to perform content risk control processing on the content to be processed according to the predicted classification information.

[0112] In an optional manner, the feature extraction network of each dimension includes: a pre-trained network and an intermediate processing network; the intermediate processing network includes multiple fine-tuning networks and a feature fusion network;

[0113] The multiple fine-tuning networks are respectively connected to the multiple encoding layers of the pre-trained network to perform fine-tuning processing on the initial features output by the multiple encoding layers to obtain multiple fine-tuning features;

[0114] The feature fusion network is connected to multiple fine-tuning networks to fuse multiple fine-tuning features to obtain the target features of this dimension.

[0115] In an optional manner, the initial features output by the multiple encoding layers include: bottom-level features, middle-level features, and high-level features.

[0116] In an optional manner, the device further comprises:

[0117] The training module is suitable for fixing the network parameters of the pre-trained network and fine-tuning the network parameters of the intermediate processing network of the dimension according to the loss function of the dimension during the training of the feature extraction network of each dimension in the clothing recognition model; and fixing the network parameters of the pre-trained network in the feature extraction network of each dimension and fine-tuning the network parameters of the intermediate processing network and the feature derivation network in the feature extraction network of each dimension according to the joint loss function during the training of the clothing recognition model as a whole;

[0118] Among them, the joint loss function is determined according to the loss functions of each dimension.

[0119] In an optional manner, the multiple dimensions include: an overall clothing dimension, a head accessories dimension, an upper body clothing dimension, and a lower body clothing dimension.

[0120] In an optional manner, the processing module 43 is further adapted to:

[0121] Determine whether the predicted classification information belongs to the control classification list;

[0122] If so, conduct risk control management on the content to be processed;

[0123] If not, release the pending content.

[0124] In summary, according to the clothing recognition-based content risk control processing device provided in this embodiment, by extracting feature information of multiple dimensions of clothing images for fusion classification, more accurate clothing recognition can be achieved, thereby improving content risk control capabilities.

[0125] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, which enables a processor to perform operations corresponding to the content risk control processing method based on clothing recognition in any of the above-mentioned method embodiments.

[0126] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, which can enable a processor to perform operations corresponding to the content risk control processing method based on clothing recognition in any of the above-mentioned method embodiments.

[0127] Figure 5 A schematic diagram of the structure of a computing device provided in an embodiment of the present application is shown, and the specific embodiment of the present application does not limit the specific implementation of the computing device.

[0128] like Figure 5 As shown, the computing device may include: a processor (processor) 502 , a communications interface (Communications Interface) 504 , a memory (memory) 506 , and a communication bus 508 .

[0129] The processor 502, the communication interface 504, and the memory 506 communicate with each other via the communication bus 508. The communication interface 504 is used to communicate with other devices such as a client or other server network elements. The processor 502 is used to execute the program 510, which can specifically execute the relevant steps in the embodiment of the content risk control processing method based on clothing recognition for computing devices.

[0130] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0131] The processor 502 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0132] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0133] The program 510 can be specifically used to enable the processor 502 to execute the content risk control processing method based on clothing recognition in any of the above method embodiments. The specific implementation of each step in the program 510 can refer to the corresponding description of the corresponding steps and units in the content risk control processing embodiment based on clothing recognition, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0134] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application.

[0135] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0136] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0137] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0138] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0139] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0140] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A content risk control processing method based on clothing recognition, characterized in that: Methods include: Obtain clothing images in the content to be processed; Inputting the clothing image into a trained clothing recognition model, so that the clothing recognition model extracts target features of each dimension of the clothing image through feature extraction networks of each dimension, fuses the target features of each dimension through a feature derivation network, and obtains prediction classification information based on the fused features; The content to be processed is subjected to content risk control processing according to the predicted classification information.

2. The method according to claim 1, characterized in that: The feature extraction network of each dimension includes: a pre-training network and an intermediate processing network; the intermediate processing network includes multiple fine-tuning networks and a feature fusion network; The multiple fine-tuning networks are respectively connected to the multiple encoding layers of the pre-trained network, and are used to perform fine-tuning processing on the initial features output by the multiple encoding layers to obtain multiple fine-tuning features; The feature fusion network is connected to the multiple fine-tuning networks and is used to fuse the multiple fine-tuning features to obtain the target feature of the dimension.

3. The method according to claim 2, characterized in that The initial features output by the multiple encoding layers include: low-level features, middle-level features, and high-level features.

4. The method according to claim 2, characterized in that: Before inputting the clothing image into the trained clothing recognition model, the method further comprises: In the process of training the feature extraction network of each dimension in the clothing recognition model, the network parameters of the pre-trained network are fixed, and the network parameters of the intermediate processing network of the dimension are fine-tuned according to the loss function of the dimension; In the process of training the clothing recognition model as a whole, the network parameters of the pre-trained network in the feature extraction network of each dimension are fixed, and the network parameters of the intermediate processing network and the feature derivation network in the feature extraction network of each dimension are fine-tuned according to the joint loss function; The joint loss function is determined according to the loss functions of each dimension.

5. The method according to any one of claims 1 to 4, characterized in that The multiple dimensions include: overall clothing dimension, head accessories dimension, upper body clothing dimension, and lower body clothing dimension.

6. The method according to any one of claims 1 to 5, characterized in that The performing content risk control processing on the content to be processed according to the predicted classification information further includes: Determine whether the predicted classification information belongs to the control classification list; If yes, perform risk control management on the pending content; If not, release the pending content.

7. A content risk control processing device based on clothing recognition, characterized in that: The device includes: An acquisition module, adapted to acquire clothing images in the content to be processed; A recognition module, adapted to input the clothing image into a trained clothing recognition model, so that the clothing recognition model can respectively extract target features of each dimension of the clothing image through feature extraction networks of each dimension, fuse the target features of each dimension through a feature derivation network, and obtain prediction classification information based on the fused features; The processing module is adapted to perform content risk control processing on the content to be processed according to the predicted classification information.

8. A computing device, characterized in that include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the content risk control processing method based on clothing recognition as described in any one of claims 1-6.

9. A computer storage medium, characterized in that The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the content risk control processing method based on clothing recognition as described in any one of claims 1-6.

10. A computer program product, characterized in that It includes at least one executable instruction, which enables the processor to execute operations corresponding to the content risk control processing method based on clothing recognition as described in any one of claims 1-6.

Images