Content auditing model training method and device, equipment and storage medium

Abstract

The invention relates to a content auditing model training method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps of obtaining training data sets corresponding to a plurality of content auditing tasks respectively; based on the training data set, training the initial content auditing model, adjusting global multi-modal alignment parameters in the initial content auditing model until a first convergence condition is met, and obtaining an intermediate content auditing model; based on an intermediate content auditing model, grouping the plurality of content auditing tasks to obtain a plurality of task groups; and based on the training data set, training the initial content auditing model, adjusting intra-group multi-modal alignment parameters corresponding to each task group in the initial content auditing model, adjusting task multi-modal alignment parameters corresponding to each content auditing task in the initial content auditing model until a second convergence condition is met, and obtaining a target content auditing model. By adopting the method, the model accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a content review model training method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, the content creation industry has ushered in a new era of rapid development. Both individuals and enterprises are using various forms of content to express themselves, convey information, and shape their brand image. In this process, high-quality content has become a key factor in attracting user attention and enhancing interactivity.

[0003] In order to improve the quality of content, the content can be audited and low-quality content can be filtered out. The content audit model can be trained to perform content audit. At present, multi-task learning is usually performed directly on the content audit model based on the training data set. However, multi-task learning is difficult and it is difficult to train a content audit model with high accuracy. Summary of the invention

[0004] Based on this, it is necessary to provide a content review model training method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of the model in response to the above-mentioned technical problems.

[0005] This application provides a content review model training method, including:

[0006] Obtain training data sets corresponding to multiple content review tasks respectively; the training data sets include training samples and training labels corresponding to the training samples, the training samples include training images and training texts corresponding to the training images, and the training labels are content review labels corresponding to the training images;

[0007] Inputting a plurality of training samples into an initial content review model to obtain first prediction labels corresponding to the plurality of training samples, and adjusting a global multimodal alignment parameter in the initial content review model based on the first prediction labels and training labels corresponding to the plurality of training samples until a first convergence condition is met, thereby obtaining an intermediate content review model;

[0008] Inputting a plurality of training samples into the intermediate content review model to obtain second prediction labels corresponding to the plurality of training samples, and grouping the plurality of content review tasks based on the second prediction labels and training labels corresponding to the plurality of training samples to obtain a plurality of task groups;

[0009] Input multiple training samples into the initial content review model to obtain third prediction labels corresponding to the multiple training samples, and obtain task group losses corresponding to the multiple task groups based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same task group, and obtain task losses corresponding to the multiple content review tasks based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same content review task;

[0010] Based on the task group loss, adjust the intra-group multimodal alignment parameters corresponding to the corresponding task group in the initial content audit model; based on the task loss, adjust the task multimodal alignment parameters corresponding to the corresponding content audit task in the initial content audit model until the second convergence condition is met to obtain the target content audit model.

[0011] The present application also provides a content review model training device, comprising:

[0012] A data set acquisition module is used to acquire training data sets corresponding to multiple content review tasks respectively; the training data sets include training samples and training labels corresponding to the training samples, the training samples include training images and training texts corresponding to the training images, and the training labels are content review labels corresponding to the training images;

[0013] A first training module is used to input a plurality of training samples into an initial content review model to obtain first prediction labels corresponding to the plurality of training samples, and adjust a global multimodal alignment parameter in the initial content review model based on the first prediction labels and training labels corresponding to the plurality of training samples until a first convergence condition is met, thereby obtaining an intermediate content review model;

[0014] A task grouping module, used to input a plurality of training samples into the intermediate content review model, obtain second prediction labels corresponding to the plurality of training samples, and group the plurality of content review tasks based on the second prediction labels and training labels corresponding to the plurality of training samples to obtain a plurality of task groups;

[0015] A second training module is used to input multiple training samples into the initial content review model to obtain third prediction labels corresponding to the multiple training samples, and obtain task group losses corresponding to the multiple task groups based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same task group, and obtain task losses corresponding to the multiple content review tasks based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same content review task;

[0016] The second training module is also used to adjust the intra-group multimodal alignment parameters corresponding to the corresponding task group in the initial content review model based on the task group loss, and adjust the task multimodal alignment parameters corresponding to the corresponding content review task in the initial content review model based on the task loss, until the second convergence condition is met to obtain the target content review model.

[0017] The present application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the above-mentioned content review model training method when executing the computer program.

[0018] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the above-mentioned content review model training method.

[0019] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps described in the above-mentioned content review model training method.

[0020] The above-mentioned content audit model training method, device, computer equipment, storage medium and computer program product obtain training data sets corresponding to multiple content audit tasks respectively, and obtain the target content audit model based on the training of multiple training data sets. First, the initial content audit model is trained based on multiple training data sets, and the global multimodal alignment parameters shared by all tasks in the initial content audit model are adjusted during training, so as to obtain an intermediate content audit model, so that the intermediate content audit model has a certain content audit capability. Then, multiple content audit tasks are grouped based on the intermediate content audit model, and similar content audit tasks are classified into one task group to obtain multiple task groups. The initial content audit model is grouped and trained based on multiple training data sets. During training, different task groups each calculate their own losses, and the intra-group multimodal alignment parameters shared by each task in the same task group are adjusted based on the task group losses corresponding to the task groups, so as to strengthen the mutual learning of tasks within the group and improve the complementarity between similar tasks. Different tasks each calculate their own losses, and the task multimodal alignment parameters of the tasks themselves are adjusted based on the task losses corresponding to the tasks, so as to improve the uniqueness of each task, and finally the target content audit model is obtained, so that the target content audit model can have high accuracy on multiple content audit tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] FIG1 is an application environment diagram of a content audit model training method in one embodiment;

[0023] FIG2 is a flow chart of a content review model training method in one embodiment;

[0024] FIG3 is a schematic diagram of task grouping in one embodiment;

[0025] FIG4 is a schematic diagram of joint training of multiple tasks in one embodiment;

[0026] FIG5 is a structural block diagram of a content audit model training device in one embodiment;

[0027] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] The content review model training method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other devices. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0030] It is understandable that the terminal and the server can be used alone to execute the content audit model training method provided in the embodiment of the present application. The terminal and the server can also be used together to execute the content audit model training method provided in the embodiment of the present application.

[0031] For example, the server obtains training data sets corresponding to multiple content audit tasks, trains the initial content audit model based on the multiple training data sets, and obtains the target content audit model. The training data sets include training samples and training labels corresponding to the training samples, the training samples include training images and training texts corresponding to the training images, and the training labels are content audit labels corresponding to the training images.

[0032] For model training, the server inputs multiple training samples into the initial content audit model, obtains first prediction labels corresponding to the multiple training samples, and adjusts the global multimodal alignment parameters in the initial content audit model based on the first prediction labels and training labels corresponding to the multiple training samples until the first convergence condition is met to obtain an intermediate content audit model. The server inputs multiple training samples into the intermediate content audit model, obtains second prediction labels corresponding to the multiple training samples, and groups multiple content audit tasks based on the second prediction labels and training labels corresponding to the multiple training samples to obtain multiple task groups. The server inputs multiple training samples into the initial content audit model, obtains third prediction labels corresponding to the multiple training samples, obtains task group losses corresponding to the multiple task groups based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same task group, and obtains task losses corresponding to the multiple content audit tasks based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same content audit task. The server adjusts the intra-group multimodal alignment parameters corresponding to the corresponding task groups in the initial content audit model based on the task group losses, and adjusts the task multimodal alignment parameters corresponding to the corresponding content audit tasks in the initial content audit model based on the task losses until the second convergence condition is met to obtain a target content audit model.

[0033] The server can store the target content audit model locally, or send the target content audit model to other devices for storage. The terminal sends a content audit request to the server, so that the server processes the content audit request based on the target content audit model.

[0034] In one embodiment, as shown in FIG2 , a content audit model training method is provided, and the method is applied to a computer device as an example, and the computer device can be a terminal or a server. It can be understood that the method can be executed by the terminal or the server itself alone, or can be implemented through the interaction between the terminal and the server. Wherein:

[0035] Step S202, obtain training data sets corresponding to multiple content review tasks respectively; the training data sets include training samples and training labels corresponding to the training samples, the training samples include training images and training texts corresponding to the training images, and the training labels are content review labels corresponding to the training images.

[0036] Among them, content review tasks refer to tasks for reviewing published content or content to be published to ensure that the content complies with relevant regulations. There are multiple content review tasks, and different content review tasks correspond to different review objectives. For example, multiple content review tasks include sensitive topic review tasks, illegal topic review tasks, vulgar topic review tasks, and other bad topic review tasks. Content review tasks are used to identify content that does not comply with regulations and rectify the content that does not comply with regulations.

[0037] Each content review task has its own training data set. For any content review task, the training data set includes multiple training samples and training labels corresponding to each of the multiple training samples. The training samples include training images and training texts corresponding to the training images. The training text refers to the text that describes the training image. The training label corresponding to the training sample is the content review label corresponding to the training image. The content review label corresponding to the training image is used to indicate the review result of the training image in the content review task. For example, for a sensitive topic review task, the content review label corresponding to the training image is used to indicate whether the training image belongs to a sensitive topic.

[0038] Specifically, the computer device can obtain training data sets corresponding to multiple content review tasks locally or from other devices, and perform supervised training on the initial content review model based on the multiple training data sets to obtain a target content review model.

[0039] Step S204: Input multiple training samples into the initial content review model to obtain first prediction labels corresponding to the multiple training samples. Based on the first prediction labels and training labels corresponding to the multiple training samples, adjust the global multimodal alignment parameters in the initial content review model until the first convergence condition is met to obtain an intermediate content review model.

[0040] Among them, the content audit model is an artificial intelligence model used to perform content audit on the input content. The initial content audit model refers to the content audit model to be trained. The initial content audit model can be a pre-trained content audit model. For example, based on the pre-training data set, a randomly initialized content audit model can be supervised trained to obtain an initial content audit model. The pre-training data set includes multiple image-text pairs and matching labels corresponding to the multiple image-text pairs. An image-text pair includes an image and a text. The matching labels corresponding to the image-text pair are divided into matching and mismatching. If the matching label is matching, it means that the image and text in the image-text pair match each other. If the matching label is mismatching, it means that the image and text in the image-text pair do not match each other.

[0041] The initial content audit model includes global multimodal alignment parameters. Global multimodal alignment parameters are multimodal alignment parameters shared between various content audit tasks. Multimodal alignment parameters are used to align the two modalities of image and text. Multimodal alignment parameters are learnable parameters during model training.

[0042] It can be understood that for any multimodal alignment parameter, the initial multimodal alignment parameter can be randomly generated. During the model training process, the multimodal alignment parameter is adjusted based on the model loss, and the parameters are continuously adjusted through multiple iterative training until a certain convergence condition is met to obtain the final multimodal alignment parameter. The convergence condition is a condition for judging whether the model has reached convergence. The convergence condition includes but is not limited to at least one of the model loss being greater than a preset loss value, the number of model iterations being greater than a preset number of iterations, and the rate of change of the model loss being less than a preset rate of change.

[0043] The intermediate content audit model is a content audit model obtained by adjusting the global multimodal alignment parameters in the initial content audit model.

[0044] Specifically, the computer device inputs multiple training samples into the initial content review model, obtains the first prediction labels corresponding to each of the multiple training samples, calculates the total model loss based on the first prediction labels and training labels corresponding to the multiple training samples, back-propagates the model loss to adjust the global multimodal alignment parameters in the initial content review model, and iteratively adjusts the global multimodal alignment parameters through iterative training until the first convergence condition is met to obtain an intermediate content review model. For example, if the first convergence condition is that the number of model iterations is 100, the content review model obtained after adjusting the global multimodal alignment parameters for the 100th time is used as the intermediate content review model.

[0045] The initial content review model includes a text encoder and an image encoder. The text encoder is used to process text to extract text features, and the image encoder is used to process images to extract image features. The global multimodal alignment parameters are used to assist the text encoder and image encoder in feature extraction to obtain better text features and image features.

[0046] It can be understood that the multiple training samples include multiple training samples corresponding to each content review task.

[0047] Step S206, input multiple training samples into the intermediate content review model, obtain second prediction labels corresponding to the multiple training samples, and group multiple content review tasks based on the second prediction labels and training labels corresponding to the multiple training samples to obtain multiple task groups.

[0048] Among them, a task group includes at least one content review task.

[0049] Specifically, after obtaining the intermediate content review model, tasks can be grouped based on the intermediate content review model. The computer device can input multiple training samples into the intermediate content review model to obtain second prediction labels corresponding to the multiple training samples, calculate the model loss corresponding to each content review task based on the second prediction labels and training labels corresponding to the multiple training samples, and group each content review task based on the model loss corresponding to each content review task to obtain multiple task groups.

[0050] It can be understood that the multiple training samples input into the initial content review model and the multiple training samples input into the intermediate content review model may include the same training samples or different training samples.

[0051] Step S208, input multiple training samples into the initial content review model, obtain third prediction labels corresponding to the multiple training samples, and obtain task group losses corresponding to the multiple task groups based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same task group. Based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same content review task, obtain task losses corresponding to the multiple content review tasks.

[0052] Step S210: Based on the task group loss, adjust the intra-group multimodal alignment parameters corresponding to the corresponding task group in the initial content review model; based on the task loss, adjust the task multimodal alignment parameters corresponding to the corresponding content review task in the initial content review model until the second convergence condition is met to obtain the target content review model.

[0053] Among them, the task group loss refers to the model loss corresponding to a task group. The task loss refers to the model loss corresponding to a content review task. The initial content review model includes intra-group multimodal alignment parameters corresponding to each task group and task multimodal alignment parameters corresponding to each content review task. It can be understood that the intra-group multimodal alignment parameters are multimodal alignment parameters unique to a single task group, the intra-group multimodal alignment parameters are multimodal alignment parameters shared between various content review tasks within a task group, and the task multimodal alignment parameters are multimodal alignment parameters unique to a single task.

[0054] The target content audit model refers to the content audit model that is finally trained.

[0055] Specifically, the computer device can input multiple training samples into the initial content review model, obtain the third prediction labels corresponding to each of the multiple training samples, calculate the model loss corresponding to the task group based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same task group, and obtain the task group loss corresponding to each task group respectively, calculate the model loss corresponding to the content review task based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same content review task, and obtain the task loss corresponding to each content review task respectively. For example, a task group includes content review task A and content review task B. Based on the training labels and third prediction labels corresponding to each training sample belonging to content review task A and the training labels and third prediction labels corresponding to each training sample belonging to content review task B, the task group loss corresponding to the task group is calculated, based on the training labels and third prediction labels corresponding to each training sample belonging to content review task A, the task loss corresponding to content review task A is calculated, and based on the training labels and third prediction labels corresponding to each training sample belonging to content review task B, the task loss corresponding to content review task B is calculated.

[0056] The computer device adjusts the intra-group multimodal alignment parameters corresponding to the corresponding task groups in the initial content audit model based on the losses of each task group, and adjusts the task multimodal alignment parameters corresponding to the corresponding content audit tasks in the initial content audit model based on the losses of each task, and iteratively adjusts the intra-group multimodal alignment parameters and the task multimodal alignment parameters through iterative training until the second convergence condition is met to obtain the target content audit model. For example, if the task group includes task group 1 and task group 2, the intra-group multimodal alignment parameters corresponding to task group 1 in the initial content audit model are adjusted based on the task group loss corresponding to task group 1, and the intra-group multimodal alignment parameters corresponding to task group 2 in the initial content audit model are adjusted based on the task group loss corresponding to task group 2. If the content review task includes content review task 1, content review task 2 and content review task 3, then the task multimodal alignment parameters corresponding to content review task 1 in the initial content review model are adjusted based on the task loss corresponding to content review task 1, the task multimodal alignment parameters corresponding to content review task 2 in the initial content review model are adjusted based on the task loss corresponding to content review task 2, and the task multimodal alignment parameters corresponding to content review task 3 in the initial content review model are adjusted based on the task loss corresponding to content review task 3.

[0057] The initial content review model includes a text encoder and an image encoder. The text encoder is used to process text to extract text features, and the image encoder is used to process images to extract image features. The intra-group multimodal alignment parameters and the task multimodal alignment parameters are used to assist the text encoder and the image encoder in feature extraction to obtain better text features and image features.

[0058] It can be understood that the first convergence condition and the second convergence condition can be different convergence conditions. The initial content review model may include the same training samples or different training samples between the multiple training samples used in the first stage and the multiple training samples used in the second stage. The first stage is used to train to obtain the intermediate content review model, and the second stage is used to train to obtain the target content review model. For example, each training data set includes 1,000 training samples, the first stage uses the first 500 training samples, and the second stage uses the last 500 training samples.

[0059] It can be understood that in the first stage, the input data of the text encoder includes text and global multimodal alignment parameters, and the input data of the image encoder includes images and global multimodal alignment parameters. In the second stage, the input data of the text encoder includes text, intra-group multimodal alignment parameters and task multimodal alignment parameters, and the input data of the image encoder includes images, intra-group multimodal alignment parameters and task multimodal alignment parameters. In the second stage, no global multimodal alignment parameters are required.

[0060] In one embodiment, the convergence difficulty of the first convergence condition is less than the convergence difficulty of the second convergence condition to improve the accuracy of the model. For example, the convergence condition includes the number of model iterations, and the preset number of iterations in the first convergence condition is less than the preset number of iterations in the second convergence condition.

[0061] In one embodiment, the model loss may be calculated based on the predicted labels and training labels corresponding to the plurality of training samples by using a cross entropy loss function, and the multimodal alignment parameters may be adjusted based on the model loss by using a gradient descent algorithm.

[0062] In one embodiment, for any type of multimodal alignment parameters, the multimodal alignment parameters include a multimodal alignment hint and a scaling matrix.

[0063] In the above content audit model training method, training data sets corresponding to multiple content audit tasks are obtained, and the target content audit model is obtained based on the training of the multiple training data sets. First, the initial content audit model is trained based on multiple training data sets, and the global multimodal alignment parameters shared by all tasks in the initial content audit model are adjusted during training, so as to obtain an intermediate content audit model, so that the intermediate content audit model has a certain content audit capability. Then, multiple content audit tasks are grouped based on the intermediate content audit model, and similar content audit tasks are classified into one task group to obtain multiple task groups. The initial content audit model is grouped and trained based on multiple training data sets. During training, different task groups calculate their own losses, and the intra-group multimodal alignment parameters shared by each task in the same task group are adjusted based on the task group losses corresponding to the task groups, so as to strengthen the mutual learning of tasks within the group and improve the complementarity between similar tasks. Different tasks calculate their own losses, and the task multimodal alignment parameters of the tasks themselves are adjusted based on the task losses corresponding to the tasks, so as to improve the uniqueness of each task, and finally the target content audit model is obtained, so that the target content audit model can have high accuracy on multiple content audit tasks.

[0064] In one embodiment, multiple training samples are input into an initial content review model to obtain first prediction labels corresponding to the multiple training samples, including:

[0065] Input the training samples into the initial content review model; the initial content review model includes a text encoder and an image encoder;

[0066] Generate global multimodal alignment parameters shared by text encoder and image encoder;

[0067] Extracting features of training texts in training samples by using a text encoder to obtain initial text features, and obtaining first training text features based on global multimodal alignment parameters and the initial text features;

[0068] Extracting features of the training images in the training samples through an image encoder to obtain initial image features, and obtaining first training image features based on global multimodal alignment parameters and the initial image features;

[0069] Based on the first training text feature and the first training image feature, a first predicted label corresponding to the training sample is obtained.

[0070] Specifically, the computer device inputs the training sample into the initial content audit model, and in the initial content audit model, generates the global multimodal alignment parameters shared by the text encoder and the image encoder, that is, initializes the global multimodal alignment parameters shared by the text encoder and the image encoder. The initial content audit model includes a text encoder and an image encoder. In the initial content audit model, the text encoder is used to extract features of the training text in the training sample to obtain initial text features, and based on the global multimodal alignment parameters and the initial text features, the first training text features are obtained. For example, the global multimodal alignment parameters and the initial text features are concatenated, and the first training text features are obtained based on the concatenation result. In the initial content audit model, the image encoder is used to extract features of the training images in the training sample to obtain initial image features, and based on the global multimodal alignment parameters and the initial image features, the first training image features are obtained. For example, the global multimodal alignment parameters and the initial image features are concatenated, and the first training image features are obtained based on the concatenation result. In the initial content audit model, based on the first training text features and the first training image features, the first predicted label corresponding to the training sample is obtained. For example, features are usually represented by matrices, the outer product of the first training text feature and the first training image feature is calculated, the outer product is processed based on the softmax function (normalized exponential function), and the first predicted label corresponding to the training sample is obtained.

[0071] It can be understood that the generation process of the first prediction label corresponding to the training sample and the generation process of the second prediction label corresponding to the training sample are similar. Specifically, the training sample is input into the intermediate content review model; the training text in the training sample is feature extracted through the text encoder to obtain the initial text feature, and the training text feature is obtained based on the global multimodal alignment parameters and the initial text feature; the training image in the training sample is feature extracted through the image encoder to obtain the initial image feature, and the training image feature is obtained based on the global multimodal alignment parameters and the initial image feature; the second prediction label corresponding to the training sample is obtained based on the training text feature and the training image feature. It can be understood that the global multimodal alignment parameters in the intermediate content review model are the global multimodal alignment parameters obtained by training the initial content review model.

[0072] In one embodiment, the text encoder includes multiple text encoding layers. The training text in the training sample is converted into a text embedding vector, the text embedding vector and the corresponding global multimodal alignment parameter are spliced ​​and input into the first text encoding layer, the feature vector output by the first text encoding layer and the global multimodal alignment parameter are spliced ​​and input into the second text encoding layer, the feature vector output by the second text encoding layer and the global multimodal alignment parameter are spliced ​​and input into the third text encoding layer, and so on, and the feature vector output by the last text encoding layer is used as the first training text feature. Similarly, the image encoder can also adopt a similar method to obtain the first training image feature. It can be understood that there can be one or more global multimodal alignment parameters. If there is only one global multimodal alignment parameter, the same global multimodal alignment parameter is used for each text encoding layer; if there are multiple global multimodal alignment parameters, different global multimodal alignment parameters are used for each text encoding layer.

[0073] In the above embodiment, the initial content review model includes a text encoder and an image encoder. The global multimodal alignment parameters are used to assist the text encoder in feature extraction to obtain a first training text feature. The global multimodal alignment parameters are used to assist the image encoder in feature extraction to obtain a first training image feature. Based on the first training text feature and the first training image feature, a first predicted label corresponding to the training sample is obtained. The parameter amount of the global multimodal alignment parameters is smaller than the parameter amount of the text encoder or the image encoder. During model training, by adjusting the global multimodal alignment parameters, not only can the effect equivalent to adjusting the text encoder or the image encoder be achieved, but also the model training efficiency can be improved.

[0074] In one embodiment, based on the second prediction labels and training labels corresponding to the plurality of training samples, the plurality of content review tasks are grouped to obtain a plurality of task groups, including:

[0075] Based on the second prediction labels and training labels corresponding to multiple training samples belonging to the same content review task, the model losses corresponding to the multiple content review tasks are obtained; the loss gradient of the model loss with respect to the global multimodal alignment parameter is calculated to obtain the loss gradients corresponding to the multiple content review tasks; the content review tasks whose similarity between the loss gradients is greater than the similarity threshold are divided into the same task group to obtain multiple task groups.

[0076] Specifically, when grouping tasks, they can be grouped based on the loss gradient of the model loss.

[0077] The computer device calculates the model loss corresponding to the content review task based on the second prediction labels and training labels corresponding to multiple training samples belonging to the same content review task, and obtains the model losses corresponding to the multiple content review tasks. Furthermore, the computer device calculates the loss gradient of the model loss with respect to the global multimodal alignment parameters, obtains the loss gradients corresponding to the multiple content review tasks, calculates the similarity between the loss gradients corresponding to any two content review tasks, and groups the content review tasks whose similarity between the loss gradients is greater than the similarity threshold into the same task group, to obtain multiple task groups. It can be understood that if the loss gradients generated by two tasks during the training process are similar, then the two tasks can be considered to be similar to some extent because they have similar effects on the update of the global multimodal alignment parameters.

[0078] In the above embodiment, tasks are grouped by using the loss gradient of the model loss, so that similar tasks can be grouped into one task group, thereby ensuring the accuracy of task grouping.

[0079] In one embodiment, multiple training samples are input into the initial content review model to obtain third prediction labels corresponding to the multiple training samples, including:

[0080] Input the training samples into the initial content review model; the initial content review model includes a text encoder and an image encoder;

[0081] Generate intra-group multimodal alignment parameters corresponding to multiple task groups and task multimodal alignment parameters corresponding to multiple content review tasks;

[0082] Through the text encoder, feature extraction is performed on the training text in the training sample to obtain initial text features, and second training text features are obtained based on the intra-group multimodal alignment parameters corresponding to the task group to which the training sample belongs, the task multimodal alignment parameters corresponding to the content review task to which the training sample belongs, and the initial text features;

[0083] Perform feature extraction on the training image in the training sample through an image encoder to obtain initial image features, and obtain second training image features based on the intra-group multimodal alignment parameters corresponding to the task group to which the training sample belongs, the task multimodal alignment parameters corresponding to the content review task to which the training sample belongs, and the initial image features;

[0084] Based on the second training text feature and the second training image feature, a third predicted label corresponding to the training sample is obtained.

[0085] Specifically, the computer device inputs the training sample into the initial content audit model, and in the initial content audit model, generates intra-group multimodal alignment parameters corresponding to multiple task groups, and generates task multimodal alignment parameters corresponding to multiple content audit tasks, that is, initializes the intra-group multimodal alignment parameters and task multimodal alignment parameters. It can be understood that these multimodal alignment parameters are also multimodal alignment parameters shared by the text encoder and the image encoder. The initial content audit model includes a text encoder and an image encoder. In the initial content audit model, the training text in the training sample is feature extracted by the text encoder to obtain the initial text feature, and the second training text feature is obtained based on the intra-group multimodal alignment parameters corresponding to the task group to which the training sample belongs, the task multimodal alignment parameters corresponding to the content audit task to which the training sample belongs, and the initial text feature. For example, the intra-group multimodal alignment parameters corresponding to the task group to which the training sample belongs, the task multimodal alignment parameters corresponding to the content audit task to which the training sample belongs, and the initial text feature are spliced, and the second training text feature is obtained based on the splicing result. In the initial content review model, feature extraction is performed on the training images in the training samples through an image encoder to obtain initial image features, and second training image features are obtained based on the intra-group multimodal alignment parameters corresponding to the task group to which the training samples belong, the task multimodal alignment parameters corresponding to the content review task to which the training samples belong, and the initial image features. For example, the intra-group multimodal alignment parameters corresponding to the task group to which the training samples belong, the task multimodal alignment parameters corresponding to the content review task to which the training samples belong, and the initial image features are concatenated, and the second training image features are obtained based on the concatenation result. In the initial content review model, a third predicted label corresponding to the training sample is obtained based on the second training text features and the second training image features.

[0086] In one embodiment, the image encoder includes a plurality of image coding layers, and each image coding layer has its own corresponding intra-group multimodal alignment parameters and task multimodal alignment parameters. The training images in the training samples are converted into image embedding vectors, and the image embedding vectors, the initial image class labels, the intra-group multimodal alignment parameters corresponding to the task group to which the training samples belong, and the task multimodal alignment parameters corresponding to the content review task to which the training samples belong are spliced ​​and input into the first image coding layer, and the feature vector output by the first image coding layer, the updated image class labels output by the first image coding layer, the intra-group multimodal alignment parameters corresponding to the task group to which the training samples belong, and the task multimodal alignment parameters corresponding to the content review task to which the training samples belong are spliced ​​and input into the second image coding layer, and the feature vector output by the second image coding layer, the updated image class labels output by the second image coding layer, the intra-group multimodal alignment parameters corresponding to the task group to which the training samples belong, and the task multimodal alignment parameters corresponding to the content review task to which the training samples belong are spliced ​​and input into the third image coding layer, and so on, and the image class labels output by the last image coding layer are used as the second training image features. Image class labels are used to learn information related to categories in images.

[0087] In one embodiment, the global multimodal alignment parameters include a multimodal alignment hint and a scaling matrix shared by the text encoder and the image encoder. The outer product of the multimodal alignment hint and the scaling matrix is ​​calculated, and the outer product is concatenated with other data.

[0088] The intra-group multimodal alignment parameters include the multimodal alignment hint, the scaling matrix for the text encoder, and the scaling matrix for the image encoder. The task multimodal alignment parameters include the multimodal alignment hint shared by the text encoder and the image encoder, the scaling matrix for the text encoder, and the scaling matrix for the image encoder. In the text encoder, the outer product of the multimodal alignment hint and the scaling matrix for the text encoder is calculated, and the outer product is concatenated with other data. In the image encoder, the outer product of the multimodal alignment hint and the scaling matrix for the image encoder is calculated, and the outer product is concatenated with other data.

[0089] In the above embodiment, the parameter amount of the intra-group multimodal alignment parameter is less than the reference amount of the text encoder or the image encoder, and the parameter amount of the task multimodal alignment parameter is less than the reference amount of the text encoder or the image encoder. During model training, by adjusting the intra-group multimodal alignment parameter and the task multimodal alignment parameter, not only can the effect equivalent to adjusting the text encoder or the image encoder be achieved, but also the model training efficiency can be improved. In addition, the intra-group multimodal alignment parameter helps to strengthen the mutual learning of tasks within the group and improve the complementarity between similar tasks, and the task multimodal alignment parameter helps to improve the uniqueness of each task.

[0090] In one embodiment, obtaining training data sets corresponding to a plurality of content review tasks respectively includes:

[0091] For the current content review task, multiple normal images and multiple abnormal images are obtained as training images respectively; when the training image carries a description text, the description text corresponding to the training image is used as the training text corresponding to the training image; when the training image does not carry a description text, the training image is input into an image description generation model to obtain the training text corresponding to the training image; the training image and the training text corresponding to the training image are used as training samples, and the training label corresponding to the training sample is obtained based on the content review label corresponding to the training image; the content review label corresponding to the normal image is the normal label, and the content review label corresponding to the abnormal image is the abnormal label; based on multiple training samples and the training labels corresponding to the multiple training samples, a training data set corresponding to the current content review task is obtained.

[0092] The current content review task refers to any one of the multiple content review tasks. Each content review task has normal images and abnormal images on its own review target. Normal images refer to images without abnormalities in the review, and abnormal images refer to images with abnormalities in the review. For example, for sensitive topic review tasks, normal images refer to images that do not involve sensitive topics, and abnormal images refer to images that involve sensitive topics.

[0093] It can be understood that some images carry description text and some images do not carry description text. The description text carried by the image is the text describing the image. The image description generation model is an artificial intelligence model used to generate description text for images.

[0094] Specifically, when constructing a training dataset corresponding to a content review task, manual labeling and intelligent models can be combined to generate a data-rich training dataset.

[0095] For the current content review task, the computer device obtains multiple normal images and multiple abnormal images determined by manual review, and uses each normal image and abnormal image as a training image. When the training image carries a description text, the description text corresponding to the training image is used as the training text corresponding to the training image; when the training image does not carry a description text, the training image is input into the image description generation model, the image description generation model outputs the description text corresponding to the training image, and the result output by the image description generation model is used as the training text corresponding to the training image. It can be understood that the content review label corresponding to the normal image determined by manual review is the normal label, and the content review label corresponding to the abnormal image determined by manual review is the abnormal label. The normal label means that the review result of the image on the review target of the content review task is passed, and the abnormal label means that the review result of the image on the review target of the content review task is not passed. For example, for the sensitive topic review task, the normal label means that the image does not belong to a sensitive topic, and the abnormal label means that the image belongs to a sensitive topic. For any training image, the training image and the training text corresponding to the training image are used as training samples, and the training label corresponding to the training sample is obtained based on the content review label corresponding to the training image. According to the plurality of training images, a plurality of training samples and training labels corresponding to the plurality of training samples can be obtained. Finally, the computer device combines the plurality of training samples and the training labels corresponding to the plurality of training samples into a training data set corresponding to the current content review task.

[0096] In the above embodiment, when constructing a training data set, for training images that do not carry descriptive text, a descriptive text for the training image is generated by an image description generation model, and training samples are generated based on the training image and its descriptive text. This can improve the richness of training samples in the training data set, thereby helping to improve the quality of model training.

[0097] In one embodiment, the content audit model training method further includes:

[0098] Acquire an image to be reviewed; input the image to be reviewed into a target content review model to obtain target image features of the image to be reviewed on multiple content review tasks; obtain reference content features corresponding to reference content carrying content review labels, and determine content similarity between the image to be reviewed and the reference content based on feature similarity between the reference content features and multiple target image features; determine target content from multiple reference contents based on content similarity, and obtain a review result corresponding to the image to be reviewed based on the content review label corresponding to the target content.

[0099] The reference content may be a reference image or a reference text. The reference content carries a content review tag. If the reference content is a reference image, the content review tag carried by the reference content is the content review tag corresponding to the reference image. If the reference content is a reference text, the content review tag carried by the reference content is the content review tag corresponding to the image to which the reference text belongs. It can be understood that the content review tag corresponding to the reference content may include the content review tag of the reference content on at least one content review task.

[0100] The reference content has corresponding reference content features, which are obtained by inputting the reference content into the target content audit model. If the reference content is a reference image, the reference image is input into the target content audit model in advance, and the output data of the image encoder is used as the reference content features corresponding to the reference image. If the reference content is a reference text, the reference text is input into the target content audit model in advance, and the output data of the text encoder is used as the reference content features corresponding to the reference text.

[0101] Specifically, after the target content review model is trained, content review of the image to be reviewed can be performed based on the target content review model through feature matching.

[0102] The computer device obtains the image to be reviewed, inputs the image to be reviewed into the target content review model, and the image encoder in the target content review model outputs the target image features of the image to be reviewed on each content review task. For example, the task group includes task group 1 and task group 2, task group 1 includes content review task 1 and content review task 4, and task group 2 includes content review task 2 and content review task 3. Taking task group 1 as an example, in the target content review model, the image to be reviewed is input into the image encoder, and the image encoder is used to extract features of the image to be reviewed to obtain the initial image features. Based on the intra-group multimodal alignment parameters corresponding to task group 1, the task multimodal alignment parameters corresponding to content review task 1, and the initial image features, the target image features of the image to be reviewed on content review task 1 are obtained. Based on the intra-group multimodal alignment parameters corresponding to task group 1, the task multimodal alignment parameters corresponding to content review task 4, and the initial image features, the target image features of the image to be reviewed on content review task 4 are obtained.

[0103] The computer device can determine the reference content features corresponding to multiple reference contents respectively based on the target content review model in advance, and the reference contents carry content review tags. After determining the target image features corresponding to the image to be reviewed, for any reference content features, the computer device can calculate the feature similarities between the reference content features and the target image features of the image to be reviewed on each content review task, and determine the content similarity between the image to be reviewed and the reference content based on each feature similarity. For example, the maximum feature similarity is obtained from each feature similarity as the content similarity. The computer device can obtain the content similarity between the image to be reviewed and each reference content respectively, and then determine the target content from multiple reference contents based on the content similarity. For example, the reference content with the greatest content similarity is obtained as the target content. Finally, the computer device uses the content review tag corresponding to the target content as the review result corresponding to the image to be reviewed.

[0104] In one embodiment, the reference content may be determined based on the training samples in the training data set, and the training images and training texts in the training samples may be used as the reference content respectively.

[0105] In the above embodiment, the image features of the image to be reviewed are extracted through the target content review model, and the target content is determined from the reference content through feature matching between the image to be reviewed and the reference content, and the content review label corresponding to the target content is used as the review result corresponding to the image to be reviewed. The target content review model has high accuracy. Based on the features extracted by the target content review model, the review result corresponding to the image to be reviewed is determined by image search or image search, which can ensure the accuracy of the review result.

[0106] In one embodiment, the content audit model training method further includes:

[0107] Obtain the image to be reviewed; input the image to be reviewed into the target content review model to obtain the predicted label of the image to be reviewed on multiple content review tasks; based on the predicted label, determine the review result corresponding to the image to be reviewed.

[0108] Specifically, after the target content review model is trained, content review of the image to be reviewed can be performed based on the target content review model.

[0109] The computer device obtains the image to be reviewed, inputs the image to be reviewed into the target content review model, obtains the predicted label of the image to be reviewed on multiple content review tasks, and determines the review result corresponding to the image to be reviewed based on the predicted label. For example, the predicted label is a predicted probability. If the predicted probability of the image to be reviewed on a certain content review task is greater than the probability threshold, then the review result of the image to be reviewed on the content review task is determined to be passed. Otherwise, the review result is failed. That is, if the predicted probability of the image to be reviewed on a certain content review task is greater than the probability threshold, then it is determined that the image to be reviewed is a normal image on the content review task. Otherwise, it is determined that the image to be reviewed is an abnormal image on the content review task.

[0110] In the above embodiment, the target content review model has high accuracy. The target content review model is used to determine the predicted labels of the image to be reviewed on multiple content review tasks. Based on the predicted labels, the review results corresponding to the image to be reviewed are determined, thereby ensuring the accuracy of the review results.

[0111] In a specific embodiment, the method of the present application is applied to a content review scenario. In the content review scenario, there will be multiple different content review tasks (referred to as tasks), and content review is performed by training a content review model for multiple tasks. The training process of the content review model is divided into two stages.

[0112] Referring to Figure 3, in the first stage, tasks are grouped by training an initial content moderation model. Initialize the global multimodal alignment prompts for all tasks , train the initial content review model based on the training data set, and adjust the model during training , until the model converges, and the intermediate content review model is obtained. The loss gradient cosine similarity between the i-th task and the j-th task is calculated by formula (1). When , it means that the two tasks can gain each other. The tasks whose loss gradient cosine similarity is greater than 0 are grouped together.

[0113] (1)

[0114] in, Indicates the task of loss, Indicates the task loss.

[0115] After combining similar tasks together, multimodal alignment parameters were designed for each group to strengthen the mutual learning of tasks within the group. At the same time, in order to ensure the uniqueness of each task in its own task, multimodal alignment parameters were also designed for each task separately.

[0116] Referring to Figure 4, the initial content audit model includes a text encoder and an image encoder. The text encoder and the image encoder each include k transformer layers. A matching text-image pair is obtained. The text in the text-image pair is converted into a text embedding vector and input into the text encoder. The image in the text-image pair is converted into an image embedding vector and input into the image encoder. The output data of the model is calculated based on the output data of the text encoder and the image encoder. G is represented as a task group. The intra-group multimodal alignment parameters corresponding to the task group include , , .in, represents a multimodal alignment cue, represents the scaling matrix of the text modality, Represents the scaling matrix of the image modality. Similarly, the task multimodal alignment parameters corresponding to the task include , and The calculation of the kth layer in the text encoder is shown in formula (2), and the calculation of the kth layer in the image encoder is shown in formula (3).

[0117] (2)

[0118] in, represents the text vector output by the kth layer of the text encoder, Represents the text vector output by the k-1th layer of the text encoder. , which means performing matrix multiplication on the multimodal alignment hint and scaling matrix corresponding to the k-th layer on the task group. , which means performing matrix multiplication on the multimodal alignment hint and scaling matrix corresponding to the k-th layer in the task. Indicates that , , Splicing. Represents the data processing of the k-th layer of the text encoder.

[0119] (3)

[0120] in, represents the image vector output by the kth layer of the image encoder, Represents the image vector output by the k-1th layer of the image encoder. represents the image class label output by the kth layer, Represents the image class label output by the k-1th layer. , which means performing matrix multiplication on the multimodal alignment hint and scaling matrix corresponding to the k-th layer on the task group. , which means performing matrix multiplication on the multimodal alignment hint and scaling matrix corresponding to the k-th layer in the task. Indicates that , , , Splicing. Represents the data processing of the kth layer of the image encoder.

[0121] The intra-group multimodal alignment parameters are updated by formula (4), and the task multimodal alignment parameters are updated by formula (5).

[0122] (4)

[0123] (5)

[0124] Initialize the intra-group multimodal alignment parameters for all task groups ( , , ), initialize the task multimodal alignment parameters of all tasks ( , and ), train the initial content audit model based on the training data set, adjust the intra-group multimodal alignment parameters and task multimodal alignment parameters in the model during training until the model converges to obtain the target content audit model.

[0125] By training the model using the above method, the quality of model training can be improved, so that the target content review model obtained through training has higher accuracy. In addition, during the model training process, the efficiency of model training can be improved by adjusting the multimodal alignment parameters instead of adjusting the text encoder and image encoder.

[0126] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0127] Based on the same inventive concept, the embodiment of the present application also provides a content audit model training device for implementing the content audit model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more content audit model training device embodiments provided below can refer to the limitations on the content audit model training method above, and will not be repeated here.

[0128] In one embodiment, as shown in FIG. 5 , a content audit model training device is provided, including: a data set acquisition module 502, a first training module 504, a task grouping module 506, and a second training module 508, wherein:

[0129] The data set acquisition module 502 is used to obtain training data sets corresponding to multiple content review tasks respectively; the training data sets include training samples and training labels corresponding to the training samples, the training samples include training images and training texts corresponding to the training images, and the training labels are content review labels corresponding to the training images.

[0130] The first training module 504 is used to input multiple training samples into the initial content review model to obtain first prediction labels corresponding to the multiple training samples, and adjust the global multimodal alignment parameters in the initial content review model based on the first prediction labels and training labels corresponding to the multiple training samples until the first convergence condition is met to obtain an intermediate content review model.

[0131] The task grouping module 506 is used to input multiple training samples into the intermediate content review model to obtain second prediction labels corresponding to the multiple training samples, and group multiple content review tasks based on the second prediction labels and training labels corresponding to the multiple training samples to obtain multiple task groups.

[0132] The second training module 508 is used to input multiple training samples into the initial content review model to obtain third prediction labels corresponding to the multiple training samples, and obtain task group losses corresponding to the multiple task groups based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same task group, and obtain task losses corresponding to the multiple content review tasks based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same content review task.

[0133] The second training module 508 is also used to adjust the intra-group multimodal alignment parameters corresponding to the corresponding task group in the initial content review model based on the task group loss, and to adjust the task multimodal alignment parameters corresponding to the corresponding content review task in the initial content review model based on the task loss, until the second convergence condition is met to obtain the target content review model.

[0134] In one embodiment, the first training module 504 is further configured to:

[0135] Input the training samples into the initial content review model; the initial content review model includes a text encoder and an image encoder;

[0136] Generate global multimodal alignment parameters shared by text encoder and image encoder;

[0137] Extracting features of training texts in training samples by using a text encoder to obtain initial text features, and obtaining first training text features based on global multimodal alignment parameters and the initial text features;

[0138] Extracting features of the training images in the training samples through an image encoder to obtain initial image features, and obtaining first training image features based on global multimodal alignment parameters and the initial image features;

[0139] Based on the first training text feature and the first training image feature, a first predicted label corresponding to the training sample is obtained.

[0140] In one embodiment, the task grouping module 506 is further configured to:

[0141] Based on the second prediction labels and training labels corresponding to the multiple training samples belonging to the same content review task, the model losses corresponding to the multiple content review tasks are obtained respectively;

[0142] Calculate the loss gradient of the model loss with respect to the global multimodal alignment parameter, and obtain the loss gradients corresponding to multiple content review tasks.

[0143] The content review tasks whose similarity between loss gradients is greater than a similarity threshold are divided into the same task group to obtain multiple task groups.

[0144] In one embodiment, the second training module 508 is further configured to:

[0145] Input the training samples into the initial content review model; the initial content review model includes a text encoder and an image encoder;

[0146] Generate intra-group multimodal alignment parameters corresponding to multiple task groups and task multimodal alignment parameters corresponding to multiple content review tasks;

[0147] Through the text encoder, feature extraction is performed on the training text in the training sample to obtain initial text features, and second training text features are obtained based on the intra-group multimodal alignment parameters corresponding to the task group to which the training sample belongs, the task multimodal alignment parameters corresponding to the content review task to which the training sample belongs, and the initial text features;

[0148] Perform feature extraction on the training image in the training sample through an image encoder to obtain initial image features, and obtain second training image features based on the intra-group multimodal alignment parameters corresponding to the task group to which the training sample belongs, the task multimodal alignment parameters corresponding to the content review task to which the training sample belongs, and the initial image features;

[0149] Based on the second training text feature and the second training image feature, a third predicted label corresponding to the training sample is obtained.

[0150] In one embodiment, the data set acquisition module 502 is further used to:

[0151] For the current content review task, multiple normal images and multiple abnormal images are obtained as training images respectively;

[0152] When the training image carries description text, the description text corresponding to the training image is used as the training text corresponding to the training image;

[0153] When the training image does not carry description text, the training image is input into the image description generation model to obtain the training text corresponding to the training image;

[0154] The training image and the training text corresponding to the training image are used as training samples, and the training label corresponding to the training sample is obtained based on the content review label corresponding to the training image; the content review label corresponding to the normal image is the normal label, and the content review label corresponding to the abnormal image is the abnormal label;

[0155] Based on multiple training samples and training labels corresponding to the multiple training samples, a training data set corresponding to the current content review task is obtained.

[0156] In one embodiment, the content audit model training device is further used to:

[0157] Get the image to be reviewed;

[0158] Input the image to be reviewed into the target content review model to obtain the target image features of the image to be reviewed on multiple content review tasks;

[0159] Obtain reference content features corresponding to the reference content carrying the content review tag, and determine content similarity between the image to be reviewed and the reference content based on feature similarities between the reference content features and features of multiple target images;

[0160] Based on the content similarity, the target content is determined from multiple reference contents, and based on the content review label corresponding to the target content, the review result corresponding to the image to be reviewed is obtained.

[0161] In one embodiment, the content audit model training device is further used to:

[0162] Get the image to be reviewed;

[0163] Input the image to be reviewed into the target content review model to obtain the predicted labels of the image to be reviewed on multiple content review tasks;

[0164] Based on the predicted labels, the review results corresponding to the image to be reviewed are determined.

[0165] The above-mentioned content review model training device can improve the training quality, thereby improving the model accuracy of the target content review model and improving the accuracy of content review.

[0166] Each module in the above-mentioned content review model training device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0167] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the extracted descriptive information data for the promotion object. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a content audit model training method is implemented.

[0168] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0169] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0170] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0171] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0172] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0173] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0174] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A content audit model training method, characterized in that: The method comprises: Obtain training data sets corresponding to multiple content review tasks respectively; the training data sets include training samples and training labels corresponding to the training samples, the training samples include training images and training texts corresponding to the training images, and the training labels are content review labels corresponding to the training images; Inputting a plurality of training samples into an initial content review model to obtain first prediction labels corresponding to the plurality of training samples, and adjusting a global multimodal alignment parameter in the initial content review model based on the first prediction labels and training labels corresponding to the plurality of training samples until a first convergence condition is met, thereby obtaining an intermediate content review model; Inputting a plurality of training samples into the intermediate content review model to obtain second prediction labels corresponding to the plurality of training samples, and grouping the plurality of content review tasks based on the second prediction labels and training labels corresponding to the plurality of training samples to obtain a plurality of task groups; Input multiple training samples into the initial content review model to obtain third prediction labels corresponding to the multiple training samples, and obtain task group losses corresponding to the multiple task groups based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same task group, and obtain task losses corresponding to the multiple content review tasks based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same content review task; Based on the task group loss, adjust the intra-group multimodal alignment parameters corresponding to the corresponding task group in the initial content audit model; based on the task loss, adjust the task multimodal alignment parameters corresponding to the corresponding content audit task in the initial content audit model until the second convergence condition is met to obtain the target content audit model.

2. The method according to claim 1, characterized in that The step of inputting a plurality of training samples into the initial content review model to obtain first prediction labels corresponding to the plurality of training samples includes: Inputting the training samples into an initial content audit model; the initial content audit model includes a text encoder and an image encoder; generating a global multimodal alignment parameter shared by the text encoder and the image encoder; By using the text encoder, feature extraction is performed on the training text in the training sample to obtain initial text features, and based on the global multimodal alignment parameters and the initial text features, a first training text feature is obtained; Extracting features of a training image in a training sample by using the image encoder to obtain initial image features, and obtaining first training image features based on a global multimodal alignment parameter and the initial image features; Based on the first training text feature and the first training image feature, a first predicted label corresponding to the training sample is obtained.

3. The method according to claim 1, characterized in that The plurality of content review tasks are grouped based on the second prediction labels and the training labels corresponding to the plurality of training samples to obtain a plurality of task groups, including: Based on the second prediction labels and training labels corresponding to the multiple training samples belonging to the same content review task, obtain the model losses corresponding to the multiple content review tasks respectively; Calculate the loss gradient of the model loss with respect to the global multimodal alignment parameter to obtain the loss gradients corresponding to the multiple content review tasks respectively; The content review tasks whose similarity between loss gradients is greater than a similarity threshold are divided into the same task group to obtain multiple task groups.

4. The method according to claim 1, characterized in that: The step of inputting a plurality of training samples into the initial content review model to obtain third prediction labels corresponding to the plurality of training samples includes: Inputting the training samples into the initial content audit model; the initial content audit model includes a text encoder and an image encoder; Generate intra-group multimodal alignment parameters corresponding to multiple task groups and task multimodal alignment parameters corresponding to multiple content review tasks; By means of the text encoder, feature extraction is performed on the training text in the training sample to obtain initial text features, and second training text features are obtained based on the intra-group multimodal alignment parameters corresponding to the task group to which the training sample belongs, the task multimodal alignment parameters corresponding to the content review task to which the training sample belongs, and the initial text features; By means of the image encoder, feature extraction is performed on the training image in the training sample to obtain initial image features, and second training image features are obtained based on the intra-group multimodal alignment parameters corresponding to the task group to which the training sample belongs, the task multimodal alignment parameters corresponding to the content review task to which the training sample belongs, and the initial image features; Based on the second training text feature and the second training image feature, a third predicted label corresponding to the training sample is obtained.

5. The method according to claim 1, characterized in that The obtaining of training data sets corresponding to the plurality of content review tasks respectively includes: For the current content review task, multiple normal images and multiple abnormal images are obtained as training images respectively; When the training image carries description text, the description text corresponding to the training image is used as the training text corresponding to the training image; When the training image does not carry description text, the training image is input into the image description generation model to obtain the training text corresponding to the training image; The training image and the training text corresponding to the training image are used as training samples, and the training label corresponding to the training sample is obtained based on the content review label corresponding to the training image; the content review label corresponding to the normal image is the normal label, and the content review label corresponding to the abnormal image is the abnormal label; Based on multiple training samples and training labels corresponding to the multiple training samples, a training data set corresponding to the current content review task is obtained.

6. The method according to claim 1, characterized in that The method further comprises: Get the image to be reviewed; Inputting the image to be reviewed into the target content review model to obtain target image features of the image to be reviewed on the multiple content review tasks; Acquire reference content features corresponding to reference content carrying content review tags, and determine content similarity between the image to be reviewed and the reference content based on feature similarities between the reference content features and features of multiple target images; Based on the content similarity, target content is determined from multiple reference contents, and based on the content review label corresponding to the target content, the review result corresponding to the image to be reviewed is obtained.

7. The method according to claim 1, characterized in that The method further comprises: Get the image to be reviewed; Inputting the image to be reviewed into the target content review model to obtain predicted labels of the image to be reviewed on the multiple content review tasks; Based on the predicted label, a review result corresponding to the image to be reviewed is determined.

8. A content review model training device, characterized in that: The device comprises: A data set acquisition module is used to acquire training data sets corresponding to multiple content review tasks respectively; the training data sets include training samples and training labels corresponding to the training samples, the training samples include training images and training texts corresponding to the training images, and the training labels are content review labels corresponding to the training images; A first training module is used to input a plurality of training samples into an initial content review model to obtain first prediction labels corresponding to the plurality of training samples, and adjust a global multimodal alignment parameter in the initial content review model based on the first prediction labels and training labels corresponding to the plurality of training samples until a first convergence condition is met, thereby obtaining an intermediate content review model; A task grouping module, used to input a plurality of training samples into the intermediate content review model, obtain second prediction labels corresponding to the plurality of training samples, and group the plurality of content review tasks based on the second prediction labels and training labels corresponding to the plurality of training samples to obtain a plurality of task groups; A second training module is used to input multiple training samples into the initial content review model to obtain third prediction labels corresponding to the multiple training samples, and obtain task group losses corresponding to the multiple task groups based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same task group, and obtain task losses corresponding to the multiple content review tasks based on the third prediction labels and training labels corresponding to the multiple training samples belonging to the same content review task; The second training module is also used to adjust the intra-group multimodal alignment parameters corresponding to the corresponding task group in the initial content review model based on the task group loss, and adjust the task multimodal alignment parameters corresponding to the corresponding content review task in the initial content review model based on the task loss, until the second convergence condition is met to obtain the target content review model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images