Content recognition model training method and content recognition method and apparatus

Abstract

This application relates to a content recognition model training method, apparatus, computer device, storage medium, and computer program product. The method includes: in the first round of model adjustment for the content recognition model, adjusting the content recognition model using reliable training data to obtain a content recognition model after the first round of adjustment; in the current round of model adjustment, respectively calling a preset reliable large model and the content recognition model after the previous round of adjustment, recognizing the training data of the first modality in the supplementary training data based on the training data of the second modality in the supplementary training data, respectively obtaining a first recognition result and a second recognition result; updating the training data of the second modality in the supplementary training data according to the recognition accuracy of the first recognition result and the second recognition result, and adjusting the content recognition model after the previous round of adjustment based on the updated supplementary training data. This method can improve model training efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a content recognition model training method, content recognition method, and apparatus. Background Technology

[0002] With the development of science and technology, content recognition technology has emerged. Content recognition refers to the process of using technical means to identify and understand information content. Content recognition typically includes unimodal recognition and multimodal content recognition. Multimodal content recognition refers to the process of identifying and understanding data from multiple modalities. For example, the multimodal content recognition process can involve understanding a question belonging to the text modality, and based on the text understanding results, identifying and understanding an image belonging to the image modality, and outputting an answer corresponding to the question.

[0003] Currently, pre-trained models are primarily used for multimodal content recognition. During model training, the training dataset is typically labeled manually, and the model is then trained based on this labeled dataset. However, manual labeling incurs significant labor costs, resulting in low model training efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a content recognition model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the training efficiency of the aforementioned technical problems.

[0005] In a first aspect, this application provides a method for training a content recognition model, the method comprising:

[0006] In the first round of model adjustment for the content recognition model, reliable training data is obtained, and the content recognition model is adjusted using the reliable training data to obtain the content recognition model after the first round of adjustment.

[0007] In the current round of model adjustment following the first round of adjustments, supplementary training data is obtained;

[0008] The preset trusted large model and the content recognition model adjusted in the previous round are called respectively. Based on the training data of the second modality in the supplementary training data, the training data of the first modality in the supplementary training data are recognized to obtain the first recognition result and the second recognition result respectively.

[0009] Based on the respective recognition accuracy of the first recognition result and the second recognition result, the training data of the second modality in the supplementary training data is updated, and the content recognition model adjusted in the previous round is adjusted based on the updated supplementary training data.

[0010] Secondly, this application also provides a content recognition model training apparatus, the apparatus comprising:

[0011] The first-round adjustment module is used to obtain reliable training data in the first round of model adjustment for the content recognition model, and use the reliable training data to adjust the content recognition model to obtain the content recognition model after the first round of adjustment.

[0012] The comparison module is used to obtain supplementary training data in the current round of model adjustment after the first round of adjustment;

[0013] The preset trusted large model and the content recognition model adjusted in the previous round are called respectively. Based on the training data of the second modality in the supplementary training data, the training data of the first modality in the supplementary training data are recognized to obtain the first recognition result and the second recognition result respectively.

[0014] The current round adjustment module is used to update the training data of the second modality in the supplementary training data according to the respective recognition accuracy of the first recognition result and the second recognition result, and to adjust the content recognition model after the previous round adjustment based on the updated supplementary training data.

[0015] In one embodiment, the content recognition model training device further includes a pre-training module, which is used to acquire open-source training data and an initial content recognition model; the training data of the second modality in the open-source training data includes open-source questions and corresponding open-source answers; the initial content recognition model is invoked to perform content recognition on the training data of the first modality in the open-source training data according to the guidance of the open-source questions, to obtain the recognition prediction results of the pre-training stage; the feature space transformation module in the initial content recognition model is allowed to participate in training, and the parameters of the feature space transformation module in the initial content recognition model are adjusted based on the difference between the open-source answers and the recognition prediction results of the pre-training stage, to obtain the content recognition model.

[0016] In one embodiment, the trusted training data includes training data for a first modality and training data for a second modality; the training data for the second modality in the trusted training data includes a training question and a training answer corresponding to the training question; the first-round adjustment module is further configured to call the content recognition model to perform content recognition on the training data of the first modality in the trusted training data according to the guidance of the training question, and obtain the recognition prediction result of the first round; based on the difference between the training answer and the recognition prediction result of the first round, the parameters of the content recognition model are adjusted to obtain the content recognition model after the first round adjustment.

[0017] In one embodiment, the training problem includes a classification training problem and an inference training problem; the recognition prediction result of the first round includes the classification prediction answer of the first round and the inference prediction answer of the first round; the training answer includes the standard classification result and the standard inference result; the first-round adjustment module is further configured to adjust the parameters of the discriminative module in the content recognition model based on a first difference between the classification prediction answer of the first round and the standard classification result; adjust the parameters of the generative module in the content recognition model based on a second difference between the inference prediction answer of the first round and the standard inference result; and adjust the parameters of the feature space transformation module in the content recognition model based on the first difference and the second difference.

[0018] In one embodiment, the current round adjustment module is further configured to determine a first recognition accuracy of the first recognition result; determine a second recognition accuracy of the second recognition result; filter out a target recognition result from the first recognition result and the second recognition result based on the first recognition accuracy and the second recognition accuracy; and update the training data of the second modality in the supplementary training data based on the target recognition result.

[0019] In one embodiment, the supplementary training data is stored in a supplementary training dataset; the current round adjustment module is further configured to determine the classification accuracy of the reliable large model on the supplementary training dataset; determine the output accuracy of the first recognition result; determine the keyword matching degree of the first recognition result; and determine the first recognition accuracy of the first recognition result based on the classification accuracy, the output accuracy of the first recognition result, and the keyword matching degree.

[0020] In one embodiment, the current round adjustment module is further configured to: filter samples from the supplementary training dataset that are classified as positive by the trusted large model; determine precision based on the number of correctly classified samples and the number of samples classified as positive; determine the number of samples labeled as positive in the supplementary training dataset; determine recall based on the number of samples labeled as positive and the number of samples labeled as positive that are classified as positive by the trusted large model; and determine the classification accuracy of the trusted large model on the supplementary training dataset based on precision and recall.

[0021] In one embodiment, the current round adjustment module is further configured to obtain a pre-trained multimodal matching degree determination model; and call the multimodal matching degree determination model to determine the degree of matching between the first recognition result and the training data of the first modality in the supplementary training data.

[0022] In one embodiment, the output accuracy includes text similarity; the current round adjustment module is further configured to determine the inference reference answer in the training data of the second modality in the supplementary training data; and determine the text similarity between the inference reference answer and the inference prediction answer in the first recognition result.

[0023] In one embodiment, the current round adjustment module is further configured to determine the inference reference answer in the training data of the second modality in the supplementary training data; determine the degree of overlap between the inference reference answer and the inference prediction answer in the first recognition result; and determine the keyword matching degree of the first recognition result based on the degree of overlap.

[0024] In one embodiment, the supplementary training data is stored in a supplementary training dataset; the current round adjustment module is further configured to determine the classification accuracy of the reliable large model on the supplementary training dataset; determine the output accuracy of the second recognition result; determine the keyword matching degree of the second recognition result; and determine the second recognition accuracy of the second recognition result based on the classification accuracy, the output accuracy of the second recognition result, and the keyword matching degree.

[0025] In one embodiment, the training data for the second modality in the supplementary training data includes a supplementary inference question and an inference reference answer; the current round adjustment module is further configured to add the classification prediction answer in the target recognition result to the supplementary inference question, and replace the inference reference answer with the inference prediction answer in the target recognition result.

[0026] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in any of the content recognition model training methods provided in the embodiments of this application.

[0027] Fourthly, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in any of the content recognition model training methods provided in the embodiments of this application.

[0028] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the content recognition model training methods provided in the embodiments of this application.

[0029] The aforementioned content recognition model training method, apparatus, computer equipment, storage medium, and computer program product, by acquiring reliable training data, can perform the first round of model adjustment based on the reliable training data. Since the reliable training data is trustworthy, the content recognition model trained on the reliable training data in the first round of adjustment can output relatively accurate recognition results. In subsequent rounds, by acquiring supplementary training data and a reliable large model, the first recognition result output by the reliable large model based on the supplementary training data can be compared with the second recognition result output by the content recognition model adjusted in the previous round based on the supplementary training data. This allows the more accurate recognition result (the first or second recognition result) to be used as a label to update the training data for the second modality, thereby eliminating the need for manual annotation, saving labor costs, and improving model training efficiency.

[0030] Furthermore, since the supplementary training data is updated based on the respective recognition accuracy rates of the first and second recognition results, the updated supplementary training data can be made more accurate, thereby allowing for more accurate adjustments to the content recognition model based on the more accurate supplementary training data.

[0031] Firstly, this application provides a content recognition method, the method comprising:

[0032] Acquire first data for a first modality and second data for a second modality; the second data includes at least one question related to the first data;

[0033] Feature extraction is performed on the first data to obtain the first feature in the feature space of the first modality;

[0034] Feature extraction is performed on the second data to obtain the second feature in the feature space of the second modality;

[0035] The first feature and the second feature are concatenated in the same feature space to obtain the concatenated feature.

[0036] When the second data includes a category problem related to the first data, category identification is performed based on the splicing features to obtain the category of the first data in the first modality;

[0037] When the second data includes a reasoning question about the first data, data reasoning is performed based on the splicing features to obtain the answer data corresponding to the reasoning question.

[0038] Secondly, this application also provides a content recognition device, the device comprising:

[0039] A feature extraction module is used to acquire first data of a first modality and second data of a second modality; the second data includes at least one question related to the first data; feature extraction is performed on the first data to obtain a first feature in the feature space of the first modality; feature extraction is performed on the second data to obtain a second feature in the feature space of the second modality;

[0040] The feature splicing module is used to splice the first feature and the second feature in the same feature space to obtain spliced ​​features;

[0041] The output module is configured to, when the second data includes a category question for the first data, perform category recognition based on the splicing features to obtain the category of the first data in the first modality; and when the second data includes a reasoning question for the first data, perform data reasoning based on the splicing features to obtain the answer data corresponding to the reasoning question.

[0042] In one embodiment, the feature stitching module is further configured to obtain a projection matrix, transform the first feature into the feature space of the second modality based on the projection matrix to obtain a third feature in the feature space of the second modality, and stitch the third feature and the second feature together in the feature space of the second modality to obtain a stitched feature.

[0043] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in any of the content recognition methods provided in the embodiments of this application.

[0044] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any content recognition method provided in the embodiments of this application.

[0045] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any content recognition method provided in the embodiments of this application.

[0046] The aforementioned content recognition methods, apparatus, computer devices, storage media, and computer program products, by acquiring first data of a first modality and second data of a second modality, can extract features from the first and second data respectively, obtaining first and second features in their respective feature spaces. By obtaining the first and second features, they can be concatenated in the same feature space to obtain concatenated features. Based on these concatenated features, the category of the first data in the first modality and the answer data corresponding to the reasoning question can be output. Since not only can classification results be output, but also answer data corresponding to the reasoning question, this answer data can include the reasoning process for outputting the classification result. Therefore, users can understand the reasoning process for the model to output the classification result, improving the user experience. Furthermore, by concatenating the first and second features in the same feature space, data from different modalities can be aligned. This allows the discriminative and generative modules to mine deep correlations between data from different modalities based on the alignment results, outputting more accurate classification and reasoning results.

[0047] Furthermore, for some specialized content classification tasks, discriminative models offer superior training and inference speeds as well as better classification performance. Secondly, traditional machine learning models, due to limitations in their network structure, cannot effectively integrate classification and generation tasks. Therefore, this application's embodiments add a discriminative module to the content recognition model, enabling rapid output of classification results, and also add a generation module, allowing the model to output inference results simultaneously with classification results. Attached Figure Description

[0048] Figure 1 This is a diagram illustrating the application environment of a content recognition model training method in one embodiment.

[0049] Figure 2 This is a flowchart illustrating a content recognition model training method in one embodiment;

[0050] Figure 3 This is a schematic diagram of multimodal training data in one embodiment;

[0051] Figure 4 This is a schematic diagram of trusted training data in one embodiment;

[0052] Figure 5 This is a schematic diagram of the model structure of a content recognition model in one embodiment;

[0053] Figure 6 This is a flowchart illustrating the pre-training and adjustment phases in one embodiment;

[0054] Figure 7 This is a schematic diagram of the process for updating supplementary training data in one embodiment;

[0055] Figure 8 This is a schematic diagram illustrating the iterative training of a content recognition model in one embodiment;

[0056] Figure 9 This is a schematic diagram of the overall training process of a content recognition model in one embodiment;

[0057] Figure 10 This is a flowchart illustrating a content recognition method in one embodiment;

[0058] Figure 11 This is a schematic diagram illustrating the overall process of training a content recognition model in one embodiment;

[0059] Figure 12 This is a structural block diagram of a content recognition model training device in one embodiment;

[0060] Figure 13 This is a structural block diagram of a content recognition device in one embodiment;

[0061] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The content recognition model training method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed in the cloud or on other servers. Both terminal 102 and server 104 can be used independently to execute the content recognition model training method provided in this embodiment. Terminal 102 and server 104 can also be used collaboratively to execute the content recognition model training method provided in this embodiment. Taking the example of terminal 102 and server 104 collaboratively executing the content recognition model training method provided in this embodiment, terminal 102 can send trusted training data to server 104 so that server 104 can perform the first round of model adjustment. Afterwards, server 104 can obtain supplementary training data from the network and adjust the content recognition model again based on the supplementary training data. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0064] It should be noted that the terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Unless the context clearly indicates otherwise, the singular forms "a," "one," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. The quantities of "multiple" or "multiple copies" mentioned in the embodiments of this application all refer to a quantity of "at least two," for example, "multiple" means "at least two," and "multiple copies" means "at least two copies."

[0065] In order to clearly describe the technical solution of this application and facilitate understanding of the technical solution of this application, the key concepts involved in this application will be explained below.

[0066] Training data refers to the training data for machine learning models, which includes data from the first modality and data from the second modality. The first and second modalities belong to different modalities; for example, the training data for the first modality can be images, audio, or video. The training data for the second modality can be text.

[0067] Content recognition models are machine learning models used to identify data from a first modality based on data from a second modality.

[0068] Classification accuracy refers to the degree to which a trusted large model classifies the first modality of the training data in the supplementary training dataset. For example, classification accuracy is used to measure how accurately a trusted large model classifies images in the supplementary training dataset.

[0069] Output accuracy is the correlation between the recognition result and the training data of the first modality in the corresponding supplementary training data. For example, the output accuracy of the first recognition result can be the correlation between the model output and the images in the supplementary training data.

[0070] The keyword matching degree of the recognition result refers to the hit rate of the model output result in the keyword. For example, the keyword matching degree of the first recognition result can be used to measure whether the model output result hits the keyword.

[0071] In one embodiment, such as Figure 2 As shown, a content recognition model training method is provided. The method is illustrated using a computer device as an example. The computer device can provide... Figure 1 The server or terminal in the process. The content recognition model training method includes the following steps:

[0072] Step 202: In the first round of model adjustment for the content recognition model, obtain reliable training data, use the reliable training data to adjust the content recognition model, and obtain the content recognition model after the first round of adjustment.

[0073] The reliable training data may include training data for the first modality and training data for the second modality. The first and second modalities belong to different modalities; for example, the training data for the first modality may be images, audio, or video. The training data for the second modality may be text. The content recognition model is used to recognize the data of the first modality based on the data of the second modality.

[0074] Specifically, the content recognition model can be a pre-trained machine learning model. To improve the model's ability to understand, judge, and generate multimodal content in professional scenarios, multiple rounds of model adjustments can be performed after pre-training. More specifically, in the first round of model adjustments during these multiple rounds, the computer device can acquire reliable training data to perform the initial model adjustments. This reliable training data includes training data for the first modality and training data for the second modality, for example, referencing... Figure 3 The training data for the first modality can be images, and the training data for the second modality can be text, so the reliable training data can include images 301 and text 302. Figure 3 A schematic diagram of multimodal training data is shown in one embodiment.

[0075] Among them, the content recognition model is used to recognize the data of the first modality based on the data of the second modality. For example, the content recognition model can be used to recognize and understand images based on questions in the text.

[0076] In one embodiment, the data for the first modality can be images, and the data for the second modality can be text. Multiple images can be acquired, and for each image, text corresponding to that image can be manually input to form the corresponding text. Combining the images and text yields reliable training data. For example, refer to... Figure 4 For an image with the path XXX.1.1.1, you can input: "Question: Please determine which of the following four categories this image belongs to: 0 (Cartoon), 1 (Advertisement), 2 (Chart), 3 (Other)? Answer: 0" or "Question: Please briefly describe this image. Answer: This is a normal cartoon image. The image shows a group of cartoon characters sitting on a green grassy hillside with a clear blue sky in the background. These characters appear to be children's cartoon characters. Question: Please provide the basis for your judgment. Answer: These characters appear to be characters from an animation. They seem to be enjoying outdoor activities or resting, and the overall image gives a relaxed and pleasant feeling." By combining the image with the path XXX.1.1.1 and the input text, a reliable training data is generated. Because the reliable training data is generated manually, it is accurate training data. Figure 4 A schematic diagram of trusted training data in one embodiment is shown. Alternatively, the data for the first modality can also be audio, and the data for the second modality can be text. For each audio file, trusted training data can be generated by manually inputting the text corresponding to that audio file.

[0077] In one embodiment, a trusted large model can be obtained, and the data of the second modality output by the trusted large model can be invoked. The data of the second modality output by the trusted large model is then validated. If the validation passes, trusted training data is generated based on the data of the second modality. For example, an image and a question can be input into the trusted large model, which outputs the corresponding answer to the question. The answer is then validated manually to determine its correctness. Based on the image, question, and validated answer, trusted training data is generated.

[0078] In one embodiment, trusted training data can be input into a content recognition model. The content recognition model processes the trusted training data, and the parameters of the content recognition model are adjusted based on the processing results to obtain the content recognition model after the first round of adjustments. For example, refer to... Figure 5 , Figure 5A schematic diagram of the model structure of a content recognition model in one embodiment is shown. The content recognition model includes a first feature extraction module, a second feature extraction module, a feature space transformation module, a feature concatenation module, a discriminative module, and a generative module. Specifically, the first feature extraction module extracts features from the data of the first modality; the second feature extraction module extracts features from the data of the second modality; the feature space transformation module performs spatial transformation on the features; the feature concatenation module concatenates multiple features; the discriminative module outputs the classification category, for example, the category to which an image belongs; and the generative module outputs the inference result, such as the image content, the reason for determining the image's category, and the basis for determining the image content. Training data of the first modality from trusted training data can be input into the content recognition model, and the first feature extraction module in the content recognition model performs feature extraction processing on the training data of the first modality to obtain a fourth feature; similarly, training data of the second modality from trusted training data can be input into the content recognition model, and the second feature extraction module in the content recognition model performs feature extraction processing on the training data of the second modality to obtain a fifth feature. The fourth and fifth features can be transformed into the same feature space using the feature space transformation module in the content recognition model. Then, the features transformed into the same feature space are concatenated using the feature concatenation module to obtain the first concatenated feature. This first concatenated feature is input into the discriminative and generative modules. Based on the differences between the outputs of the discriminative and generative models and the training data of the second modality in the reliable training data, the content recognition model is adjusted. During training, the output of the generative module is called the inference prediction answer, and the output of the discriminative module is called the classification prediction answer.

[0079] For example, in the above example, when the data of the first modality is an image and the data of the second modality is text, the features of the image in the reliable training data can be extracted by the first feature extraction module in the content recognition model, and the features of the text corresponding to the image in the reliable training data can be extracted by the second feature extraction module in the content recognition model. The extracted image features and extracted text features are transformed to the same feature space by the feature space transformation module, and the features transformed to the same feature space are concatenated by the concatenation module to obtain the first concatenated feature. The first concatenated feature is input to the discriminative module so that the discriminative module outputs the category to which the image belongs, for example, outputting 0, which is the classification prediction answer for the first round. The first concatenated feature is input to the generative module so that the generative module outputs the image content, the reason for judging the category to which the image belongs, and the basis for judging the image content. The image content, the reason for judging the category to which the image belongs, and the basis for judging the image content constitute the inference prediction answer for the first round. Since the text in the reliable training data already specifies the image category, image content, the reason for judging the category to which the image belongs, and the basis for judging the image content (e.g., ...), Figure 4 Therefore, the content recognition model can be adjusted based on the first round of classification prediction answers output by the discriminative module, the first round of inference prediction answers output by the generative module, and the image category, image content, reason for the image category, and basis for the judgment of the image content stated in the text of the reliable training data, so as to obtain the content recognition model after the first round of adjustment.

[0080] In one embodiment, the content recognition model can be adjusted using a trusted training dataset that includes multiple trusted training data to obtain the content recognition model after the first round of adjustment.

[0081] Step 204: In the current round of model adjustment following the first round of adjustment, obtain supplementary training data.

[0082] Specifically, in the current round of model adjustment (excluding the first round) during multi-round model adjustments of the content recognition model, the computer device can acquire supplementary training data. This supplementary training data can be training data obtained from the network. Understandably, since the training data is obtained from the network, the training data for the second training modality in the supplementary training data may be inaccurate; for example, the answers in the supplementary training data may not be accurate. In one embodiment, the training data for the second modality in the supplementary training data may only contain questions, without including the corresponding answers.

[0083] Step 206: Call the preset trusted large model and the content recognition model adjusted in the previous round respectively, and recognize the training data of the first modality in the supplementary training data based on the training data of the second modality in the supplementary training data, and obtain the first recognition result and the second recognition result respectively.

[0084] The trusted large model can be an open-source large model used to compare with the content recognition model adjusted in the previous round. For example, the trusted large model could be ChatGPT-4.

[0085] Specifically, supplementary training data can be input into the trusted large model and the content recognition model adjusted in the previous round, respectively. The supplementary training data can be processed by the trusted large model and the content recognition model adjusted in the previous round to obtain the first recognition result output by the trusted large model and the second recognition result output by the content recognition model adjusted in the previous round.

[0086] For example, the data of the first modality is an image, and the data of the second modality is text. Supplementary training data may include an image and the text corresponding to the image, which contains a question. When the image and the question are input into the trusted big data model, the trusted big data model can output the corresponding answer, which is the first recognition result. When the image and the question are input into the content recognition model adjusted in the previous round, the content recognition model adjusted in the previous round can also output the corresponding answer, which is the second recognition result.

[0087] Step 208: Based on the recognition accuracy of the first recognition result and the second recognition result, update the training data of the second modality in the supplementary training set, and adjust the content recognition model adjusted in the previous round based on the updated supplementary training set.

[0088] Specifically, the computer device can determine the recognition accuracy of the first recognition result and the recognition accuracy of the second recognition result. Based on the recognition accuracy of the first recognition result and the recognition accuracy of the second recognition result, a more accurate recognition result is selected. Based on the more accurate recognition result, the training data of the second modality in the supplementary training data is updated to obtain updated supplementary training data. Based on the updated supplementary training data, the content recognition model adjusted in the previous round is adjusted.

[0089] In one embodiment, the computer device can determine a first recognition accuracy rate of the first recognition result and a second recognition accuracy rate of the second recognition result. Based on the first and second recognition accuracy rates, a target recognition result with a higher recognition accuracy rate is selected from the first and second recognition results, and this target recognition result is added as a label to supplementary training data. For example, supplementary training data A includes image A and text A, where text A includes image classification and image reasoning problems. The reliable large model can output the first recognition result A through image A and text A: "Cartoon category; This is a normal cartoon image, showing a group of cartoon characters sitting on a green grassy hillside; these characters appear to be characters from animation A, therefore it belongs to the cartoon category." The content recognition model after the previous round of adjustments can output the second recognition result B through image A and text A: "Advertisement category; This is an advertisement image, showing a group of people sitting on a green grassy hillside; these characters appear to be characters from advertisement A, therefore it belongs to the advertisement category." If the first recognition accuracy of the first recognition result A is higher than the second recognition accuracy of the second recognition result B, then the first recognition result A can be added to text A as the answer to the question in text A.

[0090] In one embodiment, when updated supplementary training data is obtained, the content recognition model adjusted in the previous round can be adjusted by combining the updated supplementary training data and reliable training data to obtain the content recognition model adjusted in the current round.

[0091] In the aforementioned content recognition model training method, by acquiring reliable training data, the content recognition model can be adjusted in the first round based on this data. Since the reliable training data is trustworthy, the content recognition model trained on it in the first round can output relatively accurate recognition results. In subsequent rounds, by acquiring supplementary training data and a reliable large model, the first recognition result output by the reliable large model based on the supplementary training data can be compared with the second recognition result output by the content recognition model adjusted in the previous round based on the supplementary training data. This allows the more accurate recognition result (the one from the first and second rounds) to be used as a label to update the training data for the second modality, thus eliminating the need for manual annotation, saving labor costs, and improving model training efficiency. Furthermore, since the supplementary training data is updated based on the respective recognition accuracies of the first and second recognition results, the updated supplementary training data becomes more accurate, allowing for more precise adjustments to the content recognition model based on this more accurate supplementary training data.

[0092] In one embodiment, before acquiring reliable training data in the first round of model adjustment for the content recognition model, the method further includes: acquiring open-source training data and an initial content recognition model; the training data of the second modality in the open-source training data includes open-source questions and corresponding open-source answers; calling the initial content recognition model to perform content recognition on the training data of the first modality in the open-source training data according to the guidance of the open-source questions, and obtaining the recognition prediction results in the pre-training stage; allowing the feature space transformation module in the initial content recognition model to participate in training, and adjusting the parameters of the feature space transformation module in the initial content recognition model based on the difference between the open-source answers and the recognition prediction results in the pre-training stage, to obtain the content recognition model.

[0093] Specifically, before performing multiple rounds of model adjustments on the content recognition model, the initial content recognition model can be pre-trained using open-source training data to obtain the final model. For example, the open-source training data can include training data for both the first and second modalities. The training data for the second modality can include questions and their corresponding answers. For ease of description, the questions in the open-source training data will be referred to as open-source questions, and the answers as open-source answers. The training data for the first modality and the open-source questions from the open-source training data can be input into the initial content recognition model, which will then output the recognition prediction results from the pre-training stage. During the pre-training stage, the parameters of the first feature extraction module, the discriminative module, and the generative module are frozen, while only the feature space transformation module is unfrozen. Based on the difference between the open-source answers and the recognition prediction results from the pre-training stage, the parameters of the feature space transformation module in the initial content recognition model are adjusted.

[0094] In one embodiment, the open-source training data may be... The image "dem / 000.jpg" can be briefly described as follows: This is a very beautiful landscape painting, with mountains that are often layered, some majestic and some winding. "dem / 000.jpg" indicates an image and represents the first modality of training data in the open-source training dataset. "Please briefly describe the image" is the open-source question. "This is a very beautiful landscape painting, with mountains that are often layered, some majestic and some winding" is the open-source answer. Guided by this open-source question, the initial content recognition model can generate the current text character and, based on the currently generated text character and previously generated text characters, output the next text character. This yields the recognition and prediction results for the pre-training stage.

[0095] In the above embodiments, since the weights of the feature space transformation module are deferred to participate in training during the pre-training phase, the alignment ability between the features of the first modality data and the features of the second modality data can be improved through pre-training. For example, pre-training can improve the model's ability to align image features with text features. By improving feature alignment ability, the model can better understand multimodal data based on aligned features, such as better understanding text semantics and image semantics, thereby improving the understanding ability of multimodal data, uncovering complex relationships between multimodal data, and improving the accuracy of the model's output content.

[0096] In one embodiment, the training data of the second modality in the trusted training data includes a training question and a training answer corresponding to the training question; adjusting the content recognition model using the trusted training data to obtain the content recognition model after the first round of adjustment includes: performing content recognition on the training data of the first modality in the trusted training data according to the guidance of the training question using the content recognition model to obtain the recognition prediction result of the first round; adjusting the parameters of the content recognition model based on the difference between the training answer and the recognition prediction result of the first round to obtain the content recognition model after the first round of adjustment.

[0097] Specifically, the training data for the second modality in the trusted training data also includes questions and their corresponding answers. For ease of description, the questions in the trusted training data will be referred to as training questions, and the answers in the trusted training data will be referred to as training answers. The trusted training data can be input into the content recognition model. Guided by the training questions, the content recognition model performs content recognition on the training data for the first modality in the trusted training data, obtaining the recognition prediction result for the first round. For example, the recognition prediction result for the first round may include the image category, image content, and the reason for outputting the image content, etc. The difference between the training answer and the recognition prediction result for the first round can be determined. Based on this difference, the content recognition model is adjusted to obtain the adjusted content recognition model for the first round.

[0098] In one embodiment, during multi-round model adjustment, the first feature extraction module can be frozen, while only the parameters of the feature space transformation module, generative module, and discriminative module are allowed to participate in training. For example, the parameters of the feature space transformation module, generative module, and discriminative module can be adjusted based on the difference between the training response and the recognition prediction result of the first round.

[0099] In one embodiment, reference Figure 6 , Figure 6The diagram illustrates the flowchart of the pre-training and tuning phases in one embodiment. During the pre-training phase, the first feature extraction module and the LLM (Large Language Model) can be frozen, while only the feature space transformation module is enabled. The LLM can be either a discriminative or generative module. During the tuning phase, the first feature extraction module can be frozen, while only the feature space transformation module, the generative LLM, and the discriminative LLM are enabled. The generative LLM can be any of the aforementioned generative modules, and the discriminative LLM can be any of the aforementioned discriminative modules.

[0100] In the above embodiments, in the first round, the content recognition model is adjusted using reliable training data, so that the content recognition model adjusted in the first round can output more accurate results.

[0101] In one embodiment, the training problem includes a classification training problem and an inference training problem; the recognition prediction result of the first round includes the classification prediction answer of the first round and the inference prediction answer of the first round; the training answer includes the standard classification result and the standard inference result; based on the difference between the training answer and the recognition prediction result of the first round, the parameters of the content recognition model are adjusted to obtain the content recognition model adjusted for the first round, including: adjusting the parameters of the discriminative module in the content recognition model according to a first difference between the classification prediction answer of the first round and the standard classification result; adjusting the parameters of the generative module in the content recognition model according to a second difference between the inference prediction answer of the first round and the standard inference result; and adjusting the parameters of the feature space transformation module in the content recognition model based on the first difference and the second difference.

[0102] Specifically, a first difference can be identified between the classification prediction answer in the first round and the standard classification result, and the parameters of the discriminative module in the content recognition model can be adjusted based on this first difference. A second difference can be identified between the inference prediction answer in the first round and the standard inference result, and the parameters of the generative module in the content recognition model can be adjusted based on this second difference. The parameters of the feature space transformation module in the content recognition model can be adjusted based on both the first and second differences.

[0103] In one embodiment, the discriminative module may specifically be a discriminative large model (LLM), and the generative module may specifically be a generative large model (LLM). In one embodiment, the discriminative module and the generative module may be trained synchronously and share attention layer parameters.

[0104] In one embodiment, during the content recognition model adjustment process, the model can be trained using the following loss function.

[0105]

[0106] In the discriminant module, during the adjustment of its parameters based on the aforementioned loss function formula, C represents the number of preset categories. Represents the true classification category (if it belongs to category i, then...). =1, otherwise =0), This represents the classification prediction answer output by the discriminant module.

[0107] For the generative module, during the adjustment of the parameters of the discriminative module based on the above loss function formula, C represents the length of the preset vocabulary. Represents the i-th real character. This represents the i-th character output by the generative module.

[0108] In the above embodiments, by adjusting the parameters of the content recognition model, the content recognition model with adjusted parameters can output more accurate content.

[0109] In one embodiment, updating the training data of the second modality in the supplementary training data corresponding to the current round based on the respective recognition accuracy of the first recognition result and the second recognition result includes: determining a first recognition accuracy of the first recognition result; determining a second recognition accuracy of the second recognition result; selecting a target recognition result from the first recognition result and the second recognition result based on the first recognition accuracy and the second recognition accuracy; and updating the training data of the second modality in the supplementary training data based on the target recognition result.

[0110] Specifically, during the current round of model adjustment, after obtaining the first recognition result output by the trusted large model and the second recognition result output by the content recognition model after the previous round of adjustment, the first recognition accuracy of the first recognition result and the second recognition accuracy of the second recognition result can be determined. For example, the first and second recognition accuracies can be determined manually, or the first and second recognition results, along with supplementary training data, can be input into other trusted large models, which can then use the supplementary training data to determine the respective recognition accuracies of the first and second recognition results. Further, after obtaining the first and second recognition accuracies, the target recognition result can be selected from the first and second recognition results based on these accuracies. For example, the predicted recognition result with the higher recognition accuracy can be used as the target recognition result. Once the target recognition result is obtained, the training data for the second modality in the supplementary training data can be updated using the target recognition result.

[0111] For example, refer to Figure 7 , Figure 7The diagram illustrates a flowchart of updating supplementary training data in one embodiment. A first identification result may include a classification prediction answer and an inference prediction answer; a second identification result may also include a classification prediction answer and an inference prediction answer. A first identification accuracy can be determined by combining the classification prediction answer and the inference prediction answer from the first identification result, and a second identification accuracy can be determined by combining the classification prediction answer and the inference prediction answer from the second identification result. The identification result with the higher accuracy is taken as the target identification result, and then the target identification result is added as a label to the training data of the second modality in the supplementary training data.

[0112] In this embodiment, by adding the recognition results with higher recognition accuracy as labels to the training data of the second modality in the supplementary training data, the content recognition model trained based on the updated supplementary training set can output more accurate content.

[0113] In one embodiment, supplementary training data is stored in a supplementary training dataset; determining the first recognition accuracy of the first recognition result includes: determining the classification accuracy of the credible large model on the supplementary training dataset; determining the output accuracy of the first recognition result; determining the keyword matching degree of the first recognition result; and determining the first recognition accuracy of the first recognition result based on the classification accuracy, output accuracy, and keyword matching degree.

[0114] Specifically, supplementary training data is stored in a supplementary training dataset. The trusted large model and the content recognition model adjusted in the previous round can process each piece of supplementary training data in the supplementary training dataset to obtain multiple first recognition results and multiple second recognition results. The computer device can determine the classification accuracy of the trusted large model on the supplementary training dataset. For each first recognition result, when it is necessary to determine the first recognition accuracy of the first recognition result, the computer device can obtain the classification accuracy of the trusted large model on the supplementary training dataset, determine the output accuracy of the first recognition result, determine the keyword matching degree of the first recognition result, and combine the classification accuracy of the trusted large model on the supplementary training dataset, the output accuracy of the first recognition result, and the keyword matching degree to determine the first recognition accuracy of the first recognition result.

[0115] In one embodiment, determining the second recognition accuracy of the second recognition result includes: determining the classification accuracy of the credible large model on the supplementary training dataset; determining the output accuracy of the second recognition result; determining the keyword matching degree of the second recognition result; and determining the second recognition accuracy of the second recognition result based on the classification accuracy, the output accuracy of the second recognition result, and the keyword matching degree.

[0116] Here, classification accuracy refers to the accuracy with which the trusted large model classifies the first modality training data in the supplementary training dataset. For example, classification accuracy measures how accurately the trusted large model classifies images in the supplementary training dataset. The output accuracy of the first recognition result refers to the correlation between the first recognition result and the corresponding first modality training data in the supplementary training dataset. For example, the output accuracy of the first recognition result can be the correlation between the model's output and the images in the supplementary training data. The keyword matching degree of the first recognition result refers to the hit rate of the model's output on keywords. For example, the keyword matching degree of the first recognition result can be used to measure whether the model's output matches the keywords.

[0117] In this embodiment, the first recognition accuracy rate of the first recognition result is determined by combining classification accuracy, output accuracy and keyword matching degree, so that the determined first recognition accuracy rate is more accurate.

[0118] In one embodiment, determining the classification accuracy of the trusted large model on the supplementary training dataset includes: selecting samples from the supplementary training dataset that are classified as positive by the trusted large model, and determining precision based on the number of correctly classified samples and the number of samples classified as positive; determining the number of samples labeled as positive in the supplementary training dataset, and determining recall based on the number of samples labeled as positive and the number of samples labeled as positive that are classified as positive by the trusted large model; and determining the classification accuracy of the trusted large model on the supplementary training dataset based on precision and recall.

[0119] Specifically, it can be defined which categories are positive and which are negative, thus allowing the selection of supplementary training data from the supplementary training dataset that are classified as positive by the credible large model. For ease of description, the supplementary training data classified as positive will be referred to as positive-classified samples. For example, if the cartoon category is defined as positive, samples classified as cartoon by the credible large model can be selected from the supplementary training dataset. The computer device determines the number of correctly classified samples in the positive category based on the labels. Based on the number of correctly classified samples in the positive category and the total number of positive-classified samples, the precision is determined; that is, precision reflects how many samples predicted as positive are actually positive. Further, the computer device can identify supplementary training data labeled as positive in the supplementary training dataset. For ease of description, the supplementary training data labeled as positive in the supplementary training dataset will be referred to as positive-classified samples. The computer device then identifies the positive-classified samples among the positive-classified samples that are classified as positive by the credible large model. The recall rate is determined based on the number of samples labeled as positive and the number of samples labeled as positive that are classified as positive by the reliable large model.

[0120] Furthermore, the computer equipment determines the classification accuracy of the credible large model on the supplementary training dataset based on precision and recall.

[0121] In one embodiment, the accuracy of classification can be determined using the following formula:

[0122]

[0123]

[0124]

[0125] Where Pre represents precision; Recall represents recall; TP represents the number of samples that are actually positive and are classified as positive by the reliable large model; FP represents the number of samples that are actually negative and are classified as positive by the reliable large model; FN represents the number of samples that are actually positive but are classified as negative by the reliable large model; and F1 represents classification accuracy.

[0126] In this embodiment, F1 is the harmonic mean of precision and recall, which provides a comprehensive performance metric that makes the determined classification accuracy more accurate.

[0127] In one embodiment, the output accuracy includes the degree of matching; determining the output accuracy of the first recognition result includes: acquiring a pre-trained multimodal matching degree determination model; invoking the multimodal matching degree determination model to determine the degree of matching between the training data of the first modality in the supplementary training data and the first recognition result.

[0128] Specifically, a pre-trained multimodal matching degree determination model can be obtained. The training data of the first modality and the first recognition result from the supplementary training data are input into the multimodal matching degree determination model, and the model outputs the matching degree. For example, when the data of the first modality is an image, the data of the second modality is text, and the first recognition result is the data of the second modality, the pre-trained multimodal matching degree determination model can be a CLIP model (Contrastive Language-Image Pre-Training, a multimodal pre-trained neural network). The training data of the first modality and the first recognition result from the supplementary training data are input into the multimodal matching degree determination model. That is, the original image used to generate the first recognition result and the first recognition result of the text type are jointly input into the multimodal matching degree determination model, and the model performs image-text matching and outputs the matching degree.

[0129] In one embodiment, the matching degree CL can be output using the following formula:

[0130]

[0131] in, As the first identification result, To supplement the training data of the first mode in the training data, || represents the norm.

[0132] In this embodiment, by determining the degree of matching between the training data of the first modality and the first recognition result in the supplementary training data, the accuracy of the first recognition result output by the reliable large model can be accurately measured based on the degree of matching.

[0133] In one embodiment, it is readily understood that the degree of matching between the training data of the first modality in the supplementary training data and the second recognition result can also be determined in the manner described above, and then the second recognition accuracy of the second recognition result can be determined based on the degree of matching.

[0134] In one embodiment, output accuracy includes text similarity; determining the output accuracy of the first recognition result includes: determining the inference reference answer in the training data of the second modality in the supplementary training data; and determining the text similarity between the inference reference answer and the inference prediction answer of the first recognition result.

[0135] Specifically, the training data for the second modality in the supplementary training data may include answers to questions, which serve as reference answers. Answers to classification questions in the second modality training data are the classification reference answers, and answers to inference questions are the inference reference answers. For example, the training data for the second modality in the supplementary training data could be: "Question: Please determine which of the four categories this image belongs to: 0 (cartoon), 1 (advertisement), 2 (chart), 3 (other)? Answer: 0. Question: Please briefly describe this image. Answer: This is a normal cartoon image showing a group of cartoon characters sitting on a green grassy hillside. Question: Please provide the basis for your judgment. Answer: These characters look like characters from animation A; they seem to be enjoying outdoor activities." Here, "Please determine which of the four categories this image belongs to: 0 (cartoon), 1 (advertisement), 2 (chart), 3 (other)?" is the classification question in the supplementary training data; "Please briefly describe this image" and "Please provide the basis for your judgment" are the inference questions in the supplementary training data. For ease of description, the classification problem in the supplementary training data will be referred to as the supplementary classification problem, and the reasoning problem in the supplementary training data will be referred to as the supplementary reasoning problem. Therefore, "This is a normal cartoon image showing a group of cartoon characters sitting on a green grassy hillside" and "These characters look like characters from animation A; they seem to be enjoying outdoor activities" are the reasoning reference answers in the supplementary training data corresponding to the supplementary reasoning problems.

[0136] In one embodiment, the first identification result may include a predicted inference answer output for the supplementary inference question. For example, the first identification result may include a predicted inference answer output by the credible large model for "Please briefly describe this image," such as "This is a cartoon image showing a group of anime characters sitting on a hillside," and a predicted inference answer output for "Please provide the basis for your judgment," such as "These characters look like characters from animation A." Furthermore, the computer device can determine the text similarity between the predicted inference answer and the inference reference answer in the first identification result using a preset text similarity calculation formula.

[0137] In one embodiment, the text similarity BL can be determined using the following text similarity calculation formula:

[0138]

[0139] Where BP is the penalty factor; N is the maximum length of the n-gram; These are weighting coefficients; It refers to the precision of n-grams. The upper limit of N is 4, meaning that at most 4-gram precision is counted; an n-gram refers to N consecutive characters or words in the text.

[0140] In one embodiment, it is readily understood that the text similarity between the inference reference answer and the inference prediction answer in the second recognition result can also be determined in the manner described above, and then the second recognition accuracy of the second recognition result can be determined based on the text similarity.

[0141] In this embodiment, by determining the text similarity between the inference reference answer and the inference prediction answer in the first recognition result, the accuracy of the first recognition result output by the credible large model can be accurately measured based on the text similarity.

[0142] In one embodiment, determining the keyword matching degree of the first identification result includes: determining the inference reference answer in the training data of the second modality in the supplementary training data; determining the degree of overlap between the inference reference answer and the inference prediction answer in the first identification result; and determining the keyword matching degree of the first identification result based on the degree of overlap.

[0143] Specifically, the images involve cartoon characters, advertising slogans, etc., and whether these are mentioned in the generated text is crucial. Therefore, the keyword matching degree of the first recognition result can be determined. The computer device determines the inference reference answer in the training data of the second modality in the supplementary training data, and determines the inference prediction answer in the first recognition result, and determines the degree of overlap between the inference reference answer and the inference prediction answer. Once the degree of overlap is determined, the computer device can determine the word matching degree of the first recognition result based on the degree of overlap.

[0144] In one embodiment, the keyword matching degree RO can be determined by the following formula:

[0145]

[0146] Wherein, reference represents a reasoning-based reference answer; candidate represents a reasoning-based predicted answer. This represents the degree of overlap between the inference-predicted answer and the inference-referenced answer.

[0147] In one embodiment, it is readily understood that the keyword matching degree of the second identification result can also be determined in the manner described above.

[0148] In this embodiment, by determining the keyword matching degree of the first identification result, the accuracy of the first identification result output by the reliable large model can be accurately measured based on the keyword matching degree of the first identification result.

[0149] In one embodiment, the output accuracy includes the degree of matching and text similarity; determining the first recognition accuracy of the first recognition result based on the classification accuracy, output accuracy, and keyword matching degree includes: performing a weighted summation of the classification accuracy, matching degree, text similarity, and keyword matching degree to obtain the first recognition accuracy of the first recognition result. It is readily understood that the second recognition accuracy of the second recognition result can also be determined using this weighted summation method.

[0150] In one embodiment, once the first recognition accuracy and the second recognition accuracy are determined, the recognition result with the higher recognition accuracy can be used as the target recognition result. For example, when the first recognition accuracy of the first recognition result is higher than the second recognition accuracy of the second recognition result, the first recognition result is used as the target recognition result.

[0151] In one embodiment, the training data of the second modality in the supplementary training data includes a supplementary inference question and an inference reference answer; updating the training data of the second modality in the supplementary training data according to the target recognition result includes: adding the classification prediction answer in the target recognition result to the supplementary inference question, and replacing the inference reference answer in the supplementary training data with the inference prediction answer in the target recognition result.

[0152] Specifically, the training data for the second modality in the supplementary training data includes supplementary inference questions and inference reference answers. The classification prediction answer from the object recognition result can be added to the supplementary inference question, and the inference prediction answer from the object recognition result can replace the inference reference answer. For example, if the supplementary training data includes the supplementary inference question "Please describe this picture," then the classification prediction answer "cartoon" from the object recognition result can be added to the supplementary inference question to obtain the new supplementary inference question "Please describe this cartoon image." The supplementary training data may also include inference reference answers. For example, it may include "This is a cartoon picture, showing a group of cartoon characters sitting on a hillside." Then, the inference prediction answer "This is a normal cartoon picture, showing a group of cartoon characters sitting on a green grassy hillside, they seem to be enjoying outdoor activities" from the object recognition result can be replaced with the inference reference answer "This is a cartoon picture, showing a group of cartoon characters sitting on a hillside," to obtain a new inference reference answer.

[0153] In this embodiment, since the target recognition result is more accurate, the training data of the second modality in the supplementary training data is updated based on the target recognition result, so that the updated supplementary training data is more accurate. As a result, the content recognition model trained based on the more accurate supplementary training data can output more accurate content.

[0154] In one embodiment, reference Figure 8 , Figure 8 This diagram illustrates the iterative training of a content recognition model in one embodiment. Iw represents an image obtained from the network, Tw represents text obtained from the network, Ih represents an image with manual annotation, Th represents manually annotated text, and Ti represents the text output by the machine learning model. Image Iw and text Tw can be obtained from the network, and the initial content recognition model can be pre-trained using these images to obtain the content recognition model (i.e., the pre-trained content recognition model). The content recognition model can be adjusted in the first round using Ih and Th to obtain the content recognition model after the first round of adjustment. Content recognition can be performed on Iw using the credible large model and the content recognition model adjusted in the previous round, respectively, to obtain a first recognition result and a second recognition result. Based on the first and second recognition results, the target recognition result Ti is determined, and Tw is replaced based on the target recognition result Ti to obtain training data (Iw, Ti). Based on the training data (Ih, Th) and (Iw, Ti), the content recognition model adjusted in the previous round is adjusted for the current round. Where (Ih, Th) are the aforementioned reliable training data; (Iw, Ti) are the updated supplementary training data.

[0155] In one embodiment, such as Figure 9 The diagram illustrates the overall training process of a content recognition model. Taking images as the first modality and text as the second modality as an example, images can be collected, classified, and inferred to obtain category construction, content description, and detailed inference. Based on the images, category construction, content description, and detailed inference, a multimodal dataset is generated, including supplementary training data. An open-source multimodal dataset, including open-source training data, can be obtained and pre-trained on the content recognition model. The pre-trained content recognition model is then fine-tuned multiple times based on the multimodal dataset and a trusted training dataset. This fine-tuning can be achieved through iterative updates of the content recognition model and continuous updates to the data in the multimodal dataset using a trusted large model.

[0156] In one embodiment, such as Figure 10 As shown, a content recognition method is provided. Taking the application of this method to a computer device as an example, the computer device can provide... Figure 1 The server or terminal in the process. The content recognition method includes the following steps:

[0157] Step 1002: Obtain first data of the first modality and second data of the second modality; the second data includes at least one question related to the first data.

[0158] Specifically, first data of the first modality and second data of the second modality can be acquired, for example, images and text can be acquired, wherein the text may include at least one question for the image.

[0159] Step 1004: Perform feature extraction on the first data to obtain the first feature in the feature space of the first modality. Specifically, feature extraction processing can be performed on the first data to obtain the first feature in the feature space of the first modality. For example, the image can be encoded to obtain image features in the image feature space.

[0160] Step 1006: Perform feature extraction on the second data to obtain the second features in the feature space of the second modality. Specifically, feature extraction processing can be performed on the second data to obtain the second features in the feature space of the second modality. For example, text can be encoded to obtain text features in the text feature space.

[0161] Step 1008: Concatenate the first feature and the second feature in the same feature space to obtain the concatenated feature.

[0162] Specifically, the first feature and the second feature can be concatenated in the same feature space to obtain concatenated features. For example, the first feature can be transformed into the feature space of the second modality, and the features in the feature space of the second modality can be concatenated to obtain concatenated features.

[0163] Step 1010: When the second data includes a category problem for the first data, category identification is performed based on the splicing features to obtain the category of the first data in the first modality.

[0164] Specifically, the second data may include a category question for the first data, such as "Please determine which of the four categories this image belongs to: 0 (cartoon), 1 (advertisement), 2 (chart), 3 (other)?" This allows the spliced ​​features to be input into the discriminative module in the content recognition model, which can output the category of the first data in the first modality, such as the category of the image.

[0165] Step 1012: When the second data includes a reasoning question for the first data, perform data reasoning based on the splicing features to obtain the answer data corresponding to the reasoning question.

[0166] Specifically, the second data may include inference questions based on the first data, such as "Please describe this picture" or "Please give a reason," thereby inputting the spliced ​​features into the generative module in the content recognition model. The generative module can output the answer data corresponding to the inference question, such as "This is a cartoon picture, which shows a group of anime characters sitting on a hillside."

[0167] In the aforementioned content recognition method, by acquiring first data from a first modality and second data from a second modality, features can be extracted from the first and second data respectively, yielding first and second features in their respective feature spaces. These first and second features can then be concatenated within the same feature space to obtain concatenated features. Based on these concatenated features, the category of the first data in the first modality and the corresponding answer data for the reasoning question can be output. Since not only can classification results be output, but also answer data corresponding to the reasoning question, this answer data can include the reasoning process for outputting the classification result. This allows users to understand the reasoning process behind the model's classification result, improving the user experience. Furthermore, by concatenating the first and second features within the same feature space, data from different modalities can be aligned. This allows the discriminative and generative modules to mine deeper relationships between data from different modalities based on the alignment results, outputting more accurate classification and reasoning results.

[0168] Furthermore, for some specialized content classification tasks, discriminative models offer superior training and inference speeds as well as better classification performance. Secondly, traditional machine learning models, due to limitations in their network structure, cannot effectively integrate classification and generation tasks. Therefore, this application's embodiments add a discriminative module to the content recognition model, enabling rapid output of classification results, and also add a generation module, allowing the model to output inference results simultaneously with classification results.

[0169] In one embodiment, the content recognition method is executed by a trained content recognition model, which is obtained by training the content recognition model using the content recognition model training method described above.

[0170] In one embodiment, the first feature and the second feature are concatenated in the same feature space to obtain concatenated features, including: obtaining a projection matrix; transforming the first feature into the feature space of the second modality based on the projection matrix to obtain a third feature in the feature space of the second modality; and concatenating the third feature and the second feature in the feature space of the second modality to obtain concatenated features.

[0171] Specifically, the projection matrix is ​​a parameter in the content recognition model. The content recognition model can multiply the projection matrix with the first feature in the feature space of the first modality to obtain the third feature in the feature space of the second modality. Thus, the content recognition model can concatenate the second feature and the third feature to obtain the concatenated feature.

[0172] In this embodiment, by splicing the first feature and the second feature in the same feature space, data under different modalities can be aligned. Thus, the discriminative module and the generative module can mine the deep correlation between data under different modalities based on the alignment result, and output more accurate classification results and inference results.

[0173] In one embodiment, reference Figure 11 , Figure 11 This diagram illustrates the overall training process of a content recognition model in one embodiment. Open-source training data is available for pre-training the initial content recognition model. The purpose of pre-training is to improve the model's ability to align features in the feature space of the first modality with features in the feature space of the second modality; for example, it aims to improve the model's ability to align image features with text features. Trusted training data and supplementary training data are available for multiple rounds of fine-tuning of the pre-trained content recognition model to obtain the final content recognition model. The purpose of these multiple rounds of fine-tuning is to improve the model's ability to understand, judge, and generate multimodal content in professional scenarios. During the use of the content recognition model, first data from the first modality and second data from the second modality can be input into the content recognition model, allowing the model to output category results and inference results.

[0174] In one specific embodiment, a content recognition model training method is provided, including:

[0175] The computer device acquires open-source training data and an initial content recognition model; the training data for the second modality in the open-source training data includes open-source questions and the corresponding open-source answers.

[0176] The computer device invokes the initial content recognition model and, guided by the open-source problem, performs content recognition on the training data of the first modality in the open-source training data to obtain the recognition prediction results of the pre-training stage.

[0177] The computer device allows the feature space transformation module in the initial content recognition model to participate in training. Based on the difference between the open-source answer and the recognition prediction results in the pre-training stage, the parameters of the feature space transformation module in the initial content recognition model are adjusted to obtain the content recognition model.

[0178] In the first round of model adjustments for the content recognition model, the computer device acquires credible training data, which includes training data for the first modality and training data for the second modality. The training data for the second modality in the credible training data includes training questions and training answers corresponding to the training questions. The training questions include classification training questions and inference training questions. The training answers include standard classification results and standard inference results.

[0179] Guided by the training question, the content recognition model performs content recognition on the first modality of the training data in the credible training data to obtain the recognition prediction results of the first round. The recognition prediction results of the first round include the classification prediction answer and the reasoning prediction answer of the first round.

[0180] The content recognition model adjusts the parameters of the discriminative module based on the first difference between the classification prediction answer and the standard classification result in the first round; it adjusts the parameters of the generative module based on the second difference between the inference prediction answer and the standard inference result in the first round; and it adjusts the parameters of the feature space transformation module based on the first and second differences to obtain the content recognition model after the first round of adjustments.

[0181] In the current round of model adjustments following the first round of adjustments, computer equipment acquires supplementary training data.

[0182] The computer device calls the preset trusted large model and the content recognition model adjusted in the previous round, respectively, and recognizes the training data of the first modality in the supplementary training data based on the training data of the second modality in the supplementary training data, and obtains the first recognition result and the second recognition result respectively.

[0183] The computer device selects samples from the supplementary training dataset that are classified as positive by the trusted large model, and determines the precision based on the number of correctly classified samples and the total number of samples classified as positive. It also determines the number of samples labeled as positive in the supplementary training dataset, and determines the recall based on the number of positively labeled samples and the number of positively labeled samples that are classified as positive by the trusted large model. Finally, based on the precision and recall, the classification accuracy of the trusted large model on the supplementary training dataset is determined.

[0184] The computer device acquires a pre-trained multimodal matching degree determination model; it calls the multimodal matching degree determination model to determine the degree of matching between the training data of the first modality in the supplementary training data and the first recognition result.

[0185] The computer device determines the inference reference answer in the training data of the second modality in the supplementary training data; and determines the text similarity between the inference prediction answer and the inference reference answer in the first recognition result.

[0186] The computer device determines the inference reference answer in the training data of the second modality in the supplementary training data; determines the degree of overlap between the inference prediction answer and the inference reference answer in the first recognition result, and determines the keyword matching degree of the first recognition result based on the degree of overlap.

[0187] The computer equipment determines the first recognition accuracy rate of the first recognition result based on classification accuracy, matching degree, text similarity, and keyword matching degree.

[0188] The computer device determines a second recognition accuracy rate of the second recognition result; based on the first recognition accuracy rate and the second recognition accuracy rate, it filters out the target recognition result from the first recognition result and the second recognition result;

[0189] The computer device adds the classification prediction answer from the target recognition result to the supplementary reasoning question, and replaces the reasoning reference answer with the reasoning prediction answer from the target recognition result; it then adjusts the content recognition model from the previous round of adjustments based on the updated supplementary training data.

[0190] The content recognition model acquires first data of a first modality and second data of a second modality; the second data includes at least one question related to the first data; features are extracted from the first data to obtain a first feature in the feature space of the first modality; features are extracted from the second data to obtain a second feature in the feature space of the second modality.

[0191] The content recognition model obtains a projection matrix, and based on the projection matrix, the first feature is transformed into the feature space of the second modality to obtain the third feature in the feature space of the second modality; the third feature and the second feature are concatenated in the feature space of the second modality to obtain the concatenated feature.

[0192] When the second data includes a category question for the first data, the content recognition model performs category recognition based on the splicing features to obtain the category of the first data in the first modality; when the second data includes a reasoning question for the first data, data reasoning is performed based on the splicing features to obtain the answer data corresponding to the reasoning question.

[0193] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0194] This application also provides an application scenario in which the above-described content recognition method is applied. Specifically, the content recognition method is applied in this scenario as follows:

[0195] When content recognition is required for an image, both the image and text type of question can be input into the content recognition model, which can then output the corresponding answer to the question based on the image.

[0196] This application also provides another application scenario in which the above-described content recognition method is applied. Specifically, the content recognition method is applied in this scenario as follows:

[0197] When content recognition is required for audio, audio and text-type questions can be input into the content recognition model, which can then output the corresponding answer to the question based on the audio.

[0198] The above application scenarios are merely illustrative. It is understood that the application of the content recognition model training method and content recognition method provided in the embodiments of this application is not limited to the above scenarios.

[0199] Based on the same inventive concept, this application also provides a content recognition model training apparatus for implementing the content recognition model training method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more content recognition model training apparatus embodiments provided below can be found in the limitations of the content recognition model training method described above, and will not be repeated here.

[0200] In one embodiment, such as Figure 12 As shown, a content recognition model training device 1200 is provided, including: a first-round adjustment module 1202, a comparison module 1204, and a current-round adjustment module 1206, wherein:

[0201] The first-round adjustment module 1202 is used to obtain reliable training data in the first round of model adjustment for the content recognition model, and use the reliable training data to adjust the content recognition model to obtain the content recognition model after the first round of adjustment.

[0202] The comparison module 1204 is used to obtain supplementary training data in the current round of model adjustment after the first round of adjustment; respectively call the preset reliable large model and the content recognition model after the previous round of adjustment, and identify the training data of the first modality in the supplementary training data based on the training data of the second modality in the supplementary training data, and obtain the first recognition result and the second recognition result respectively.

[0203] The current round adjustment module 1206 is used to update the training data of the second modality in the supplementary training data according to the respective recognition accuracy of the first recognition result and the second recognition result, and to adjust the content recognition model after the previous round adjustment based on the updated supplementary training data.

[0204] In one embodiment, the content recognition model training device 1200 further includes a pre-training module, which is used to acquire open-source training data and an initial content recognition model. The training data of the second modality in the open-source training data includes open-source questions and corresponding open-source answers. The initial content recognition model is invoked to perform content recognition on the training data of the first modality in the open-source training data according to the guidance of the open-source questions, thereby obtaining the recognition prediction results of the pre-training stage. The feature space transformation module in the initial content recognition model is allowed to participate in the training. Based on the difference between the open-source answers and the recognition prediction results of the pre-training stage, the parameters of the feature space transformation module in the initial content recognition model are adjusted to obtain the content recognition model.

[0205] In one embodiment, the trusted training data includes training data for a first modality and training data for a second modality; the training data for the second modality in the trusted training data includes training questions and training answers corresponding to the training questions; the first-round adjustment module 1202 is further used to call the content recognition model to perform content recognition on the training data of the first modality in the trusted training data according to the guidance of the training questions, and obtain the recognition prediction result of the first round; based on the difference between the training answers and the recognition prediction result of the first round, the parameters of the content recognition model are adjusted to obtain the content recognition model after the first round adjustment.

[0206] In one embodiment, the training problem includes a classification training problem and an inference training problem; the recognition prediction result of the first round includes the classification prediction answer of the first round and the inference prediction answer of the first round; the training answer includes the standard classification result and the standard inference result; the first-round adjustment module 1202 is further configured to adjust the parameters of the discriminative module in the content recognition model based on the first difference between the classification prediction answer of the first round and the standard classification result; adjust the parameters of the generative module in the content recognition model based on the second difference between the inference prediction answer of the first round and the standard inference result; and adjust the parameters of the feature space transformation module in the content recognition model based on the first difference and the second difference.

[0207] In one embodiment, the current round adjustment module 1206 is further configured to determine a first recognition accuracy of the first recognition result; determine a second recognition accuracy of the second recognition result; select a target recognition result from the first recognition result and the second recognition result based on the first recognition accuracy and the second recognition accuracy; and update the training data of the second modality in the supplementary training data based on the target recognition result.

[0208] In one embodiment, supplementary training data is stored in a supplementary training dataset; the current round adjustment module 1206 is further configured to determine the classification accuracy of the credible large model on the supplementary training dataset; determine the output accuracy of the first recognition result; determine the keyword matching degree of the first recognition result; and determine the first recognition accuracy rate of the first recognition result based on the classification accuracy, the output accuracy of the first recognition result, and the keyword matching degree.

[0209] In one embodiment, the current round adjustment module 1206 is further configured to filter samples classified as positive by the credible large model from the supplementary training dataset, and determine the precision based on the number of correctly classified samples and the number of samples classified as positive; determine the number of samples labeled as positive in the supplementary training dataset, and determine the recall based on the number of samples labeled as positive and the number of samples labeled as positive that are classified as positive by the credible large model; and determine the classification accuracy of the credible large model on the supplementary training dataset based on the precision and recall.

[0210] In one embodiment, the current round adjustment module 1206 is further configured to obtain a pre-trained multimodal matching degree determination model; call the multimodal matching degree determination model to determine the degree of matching between the first recognition result and the training data of the first modality in the supplementary training data.

[0211] In one embodiment, the output accuracy includes text similarity; the current round adjustment module 1206 is also used to determine the inference reference answer in the training data of the second modality in the supplementary training data; and to determine the text similarity between the inference reference answer and the inference prediction answer in the first recognition result.

[0212] In one embodiment, the current round adjustment module 1206 is further configured to determine the inference reference answer in the training data of the second modality in the supplementary training data; determine the degree of overlap between the inference reference answer and the inference prediction answer in the first recognition result; and determine the keyword matching degree of the first recognition result based on the degree of overlap.

[0213] In one embodiment, supplementary training data is stored in a supplementary training dataset; the current round adjustment module 1206 is further configured to determine the classification accuracy of the credible large model on the supplementary training dataset; determine the output accuracy of the second recognition result; determine the keyword matching degree of the second recognition result; and determine the second recognition accuracy of the second recognition result based on the classification accuracy, the output accuracy of the second recognition result, and the keyword matching degree.

[0214] In one embodiment, the training data for the second modality in the supplementary training data includes a supplementary inference question and an inference reference answer; the current round adjustment module 1206 is further configured to add the classification prediction answer in the target recognition result to the supplementary inference question, and replace the inference reference answer with the inference prediction answer in the target recognition result.

[0215] Based on the same inventive concept, this application also provides a content recognition device for implementing the content recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more content recognition device embodiments provided below can be found in the limitations of the content recognition method described above, and will not be repeated here.

[0216] In one embodiment, such as Figure 13 As shown, a content recognition device 1300 is provided, including: a feature extraction module 1302, a feature splicing module 1304, and an output module 1306, wherein:

[0217] The feature extraction module 1302 is used to acquire first data of a first modality and second data of a second modality; the second data includes at least one question related to the first data; perform feature extraction on the first data to obtain a first feature in the feature space of the first modality; and perform feature extraction on the second data to obtain a second feature in the feature space of the second modality.

[0218] The feature splicing module 1304 is used to splice the first feature and the second feature in the same feature space to obtain spliced ​​features;

[0219] The output module 1306 is used to perform category recognition based on splicing features to obtain the category of the first data in a first modality when the second data includes a category question for the first data; and to perform data reasoning based on splicing features to obtain the answer data corresponding to the reasoning question when the second data includes a reasoning question for the first data.

[0220] In one embodiment, the feature splicing module 1306 is further configured to obtain a projection matrix, transform the first feature into the feature space of the second modality based on the projection matrix, and obtain a third feature in the feature space of the second modality; and splice the third feature and the second feature in the feature space of the second modality to obtain spliced ​​features.

[0221] The aforementioned content recognition model training device and each module in the content recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0222] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 14 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores content recognition data and content recognition model training. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements a content recognition method and a content recognition model training method.

[0223] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0224] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0225] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0226] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0227] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0228] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0229] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0230] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A content recognition model training method, characterized in that, The method includes: In the first round of model adjustment for the content recognition model, reliable training data is obtained, and the content recognition model is adjusted using the reliable training data to obtain the content recognition model after the first round of adjustment. In the current round of model adjustment following the first round of adjustments, supplementary training data is obtained; The preset trusted large model and the content recognition model adjusted in the previous round are called respectively. Based on the training data of the second modality in the supplementary training data, the training data of the first modality in the supplementary training data are recognized to obtain the first recognition result and the second recognition result respectively. Based on the respective recognition accuracy of the first recognition result and the second recognition result, the training data of the second modality in the supplementary training data is updated, and the content recognition model adjusted in the previous round is adjusted based on the updated supplementary training data.

2. The method according to claim 1, characterized in that, Before obtaining reliable training data in the first round of model tuning for the content recognition model, the method further includes: Obtain open-source training data and an initial content recognition model; the training data for the second modality in the open-source training data includes open-source questions and the corresponding open-source answers to the open-source questions; The initial content recognition model is invoked to perform content recognition on the training data of the first modality in the open-source training data, guided by the open-source problem, to obtain the recognition prediction results in the pre-training stage; The feature space transformation module in the initial content recognition model is allowed to participate in training. Based on the difference between the open-source answer and the recognition prediction results in the pre-training stage, the parameters of the feature space transformation module in the initial content recognition model are adjusted to obtain the content recognition model.

3. The method according to claim 1, characterized in that, The trusted training data includes training data for the first modality and training data for the second modality; The training data for the second modality in the trusted training data includes the training question and the training answer corresponding to the training question; The step of adjusting the content recognition model using the trusted training data to obtain the content recognition model after the first round of adjustment includes: The content recognition model is invoked to perform content recognition on the training data of the first modality in the trusted training data, guided by the training question, to obtain the recognition prediction result of the first round. Based on the difference between the training response and the recognition prediction result of the first round, the parameters of the content recognition model are adjusted to obtain the content recognition model after the first round of adjustment.

4. The method according to claim 3, characterized in that, The training problem includes a classification training problem and an inference training problem; the recognition prediction result of the first round includes the classification prediction answer of the first round and the inference prediction answer of the first round; the training answer includes the standard classification result and the standard inference result; the content recognition model is adjusted based on the difference between the training answer and the recognition prediction result of the first round to obtain the content recognition model after the first round adjustment, including: Based on the first difference between the classification prediction answer in the first round and the standard classification result, the parameters of the discriminative module in the content recognition model are adjusted. Based on the second difference between the inference prediction answer of the first round and the standard inference result, the parameters of the generative module in the content recognition model are adjusted. The parameters of the feature space transformation module in the content recognition model are adjusted based on the first difference and the second difference.

5. The method of claim 1, characterized in that, The step of updating the training data of the second modality in the supplementary training data according to the respective recognition accuracy rates of the first recognition result and the second recognition result includes: Determine the first recognition accuracy of the first recognition result; Determine the second recognition accuracy of the second recognition result; Based on the first recognition accuracy and the second recognition accuracy, the target recognition result is selected from the first recognition result and the second recognition result; Based on the target recognition result, the training data of the second modality in the supplementary training data is updated.

6. The method according to claim 5, characterized in that, The supplementary training data is stored in the supplementary training dataset; determining the first recognition accuracy of the first recognition result includes: Determine the classification accuracy of the reliable large model on the supplementary training dataset; Determine the accuracy of the output of the first identification result; Determine the keyword matching degree of the first identification result; Based on the classification accuracy, the output accuracy of the first identification result, and the keyword matching degree, the first identification accuracy rate of the first identification result is determined.

7. The method according to claim 6, characterized in that, Determining the classification accuracy of the reliable large model on the supplementary training dataset includes: The samples classified as positive by the reliable large model are selected from the supplementary training dataset, and the accuracy is determined based on the number of correctly classified samples and the number of samples classified as positive. Determine the number of samples labeled as positive in the supplementary training dataset, and determine the recall rate based on the number of samples labeled as positive and the number of samples labeled as positive that are classified as positive by the reliable large model. The classification accuracy of the reliable large model on the supplementary training dataset is determined based on the precision and the recall.

8. The method according to claim 6, characterized in that, The output accuracy includes the degree of matching; determining the output accuracy of the first recognition result includes: Obtain pre-trained multimodal matching degree to determine the model; The multimodal matching degree determination model is invoked to determine the degree of matching between the first recognition result and the training data of the first modality in the supplementary training data.

9. The method according to claim 6, characterized in that, The output accuracy includes text similarity; determining the output accuracy of the first recognition result includes: Determine the inference reference answer in the training data of the second modality in the supplementary training data; Determine the text similarity between the inference reference answer and the inference prediction answer in the first recognition result.

10. The method according to claim 6, characterized in that, Determining the keyword matching degree of the first identification result includes: Determine the inference reference answer in the training data of the second modality in the supplementary training data; Determine the degree of overlap between the inference reference answer and the inference prediction answer in the first identification result, and determine the keyword matching degree of the first identification result based on the degree of overlap.

11. The method according to claim 5, characterized in that, The supplementary training data is stored in the supplementary training dataset; determining the second recognition accuracy of the second recognition result includes: Determine the classification accuracy of the reliable large model on the supplementary training dataset; Determine the accuracy of the output of the second identification result; Determine the keyword matching degree of the second identification result; The second recognition accuracy rate of the second recognition result is determined based on the classification accuracy, the output accuracy of the second recognition result, and the keyword matching degree.

12. The method according to claim 5, characterized in that, The training data for the second modality in the supplementary training data includes supplementary reasoning questions and reasoning reference answers; The step of updating the training data of the second modality in the supplementary training data according to the target recognition result includes: The classification prediction answer from the target recognition result is added to the supplementary reasoning question, and the reasoning prediction answer from the target recognition result replaces the reasoning reference answer.

13. A content recognition method, characterized in that, The method includes: Acquire first data for a first modality and second data for a second modality; the second data includes at least one question related to the first data; Feature extraction is performed on the first data to obtain the first feature in the feature space of the first modality; Feature extraction is performed on the second data to obtain the second feature in the feature space of the second modality; The first feature and the second feature are concatenated in the same feature space to obtain the concatenated feature. When the second data includes a category problem related to the first data, category identification is performed based on the splicing features to obtain the category of the first data in the first modality; When the second data includes a reasoning question about the first data, data reasoning is performed based on the splicing features to obtain the answer data corresponding to the reasoning question.

14. The method according to claim 13, characterized in that, The content recognition method is executed by a trained content recognition model, which is trained by any one of the methods in claims 1 to 12.

15. The method according to claim 13, characterized in that, The step of concatenating the first feature and the second feature in the same feature space to obtain the concatenated feature includes: Obtain the projection matrix, and based on the projection matrix, transform the first feature into the feature space of the second modality to obtain the third feature in the feature space of the second modality; The third feature and the second feature are concatenated in the feature space of the second modality to obtain the concatenated feature.

16. A content recognition model training device, characterized in that, The device includes: The first-round adjustment module is used to obtain reliable training data in the first round of model adjustment for the content recognition model, and use the reliable training data to adjust the content recognition model to obtain the content recognition model after the first round of adjustment. The comparison module is used to obtain supplementary training data in the current round of model adjustment after the first round of adjustment; it calls the preset reliable large model and the content recognition model after the previous round of adjustment respectively, and identifies the training data of the first modality in the supplementary training data based on the training data of the second modality in the supplementary training data, and obtains the first recognition result and the second recognition result respectively. The current round adjustment module is used to update the training data of the second modality in the supplementary training data according to the respective recognition accuracy of the first recognition result and the second recognition result, and to adjust the content recognition model after the previous round adjustment based on the updated supplementary training data.

17. A content recognition device, characterized in that, The device includes: A feature extraction module is used to acquire first data of a first modality and second data of a second modality; the second data includes at least one question related to the first data; feature extraction is performed on the first data to obtain a first feature in the feature space of the first modality; feature extraction is performed on the second data to obtain a second feature in the feature space of the second modality; The feature splicing module is used to splice the first feature and the second feature in the same feature space to obtain spliced ​​features; The output module is configured to, when the second data includes a category question for the first data, perform category recognition based on the splicing features to obtain the category of the first data in the first modality; and when the second data includes a reasoning question for the first data, perform data reasoning based on the splicing features to obtain the answer data corresponding to the reasoning question.

18. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 15.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 15.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images