Multi-modal model processing method and device, equipment, storage medium and program product

Abstract

The invention relates to a multi-modal model processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: generating description information of visual data according to the visual data and auditory data in multi-modal data; on the basis of the description information, sample information including questions and answer options is generated; according to the sample question information, the visual data and the auditory data, generating answers and reasoning process information corresponding to the answers; and under the condition that the answer is correct, training a pre-trained multi-modal model based on the visual data, the auditory data, and the corresponding sample information and the reasoning process information when the answer is correct to obtain a target multi-modal model. By adopting the method, the illusion problem of the multi-modal model can be effectively relieved, so that the reasoning accuracy of the multi-modal model is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal model processing method, apparatus, device, storage medium, and program product. Background Art

[0002] Multimodal models can jointly process data from multiple modalities, such as text, images, audio, and video. They have shown great potential in a variety of fields and are therefore widely used. For example, to summarize the content of a video, one can input the video and the corresponding demand information into a trained multimodal model to obtain a description of the video.

[0003] The training of multimodal models mainly includes pre-training and instruction fine-tuning stages, both of which are optimized by calculating the loss of the text output by the multimodal model and the annotated text. However, this optimization strategy only relies on data distribution and is prone to introducing noise, causing hallucination problems in the multimodal model, thereby affecting the reasoning accuracy of the multimodal model. Summary of the Invention

[0004] Based on this, it is necessary to provide a multimodal model processing method, device, equipment, storage medium and program product to address the above technical problems, which can effectively alleviate the hallucination problem of the multimodal model and thus improve the reasoning accuracy of the multimodal model.

[0005] In a first aspect, the present application provides a multimodal model processing method, the method comprising:

[0006] generating description information of the visual data based on the visual data and the auditory data in the multimodal data;

[0007] generating sample question information including questions and answer options based on the description information;

[0008] generating an answer and reasoning process information corresponding to the answer based on the sample question information, the visual data, and the auditory data;

[0009] When the answer is correct, the pre-trained multimodal model is trained based on the visual data, the auditory data, the sample question information corresponding to the correct answer, and the reasoning process information to obtain a target multimodal model.

[0010] In one embodiment, generating sample question information including questions and answer options based on the description information includes:

[0011] generating questions based on the description information;

[0012] generating corresponding answer options based on the description information and the question;

[0013] The question and the answer options are combined to obtain sample question information.

[0014] In one embodiment, the method further comprises:

[0015] determining a correct answer to the question based on the descriptive information;

[0016] When the answer is correct, performing preliminary training on the pre-trained multimodal model based on the visual data, the auditory data, the sample question information corresponding to the correct answer, and the reasoning process information includes:

[0017] When the answer is consistent with the correct answer, the visual data, the auditory data and the question in the sample question information corresponding to the correct answer are used as the second training data, and the corresponding answer and the reasoning process information are used as the fourth label information to perform preliminary training on the pre-trained multimodal model so that the multimodal model learns the reasoning process of the answer.

[0018] In a second aspect, the present application further provides a multimodal model processing device, the device comprising:

[0019] A first generating module is configured to generate description information of the visual data based on the visual data and the auditory data in the multimodal data;

[0020] A second generating module, configured to generate sample question information including questions and answer options based on the description information;

[0021] a third generating module, configured to generate an answer and reasoning process information corresponding to the answer based on the sample question information, the visual data, and the auditory data;

[0022] A training module is used to train the pre-trained multimodal model based on the visual data, the auditory data, the sample question information corresponding to the correct answer and the reasoning process information to obtain a target multimodal model when the answer is correct.

[0023] In one embodiment, the training module is further used to perform preliminary training on the pre-trained multimodal model based on the visual data, the auditory data, the sample question information corresponding to the correct answer, and the reasoning process information; after completing the preliminary training, the sample question information in which the multimodal model has reasoning errors and the corresponding visual data and auditory data are selected, and model training is performed on the multimodal model after preliminary training based on the selected data to obtain a target multimodal model.

[0024] In one embodiment, the training module is further used to extract features from the visual data and the auditory data through a pre-trained multimodal model to obtain visual frame features and auditory features; perform feature complementation on the auditory features based on the visual frame features to obtain complemented auditory features; generate summary information of the visual data based on the visual frame features and the complemented auditory features; perform question reasoning on the sample question information corresponding to the correct answer based on the summary information to obtain predicted reasoning process information and a predicted answer; and perform parameter optimization on the pre-trained multimodal model based on the predicted reasoning process information, the reasoning process information, the predicted answer and the answer.

[0025] In one embodiment, the reasoning process information is first reasoning process information;

[0026] The training module is also used to perform information reasoning on the sample question information with reasoning errors based on target visual data and target auditory data through the multimodal model after preliminary training to obtain a second reasoning result and corresponding second reasoning process information; the target visual data and the target auditory data are respectively the visual data and auditory data corresponding to the sample question information with reasoning errors; determine the correctness reward value based on the second reasoning result; determine the consistency reward value based on the language consistency between the second reasoning process information and the sample question information with reasoning errors; and optimize the parameters of the multimodal model based on the correctness reward value and the consistency reward value to obtain a target multimodal model.

[0027] In one embodiment, the training module is further used to extract visual frame features and auditory features from target visual data and target auditory data; perform feature complementation on the auditory features based on the visual frame features to obtain complemented auditory features; generate summary information of the target visual data based on the visual frame features and the complemented auditory features; perform question reasoning on the sample question information corresponding to the correct answer based on the summary information of the target visual data to obtain predicted reasoning process information and predicted answers.

[0028] In one embodiment, the device further comprises:

[0029] A pre-training module is configured to perform video decoding on the video data to obtain a video frame sequence; sample some video frames from the video frame sequence as image training data, use the audio data as audio training data, and use at least some video frames and the audio data in the video frame sequence as audio-video training data to obtain first training data including the image training data, the audio training data, and the audio-video training data; and pre-train an initial multimodal model based on the first training data to obtain a pre-trained multimodal model.

[0030] In one embodiment, the pre-training module is further used to train the initial multimodal model based on the image training data in the first training data, train the initial multimodal model based on the audio training data in the first training data, and train the initial multimodal model based on the audio and video training data in the first training data until the model converges to obtain a pre-trained multimodal model;

[0031] The order of using the image training data, the audio training data, and the audio and video training data for training is random or fixed.

[0032] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network;

[0033] The pre-training module is further used to perform feature extraction on the image training data in the first training data through the feature extraction network to obtain video frame features of the image training data; align and fuse the video frame features with preset audio features through the feature synchronization network to obtain first fusion features; perform information recognition on the first fusion features through the information recognition network to obtain recognition information; and optimize parameters of the initial multimodal model based on the loss value between the recognition information and the first label information.

[0034] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network;

[0035] The pre-training module is further used to perform feature extraction on the audio training data in the first training data through the feature extraction network to obtain audio features of the audio training data; align and fuse the audio features with preset video frame features through the feature synchronization network to obtain second fused features; perform speech recognition on the second fused features through the information recognition network to obtain speech information; and optimize the parameters of the initial multimodal model based on the loss value between the speech information and the second label information.

[0036] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network, and the audio and video training data includes video training data and audio training data;

[0037] The pre-training module is further used to perform feature extraction on the audio training data through the feature extraction network to obtain audio features of the audio training data; and to perform feature extraction on the video training data through the feature extraction network to obtain video frame features of the video training data; to align and fuse the audio features with the video frame features through the feature synchronization network to obtain a third fused feature; to perform information recognition on the third fused feature through the information recognition network to obtain target description information; and to optimize the parameters of the initial multimodal model based on the loss value between the target description information and the third label information.

[0038] In one embodiment, the pre-training module is also used to, when the duration of the video data is less than or equal to a preset duration, perform feature complementation on the audio feature according to the video frame feature of the first sampled video frame sampled from the video frame sequence according to a preset sampling interval, and obtain a complemented audio feature that is synchronized in time with the video frame feature of the first sampled video frame; wherein the feature scale of the audio feature is smaller than the feature scale of the video frame feature of the first sampled video frame; and perform feature fusion on the complemented audio feature and the video frame feature of the first sampled video frame to obtain a third fused feature.

[0039] In one embodiment, the pre-training module is further used to, when the duration of the video data is greater than a preset duration, segment the video frame sequence for the video training data, and sample a second sampling video frame from the subsequence obtained by the segmentation according to a preset sampling interval, perform feature complementation on the audio feature according to the video frame feature of the second sampling video frame to obtain a complemented audio feature that is synchronized in time with the video frame feature of the second sampling video frame; wherein the feature scale of the audio feature is smaller than the feature scale of the video frame feature of the second sampling video frame; and perform feature fusion on the complemented audio feature and the video frame feature of the second sampling video frame to obtain a third fused feature.

[0040] In one embodiment, the second generating module is further configured to generate a question based on the description information; generate corresponding answer options based on the description information and the question; and combine the question and the answer options to obtain sample question information.

[0041] In one embodiment, the device further comprises:

[0042] a determination module, configured to determine a correct answer to the question based on the description information;

[0043] The training module is also used to, when the answer is consistent with the correct answer, use the visual data, the auditory data and the question as second training data, and use the answer and the reasoning process information as fourth label information to perform preliminary training on the pre-trained multimodal model, so that the multimodal model learns the reasoning process of the answer.

[0044] In one embodiment, the training module is further used to, after completing preliminary training, perform information reasoning on the questions in the sample question information based on the visual data and auditory data in the multimodal data through the multimodal model to obtain a first reasoning result; if the first reasoning result is wrong, determine the question with the reasoning error; and select sample question information, visual data, and auditory data corresponding to the question with the reasoning error.

[0045] In one embodiment, the training module is further used to perform information inference on part of the visual data and part of the auditory data in the multimodal data through the target multimodal model to obtain descriptive information of the part of the visual data; determine a prediction loss value based on the descriptive information of the part of the visual data and the corresponding fifth label information; and fine-tune the parameters of the target multimodal model according to the prediction loss value to obtain the final target multimodal model.

[0046] In one embodiment, the device further comprises:

[0047] a reasoning module, configured to, in response to a visual data analysis request, perform information reasoning on target visual data and corresponding auditory data using the target multimodal model to obtain summary information of the target visual data; and extract keywords and abstract information from the summary information of the target visual data;

[0048] A processing module is used to use the keywords and the summary information as index information of the target visual data, so that when performing information search or information recommendation, the target visual data can be searched out when the search terms and the index information match, or the target visual data can be recommended to the target object when the index information and the interest information of the target object are matched.

[0049] In a third aspect, the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the multimodal model processing method when executing the computer program.

[0050] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the multimodal model processing method are implemented.

[0051] In a fifth aspect, the present application also provides a computer program product, which includes a computer program that implements the steps of the multimodal model processing method when executed by a processor.

[0052] The above-mentioned multimodal model processing method, device, equipment, storage medium and program product generate descriptive information of visual data based on the visual data and auditory data in the multimodal data, and generate sample information including questions and answer options based on the descriptive information. Therefore, when performing answer reasoning, in addition to using visual data and questions, corresponding answer options and auditory data are also used. On the one hand, reasoning can be performed within a limited range, which is conducive to improving the accuracy of answer reasoning. On the other hand, it enriches the data and can also improve the accuracy of answer reasoning. Therefore, based on the sample information, visual data and auditory data, the answer to the question and the reasoning process information corresponding to the answer can be accurately obtained; in addition, when the answer is correct, the pre-trained multimodal model is trained based on the visual data, auditory data, the sample information corresponding to the correct answer and the reasoning process information, so that not only can the multimodal model have the reasoning ability to output the intermediate thinking process, but also can effectively avoid the problem of noise introduction and the risk of overfitting of the model during training. It can effectively alleviate the hallucination problem of the multimodal model and improve the reasoning accuracy of the multimodal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 FIG1 is an application environment diagram of a multimodal model processing method in one embodiment;

[0054] Figure 2 1 is a flow chart of a multimodal model processing method in one embodiment;

[0055] Figure 3 is a schematic diagram of sample question information and video frames in one embodiment;

[0056] Figure 4 A schematic diagram of the structure of an audio and video model when inputting audio data in one embodiment;

[0057] Figure 5 Schematic diagram of the structure of an audio and video model when inputting video data in one embodiment;

[0058] Figure 6 A schematic diagram of the structure of an audio and video model when inputting audio and video data in one embodiment;

[0059] Figure 7 1 is a flow chart of a multimodal model processing method in one embodiment;

[0060] Figure 8is a structural block diagram of a multimodal model processing device in one embodiment;

[0061] Figure 9 is a structural block diagram of a multimodal model processing device in another embodiment;

[0062] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0064] It should be noted that in the following description, the terms "first, second, ..., fifth" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second, ..., fifth" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0065] The multimodal model processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other devices.

[0066] Server 104 can generate description information of the video data based on the video data and audio data in the multimodal data; generate sample question information including questions and answer options based on the description information; generate answers and corresponding reasoning process information based on the sample question information, video data, and audio data; and if the answer is correct, train the pre-trained multimodal model based on the video data, audio data, sample question information with correct answers, and reasoning process information to obtain a target multimodal model. Server 104 uses this target multimodal model to infer summary information of each video data, and then extracts keywords and summary information of the video data. When a user is browsing videos, the keywords and summary information of the video data, combined with the user's relevant information, can be used to recommend target videos that may be of interest to the user's terminal 102.

[0067] The terminal 102 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, and projectors. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Head-mounted devices may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like.

[0068] Server 104 can be a standalone physical server or a service node in a blockchain system. Each service node in the blockchain system forms a peer-to-peer network. The peer-to-peer protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP). Furthermore, server 104 can be a server cluster consisting of multiple physical servers. It can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0069] In one embodiment, Figure 2 As shown, a multimodal model processing method is provided, which can be Figure 1 The server or terminal in the embodiment is executed, or the server and the terminal are executed in collaboration, in which method the Figure 1 The server execution in is used as an example to illustrate the following steps:

[0070] S202: Generate description information of the visual data based on the visual data and the auditory data in the multimodal data.

[0071] Among them, multimodal data can be information obtained by integrating multiple perceptions or expressions, including visual data, auditory data, text data, tactile data (i.e., data of physical feedback), and sensor data. The data of different modes in the multimodal data can have completely different structures, be related to each other, and be fused. For example, the structure of text data can be a sequence, the structure of visual data can be a similar array, and the structure of auditory data can be a waveform or a spectrum; for example, the picture in the visual data is synchronized with the auditory data. In addition, the data of different modes in the multimodal data can be integrated into a unified representation through feature alignment or joint modeling. In the present application, the number of multimodal data can be multiple (such as two or more than two), and each multimodal data includes visual data and corresponding auditory data. If the visual data is video data, the auditory data is the audio data corresponding to the video data.

[0072] Visual data can be information collected by visual sensors (such as cameras, scanners, and depth sensors) and presented in visual form. In addition, it can also be information generated by computer vision technology and presented in visual form, including: dynamic video data, dynamic or static conventional image data, three-dimensional visual data (such as radar point clouds, three-dimensional reconstructed models of objects or people, etc.) and special imaging data (such as radar images, microscope images, remote sensing images, and infrared images, etc.).

[0073] Auditory data can be audio data collected by auditory sensors (such as microphones, sonars, and ultrasonic devices) and presented in the form of sound, including time series waveform signals or spectral features. In addition, it can also be audio data generated using speech synthesis technology and presented in the form of sound, including: speech (such as human speech, synthesized speech), music, game sound effects, and other auditory data (such as animal sounds, natural sounds, sounds emitted by electronic devices, and motor vehicle sounds).

[0074] Descriptive information can be information obtained after recognizing visual and auditory data to describe the visual data. For example, by recognizing facial expressions, eye contact, and body movements of characters in a video, as well as recognizing the corresponding speech in the video, a description of the video content can be obtained, as well as text information recognized from the visual data. This descriptive information can be multiple (e.g., two or more), meaning that each piece of visual data can have multiple descriptive information.

[0075] It should be pointed out that combining auditory data with descriptive information recognition can effectively improve the recognition accuracy of descriptive information. In addition, it can also supplement descriptive information that cannot be recognized in visual data, making the obtained descriptive information richer, which is conducive to the accuracy of sample question information construction. For example, the constructed sample question information can effectively include accurate answer options, which is conducive to the training of multimodal models and improves model reasoning capabilities.

[0076] A multimodal model can be a multimodal large model, which is a general artificial intelligence model that can simultaneously process and understand data from multiple modalities. Its characteristics include:

[0077] 1) Multimodal fusion, which is to jointly model different modal data through a unified architecture to achieve cross-modal information complementarity;

[0078] 2) Large model characteristics, that is, the model has a large number of parameters, relies on large-scale multimodal data for training, and has strong generalization capabilities and the ability to handle complex tasks.

[0079] In some specific application scenarios, the multimodal model may be a large audio and video model (ie, a large audio and video synchronization model).

[0080] In one embodiment, the server can use a first large language model to perform information recognition on visual data and auditory data in multimodal data to obtain descriptive information for the visual data. The first large language model can be a large language model capable of performing complex tasks, i.e., a large language model with deep knowledge reasoning capabilities. Furthermore, the first large language model can perform information recognition based on recognition requirements, thereby enabling the first large language model to perform targeted information recognition. The recognition requirements can be input or set based on actual application scenarios.

[0081] For example, the server can use an open-source general-purpose large language model to identify video data and audio data based on input recognition requirements or pre-set recognition requirements, thereby obtaining descriptive information for the video data. Specifically, the server uses the open-source general-purpose large language model to identify the video image of the video data to obtain first descriptive information and to identify the audio data corresponding to the video data to obtain second descriptive information. On the one hand, the second descriptive information can be used to optimize the first descriptive information, and on the other hand, the portion of the second descriptive information that does not appear in the first descriptive information can be integrated to obtain descriptive information for the video data.

[0082] S204: Generate sample question information including questions and answer options based on the description information.

[0083] The question, also known as the title, is a proposition that requires an answer. The answer options may be the various options for selecting the correct answer to the question. A question typically has multiple answer options. The sample question information may be a multiple-choice question that includes the question and answer options.

[0084] For the same visual data, one or more sample question information may be generated. Given the large number of multimodal data and visual data, multiple sample question information may also be generated. For example, based on the descriptive information of each visual data item, the server generates sample question information for each visual data item, including questions and answer options.

[0085] In one embodiment, the server may generate questions based on the description information; generate corresponding answer options based on the description information and the questions; and combine the questions and the answer options to obtain sample question information.

[0086] When generating questions, the server can analyze the description information to obtain a question that summarizes the description information. When generating answer options, the description information can be adjusted based on the question. For example, for each description information, key information specific to the question can be extracted and used as an answer option. After obtaining the question and answer options, the question and answer options are combined to obtain sample question information.

[0087] In the process of generating questions and answer options, the questions and answer options can be generated in combination with input or set construction requirements.

[0088] For example, the server can input the description information and construction requirements into the second largest language model, and the second largest language model can generate questions based on the input description information and construction requirements, and generate answer options based on the questions and description information, thereby constructing a multiple-choice question containing questions and answer options.

[0089] The second-largest language model can be a large language model that supports high-volume, high-frequency tasks, or a large language model capable of performing tasks of medium complexity. Furthermore, the second-largest language model can construct sample question information based on construction requirements, allowing the second-largest language model to construct sample question information in a targeted manner. This construction requirement can be input or set based on the actual application scenario.

[0090] As an example, suppose the boy in the video walks towards his mother and sees her wearing dungarees. He thinks the dungarees look good and expresses his liking for them. After completing the video recognition and obtaining multiple description information, we can use this description information to construct a multiple-choice question, such as Figure 3As shown, the question is titled "In the video, when the boy expresses his love for the overalls, what specific actions does he take?" The corresponding answer options are: A. Nodding in affirmation, B. Saying "I want the same pair of pants," C. Approaching his mother, D. Complimenting her on her taste, E. Looking with admiration, F. Willingly cooperating with her, G. Glancing at her from time to time while walking. It should be noted that the above answer options are only examples and are not exhaustive, and include but are not limited to the above-mentioned answer options.

[0091] In one embodiment, during or after generating the sample question information, the server may also determine the correct answer to the question based on the description information. The correct answer may be given separately, or the correct answer options in the sample question information may be marked with the correct answer.

[0092] For example, during or after generating sample question information using the second largest language model, the server may also use the second largest language model to determine the correct answer to the question based on the description information.

[0093] S206 , generating answers and reasoning process information corresponding to the answers based on the sample question information, visual data, and auditory data.

[0094] The answer may be an answer inferred based on sample question information, visual data, and auditory data, and the answer is one of the answer options.

[0095] The reasoning process information can be information about the reasoning process of the answer, that is, the thinking process information of using visual data and auditory data to solve the sample question information. It should be noted that the reasoning process information is generated first, and then the corresponding answer is obtained by combining the reasoning process information.

[0096] In one embodiment, the server may group the same visual data, the auditory data corresponding to the visual data, and the sample question information, and use the sample question information, visual data, and auditory data of each group to generate an answer and the reasoning process information corresponding to the answer.

[0097] In one embodiment, for each group of sample question information, visual data, and auditory data, the server may first identify the visual data and the corresponding auditory data to obtain descriptive information of the visual data; using the sample question information as a constraint condition, the server may infer the descriptive information to obtain reasoning process information, and use the reasoning process information to obtain the answer to the sample question information (i.e., the answer to the question in the sample question information).

[0098] For example, a group of sample question information, visual data, and auditory data are input into an open-source third-party language model, and the visual data and the corresponding auditory data are identified using the third-party language model to obtain descriptive information of the visual data. Then, the descriptive information is inferred based on the sample question information, such as analyzing the descriptive information and giving a specific analysis process (i.e., reasoning process information), and the answer is obtained based on the analysis process.

[0099] Among them, the third largest language model can be a large language model with an inference process. By using the third largest language model to identify visual data and corresponding auditory data, the inference process information and corresponding description information of the recognition stage can be provided.

[0100] In another embodiment, for each group of sample question information, visual data and auditory data, the server may first identify the visual data and the corresponding auditory data to obtain description information of the visual data; and then infer the sample question information based on the description information to obtain reasoning process information and the corresponding answer.

[0101] For example, a group of sample question information, visual data and auditory data are input into the open source third-largest language model, and the third-largest language model is used to identify the visual data and the corresponding auditory data to obtain descriptive information of the visual data; then, the descriptive information is used as the basic data for solving the problem to infer the sample question information and obtain the reasoning process information and the corresponding answer.

[0102] In one embodiment, after the server obtains the answer and the corresponding reasoning process information, if the answer to a sample question is correct (such as the answer is consistent with the correct answer), the server retains the reasoning process information of the answer, the question and the correct answer options in the sample question information, and the visual data and auditory data corresponding to the sample question information, and deletes other data, thereby obtaining the question with the correct answer, the correct answer options, and the corresponding reasoning process information, visual data, and auditory data.

[0103] For example, Figure 3 As shown, if option A is correct, then the question and the correct answer option (i.e. option A) can be retained, the reasoning process information of the answer can be retained, and Figure 3 The video frame (or the entire video data corresponding to the video frame) and the corresponding audio data.

[0104] In another embodiment, after obtaining the answer and the corresponding reasoning process information, if the answer to a sample question is correct, the server may also retain the reasoning process information of the answer, retain the sample question information with the correct answer, and retain the visual data and auditory data corresponding to the sample question information, and delete other data, thereby obtaining the sample question information with the correct answer and the corresponding reasoning process information, visual data and auditory data.

[0105] For example, Figure 3 As shown, if option A is correct, you can keep the multiple-choice question, keep the reasoning process information of the answer, and keep Figure 3 If the correct answer is not given to the question in another sample question and the corresponding video data only contains this sample question, then the sample question information, reasoning process information, video data, and corresponding audio data will all be deleted.

[0106] S208, when the answer is correct, the pre-trained multimodal model is trained based on the visual data, the auditory data, the sample question information corresponding to the correct answer, and the reasoning process information to obtain a target multimodal model.

[0107] The pre-trained multimodal model can be a model obtained by pre-training an initial multimodal model using multimodal data, such as pre-training the initial multimodal model using visual data and auditory data from the multimodal data. This pre-trained multimodal model has conventional recognition capabilities but does not have the ability to output information about the reasoning process.

[0108] In one embodiment, the server can perform preliminary training on the pre-trained multimodal model based on visual data, auditory data, sample question information corresponding to correct answers, and reasoning process information; after completing the preliminary training, the server selects sample question information in which the multimodal model has reasoning errors and the corresponding visual data and auditory data, and performs model training on the pre-trained multimodal model based on the selected data to obtain the target multimodal model.

[0109] The initial training of the multimodal model can stimulate the model's ability to learn to solve problems by outputting intermediate thinking processes. It can also standardize the model's reasoning behavior and output format. Furthermore, the model training of the multimodal model after the initial training can be reinforcement training.

[0110] In one embodiment, the server extracts features from visual data and auditory data through a pre-trained multimodal model to obtain visual frame features and auditory features; performs feature complementation on the auditory features based on the visual frame features to obtain complemented auditory features; generates summary information of the visual data based on the visual frame features and the complemented auditory features; performs question reasoning on sample question information corresponding to the correct answer based on the summary information to obtain predicted reasoning process information and predicted answers; and optimizes parameters of the pre-trained multimodal model based on the predicted reasoning process information, the reasoning process information, the predicted answers, and the answers.

[0111] The visual data may include video data, and the auditory data may include audio data corresponding to the video data. The visual frame data may include video frame features, and the auditory features may include audio features.

[0112] In one embodiment, when the duration of the video data is less than or equal to a preset duration, the server samples a sampled video frame from a video frame sequence of the video data, and then performs feature extraction on the sampled video frame and audio data through a pre-trained multimodal model to obtain video frame features and audio features; the server performs feature complementation on the audio features according to the video frame features of the sampled video frame to obtain complemented audio features that are synchronized in time with the video frame features of the sampled video frame; and then, the video frame features and the complemented audio features are fused to obtain fused features.

[0113] For example, each sampled video frame passes through an image encoder to obtain the corresponding video frame features, with each video frame having 112 tokens. The corresponding audio data passes through an audio encoder to obtain audio features, with 50 tokens per second of audio data. For a synchronized audio and video network, if the video data duration does not exceed 150 seconds, the audio features per second are combined with 62 (112 minus 50) all-zero tokens to achieve padded audio features. These padded audio features are then added to the 112 tokens corresponding to the video frame to obtain the fused audio and video features.

[0114] In another embodiment, when the duration of the video data is greater than a preset duration, the server segments the video frame sequence corresponding to the video data, and samples the sampled video frames from the subsequence obtained by the segmentation according to the preset sampling interval, and performs feature complementation on the audio features according to the video frame features of the sampled video frames to obtain the complemented audio features that are synchronized in time with the video frame features of the sampled video frames; and performs feature fusion on the complemented audio features with the video frame features of the second sampled video frame to obtain fused features.

[0115] For example, each sampled video frame is passed through an image encoder to obtain corresponding video frame features, with each video frame having 112 tokens. The corresponding audio data is passed through an audio encoder to obtain audio features, with 50 tokens per second of audio data. If the video data duration exceeds 150 seconds, the duration of each sampled video frame is first calculated. For example, if a 225-second video data is sampled with 150 video frames, each video frame corresponds to 225 / 150 = 1.5 seconds. Then, the 1.5*50 = 75 audio features corresponding to 1.5 seconds of audio data can be combined with 37 (i.e., 112 - 75) all-zero tokens to complete the audio features and obtain the padded audio features. The padded audio features are then added to the 112 tokens corresponding to the video frame to obtain the fused audio and video features, ensuring that the 112 tokens of each video frame and the tokens of the corresponding audio data have the same time range.

[0116] In one embodiment, the server can determine the correct answer to the question based on the description information; when the answer is consistent with the correct answer, the visual data, auditory data and the question in the sample question information corresponding to the correct answer are used as the second training data, and the corresponding answer and reasoning process information are used as the fourth label information to perform preliminary training on the pre-trained multimodal model so that the multimodal model learns the reasoning process of the answer.

[0117] In the process of preliminary training of the pre-trained multimodal model, the loss between the output of the multimodal model and the fourth label information (such as cross entropy loss) is calculated, and the parameters of the multimodal model are optimized through back propagation, such as fine-tuning the parameters of the multimodal model to ensure the readability and logic of the model output.

[0118] For example, the server may input questions with correct answers, correct answer options, and corresponding reasoning process information, visual data, and auditory data into a pre-trained multimodal model. The multimodal model may use the visual data, auditory data, and questions as second training data, and the answers and reasoning process information as fourth label information, to perform preliminary training on the pre-trained multimodal model so that the multimodal model learns the reasoning process of the answers. Alternatively, the server may input sample question information corresponding to the correct answers and the reasoning process information corresponding to the sample question information, visual data, and auditory data into a pre-trained multimodal model. The multimodal model may use the visual data, auditory data, and sample question information including the questions as second training data, and the answers and reasoning process information as fourth label information, to perform preliminary training on the pre-trained multimodal model so that the multimodal model learns the reasoning process of the answers.

[0119] In one embodiment, after completing the preliminary training, the server selects sample question information in which the multimodal model has reasoning errors and the corresponding visual data and auditory data, and performs intensive training on the multimodal model after preliminary training based on the selected data to obtain a target multimodal model for reasoning and summarizing information.

[0120] The summary information can accurately express the content directly presented (or implied) by the visual data, and can be briefer than the description information and more accurately express the content directly presented (or implied) by the visual data.

[0121] In one embodiment, after completing preliminary training, the server can use a multimodal model to perform information reasoning on questions in the sample question information based on the visual data and auditory data in the multimodal data to obtain a first reasoning result; if the first reasoning result is wrong, determine the question for which the reasoning is wrong; and select sample question information, visual data, and auditory data corresponding to the question for which the reasoning is wrong.

[0122] For example, after completing the preliminary training, the sample question information with correct answers, the visual data and the auditory data in the multimodal data are input into the multimodal model after preliminary training. The multimodal model performs information reasoning on the questions in the sample question information based on the visual data and the auditory data in the multimodal data to obtain a first reasoning result; when the first reasoning result is wrong, the question with incorrect reasoning is determined; the sample question information, visual data and auditory data corresponding to the question with incorrect reasoning are selected; in addition, the answer options corresponding to the questions with incorrect reasoning by the multimodal model can also be selected from the sample question information with correct answers, and the corresponding visual data and auditory data can be obtained.

[0123] In one embodiment, after completing the preliminary training, the server can use the multimodal model after preliminary training to infer sample question information with correct answers and corresponding visual data and auditory data, or infer questions with correct answers and corresponding visual data and auditory data to obtain answers and corresponding reasoning process information; select sample question information including questions and answer options with incorrect answers, or select questions with incorrect answers and corresponding answer options, and then perform intensive training on the multimodal model based on the selected data to obtain a target multimodal model for inferring and summarizing information.

[0124] In one embodiment, the reasoning process information is first reasoning process information; therefore, during intensive training, the multimodal model after preliminary training performs information reasoning on the sample information with reasoning errors based on the target visual data and target auditory data to obtain a second reasoning result and corresponding second reasoning process information; the target visual data and the target auditory data are the visual data and auditory data corresponding to the sample information with reasoning errors, respectively; a consistency reward value is determined based on the language consistency between the second reasoning process information and the sample information with reasoning errors; the multimodal model parameters are optimized based on the correctness reward value and the consistency reward value to obtain a target multimodal model for reasoning summary information. In addition, a consistency reward value can also be determined based on the language consistency between the second reasoning process information and the question with reasoning errors.

[0125] Among them, the correctness reward value can be a reward value for whether the second inference result output by the multimodal model is correct. For example, if the multimodal model is used for reasoning to obtain summary information of a certain video data, if the summary information is verified to be correct based on the reliability of the rules, a corresponding reward value will be given.

[0126] The consistency reward value can be obtained by reasoning in a language consistent with the question (or sample question information). That is, the language of the reasoning process information is consistent with the language of the question, and the generated reasoning result is also consistent with the language of the question. In this case, a positive consistency reward will be obtained.

[0127] In one embodiment, when performing reasoning, the server can use a multimodal model to perform information reasoning on the problem with incorrect reasoning based on the visual data and auditory data corresponding to the problem with incorrect reasoning, and obtain a second reasoning result and corresponding second reasoning process information; and determine the correctness reward value based on the second reasoning result.

[0128] In one embodiment, the server may add the correctness reward value and the consistency reward value, and use the added reward value to optimize the parameters of the multimodal model to obtain a target multimodal model for reasoning and summarizing information.

[0129] In one embodiment, the server can extract visual frame features and auditory features from the target visual data and target auditory data; perform feature complementation on the auditory features based on the visual frame features to obtain complemented auditory features; generate summary information of the target visual data based on the visual frame features and the complemented auditory features; perform question reasoning on the sample question information corresponding to the correct answer based on the summary information of the target visual data to obtain predicted reasoning process information and predicted answers.

[0130] Before generating the summary information, the video frame features and the padded audio features can be fused, and then the fused features can be used to generate the summary information of the target visual data. For the feature padded and fusion during the enhanced training process, please refer to the feature padded and fusion during the initial training process.

[0131] In the above embodiment, description information of the visual data is generated based on the visual data and auditory data in the multimodal data, and sample question information including questions and answer options is generated based on the description information. Therefore, when performing answer reasoning, in addition to using visual data and questions, corresponding answer options and auditory data are also given. On the one hand, reasoning can be performed within a limited range, which is conducive to improving the accuracy of answer reasoning. On the other hand, the data is enriched, which can also improve the accuracy of answer reasoning. Therefore, based on the sample question information, visual data and auditory data, the answer to the question and the reasoning process information corresponding to the answer can be accurately obtained; in addition, when the answer is correct, the pre-trained multimodal model is trained based on the visual data, auditory data, the corresponding sample question information when the answer is correct, and the reasoning process information. This not only enables the multimodal model to have the reasoning ability to output the intermediate thinking process, but also effectively avoids the problem of noise introduction and the risk of overfitting of the model during training. It can effectively alleviate the hallucination problem of the multimodal model and improve the reasoning accuracy of the multimodal model.

[0132] In one embodiment, the visual data includes video data, and the auditory data includes audio data corresponding to the video data. The method also includes: the server can perform video decoding on the video data to obtain a video frame sequence; sampling part of the video frames from the video frame sequence as image training data, using the audio data as audio training data, and using at least part of the video frames and audio data in the video frame sequence as audio and video training data to obtain first training data including image training data, audio training data, and audio and video training data; and pre-training the initial multimodal model based on the first training data to obtain a pre-trained multimodal model.

[0133] The video frame sequence may be a sequence obtained by sequentially combining video frames obtained by decoding video data. For example, 10 video frames are decoded from the video data and these 10 video frames are combined in the order of playback time to obtain a video frame sequence.

[0134] For sampling of video frames, the following methods can be used:

[0135] Method 1: Sampling is performed directly according to the preset sampling interval.

[0136] In one embodiment, when the duration of the video data is less than or equal to a preset duration, the server may directly sample from the video frame sequence according to a preset sampling interval and use the sampled video frames as image training data.

[0137] The preset duration may be a pre-set time length, such as 150 seconds.

[0138] For example, for a video with a duration of 150 seconds, one frame can be sampled from the video frame sequence per second, thereby obtaining 150 video frames, and these 150 video frames are used as image training data.

[0139] Method 2: Segment first, then sample.

[0140] In one embodiment, when the duration of the video data is greater than a preset duration, the server may first segment the video frame sequence, then sample the subsequences obtained by the segmentation, and use the sampled video frames as image training data.

[0141] For example, for a video longer than 150 seconds, the video frame sequence can be segmented, such as into 150 segments, and one video frame can be sampled from each segment, thereby obtaining 150 video frames, which are used as image training data.

[0142] In one embodiment, for pre-training of a multimodal model, the server may use image training data, audio training data, and audio-video training data for training. Specifically, the server trains the initial multimodal model based on the image training data in the first training data, trains the initial multimodal model based on the audio training data in the first training data, and trains the initial multimodal model based on the audio-video training data in the first training data until the model converges to obtain a pre-trained multimodal model.

[0143] The order of using image training data, audio training data, and audio and video training data for training is random or fixed.

[0144] For example, in each round of training, one of the unused image training data, audio training data, or audio-video training data can be randomly used for training. Let's assume that the image training data includes image training data 1~3, the image training data includes image training data 1~3, and the audio-video training data includes audio-video training data 1~3. Then, during training, one can be randomly selected from image training data 1~3, image training data 1~3, and audio-video training data 1~3 for training. After completing the current round of training, one type of data is randomly selected from the remaining training data for training, and so on.

[0145] Alternatively, in each round of training, one of the image training data, audio training data, or audio-video training data can be used for training in a fixed order, such as first using image training data for training, then using audio training data for training, and finally using audio-video training data for training.

[0146] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network; therefore, the steps of training using image training data may include: the server may perform feature extraction on the image training data in the first training data through the feature extraction network to obtain video frame features of the image training data; align and fuse the video frame features with the preset audio features through the feature synchronization network to obtain first fusion features; perform information recognition on the first fusion features through the information recognition network to obtain recognition information; and optimize the parameters of the initial multimodal model based on the loss value between the recognition information and the first label information.

[0147] Among them, the feature extraction network may include an image encoder and an audio encoder. The image encoder may be a ViT network, a network for extracting image features (or video frame features); the audio encoder may be a Whisper network (such as a Whisper-V3 network), a network for extracting audio features; in addition, in addition to the ViT network and the Whisper network, the image encoder and the audio encoder may also be other networks that can extract image features and audio features.

[0148] The feature synchronization network may be a network for aligning and fusing different features, such as an audio and video synchronization network, which may be a temporal fine-grained synchronization network.

[0149] The information recognition network may be a large language model (LLM) for performing recognition and inference using features of various input models.

[0150] The identification information may include text information displayed in the image training data and content presented by the image training data expressed in text information, such as text information displayed in a video frame and content presented by the video frame.

[0151] For example, Figure 4As shown, image training data (i.e., the sampled video frames) can be input into an image encoder, which can extract video frame features from each video frame. The video frame features of each video frame are then input into an audio-video synchronization network, which aligns and fuses the video frame features of each video frame with preset audio features to obtain a first fused feature. The first fused feature is then recognized through a large language model to obtain recognition information of the video frame.

[0152] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network; therefore, the step of training using audio training data may include: the server may perform feature extraction on the audio training data in the first training data through the feature extraction network to obtain audio features of the audio training data; align and fuse the audio features with the preset video frame features through the feature synchronization network to obtain second fused features; perform speech recognition on the second fused features through the information recognition network to obtain speech information; and optimize the parameters of the initial multimodal model based on the loss value between the speech information and the second label information.

[0153] The voice information may be voice content recognized from audio training data (ie, audio data).

[0154] In one embodiment, the server may segment the audio data according to the sampling method of the video frame sequence, and then use the segmented audio data as audio training data. For example, if 150 video frames are sampled from a video frame sequence of 150 seconds of video data, the corresponding audio data may be segmented into 150 segments.

[0155] For example, Figure 5 As shown, the audio training data (i.e., the segmented audio data) can be input into an audio encoder, which can extract audio features from each segment of audio data. The audio features of each segment of audio data are then input into an audio-video synchronization network, which aligns and fuses the audio features of each segment of audio data with the preset video frame features to obtain a second fused feature. The second fused feature is then used to perform speech recognition using a large language model to obtain speech information. It should be noted that Figure 4 and Figure 5 The models are all the same audio and video model, but the input data is different. The encoder corresponding to the modal data that is not input is not shown in the figure.

[0156] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network and an information recognition network, and the audio and video training data includes video training data and audio training data; therefore, the step of training using the audio and video training data may include: the server may perform feature extraction on the audio training data through the feature extraction network to obtain audio features of the audio training data; and, perform feature extraction on the video training data through the feature extraction network to obtain video frame features of the video training data; align and fuse the audio features with the video frame features through the feature synchronization network to obtain a third fused feature; perform information recognition on the third fused feature through the information recognition network to obtain target description information; and optimize the parameters of the initial multimodal model based on the loss value between the target description information and the third label information.

[0157] Among them, the target description information can be used to describe the information of visual data, such as by identifying the facial expressions, eye expressions and body movements of the characters in the video, and identifying the corresponding voice of the video, so as to obtain the content used to describe the video and the text information identified from the visual data.

[0158] For the extraction of audio features and video frame features, reference may be made to the extraction process of the above embodiment.

[0159] For example, Figure 6 As shown, audio training data (i.e., segmented audio data) can be input into an audio encoder, which can extract audio features from each segment of audio data. In addition, image training data (i.e., sampled video frames) can be input into an image encoder, which can extract video frame features from each video frame. The audio features of each segment of audio data and the video frame features of each video frame are then input into an audio-video synchronization network, which aligns and fuses the audio features of each segment of audio data with the video frame features of each video frame to obtain a third fused feature. The third fused feature is then identified through a large language model to obtain target description information. It should be pointed out that Figure 4 、 Figure 5 and Figure 6 The models are all the same audio and video model.

[0160] In one embodiment, when the duration of the video data of the server is less than or equal to a preset duration, the video training data is a first sampled video frame sampled from a video frame sequence according to a preset sampling interval, and the audio features are feature-padded according to the video frame features of the first sampled video frame to obtain padded audio features that are synchronized with the video frame features of the first sampled video frame in time sequence; wherein the feature scale of the audio features is smaller than the feature scale of the video frame features of the first sampled video frame; and the padded audio features are feature-fused with the video frame features of the first sampled video frame to obtain a third fused feature.

[0161] For example, each sampled video frame passes through an image encoder to obtain the corresponding video frame features, with each video frame having 112 tokens. The corresponding audio data passes through an audio encoder to obtain audio features, with 50 tokens per second of audio data. For audio and video synchronization networks, if the video data duration does not exceed 150 seconds, the audio features per second are combined with 62 (112 minus 50) all-zero tokens to achieve padded audio features. These padded audio features are then added to the 112 tokens corresponding to the video frame to obtain the fused audio and video features, which are used as input to the LLM model.

[0162] In another embodiment, when the duration of the video data of the server is greater than a preset duration, the video training data is segmented into a video frame sequence, and a second sampled video frame is sampled from the subsequence obtained by the segmentation according to a preset sampling interval, and the audio feature is feature-padded according to the video frame feature of the second sampled video frame to obtain a padded audio feature that is synchronized with the video frame feature of the second sampled video frame in time sequence; wherein the feature scale of the audio feature is smaller than the feature scale of the video frame feature of the second sampled video frame; and the padded audio feature is feature-fused with the video frame feature of the second sampled video frame to obtain a third fused feature.

[0163] For example, each sampled video frame is passed through an image encoder to obtain corresponding video frame features, with each video frame having 112 tokens. The corresponding audio data is passed through an audio encoder to obtain audio features, with 50 tokens per second of audio data. For an audio-video synchronization network, if the video data duration exceeds 150 seconds, the duration of each sampled video frame is first calculated. For example, if a 225-second video data consists of 150 sampled video frames, each video frame corresponds to 225 / 150 = 1.5 seconds. Then, the 1.5*50 = 75 audio features corresponding to 1.5 seconds of audio data can be combined with 37 (i.e., 112 - 75) all-zero tokens to achieve padded audio features. The padded audio features are then added to the 112 tokens corresponding to the video frame to obtain a fused audio and video feature, ensuring that the 112 tokens of each video frame and the corresponding audio data tokens have the same time range. The resulting fused feature is then used as the input to the LLM model.

[0164] In one embodiment, after S210, a small amount of video data can also be used to fine-tune the target multimodal model, which may specifically include: the server uses the target multimodal model to perform information inference on part of the visual data and part of the auditory data in the multimodal data to obtain description information of part of the visual data; based on the description information of the part of the visual data and the corresponding fifth label information, a predicted loss value is determined; and according to the predicted loss value, parameters of the target multimodal model are fine-tuned to obtain the final target multimodal model.

[0165] Among them, the final target multimodal model can be a multimodal model that completes pre-training and post-training.

[0166] For example, a small amount of visual data and auditory data is selected from the multimodal data, and the selected visual data and auditory data are input into the target multimodal model. The target multimodal model performs information inference on the input visual data and auditory data to obtain descriptive information of the selected visual data; the prediction loss value is determined based on the descriptive information of the selected visual data and the corresponding fifth label information; the parameters of the target multimodal model are fine-tuned according to the predicted loss value to obtain the final target multimodal model.

[0167] In one embodiment, after obtaining the target multimodal model, the server can respond to the visual data analysis request, perform information inference on the target visual data and the corresponding auditory data through the target multimodal model, and obtain summary information of the target visual data; extract keywords and summary information from the summary information of the target visual data; use the keywords and summary information as index information of the target visual data, so that when performing information search or information recommendation, the target visual data can be searched out when the search term and the index information match, or the target visual data can be recommended to the target object when the index information and the interest information of the target object are matched.

[0168] For example, by inputting the audio data corresponding to the video data to be processed into a trained multimodal large model, summary information of the video data can be obtained, and then keywords and summary information of the video data can be further extracted for searching and recommending video data.

[0169] In one embodiment, Figure 7 As shown in the figure, another multimodal model processing method is provided, which can be Figure 1 The server or terminal in the embodiment is executed, or the server and the terminal are executed in collaboration, in which method the Figure 1 The server execution in is used as an example to illustrate the following steps:

[0170] S702 , performing video decoding on the video data in the multimodal data to obtain a video frame sequence; the multimodal data includes visual data and auditory data, the visual data includes video data, and the auditory data includes audio data corresponding to the video data.

[0171] S704: Sampling a portion of video frames from the video frame sequence as image training data, using audio data in the multimodal data as audio training data, and using at least a portion of the video frames and audio data in the video frame sequence as audio-video training data, to obtain first training data including the image training data, the audio training data, and the audio-video training data;

[0172] S706: Pre-train the initial multimodal model based on the first training data to obtain a pre-trained multimodal model.

[0173] In one embodiment, the above-mentioned pre-training of the initial multimodal model based on the first training data to obtain the pre-trained multimodal model includes: the server training the initial multimodal model based on the image training data in the first training data, training the initial multimodal model based on the audio training data in the first training data, and training the initial multimodal model based on the audio and video training data in the first training data until the model converges to obtain the pre-trained multimodal model; wherein the order of using the image training data, the audio training data, and the audio and video training data for training is random or fixed.

[0174] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network;

[0175] The above-mentioned training of the initial multimodal model based on the image training data in the first training data includes: extracting features of the image training data in the first training data through a feature extraction network to obtain video frame features of the image training data; aligning and fusing the video frame features with the preset audio features through a feature synchronization network to obtain first fusion features; performing information recognition on the first fusion features through an information recognition network to obtain recognition information; and optimizing parameters of the initial multimodal model based on the loss value between the recognition information and the first label information.

[0176] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network;

[0177] The above-mentioned training of the initial multimodal model based on the audio training data in the first training data includes: the server extracts features of the audio training data in the first training data through a feature extraction network to obtain audio features of the audio training data; aligns and fuses the audio features with preset video frame features through a feature synchronization network to obtain second fused features; performs speech recognition on the second fused features through an information recognition network to obtain speech information; and optimizes parameters of the initial multimodal model based on the loss value between the speech information and the second label information.

[0178] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network, and the audio and video training data includes video training data and audio training data;

[0179] The above-mentioned training of the initial multimodal model based on the audio and video training data in the first training data includes: the server extracts features from the audio training data through a feature extraction network to obtain audio features of the audio training data; and extracts features from the video training data through a feature extraction network to obtain video frame features of the video training data; aligns and fuses the audio features with the video frame features through a feature synchronization network to obtain a third fused feature; performs information recognition on the third fused feature through an information recognition network to obtain target description information; and optimizes the parameters of the initial multimodal model based on the loss value between the target description information and the third label information.

[0180] In one embodiment, the above-mentioned alignment and fusion processing of audio features and video frame features through a feature synchronization network to obtain a third fused feature includes: when the duration of the video data is less than or equal to the preset duration, the video training data is a first sampled video frame sampled from the video frame sequence according to a preset sampling interval, and the audio features are feature-padded according to the video frame features of the first sampled video frame to obtain padded audio features that are synchronized in time with the video frame features of the first sampled video frame; wherein the feature scale of the audio features is smaller than the feature scale of the video frame features of the first sampled video frame; and the padded audio features are feature-fused with the video frame features of the first sampled video frame to obtain the third fused feature.

[0181] In one embodiment, the above-mentioned alignment and fusion processing of audio features and video frame features through a feature synchronization network to obtain a third fused feature includes: when the duration of the video data is greater than a preset duration, the video training data is segmented into a video frame sequence, and a second sampled video frame is sampled from the subsequence obtained by the segmentation according to a preset sampling interval, and the audio features are feature-padded according to the video frame features of the second sampled video frame to obtain a padded audio feature that is synchronized in time with the video frame features of the second sampled video frame; wherein the feature scale of the audio feature is smaller than the feature scale of the video frame features of the second sampled video frame; and the padded audio feature is feature-fused with the video frame features of the second sampled video frame to obtain the third fused feature.

[0182] S708, generating description information of the visual data based on the visual data and the auditory data in the multimodal data;

[0183] S710, generating sample question information including questions and answer options based on the description information;

[0184] In one embodiment, the above-mentioned generation of sample question information including questions and answer options based on the description information includes: generating questions based on the description information; generating corresponding answer options based on the description information and the questions; and combining the questions and answer options to obtain sample question information.

[0185] S712, generating answers and reasoning process information corresponding to the answers based on the sample question information, visual data, and auditory data;

[0186] S714, when the answer is correct, performing preliminary training on the pre-trained multimodal model based on the visual data, the auditory data, the sample question information corresponding to the correct answer, and the reasoning process information;

[0187] In one embodiment, the server may further perform feature extraction on the visual data and the auditory data using a pre-trained multimodal model to obtain visual frame features and auditory features; perform feature complementation on the auditory features based on the visual frame features to obtain complemented auditory features; generate summary information of the visual data based on the visual frame features and the complemented auditory features; perform question reasoning on the sample question information corresponding to the correct answer based on the summary information to obtain predicted reasoning process information and a predicted answer;

[0188] Based on the predicted reasoning process information, the reasoning process information, the predicted answer and the answer, the parameters of the pre-trained multimodal model are optimized.

[0189] In one embodiment, the server can also determine the correct answer to the question based on the description information; therefore, the above-mentioned preliminary training of the pre-trained multimodal model based on the visual data, auditory data, reasoning process information, questions and correct answer options when the answer is correct includes: when the answer is consistent with the correct answer, the visual data, auditory data and questions are used as second training data, and the answer and reasoning process information are used as fourth label information, and the pre-trained multimodal model is preliminarily trained to enable the multimodal model to learn the reasoning process of the answer.

[0190] S716, after completing the preliminary training, select sample question information with multimodal model reasoning errors and corresponding visual data and auditory data, and perform model training on the multimodal model after preliminary training based on the selected data to obtain the target multimodal model.

[0191] In one embodiment, the server uses a preliminarily trained multimodal model to perform information reasoning on the sample question information with reasoning errors based on the target visual data and the target auditory data to obtain a second reasoning result and corresponding second reasoning process information; the target visual data and the target auditory data are the visual data and auditory data corresponding to the sample question information with reasoning errors, respectively; the correctness reward value is determined based on the second reasoning result; the consistency reward value is determined based on the language consistency between the second reasoning process information and the sample question information with reasoning errors; the multimodal model parameters are optimized based on the correctness reward value and the consistency reward value to obtain the target multimodal model.

[0192] In one embodiment, the server extracts visual frame features and auditory features from the target visual data and the target auditory data; performs feature complementation on the auditory features based on the visual frame features to obtain complemented auditory features; generates summary information of the target visual data based on the visual frame features and the complemented auditory features; performs question reasoning on the sample question information corresponding to the correct answer based on the summary information of the target visual data to obtain predicted reasoning process information and a predicted answer.

[0193] In one embodiment, after completing the preliminary training, the above-mentioned selection of questions with multimodal model reasoning errors and corresponding answer options, visual data, and auditory data includes: after completing the preliminary training, the server uses the multimodal model to perform information reasoning on the questions in the sample question information based on the visual data and auditory data in the multimodal data to obtain a first reasoning result; when the first reasoning result is wrong, determining the question with reasoning errors; and selecting sample question information, visual data, and auditory data corresponding to the question with reasoning errors.

[0194] In one embodiment, the reasoning process information is first reasoning process information; therefore, the above-mentioned intensive training of the multimodal model based on the selected data to obtain the target multimodal model for reasoning summary information includes: using the multimodal model, based on the visual data and auditory data corresponding to the problem with reasoning error, information reasoning is performed on the problem with reasoning error to obtain a second reasoning result and corresponding second reasoning process information; determining a correctness reward value based on the second reasoning result; determining a consistency reward value based on the language consistency between the second reasoning process information and the sample question information with reasoning error; and optimizing the parameters of the multimodal model based on the correctness reward value and the consistency reward value to obtain the target multimodal model for reasoning summary information.

[0195] S718, performing information reasoning on the partial visual data and the partial auditory data in the multimodal data using the target multimodal model to obtain descriptive information of the partial visual data;

[0196] S720, determining a prediction loss value based on the description information of the partial visual data and the corresponding fifth label information;

[0197] S722: Fine-tune the parameters of the target multimodal model according to the predicted loss value to obtain the final target multimodal model.

[0198] In one embodiment, the server can respond to a visual data analysis request, perform information inference on the target visual data and the corresponding auditory data through a target multimodal model, and obtain summary information of the target visual data; extract keywords and summary information from the summary information of the target visual data; and use the keywords and summary information as index information of the target visual data, so that when performing information search or information recommendation, the target visual data can be searched out when the search term and the index information match, or the target visual data can be recommended to the target object when the index information and the interest information of the target object are matched.

[0199] The above-mentioned multimodal model processing method, device, equipment, storage medium and program product generate description information of visual data based on the visual data and auditory data in the multimodal data, and generate sample question information including questions and answer options based on the description information. Therefore, when performing answer reasoning, in addition to using visual data and questions, corresponding answer options and auditory data are also used. On the one hand, reasoning can be performed within a limited range, which is conducive to improving the accuracy of answer reasoning. On the other hand, it enriches the data and can also improve the accuracy of answer reasoning. Therefore, based on the sample question information, visual data and auditory data, the answer to the question and the reasoning process information corresponding to the answer can be accurately obtained; in addition, when the answer is correct, based on the visual data, auditory data, and sample questions with correct answers The pre-trained multimodal model is preliminarily trained based on the information and reasoning process information, which not only enables the multimodal model to have the reasoning ability to output the intermediate thinking process, but also effectively avoids the problem of noise introduction and avoids the risk of overfitting of the model during the training process; moreover, after completing the preliminary training, the problems with the multimodal model reasoning errors and the corresponding answer options, visual data and auditory data are selected, and the multimodal model is strengthened based on the selected data. This can enhance the reasoning ability of the multimodal model for problems that are prone to errors, and avoid the problem of noise introduction during the strengthening training, thereby avoiding the risk of overfitting of the model during the strengthening training process, effectively alleviating the hallucination problem of the multimodal model, and improving the reasoning accuracy of the multimodal model.

[0200] In order to better understand the solution of this application, take the multimodal model as an audio and video synchronization model as an example, and combine Figure 3-6 A brief description is as follows:

[0201] (1) The architecture of the audio and video synchronization model can be referenced Figure 4-5 Given a video, if the video length does not exceed 150 seconds, one video frame is sampled per second; if the video length exceeds 150 seconds, the video is evenly divided into 150 segments, and one video frame is sampled in each segment.

[0202] Each video frame passes through the image encoder to obtain visual features, and each video has 112 tokens; the corresponding audio passes through the audio encoder to obtain audio features, and there are 50 tokens per second of audio.

[0203] For audio-video synchronization networks, if the video duration does not exceed 150 seconds, the audio features per second can be combined with 62 (112-50) all-zero tokens, the resulting combination can be added to the 112 tokens of the corresponding video frame, and the resulting fused feature can be used as the input to the LLM model. If the video duration exceeds 150 seconds, the duration of each video frame can be calculated. For example, for a 225-second video, 150 frames are sampled, and each frame corresponds to 225 / 150 = 1.5 seconds. Therefore, 1.5 seconds of audio corresponding to 1.5 seconds of audio can be combined with 1.5*50 = 75 audio features and 37 (112-75) all-zero tokens. The resulting combination can then be added to the 112 tokens of the corresponding video frame, and the resulting fused feature can be used as the input to the LLM model. The above alignment process ensures that the 112 tokens of each video frame and the corresponding audio tokens have the same time range.

[0204] (2) The form of training data.

[0205] In order to enable the model to understand the input of video frames and audio, a variety of training data forms are used for training. The training data forms include:

[0206] 1. Input of a single video frame. In this case, the model needs to predict the image content of a single video frame and the text information on the image. The image content is expressed in text form.

[0207] 2. Only audio input, in this case the model needs to predict the speech corresponding to the audio;

[0208] 3. Input of multiple video frames and corresponding audio. At this time, the model needs to obtain summary information of the video.

[0209] (3) Training of large audio and video synchronization models.

[0210] Model training consists of two stages:

[0211] The first stage is pre-training of the large audio and video synchronization model. The model is optimized by calculating the loss using the text output by the model and the text annotated with the data. After the optimization is completed, the quantitative score of the large audio and video synchronization model is 6.89.

[0212] The second stage is post-training using GRPO. The specific process is as follows:

[0213] a) Use the open-source large language model a to identify the video description information, and use the open-source large language model b to construct multiple-choice questions based on the video description information. Sample questions are as follows: Figure 3 shown.

[0214] b) Use the open-source large language model C to obtain training data with thought chains. That is, multiple-choice questions, videos, and corresponding audio are input into the large language model C, and it is asked to first output information about the thought process and then output the answer. If the answer predicted by the model is consistent with the correct answer, this part of the data is retained.

[0215] c) Based on the training data obtained in the previous step, perform preliminary training (i.e., cold start) on the pre-trained audio and video synchronization model to stimulate the model to achieve the ability to answer multiple-choice questions by outputting the intermediate thinking process.

[0216] d) After the cold start is completed, the questions that the audio and video synchronization large model cannot solve after the cold start are retained and GRPO training is performed.

[0217] e) Based on a small amount of video data, fine-tune the large audio and video synchronization model after GRPO training, where the input is video and audio, and the output is the summary information of the video.

[0218] After completing the second phase of training, the large audio and video synchronization model's quantitative evaluation score was further improved to 7.35 points, and the hallucination problem was effectively alleviated in the qualitative evaluation.

[0219] Therefore, the audio and video synchronization large model based on this application can significantly improve the accuracy of model summary. The specific comparison with other open source large language models is shown in Table 1 below:

[0220] Table 1

[0221] Model Name Score Internvl-2.5-8B 5.09 Internvl-3-8B 5.10 Qwen2.5-VL-7B 6.21 Qwen2.5-Omni-7B 5.54 The audio and video synchronization model of this application 7.35

[0222] Based on the audio and video synchronization model of this application, the summary information of the video is obtained, and the qualification rate on the business side is increased from 70% to 87%.

[0223] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0224] Based on the same inventive concept, the present application also provides a multimodal model processing device for implementing the multimodal model processing method mentioned above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more multimodal model processing device embodiments provided below can be found in the above-mentioned limitations of the multimodal model processing method and will not be repeated here.

[0225] In one embodiment, Figure 8 As shown, a multimodal model processing device is provided, including: a first generation module 802, a second generation module 804, a third generation module 806 and a training module 808, wherein:

[0226] A first generating module 802 is configured to generate description information of the visual data based on the visual data and the auditory data in the multimodal data;

[0227] A second generating module 804 is configured to generate sample question information including questions and answer options based on the description information;

[0228] The third generation module 806 is used to generate answers and reasoning process information corresponding to the answers based on the sample question information, visual data and auditory data;

[0229] The training module 808 is used to train the pre-trained multimodal model based on the visual data, auditory data, sample question information corresponding to the correct answer, and reasoning process information to obtain the target multimodal model when the answer is correct.

[0230] In one embodiment, the training module 808 is also used to perform preliminary training on the pre-trained multimodal model based on visual data, auditory data, sample question information corresponding to correct answers, and reasoning process information; after completing the preliminary training, sample question information in which the multimodal model reasoned incorrectly and the corresponding visual data and auditory data are selected, and model training is performed on the pre-trained multimodal model based on the selected data to obtain a target multimodal model.

[0231] In one embodiment, the training module 808 is further used to extract features from visual data and auditory data using a pre-trained multimodal model to obtain visual frame features and auditory features; perform feature complementation on auditory features based on the visual frame features to obtain complemented auditory features; generate summary information of the visual data based on the visual frame features and the complemented auditory features; perform question reasoning on sample question information corresponding to the correct answer based on the summary information to obtain predicted reasoning process information and a predicted answer; and optimize parameters of the pre-trained multimodal model based on the predicted reasoning process information, the reasoning process information, the predicted answer, and the answer.

[0232] In one embodiment, the reasoning process information is first reasoning process information;

[0233] The training module 808 is also used to perform information reasoning on the sample question information with reasoning errors based on the target visual data and the target auditory data through the multimodal model after preliminary training, to obtain a second reasoning result and corresponding second reasoning process information; the target visual data and the target auditory data are respectively the visual data and the auditory data corresponding to the sample question information with reasoning errors; determine the correctness reward value according to the second reasoning result; determine the consistency reward value according to the language consistency between the second reasoning process information and the sample question information with reasoning errors; optimize the parameters of the multimodal model based on the correctness reward value and the consistency reward value to obtain the target multimodal model.

[0234] In one embodiment, the training module 808 is further used to extract visual frame features and auditory features from the target visual data and the target auditory data; perform feature complementation on the auditory features based on the visual frame features to obtain complemented auditory features; generate summary information of the target visual data based on the visual frame features and the complemented auditory features; perform question reasoning on the sample question information corresponding to the correct answer based on the summary information of the target visual data to obtain predicted reasoning process information and predicted answers.

[0235] In one embodiment, Figure 9 As shown, the device also includes:

[0236] The pre-training module 810 is used to decode the video data to obtain a video frame sequence; sample some video frames from the video frame sequence as image training data, use audio data as audio training data, and use at least some video frames and audio data in the video frame sequence as audio and video training data to obtain first training data including image training data, audio training data, and audio and video training data; and pre-train the initial multimodal model based on the first training data to obtain a pre-trained multimodal model.

[0237] In one embodiment, the pre-training module 810 is further configured to train the initial multimodal model based on image training data in the first training data, train the initial multimodal model based on audio training data in the first training data, and train the initial multimodal model based on audio and video training data in the first training data until the model converges to obtain a pre-trained multimodal model.

[0238] The order of using image training data, audio training data, and audio and video training data for training is random or fixed.

[0239] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network;

[0240] The pre-training module 810 is also used to extract features from the image training data in the first training data through a feature extraction network to obtain video frame features of the image training data; align and fuse the video frame features with the preset audio features through a feature synchronization network to obtain first fusion features; perform information recognition on the first fusion features through an information recognition network to obtain recognition information; and optimize the parameters of the initial multimodal model based on the loss value between the recognition information and the first label information.

[0241] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network;

[0242] The pre-training module 810 is also used to extract features from the audio training data in the first training data through a feature extraction network to obtain audio features of the audio training data; align and fuse the audio features with preset video frame features through a feature synchronization network to obtain second fused features; perform speech recognition on the second fused features through an information recognition network to obtain speech information; and optimize the parameters of the initial multimodal model based on the loss value between the speech information and the second label information.

[0243] In one embodiment, the initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network, and the audio and video training data includes video training data and audio training data;

[0244] The pre-training module 810 is also used to extract features from audio training data through a feature extraction network to obtain audio features of the audio training data; and to extract features from video training data through a feature extraction network to obtain video frame features of the video training data; to align and fuse the audio features with the video frame features through a feature synchronization network to obtain a third fused feature; to perform information recognition on the third fused feature through an information recognition network to obtain target description information; and to optimize the parameters of the initial multimodal model based on the loss value between the target description information and the third label information.

[0245] In one embodiment, the pre-training module 810 is further used to, when the duration of the video data is less than or equal to a preset duration, perform feature complementation on the audio features according to the video frame features of the first sampled video frame sampled from the video frame sequence according to a preset sampling interval, and obtain complemented audio features that are synchronized in time with the video frame features of the first sampled video frame; wherein the feature scale of the audio features is smaller than the feature scale of the video frame features of the first sampled video frame; and perform feature fusion on the complemented audio features and the video frame features of the first sampled video frame to obtain a third fused feature.

[0246] In one embodiment, the pre-training module 810 is further used to, when the duration of the video data is greater than a preset duration, segment the video frame sequence for video training data, and sample a second sampled video frame from the subsequence obtained by the segmentation according to a preset sampling interval, perform feature complementation on the audio feature according to the video frame feature of the second sampled video frame, and obtain a complemented audio feature that is synchronized in time with the video frame feature of the second sampled video frame; wherein the feature scale of the audio feature is smaller than the feature scale of the video frame feature of the second sampled video frame; and perform feature fusion on the complemented audio feature and the video frame feature of the second sampled video frame to obtain a third fused feature.

[0247] In one embodiment, the second generating module 804 is further configured to generate a question based on the description information; generate corresponding answer options based on the description information and the question; and combine the question and the answer options to obtain sample question information.

[0248] In one embodiment, Figure 9 As shown, the device also includes:

[0249] A determination module 812 is configured to determine a correct answer to the question based on the description information;

[0250] The training module 808 is also used to perform preliminary training on the pre-trained multimodal model by using the visual data, auditory data and question as the second training data and the answer and reasoning process information as the fourth label information when the answer is consistent with the correct answer, so that the multimodal model can learn the reasoning process of the answer.

[0251] In one embodiment, the training model 808 is further used to, after completing preliminary training, perform information reasoning on questions in the sample question information based on the visual data and auditory data in the multimodal data through a multimodal model to obtain a first reasoning result; when the first reasoning result is wrong, determine the question for which the reasoning is wrong; and select sample question information, visual data, and auditory data corresponding to the question for which the reasoning is wrong.

[0252] In one embodiment, Figure 9 As shown, the device also includes:

[0253] The training module 808 is used to perform information inference on part of the visual data and part of the auditory data in the multimodal data through the target multimodal model to obtain descriptive information of the partial visual data; determine the prediction loss value based on the descriptive information of the partial visual data and the corresponding fifth label information; and fine-tune the parameters of the target multimodal model according to the predicted loss value to obtain the final target multimodal model.

[0254] In one embodiment, Figure 9As shown, the device also includes:

[0255] The reasoning module 814 is configured to, in response to a visual data analysis request, perform information reasoning on the target visual data and the corresponding auditory data using a target multimodal model to obtain summary information of the target visual data; and extract keywords and abstract information from the summary information of the target visual data;

[0256] Processing module 816 is used to use keywords and summary information as index information of target visual data, so that when performing information search or information recommendation, the target visual data can be searched out when the search terms and index information match, or the target visual data can be recommended to the target object when the index information and the target object's interest information are matched.

[0257] In the above embodiment, description information of the visual data is generated based on the visual data and auditory data in the multimodal data, and sample question information including questions and answer options is generated based on the description information. Therefore, when performing answer reasoning, in addition to using visual data and questions, corresponding answer options and auditory data are also given. On the one hand, reasoning can be performed within a limited range, which is conducive to improving the accuracy of answer reasoning. On the other hand, the data is enriched, which can also improve the accuracy of answer reasoning. Therefore, based on the sample question information, visual data and auditory data, the answer to the question and the reasoning process information corresponding to the answer can be accurately obtained; in addition, when the answer is correct, the pre-trained multimodal model is trained based on the visual data, auditory data, the corresponding sample question information when the answer is correct, and the reasoning process information. This not only enables the multimodal model to have the reasoning ability to output the intermediate thinking process, but also effectively avoids the problem of noise introduction and the risk of overfitting of the model during training. It can effectively alleviate the hallucination problem of the multimodal model and improve the reasoning accuracy of the multimodal model.

[0258] Each module in the multimodal model processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0259] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store multimodal data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multimodal model processing method is implemented.

[0260] In one embodiment, a computer device is provided, which may be a terminal, and includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and computer program stored in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication, where the wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a multimodal model processing method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0261] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0262] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned multimodal model processing method when executing the computer program.

[0263] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal model processing method are implemented.

[0264] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the multimodal model processing method when executed by a processor.

[0265] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0266] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processors (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), and other data processing logic devices.

[0267] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0268] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A multimodal model processing method, characterized in that: The method comprises: generating description information of the visual data based on the visual data and the auditory data in the multimodal data; generating sample question information including questions and answer options based on the description information; generating an answer and reasoning process information corresponding to the answer based on the sample question information, the visual data, and the auditory data; When the answer is correct, the pre-trained multimodal model is trained based on the visual data, the auditory data, the sample question information corresponding to the correct answer, and the reasoning process information to obtain a target multimodal model.

2. The method according to claim 1, characterized in that The training of the pre-trained multimodal model based on the visual data, the auditory data, the sample question information corresponding to the correct answer, and the reasoning process information to obtain a target multimodal model includes: Performing preliminary training on a pre-trained multimodal model based on the visual data, the auditory data, the sample question information corresponding to a correct answer, and the reasoning process information; After completing the preliminary training, sample question information in which the multimodal model has reasoning errors and the corresponding visual data and auditory data are selected, and model training is performed on the multimodal model after preliminary training based on the selected data to obtain a target multimodal model.

3. The method according to claim 2, characterized in that The training of the pre-trained multimodal model based on the visual data, the auditory data, the sample question information corresponding to the correct answer, and the reasoning process information includes: Extracting features from the visual data and the auditory data using a pre-trained multimodal model to obtain visual frame features and auditory features; Performing feature complementation on the auditory feature based on the visual frame feature to obtain a complemented auditory feature; generating summary information of the visual data according to the visual frame features and the padded auditory features; Performing question reasoning on the sample question information corresponding to the correct answer based on the summary information to obtain predicted reasoning process information and a predicted answer; Parameter optimization is performed on the pre-trained multimodal model based on the predicted reasoning process information, the reasoning process information, the predicted answer, and the answer.

4. The method according to claim 2, characterized in that The reasoning process information is first reasoning process information; The step of performing model training on the preliminarily trained multimodal model based on the selected data to obtain a target multimodal model includes: performing information reasoning on the sample question information with reasoning errors based on target visual data and target auditory data using the preliminarily trained multimodal model to obtain a second reasoning result and corresponding second reasoning process information; the target visual data and the target auditory data are respectively the visual data and the auditory data corresponding to the sample question information with reasoning errors; Determining a correctness reward value according to the second reasoning result; determining a consistency reward value based on the language consistency between the second reasoning process information and the sample question information with reasoning errors; Parameters of the multimodal model are optimized based on the correctness reward value and the consistency reward value to obtain a target multimodal model.

5. The method according to claim 4, characterized in that The information reasoning is performed on the sample question information with reasoning errors based on the target visual data and the target auditory data to obtain a second reasoning result and corresponding second reasoning process information, including: extracting visual frame features and auditory features from target visual data and target auditory data; Performing feature complementation on the auditory feature based on the visual frame feature to obtain a complemented auditory feature; generating summary information of the target visual data according to the visual frame features and the padded auditory features; Question reasoning is performed on the sample question information corresponding to the correct answer based on the summary information of the target visual data to obtain predicted reasoning process information and a predicted answer.

6. The method according to claim 1, characterized in that The visual data includes video data, the auditory data includes audio data corresponding to the video data, and the method further includes: Performing video decoding on the video data to obtain a video frame sequence; Sampling a portion of video frames from the video frame sequence as image training data, using the audio data as audio training data, and using at least a portion of the video frames in the video frame sequence and the audio data as audio-video training data, to obtain first training data including the image training data, the audio training data, and the audio-video training data; The initial multimodal model is pre-trained based on the first training data to obtain a pre-trained multimodal model.

7. The method according to claim 6, characterized in that Pre-training the initial multimodal model based on the first training data to obtain the pre-trained multimodal model includes: Training an initial multimodal model based on image training data in the first training data, training the initial multimodal model based on audio training data in the first training data, and training the initial multimodal model based on audio and video training data in the first training data until the model converges to obtain a pre-trained multimodal model; The order of using the image training data, the audio training data, and the audio and video training data for training is random or fixed.

8. The method according to claim 7, characterized in that The initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network; and the training of the initial multimodal model based on the image training data in the first training data includes: Performing feature extraction on the image training data in the first training data by the feature extraction network to obtain video frame features of the image training data; Aligning and fusing the video frame features with the preset audio features through the feature synchronization network to obtain a first fused feature; Performing information recognition on the first fusion feature through the information recognition network to obtain recognition information; Parameters of the initial multimodal model are optimized based on a loss value between the identification information and the first label information.

9. The method according to claim 7, characterized in that The initial multimodal model includes a feature extraction network, a feature synchronization network, and an information recognition network; and the training of the initial multimodal model based on the audio training data in the first training data includes: Performing feature extraction on the audio training data in the first training data by the feature extraction network to obtain audio features of the audio training data; Aligning and fusing the audio feature with the preset video frame feature through the feature synchronization network to obtain a second fused feature; Performing speech recognition on the second fusion feature through the information recognition network to obtain speech information; Parameters of the initial multimodal model are optimized based on a loss value between the speech information and the second label information.

10. The method according to claim 7, characterized in that The initial multimodal model includes a feature extraction network, a feature synchronization network and an information recognition network, and the audio and video training data includes video training data and audio training data; The training of the initial multimodal model based on the audio and video training data in the first training data includes: Performing feature extraction on the audio training data through the feature extraction network to obtain audio features of the audio training data; and performing feature extraction on the video training data through the feature extraction network to obtain video frame features of the video training data; Aligning and fusing the audio features with the video frame features through the feature synchronization network to obtain a third fused feature; Performing information recognition on the third fusion feature through the information recognition network to obtain target description information; Parameters of the initial multimodal model are optimized based on a loss value between the target description information and the third label information.

11. The method according to claim 10, characterized in that The aligning and fusing the audio features and the video frame features through the feature synchronization network to obtain a third fused feature includes: When the duration of the video data is less than or equal to a preset duration, the video training data is a first sampled video frame sampled from the video frame sequence according to a preset sampling interval, and feature padding is performed on the audio feature according to the video frame features of the first sampled video frame to obtain a padded audio feature that is synchronized with the video frame features of the first sampled video frame in time sequence; wherein the feature scale of the audio feature is smaller than the feature scale of the video frame feature of the first sampled video frame; The padded audio feature is fused with the video frame feature of the first sampled video frame to obtain a third fused feature.

12. The method according to claim 10, characterized in that The aligning and fusing the audio features and the video frame features through the feature synchronization network to obtain a third fused feature includes: When the duration of the video data is longer than a preset duration, the video training data is obtained by segmenting the video frame sequence and sampling second sampled video frames from the subsequences obtained by the segmentation at a preset sampling interval, and the audio features are feature-padded according to the video frame features of the second sampled video frames to obtain padded audio features that are synchronized in time with the video frame features of the second sampled video frames; wherein the feature scale of the audio features is smaller than the feature scale of the video frame features of the second sampled video frames; The padded audio feature is fused with the video frame feature of the second sampled video frame to obtain a third fused feature.

13. The method according to claim 2, characterized in that After completing the preliminary training, selecting sample question information in which the multimodal model has reasoning errors and the corresponding visual data and auditory data includes: After completing the preliminary training, performing information reasoning on the questions in the sample question information based on the visual data and the auditory data in the multimodal data through the multimodal model to obtain a first reasoning result; If the first reasoning result is wrong, determining the problem of the reasoning error; Sample question information, visual data, and auditory data corresponding to the question in which the reasoning is wrong are selected.

14. The method according to any one of claims 1 to 13, characterized in that The method further comprises: Performing information reasoning on part of the visual data and part of the auditory data in the multimodal data using the target multimodal model to obtain descriptive information of the part of the visual data; Determining a prediction loss value based on the description information of the portion of visual data and the corresponding fifth label information; Fine-tune the parameters of the target multimodal model according to the predicted loss value to obtain a final target multimodal model.

15. The method according to any one of claims 1 to 13, characterized in that The method further comprises: In response to a visual data analysis request, performing information reasoning on target visual data and corresponding auditory data using the target multimodal model to obtain summary information of the target visual data; extracting keywords and summary information from the summary information of the target visual data; The keywords and the summary information are used as index information of the target visual data, so that when performing information search or information recommendation, the target visual data can be searched out when the search terms and the index information match, or the target visual data can be recommended to the target object when the index information and the target object's interest information match.

16. A multimodal model processing device, characterized in that: The device comprises: A first generating module is configured to generate description information of the visual data based on the visual data and the auditory data in the multimodal data; A second generating module, configured to generate sample question information including questions and answer options based on the description information; a third generating module, configured to generate an answer and reasoning process information corresponding to the answer based on the sample question information, the visual data, and the auditory data; A training module is used to train the pre-trained multimodal model based on the visual data, the auditory data, the sample question information corresponding to the correct answer and the reasoning process information to obtain a target multimodal model when the answer is correct.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images