Interaction method, system and equipment based on multi-modal large model and medium

Abstract

The invention provides an interaction method, system and device based on a multi-modal large model and a medium, and relates to the technical field of multi-modal large models.The method comprises the steps that a real-time input information set is obtained; the real-time input information set is processed to obtain processed data, the processed data comprises first processing information, second processing information and third processing information, the real-time video information is processed to obtain the first processing information, the real-time audio information is processed to obtain the second processing information, and the third processing information is processed to obtain the third processing information; processing the real-time text information to obtain third processing information; performing correlation calculation on the processing data through a preset modal correlation model; performing splicing processing on the processing data according to the associated loss information to obtain a spliced data set; and decoding and outputting the spliced data set to obtain interaction response data which is used for feeding back interaction information. According to the method, cross-modal association of real-time input data is achieved, and more complex multi-modal combination can be conveniently expanded.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to an interaction method, system, device and medium based on a multimodal large model. Background Art

[0002] In recent years, the field of artificial intelligence has been undergoing a revolutionary change led by large language models. From the initial text generation to the current fusion of multimodal information such as speech, audio, images, and video, large models have become a research hotspot in the field of artificial intelligence. The initial large models mainly focused on text interaction, with the goal of achieving natural language processing tasks such as machine translation, text summarization, and question-answering systems. With technological advances, researchers have tried to incorporate multimodal information into large models to enhance their generalization capabilities in real-world scenarios. Multimodal large models combine language processing capabilities with the ability to generate multiple information modes. They can not only process text information, but also process multimodal data such as audio and video, thereby achieving richer and more natural interaction methods. However, in existing methods, multimodality often relies too much on strict modal classification, does not achieve cross-modal associations, and does not extend to more complex multimodal combinations, and cannot naturally handle complex multimodal interaction scenarios. Summary of the invention

[0003] The purpose of the present invention is to provide an interactive method, system, device and medium based on a multimodal large model to solve the problem of realizing cross-modal association of real-time input data and facilitate expansion to more complex multimodal combinations.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows: In a first aspect, the present application provides an interaction method based on a multimodal large model, the method comprising: Acquire a real-time input information set, wherein the real-time input information set includes real-time video information, real-time audio information, and real-time text information; Processing the real-time input information sets respectively to obtain processed data, the processed data including first processed information, second processed information and third processed information, wherein the real-time video information is processed to obtain the first processed information, the real-time audio information is processed to obtain the second processed information, and the real-time text information is processed to obtain the third processed information; Performing correlation calculation on the processed data through a preset modal correlation model to obtain correlation loss information of the first processed information, the second processed information, and the third processed information; splicing the processed data according to the associated loss information to obtain a spliced ​​data set; The spliced ​​data set is decoded and output to obtain interactive response data, and the interactive response data is used to feed back interactive information.

[0005] In a second aspect, the present application further provides an interactive system based on a multimodal large model, the system comprising: An acquisition module, used to acquire a real-time input information set, wherein the real-time input information set includes real-time video information, real-time audio information and real-time text information; A processing module, used to process the real-time input information set respectively to obtain processed data, wherein the processed data includes first processed information, second processed information and third processed information, wherein the real-time video information is processed to obtain the first processed information, the real-time audio information is processed to obtain the second processed information, and the real-time text information is processed to obtain the third processed information; A correlation module, used for performing correlation calculation on the processed data through a preset modal correlation model to obtain correlation loss information of the first processed information, the second processed information and the third processed information; A splicing module, used for splicing the processed data according to the associated loss information to obtain a spliced ​​data set; The decoding output module is used to decode and output the spliced ​​data set to obtain interactive response data, and the interactive response data is used to feed back interactive information.

[0006] In a third aspect, the present application also provides an interactive device based on a multimodal large model, comprising: Memory for storing computer programs; A processor is used to implement the steps of the interaction method based on the multimodal large model when executing the computer program.

[0007] In a fourth aspect, the present application also provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned interaction method based on the multimodal large model are implemented.

[0008] The beneficial effects of the present invention are: The present invention realizes complete information interaction on the basis of cross-modality by sequentially classifying, associating, splicing and expanding, and decoding and outputting the real-time input information set, and can naturally handle complex multimodal interaction scenarios. Among them: in data processing, the feature processing of different modalities is considered; in the associating calculation, the preset modal association model is introduced to realize the associating calculation of different modal processing information, breaking the traditional multimodal system's reliance on strict modal classification, solving the problem of realizing cross-modal association of real-time input data, and facilitating expansion to more complex multimodal combinations; the introduction of splicing and expansion realizes the processing capability of long contexts.

[0009] In addition, the association calculation in this method embodies progressive association, starting with bimodal associations, such as vision-text, audio-text, and then gradually expanding to more complex multimodal combinations. This association algorithm not only reduces the difficulty of model association calculation, but also helps the model establish a more solid cross-modal understanding capability.

[0010] Other features and advantages of the present invention will be set forth in the following description, and in part will become apparent from the description, or may be understood by practicing embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 It is a schematic diagram of the flow of the interactive method based on the multimodal large model described in an embodiment of the present invention; Figure 2 Schematic diagram of the structure of an interactive system based on a multimodal large model described in an embodiment of the present invention; Figure 3 Schematic diagram of the structure of an interactive device based on a multimodal large model described in an embodiment of the present invention; Markings in the figure: 1. Acquisition module; 2. Processing module; 3. Association module; 4. Stitching module; 5. Decoding output module; 31. First association unit; 32. Second association unit; 33. Third association unit; 800. Interaction device based on multimodal large model; 801. Processor; 802. Memory; 803. Multimedia component; 804. I / O interface; 805. Communication component. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0014] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0015] Embodiment 1: This embodiment provides an interaction method based on a multimodal large model.

[0016] See also Figure 1 , the figure shows that the method includes steps S1 to S5, specifically: S1: Acquire a real-time input information set, wherein the real-time input information set includes real-time video information, real-time audio information and real-time text information; Among them, the real-time video information includes not only video information such as real-time video, but also picture information.

[0017] S2: Processing the real-time input information sets respectively to obtain processed data, wherein the processed data includes first processed information, second processed information and third processed information, wherein the real-time video information is processed to obtain the first processed information, the real-time audio information is processed to obtain the second processed information, and the real-time text information is processed to obtain the third processed information; In step S2, in order to clarify the processing process of the processed data, step S2 includes steps S21 to S24, specifically: S21: Solve the real-time video information through a video sampling model to obtain a key frame data set, wherein the video sampling model is: ; (1) In the above formula (1), represents a key frame dataset, (.) represents the video sampling function, Represents real-time video information, which is the original image or video sequence, Indicates the preset number of sampling frames; In step S21, the video sampling function (.) Capable of sampling key frame data and unifying the sampling length.

[0018] S22: Process each frame of data in the key frame data set through a video modality processing model to obtain first processing information, wherein the video modality processing model is: ; (2) In the above formula (2), represents a key frame dataset, Respectively represent different key frame data, Represents key frame data Model for dynamic block processing, Represents key frame data The characteristic vector of Represents key frame data First process information, {.} represents a bridge model; S23: Processing the real-time audio information through an audio modal processing model to obtain second processing information, wherein the audio modal processing model is: ; (3) In the above formula (3), Represents real-time audio information, They respectively represent the audio sequence data of different segments after segmenting the real-time audio information. Indicates that the feature extraction of the audio sequence data of the current segment is performed through the Mel filter model. Indicates that after feature extraction through the Mel filter model The data to be convolved and downsampled, {.} indicates the encoding model, express second processing information of the audio sequence; In this method, algorithms such as fixed-length segmentation, sliding window, and feature analysis can be used to segment real-time audio information; among them, fixed-length segmentation is to segment real-time audio information according to a fixed time length, which is suitable for situations where the audio content has a fixed frequency and is high; sliding window segmentation is suitable for processing audio data containing multiple speakers; feature analysis is suitable for scenarios where detailed analysis of the audio content is required before segmentation.

[0019] S24: Process the real-time text information through a preset text modality processing model to obtain third processing information.

[0020] In this step, the preset text modality processing model can use tokenization to process the text information to obtain third processing information, wherein the third processing information is text sequence data of different segments. In the tokenization process, a byte pair encoding processing algorithm can also be used.

[0021] In existing methods, multimodality often relies too much on strict modal classification, fails to achieve cross-modal associations, and does not expand to more complex multimodal combinations. Therefore, this method performs association calculations on processed data to break the traditional multimodal system's reliance on strict modal classification.

[0022] S3: performing association calculation on the processed data through a preset modal association model to obtain association loss information of the first processing information, the second processing information and the third processing information; In step S3, in order to clarify the specific content of the association calculation, step S3 includes S31 to S33: S31: Obtaining loss information associated with the video text; In this step, the specific calculation process of the loss information associated with the video text includes steps S311 to S313, specifically: S311: performing prediction probability calculation on the first processed information and the processed information of the preset text to obtain first prediction information; The calculation formula of the first prediction information is: ; (4) In the above formula (4), represents the first prediction information, represents the probability of the first processing information when the processing information of the preset text is the first processing information, represents the first processing information, Indicates the processing information of the preset text.

[0023] S312: performing prediction probability calculation on the third processing information and the processing information of the preset video to obtain second prediction information; The calculation formula of the second prediction information is: ; (5) In the above formula (5), represents the second prediction information, represents the probability that the processing information of the preset video is the third processing information, represents the third processing information, Indicates the processing information of the preset video.

[0024] S313: Perform loss calculation on the first prediction information and the second prediction information to obtain loss information associated with the video text.

[0025] In step S313, the loss is calculated as: ; (6) In the above formula (6), represents the first prediction information, represents the probability of the first processing information when the processing information of the preset text is the first processing information, represents the first processing information, Indicates the processing information of the preset text. represents the second prediction information, represents the probability that the processing information of the preset video is the third processing information, represents the third processing information, Indicates the processing information of the preset video. Indicates the loss information of video text association.

[0026] In this method, the loss information of the video-text association is used to characterize the association between the video content (including images) and the corresponding description text.

[0027] S32: Obtain loss information associated with the audio text; In step S32, in order to clarify the specific calculation process of the loss information associated with the audio text, step S32 includes S321 to S323, specifically: S321: performing prediction probability calculation on the second processed information and the processed information of the preset text to obtain third prediction information; The calculation formula of the third prediction information is: ; (7) In the above formula (7), represents the third prediction information, represents the probability of the second processing information when the processing information of the preset text is represents the second processing information, Indicates the processing information of the preset text.

[0028] S322: performing prediction probability calculation on the third processing information and the processing information of the preset audio to obtain fourth prediction information; The calculation formula of the fourth prediction information is: ; (8) In the above formula (8), represents the fourth prediction information, represents the probability that the processing information of the preset audio is the third processing information, represents the third processing information, Indicates the processing information of preset audio.

[0029] S323: Perform loss calculation on the third prediction information and the fourth prediction information to obtain loss information associated with the audio text.

[0030] In step S323, the loss is calculated as: ; (9) In the above formula (9), represents the third prediction information, represents the probability of the second processing information when the processing information of the preset text is represents the second processing information, Indicates the processing information of the preset text. represents the fourth prediction information, represents the probability that the processing information of the preset audio is the third processing information, represents the third processing information, Indicates the processing information of preset audio. Indicates the loss information of audio text association.

[0031] The audio-text association loss information is used to associate the audio content with the text transcription.

[0032] S33: Obtain overall correlation loss information based on the loss information associated with the video text and the loss information associated with the audio text. In this step, the overall correlation loss information is calculated as: ; (10) In the above formula (10), Represents the overall associated loss information, Indicates the loss information of video text association, Indicates the loss information of audio text association, represents the regularization term.

[0033] The overall correlation loss information reflects the correlation when multiple modalities are combined. Through the feedback of the overall correlation loss information, a unified framework is established to handle the conversion and correlation between modalities, so that the multimodal model can handle complex multimodal interaction scenarios more naturally.

[0034] In this method, the loss information associated with the video text, the loss information associated with the audio text, and the overall associated loss information are compared with the preset target loss information one by one to determine whether the current degree of association between different modalities meets the standard, thereby achieving natural learning and processing of complex associations between modalities.

[0035] In addition, the association calculation in this method embodies progressive association, starting with bimodal associations, such as vision-text, audio-text, and then gradually expanding to more complex multimodal combinations. This association algorithm not only reduces the difficulty of model association calculation, but also helps the model establish a more solid cross-modal understanding capability.

[0036] In this method, in order to realize the processing capability of the multimodal large model for the long context, step S4 splicing processing is introduced to realize the joint optimization of the multimodal context.

[0037] S4: splicing the processed data according to the associated loss information to obtain a spliced ​​data set; When the loss information associated with the video text, the loss information associated with the audio text, and the overall associated loss information are compared with the preset target loss information one by one, and all meet the preset target loss information requirements, splicing processing can be performed at this time. The preset target loss information requirements are set according to the association requirements of the large model for different modes. If the large model needs to strongly associate video, audio and text, the loss information associated with the video text, the loss information associated with the audio text, and the overall associated loss information are required to be less than the preset target loss information. If any one of them is greater than the preset target loss information, it is necessary to adjust the parameters through the optimization algorithm.

[0038] In this method, in order to clarify the specific process of splicing processing, step S4 includes S41 to S44, which are: S41: performing splicing processing on the first processed information to obtain first splicing information, wherein the splicing calculation is: ; (11) In the above formula (11), Indicates the first splicing information; Indicates The first processing information of key frame data, wherein middle The value range of is 1 to n; Indicates splicing processing; S42: performing splicing processing on the second processed information to obtain second splicing information, wherein the splicing calculation is: ; (12) In the above formula (12), Indicates the second splicing information, express The second processing information of the audio sequence is middle The value range of is 1 to m; Indicates splicing processing; S43: performing splicing processing on the third processed information to obtain third splicing information, wherein the splicing calculation is: ; (13) In the above formula (13), Indicates the third splicing information, Indicates The content corresponding to the text sequence data, where middle The value range of is from 1 to w; Indicates splicing processing; S44: The first splicing information, the second splicing information and the third splicing information are spliced ​​as a whole to obtain a spliced ​​data set, wherein the overall splicing calculation is: ; (14) In the above formula (14), represents the concatenated dataset, Indicates the first splicing information, Indicates the second splicing information, Indicates the third splicing information, Indicates the maximum sequence length allowed in the preset splicing model.

[0039] The introduction of the spliced ​​dataset enables the formation of a unified sequence representation of text, image, video and audio features through splicing processing based on multimodal input. In addition, during the splicing process, the splicing strategy can be dynamically adjusted according to real-time input information, and the splicing method and length can be automatically adjusted according to different scenarios to achieve the optimal feature combination, significantly improving the performance and efficiency of multimodal large models in practical applications.

[0040] S5: Decode and output the spliced ​​data set to obtain interactive response data, where the interactive response data is used to feed back interactive information.

[0041] In the decoded output, in order to dynamically allocate the importance of different modalities, the weight of each modality is solved through the attention mechanism in the feature fusion model included in the decoded output. At this time, it can achieve: 1. Dynamically allocate the proportion of the total sequence length according to the importance of different modalities; 2. Prioritize the integrity of key interactive response data.

[0042] In step S5, the decoded output is calculated as: ; (15) In the above formula (15), Indicates interactive response data. represents the language decoding model, represents the feature fusion model, represents the feature projection model, Represents a concatenated dataset.

[0043] In formula (15), the concatenated dataset is output as text in a continuous token in each model in an autoregressive manner. This method fully utilizes the powerful sequence modeling ability of the large model and can naturally understand and generate interactive information for feedback.

[0044] During decoding output, in order to achieve a unified representation of the video, image, audio and text embedding space, the feature vectors of the embedding space may be aligned. The alignment method may be based on a gradient descent method, and the aligned feature vectors are used to characterize the feature consistency of different modalities.

[0045] In addition, the decoded output text can also be converted through a text-to-speech model, and the conversion formula is: ; (16) In the above formula (16), Represents the final interactive response audio information, represents an external conversion model, Represents interaction response data.

[0046] The introduction of an external conversion model can integrate part of the decoding process into the reasoning process of the large model and realize text-to-speech. In this process, the speech generation delay can be reduced through joint optimization.

[0047] In this method, in order to improve the deep association of different modalities and enhance the generalization ability of the model for data processing, the processed data is marked and the state mark prediction is maximized, that is, between step S3 and step S4, there are also S34 and S35, specifically: S34: performing status marking on the processed data to obtain multimodal marking information, wherein the multimodal marking information includes video marking information, audio marking information and text marking information; The status mark is used to distinguish different information, specifically: Video tag information corresponding ,The video is marked by a visual encoder, and the video marking information includes images and videos; Audio tag information correspondence , audio tagging through an audio encoder; Text tag information correspondence , text tokenization is performed through a text encoder.

[0048] S35: Predicting the multimodal tag information by using a maximization state tag prediction model, wherein the maximization state tag prediction model is calculated as: ; (17) In the above formula (17), Maximize the state label prediction loss information; Indicates the number of samples of processed data; The true label of the state mark, whose value is 1 or 0; represents the predicted state label probability, where Represents multimodal tag information, Indicates the current preset data.

[0049] In the above process, the maximization state labeling prediction model realizes the labeling feedback of the processed data. The maximization state labeling prediction loss information characterizes the accuracy of the model in labeling different information. When the maximization state labeling prediction loss information meets the preset prediction loss information, it further improves the deep correlation between different modalities and enhances the model's generalization ability for data processing.

[0050] Embodiment 2: like Figure 2 As shown, this embodiment provides an interactive system based on a multimodal large model, the system comprising: An acquisition module 1 is used to acquire a real-time input information set, wherein the real-time input information set includes real-time video information, real-time audio information and real-time text information; Processing module 2, used to process the real-time input information set respectively to obtain processed data, wherein the processed data includes first processed information, second processed information and third processed information, wherein the real-time video information is processed to obtain the first processed information, the real-time audio information is processed to obtain the second processed information, and the real-time text information is processed to obtain the third processed information; An association module 3 is used to perform association calculation on the processed data through a preset modal association model to obtain association loss information of the first processing information, the second processing information and the third processing information; A splicing module 4 is used to splice the processed data according to the associated loss information to obtain a spliced ​​data set; The decoding output module 5 is used to decode and output the spliced ​​data set to obtain interactive response data, and the interactive response data is used to feed back interactive information.

[0051] In an implementation method disclosed in the present invention, the association module 3 includes: A first associating unit 31 is used to obtain loss information associated with the video text; A second associating unit 32, used to obtain loss information associated with the audio text; The third association unit 33 is used to obtain the overall association loss information according to the loss information associated with the video text and the loss information associated with the audio text.

[0052] It should be noted that, regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0053] Embodiment 3: Corresponding to the above method embodiment, this embodiment also provides an interactive device based on a multimodal large model. The interactive device based on a multimodal large model described below and the interactive method based on a multimodal large model described above can be referenced to each other.

[0054] Figure 3 8 is a block diagram of an interactive device 800 based on a multimodal large model according to an exemplary embodiment. Figure 3 As shown, the interactive device 800 based on the multimodal big model may include: a processor 801 and a memory 802. The interactive device 800 based on the multimodal big model may also include one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0055] The processor 801 is used to control the overall operation of the multimodal large model-based interactive device 800 to complete all or part of the steps in the above-mentioned multimodal large model-based interactive method. The memory 802 is used to store various types of data to support the operation of the multimodal large model-based interactive device 800, and these data may include, for example, instructions for any application or method operating on the multimodal large model-based interactive device 800, and application-related data, such as sent and received messages, pictures, audio, video, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 802 or sent via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules, and the above-mentioned other interface modules can be keyboards, mice, buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the interactive device 800 based on the multimodal large model and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 805 can include: Wi-Fi module, Bluetooth module, NFC module.

[0056] In an exemplary embodiment, the interaction device 800 based on the multimodal big model can be implemented by one or more application specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), digital signal processors (Digital Signal Processor, referred to as DSP), digital signal processing devices (Digital Signal Processing Device, referred to as DSPD), programmable logic devices (Programmable Logic Device, referred to as PLD), field programmable gate arrays (Field Programmable Gate Array, referred to as FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned interaction method based on the multimodal big model.

[0057] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned interaction method based on the multimodal large model are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 802 including program instructions, and the above-mentioned program instructions can be executed by the processor 801 of the interaction device 800 based on the multimodal large model to complete the above-mentioned interaction method based on the multimodal large model.

[0058] Embodiment 4: Corresponding to the above method embodiment, a readable storage medium is also provided in this embodiment. The readable storage medium described below and the interactive method based on a multimodal large model described above can refer to each other.

[0059] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the interactive method based on a multimodal large model of the above-mentioned method embodiment.

[0060] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0062] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An interactive method based on a multimodal large model, characterized in that: include: Acquire a real-time input information set, wherein the real-time input information set includes real-time video information, real-time audio information, and real-time text information; Processing the real-time input information sets respectively to obtain processed data, the processed data including first processed information, second processed information and third processed information, wherein the real-time video information is processed to obtain the first processed information, the real-time audio information is processed to obtain the second processed information, and the real-time text information is processed to obtain the third processed information; Performing correlation calculation on the processed data through a preset modal correlation model to obtain correlation loss information of the first processed information, the second processed information, and the third processed information; splicing the processed data according to the associated loss information to obtain a spliced ​​data set; The spliced ​​data set is decoded and output to obtain interactive response data, and the interactive response data is used to feed back interactive information.

2. The interactive method based on a multimodal large model according to claim 1, characterized in that: The processing data is correlated and calculated by using a preset modal correlation model to obtain correlation loss information of the first processing information, the second processing information and the third processing information, including: Obtain loss information associated with video text; Get the loss information associated with the audio text; The overall associated loss information is obtained based on the loss information associated with the video text and the loss information associated with the audio text.

3. The interactive method based on a multimodal large model according to claim 2, characterized in that: The loss information calculation of the video text association includes: Performing prediction probability calculation on the first processed information and the processed information of the preset text to obtain first prediction information; Performing prediction probability calculation on the third processing information and the processing information of the preset video to obtain second prediction information; Loss calculation is performed on the first prediction information and the second prediction information to obtain loss information associated with the video text.

4. The interactive method based on a multimodal large model according to claim 3 is characterized in that: The loss calculation is performed on the first prediction information and the second prediction information to obtain the loss information associated with the video text, and the loss calculation is: ； In the above formula, represents the first prediction information, represents the probability of the first processing information when the processing information of the preset text is the first processing information, represents the first processing information, Indicates the processing information of the preset text. represents the second prediction information, represents the probability that the processing information of the preset video is the third processing information, represents the third processing information, Indicates the processing information of the preset video. Indicates the loss information of video text association.

5. The interactive method based on a multimodal large model according to claim 2, characterized in that: The loss information calculation of the audio text association includes: Performing prediction probability calculation on the second processed information and the processed information of the preset text to obtain third prediction information; Performing prediction probability calculation on the third processing information and the processing information of the preset audio to obtain fourth prediction information; Loss calculation is performed on the third prediction information and the fourth prediction information to obtain loss information associated with the audio text.

6. The interactive method based on a multimodal large model according to claim 5, characterized in that: The loss calculation is performed on the third prediction information and the fourth prediction information to obtain the loss information associated with the audio text, and the loss calculation is: ； In the above formula, represents the third prediction information, represents the probability of the second processing information when the processing information of the preset text is represents the second processing information, Indicates the processing information of the preset text. represents the fourth prediction information, represents the probability that the processing information of the preset audio is the third processing information, represents the third processing information, Indicates the processing information of preset audio. Indicates the loss information of audio text association.

7. The interactive method based on a multimodal large model according to claim 2, characterized in that: The overall correlation loss information is obtained based on the loss information of the video text association and the loss information of the audio text association. The calculation of the overall correlation loss information is: ； In the above formula, Represents the overall associated loss information, Indicates the loss information of video text association, Indicates the loss information of audio text association, represents the regularization term.

8. The interactive method based on a multimodal large model according to claim 1, characterized in that: After performing correlation calculation on the processed data through a preset modal correlation model to obtain correlation loss information of the first processed information, the second processed information and the third processed information, the method further includes: Performing status marking on the processed data to obtain multimodal marking information, wherein the multimodal marking information includes video marking information, audio marking information, and text marking information; The multimodal label information is predicted by a maximization state label prediction model, and the maximization state label prediction model is calculated as: ； In the above formula, Maximize the state label prediction loss information; Indicates the number of samples of processed data; The true label of the state mark, whose value is 1 or 0; represents the predicted state label probability, where Represents multimodal tag information, Indicates the current preset data.

9. The interactive method based on a multimodal large model according to claim 1, characterized in that: In the feature fusion model included in the decoded output, the weight of each modality is solved through the attention mechanism.

10. The interactive method based on a multimodal large model according to claim 1, characterized in that: The decoded output is calculated as: ； In the above formula, Indicates interactive response data. represents the language decoding model, represents the feature fusion model, represents the feature projection model, Represents a concatenated dataset.

11. The interactive method based on a multimodal large model according to claim 1, characterized in that: The real-time input information sets are processed respectively to obtain processed data, wherein the processed data includes first processed information, second processed information and third processed information, including: The real-time video information is solved by a video sampling model to obtain a key frame data set, wherein the video sampling model is: ； In the above formula, represents a key frame dataset, (.) represents the video sampling function, Represents real-time video information, which is the original image or video sequence, Indicates the preset number of sampling frames; Each frame of data in the key frame data set is processed by a video modality processing model to obtain first processing information, wherein the video modality processing model is: ； In the above formula, represents a key frame dataset, Respectively represent different key frame data, Represents key frame data Model for dynamic block processing, Represents key frame data The characteristic vector of Represents key frame data First process information, {.} represents a bridge model; The real-time audio information is processed by an audio modal processing model to obtain second processing information, wherein the audio modal processing model is: ； In the above formula, Represents real-time audio information, They respectively represent the audio sequence data of different segments after segmenting the real-time audio information. Indicates that the feature extraction of the audio sequence data of the current segment is performed through the Mel filter model. Indicates that after feature extraction through the Mel filter model The data for convolution downsampling, {.} indicates the encoding model, express second processing information of the audio sequence; The real-time text information is processed by a preset text modality processing model to obtain third processing information.

12. The interactive method based on a multimodal large model according to claim 1, characterized in that: The processed data are spliced ​​according to the associated loss information to obtain a spliced ​​data set, including: The first processed information is spliced ​​to obtain first spliced ​​information, wherein the splicing calculation is: ； In the above formula, Indicates the first splicing information; Indicates The first processing information of key frame data, wherein middle The value range of is 1 to n; Indicates splicing processing; The second processed information is spliced ​​to obtain second spliced ​​information, wherein the splicing calculation is: ； In the above formula, Indicates the second splicing information, express The second processing information of the audio sequence is middle The value range of is 1 to m; Indicates splicing processing; The third processed information is spliced ​​to obtain third spliced ​​information, wherein the splicing calculation is: ； In the above formula, Indicates the third splicing information, Indicates The content corresponding to the text sequence data, where middle The value range of is from 1 to w; Indicates splicing processing; The first splicing information, the second splicing information and the third splicing information are spliced ​​as a whole to obtain a spliced ​​data set, wherein the overall splicing calculation is: ； In the above formula, represents the concatenated dataset, Indicates the first splicing information, Indicates the second splicing information, Indicates the third splicing information, Indicates the maximum sequence length allowed in the preset splicing model.

13. An interactive system based on a multimodal large model, characterized in that: include: An acquisition module, used to acquire a real-time input information set, wherein the real-time input information set includes real-time video information, real-time audio information and real-time text information; A processing module, used to process the real-time input information set respectively to obtain processed data, wherein the processed data includes first processed information, second processed information and third processed information, wherein the real-time video information is processed to obtain the first processed information, the real-time audio information is processed to obtain the second processed information, and the real-time text information is processed to obtain the third processed information; A correlation module, used for performing correlation calculation on the processed data through a preset modal correlation model to obtain correlation loss information of the first processed information, the second processed information and the third processed information; A splicing module, used for splicing the processed data according to the associated loss information to obtain a spliced ​​data set; The decoding output module is used to decode and output the spliced ​​data set to obtain interactive response data, and the interactive response data is used to feed back interactive information.

14. An interactive device based on a multimodal large model, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the interaction method based on a multimodal large model as claimed in any one of claims 1 to 12 when executing the computer program.

15. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the interaction method based on a multimodal large model as claimed in any one of claims 1 to 12 are implemented.

Images