A multimodal model inference method, training method and device

Abstract

The application discloses a multimodal model reasoning method and device. The method comprises the following steps: determining a reference code corresponding to reference data; taking other modal input data except the reference data in at least two kinds of input data as to-be-aligned data, determining a first code number generated by encoding each to-be-aligned data, and determining a modal code corresponding to the to-be-aligned data based on the first code number; fusing each modal code and the reference code to generate a fusion code, inputting the fusion code into a multimodal model, and obtaining a reasoning result corresponding to the fusion code output by the multimodal model; wherein, in the process of obtaining the reasoning result, the actual modal code corresponding to the to-be-aligned data generated by the multimodal encoder is used to replace the preset placeholder code between the preset starting code and the preset terminal code in the fusion code. The scheme improves the accuracy of the encoding alignment of the input data of each modal of the multimodal model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal model reasoning method, training method and apparatus. Background Technology

[0002] In inference based on multimodal models, these models typically receive input data from different modalities, such as text and images, and output inference results according to the inference request. This process involves encoding the input data for different modalities using different encoders, aligning these codes, and then performing inference based on the aligned codes. However, this modal alignment is often not precise enough, resulting in inaccurate inference results. Summary of the Invention

[0003] This invention provides a multimodal model inference method, training method, and apparatus, which improves the accuracy of multimodal model inference on input data of various modalities.

[0004] According to one aspect of the present invention, a multimodal model inference method is provided, comprising:

[0005] Acquire input data from at least two modalities, and use any one of the input data from the at least two modalities as reference data to determine the reference code corresponding to the reference data;

[0006] The modal input data other than the reference data in at least two modal input data are taken as the data to be aligned. The first number of codes generated by encoding each type of data to be aligned is determined, and the modal code corresponding to the data to be aligned is determined based on the first number of codes. The modal code includes a preset start code, a preset end code, and the first number of preset placeholder codes located between the preset start code and the preset end code.

[0007] Each modal encoding is fused with the baseline encoding to generate a fused encoding, and the fused encoding is input into a pre-trained multimodal model to obtain the inference result corresponding to the fused encoding output by the multimodal model; wherein, in the process of obtaining the inference result based on the multimodal model, the actual modal encoding corresponding to the data to be aligned generated by the multimodal encoder in the multimodal model replaces the preset placeholder encoding between the preset start encoding and the preset end encoding in the fused encoding.

[0008] According to another aspect of the present invention, a multimodal model training method is provided, comprising:

[0009] Acquire at least two sets of multimodal sample data; wherein each set of multimodal sample data contains sample data of at least two modalities;

[0010] For each of the multimodal sample data groups, the reference code corresponding to the reference data is determined by taking the sample data of any modality in the multimodal sample data group as reference data.

[0011] The sample data other than the reference data in the multimodal sample data group are taken as the data to be processed. The number of second codes generated by encoding the data to be processed for each modality is determined respectively, and the sample code corresponding to the data to be processed is determined based on the number of second codes. The sample code includes a preset start code, a preset end code, and a number of preset placeholder codes of the number of second codes located between the preset start code and the preset end code.

[0012] The sample code for each modality involved in each of the multimodal sample data groups is fused with the reference code to generate a data group code;

[0013] A preset large language model is trained based on at least two data group codes to generate a multimodal model; wherein, during the training of the preset large language model based on at least two data group codes, the preset placeholder code between the preset start code and the preset end code in the data group codes is replaced by the actual sample code corresponding to the data to be processed generated by the multimodal encoder in the large language model.

[0014] According to another aspect of the present invention, a multimodal model inference apparatus is provided, comprising:

[0015] A reference code determination module is used to acquire input data of at least two modalities, and use any one of the input data of the at least two modalities as reference data to determine the reference code corresponding to the reference data;

[0016] A modal encoding determination module is used to take input data of at least two modalities other than the reference data as data to be aligned, determine the first number of codes generated by encoding each type of data to be aligned, and determine the modal encoding corresponding to the data to be aligned based on the first number of codes; wherein, the modal encoding includes a preset start code, a preset end code, and the first number of preset placeholder codes located between the preset start code and the preset end code;

[0017] The inference result generation module is used to fuse each modal encoding with the baseline encoding to generate a fused encoding, and input the fused encoding into a pre-trained multimodal model to obtain the inference result corresponding to the fused encoding output by the multimodal model; wherein, in the process of obtaining the inference result based on the multimodal model, the actual modal encoding corresponding to the data to be aligned generated based on the multimodal encoder in the multimodal model replaces the preset placeholder encoding between the preset start encoding and the preset end encoding in the fused encoding.

[0018] According to another aspect of the present invention, a multimodal model training apparatus is provided, comprising:

[0019] A multimodal sample data set acquisition module is used to acquire at least two multimodal sample data sets; wherein each multimodal sample data set contains sample data of at least two modalities;

[0020] The reference encoding determination module is used to determine the reference encoding corresponding to the reference data for each multimodal sample data group, using the sample data of any modality in the multimodal sample data group as reference data.

[0021] The sample encoding determination module is used to take the sample data other than the reference data in the multimodal sample data group as the data to be processed, determine the number of second codes generated by encoding the data to be processed for each modality, and determine the sample code corresponding to the data to be processed based on the number of second codes; wherein, the sample code includes a preset start code, a preset end code, and a number of preset placeholder codes of the number of second codes located between the preset start code and the preset end code;

[0022] The data group encoding generation module is used to fuse the sample encoding of each modality involved in each multimodal sample data group with the reference encoding to generate data group encoding;

[0023] A multimodal model generation module is used to train a preset large language model based on at least two data group codes to generate a multimodal model; wherein, during the training of the preset large language model based on at least two data group codes, the preset placeholder code between the preset start code and the preset end code in the data group codes is replaced by the actual sample code corresponding to the data to be processed generated by the multimodal encoder in the large language model.

[0024] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0025] At least one processor; and

[0026] A memory communicatively connected to the at least one processor; wherein,

[0027] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to execute the multimodal model inference method or multimodal model training method according to any embodiment of the present invention.

[0028] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the multimodal model inference method or multimodal model training method according to any embodiment of the present invention.

[0029] The multimodal model inference scheme of this invention acquires input data from at least two modalities, and uses any one of the input data from the at least two modalities as reference data to determine the reference code corresponding to the reference data; other modal input data from the at least two modalities, excluding the reference data, are used as data to be aligned, and a first code quantity generated by encoding each type of data to be aligned is determined, and a modal code corresponding to the data to be aligned is determined based on the first code quantity; wherein, the modal code includes a preset start code, a preset end code, and a preset number of preset placeholder codes located between the preset start code and the preset end code; each modal code is fused with the reference code to generate a fused code, and the fused code is input into a pre-trained multimodal model to obtain the inference result corresponding to the fused code output by the multimodal model; wherein, in the process of obtaining the inference result based on the multimodal model, the preset placeholder code between the preset start code and the preset end code in the fused code is replaced by the actual modal code corresponding to the data to be aligned generated by the multimodal encoder in the multimodal model. The technical solution provided by this invention adds a preset start code, a preset placeholder code, and a preset end code corresponding to each type of data to be aligned in the fusion encoding. This allows the actual modal encoding corresponding to the data to be aligned, generated by the multimodal encoder in the multimodal model, to only replace the preset placeholder code between the preset start code and the preset end code in the fusion encoding during the process of obtaining inference results based on the multimodal model. This effectively avoids the influence of preset placeholder codes generated by model inference at other positions in the fusion encoding, improves the accuracy of encoding alignment of the input data of each modality in the multimodal model, and thus improves the accuracy of multimodal model inference.

[0030] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart of a multimodal model inference method provided in an embodiment of the present invention;

[0033] Figure 2 A flowchart of a multimodal model training method provided in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the structure of a multimodal model inference device provided in an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the structure of a multimodal model training device provided in an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] Example 1

[0040] Figure 1 This is a flowchart of a multimodal model inference method provided in Embodiment 1 of the present invention. This embodiment is applicable to inference based on a multimodal model. The method can be executed by a multimodal model inference device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0041] S110. Obtain input data of at least two modalities, and use the input data of any one of the at least two modalities as reference data to determine the reference code corresponding to the reference data.

[0042] In this embodiment of the invention, in response to a multimodal model inference event being triggered, input data of at least two modalities are acquired. For example, when a multimodal model inference request is detected, it is determined that a multimodal model inference event has been triggered, whereby the multimodal model inference request can be understood as a request to perform inference through a multimodal model. For instance, the multimodal model can be a model used to infer the types of content the user is interested in (such as image types, music types, short video types, etc.). It should be noted that this embodiment of the invention does not limit the application scenarios of the multimodal model. Optionally, the input data of at least two modalities may include text modal input data (i.e., text data), audio modal input data (i.e., audio data), video modal input data (i.e., video data), and image modal input data (i.e., image data), etc. It should be noted that this embodiment of the invention does not limit the modal types or number of input data.

[0043] In this embodiment of the invention, the reference code corresponding to the reference data is determined by using input data from any one of at least two modalities as reference data. For example, the input data from at least two modalities includes text modal input data, audio modal input data, and image modal input data. Therefore, the reference data can be text modal input data, audio modal input data, or image modal input data. The reference data is input into the corresponding encoder, and the reference code corresponding to the reference data is obtained through the encoder. Optionally, the reference data is text data. Text data has a relatively small data volume, low compression rate, fast encoding speed, and high inference efficiency for multimodal models. Other modal data, however, require highly compressed encoding. For example, for 44.1kHz audio, there are 44,100 sampling points per second. Each 441 sampling points of audio information is compressed to generate one audio code, resulting in 100 codes per second of audio. Similarly, image data and video data also require highly compressed encoding, leading to slow encoding generation speeds. Therefore, using text data as the baseline data can effectively improve the efficiency of encoding generation, thereby further improving the efficiency of reasoning based on multimodal models.

[0044] S120. Take the input data of at least two modalities other than the reference data as the data to be aligned, determine the first number of codes generated by encoding each type of data to be aligned, and determine the modal code corresponding to the data to be aligned based on the first number of codes; wherein, the modal code includes a preset start code, a preset end code, and the first number of preset placeholder codes located between the preset start code and the preset end code.

[0045] In this embodiment of the invention, input data from at least two modalities, excluding the reference data, are used as data to be aligned. For example, the input data from at least two modalities includes text modal input data, audio modal input data, and image modal input data. If the text modal input data is used as the reference data, then both the audio modal input data and the image modal input data are data to be aligned. For each type of data to be aligned, the first number of codes generated by encoding the data to be aligned is determined based on the size of the data to be aligned. For example, if the data to be aligned is a 128*128 image, and one code is generated for every 16*16 pixels, then the first number of codes generated by encoding the data to be aligned is 64 if there is no repetition in the encoding. Another example is 44.1kHz audio, where the audio has 44,100 sampling points per second. If one audio code is generated by compressing every 441 sampling points of audio information, then the first number of codes generated by encoding one second of audio is 100 if there is no repetition in the encoding.

[0046] In this embodiment of the invention, the modal encoding corresponding to the data to be aligned is determined based on a first encoding quantity M. The modal encoding includes a preset start encoding, a preset end encoding, and a first number of preset placeholder encodings located between the preset start encoding and the preset end encoding. It can be understood that for each type of data to be aligned, a preset start encoding is added at the start position of the data to be aligned, and a preset end encoding is added at the end position. Based on the first encoding quantity generated by encoding each type of data to be aligned, the data to be aligned is converted into the first encoding quantity M of preset placeholder encodings, thereby generating the modal encoding corresponding to the data to be aligned. For example, if the data to be aligned is an image, the preset placeholder encoding can be...<img_1> ,<img_feat> Special encodings, etc. It should be noted that the preset start encoding, preset end encoding, and preset placeholder encoding in the modal encoding are different for different types of data to be aligned.

[0047] S130. Fuse each modal encoding with the baseline encoding to generate a fused encoding, and input the fused encoding into a pre-trained multimodal model to obtain the inference result corresponding to the fused encoding output by the multimodal model; wherein, in the process of obtaining the inference result based on the multimodal model, the actual modal encoding corresponding to the data to be aligned generated by the multimodal encoder in the multimodal model replaces the preset placeholder encoding between the preset start encoding and the preset end encoding in the fused encoding.

[0048] In this embodiment of the invention, the modal codes corresponding to each type of data to be aligned are fused with the baseline codes based on a preset encoding fusion strategy to generate fused codes. These fused codes are then input into a pre-trained multimodal model, which analyzes the fused codes to obtain the inference results corresponding to the fused codes output by the multimodal model. During the process of obtaining inference results based on the multimodal model, the multimodal encoder in the multimodal model continuously generates actual modal codes corresponding to the data to be aligned. Since the multimodal model is an autoregressive model, its characteristics dictate that all previous input codes and model-generated codes will be used as input codes in subsequent model inference steps for model inference. Therefore, in this embodiment of the invention, the actual modal codes corresponding to the data to be aligned generated by the multimodal encoder replace the preset placeholder codes between the preset start and preset end codes in the fused codes. This avoids the influence of preset placeholder codes generated by model inference in other positions of the fused codes, improving the accuracy of the replacement of the preset placeholder codes. For example, if the data to be aligned includes image data and audio data, then the image modal encoding (i.e., actual modal encoding) corresponding to the image data generated by the image encoder in the multimodal model and the audio modal encoding (i.e., actual modal encoding) corresponding to the audio data generated by the audio encoder in the multimodal model are obtained. Then, the preset image placeholder encoding between the preset image start encoding and the preset image end encoding corresponding to the image data in the fusion encoding is replaced based on the image modal encoding, and the preset audio placeholder encoding between the preset audio start encoding and the preset audio end encoding corresponding to the audio data in the fusion encoding is replaced based on the audio modal encoding.

[0049] The multimodal model inference method of this invention acquires input data of at least two modalities, and uses any one of the modal input data of the at least two modalities as reference data to determine the reference code corresponding to the reference data; takes the other modal input data of the at least two modalities besides the reference data as data to be aligned, determines the first code quantity generated by encoding each type of data to be aligned, and determines the modal code corresponding to the data to be aligned based on the first code quantity; wherein, the modal code includes a preset start code, a preset end code, and the first code quantity of preset placeholder codes located between the preset start code and the preset end code; each modal code is fused with the reference code to generate a fused code, and the fused code is input into a pre-trained multimodal model to obtain the inference result corresponding to the fused code output by the multimodal model; wherein, in the process of obtaining the inference result based on the multimodal model, the preset placeholder code between the preset start code and the preset end code in the fused code is replaced by the actual modal code corresponding to the data to be aligned generated by the multimodal encoder in the multimodal model. The technical solution provided by this invention adds a preset start code, a preset placeholder code, and a preset end code corresponding to each type of data to be aligned in the fusion encoding. This allows the actual modal encoding corresponding to the data to be aligned, generated by the multimodal encoder in the multimodal model, to only replace the preset placeholder code between the preset start code and the preset end code in the fusion encoding during the process of obtaining inference results based on the multimodal model. This effectively avoids the influence of preset placeholder codes generated by model inference at other positions in the fusion encoding, improves the accuracy of encoding alignment of the input data of each modality in the multimodal model, and thus improves the accuracy of multimodal model inference.

[0050] In some embodiments, inputting the fusion encoding into a pre-trained multimodal model to obtain an inference result corresponding to the fusion encoding output by the multimodal model includes: inputting the fusion encoding into the pre-trained multimodal model to determine multiple first core intermediate results output by the multimodal model; feeding back the multiple first core intermediate results to the input of the multimodal model to determine multiple second core intermediate results output by the multimodal model; and obtaining an inference result corresponding to the fusion encoding output by the multimodal model based on the multiple second core intermediate results, or at least one first core intermediate result and at least one second core intermediate result.

[0051] In this embodiment of the invention, after the fused encoding is input into a pre-trained multimodal model, multiple first core intermediate results Logits are determined from the output of the multimodal model. The first core intermediate results Logits are the original values ​​of the last layer output during the inference process of the multimodal model. It can be understood that the first core intermediate result vector, composed of the first core intermediate results, is an unnormalized real number vector, and each value in this vector corresponds to a category. The values ​​of the multiple first core intermediate results Logits can be any real number, including positive, negative, and even very large or very small values, representing the "bias" of the multimodal model towards a certain category during inference. The larger the value of the first core intermediate results Logits, the stronger the "bias" of the multimodal model towards that category.

[0052] In this embodiment of the invention, multiple first core intermediate results are fed back to the input of the multimodal model, enabling the multimodal model to infer again based on the multiple first core intermediate results and the fusion encoding, thereby determining multiple second core intermediate results output by the multimodal model. Then, the inference result corresponding to the fusion encoding output by the multimodal model is obtained based on the multiple second core intermediate results. That is, the multimodal model performs subsequent calculations, outputs, and / or normalization processing based on the multiple second core intermediate results to obtain the corresponding inference result. Optionally, at least one first core intermediate result and at least one second core intermediate result are used to obtain the inference result corresponding to the fusion encoding output by the multimodal model. That is, the multimodal model performs subsequent calculations, outputs, and / or normalization processing based on at least one first core intermediate result and at least one second core intermediate result to obtain the corresponding inference result. The advantage of this setup is that it can effectively improve the accuracy of multimodal model inference and avoid re-inferring all input data from scratch. Only the first core intermediate results output in the latter half need to be re-inferred to obtain the inference result, avoiding waste of computing resources and improving inference efficiency.

[0053] Optionally, before feeding back the multiple first core intermediate results to the input of the multimodal model, the method further includes: aligning the multiple first core intermediate results with a reference code; feeding back the multiple first core intermediate results to the input of the multimodal model to determine the multiple second core intermediate results output by the multimodal model, including: feeding back the aligned reference code and the multiple first core intermediate results to the input of the multimodal model to determine the multiple second core intermediate results output by the multimodal model. For example, aligning the multiple first core intermediate results with a reference code and feeding back the aligned reference code and the multiple first core intermediate results to the input of the multimodal model allows the multimodal model to perform inference again based on the aligned reference code and the multiple first core intermediate results to obtain the multiple second core intermediate results output by the multimodal model. The advantage of this setup is that it effectively solves the problem of incorrect alignment and inaccurate inference caused by gradually replacing the preset placeholder code with the generated actual modality code during the first inference process of the multimodal model, thereby further improving the accuracy of multimodal model inference.

[0054] Optionally, before feeding back multiple first core intermediate results to the input of the multimodal model, the method further includes: determining a target code in the multiple first core intermediate results and deleting the target code from the multiple first core intermediate results; wherein the target code includes a start-end code and / or placeholder code generated by the multimodal model during the determination of the first core intermediate results; the start-end code includes the preset start code and the preset end code; feeding back multiple first core intermediate results to the input of the multimodal model and determining multiple second core intermediate results output by the multimodal model includes: feeding back the first core intermediate results after deleting the target code to the input of the multimodal model and determining multiple second core intermediate results output by the multimodal model.

[0055] In this embodiment of the invention, during the process of determining the first core intermediate result by the multimodal model, the multimodal encoder in the multimodal model continuously generates actual modal codes. These actual modal codes may incorrectly generate start-end codes and / or placeholder codes. Simultaneously, during the inference process of determining the first core intermediate result, the multimodal model may also incorrectly generate start-end codes and / or placeholder codes. The start-end codes include a preset start code and a preset end code, which are used as the target codes. It is determined whether the target code exists among the multiple first core intermediate results. If so, the target code is deleted from the multiple first core intermediate results, and the multiple first core intermediate results after deleting the target code are fed back to the input of the multimodal model. This allows the multimodal model to perform inference again based on the multiple first core intermediate results after deleting the target code, obtaining multiple second core intermediate results output by the multimodal model. The advantage of this setup is that it ensures that only meaningful codes are retained in the input data of the multimodal model, guaranteeing that the codes of each modality in the input data of the second inference can be accurately aligned. This avoids the situation where the multimodal model inference is not accurate due to the misalignment of modalities, thereby improving the inference accuracy of the multimodal model.

[0056] Optionally, deleting the target code from multiple first core intermediate results includes: determining a portion of the first core intermediate results from the multiple first core intermediate results, and deleting the target code from the portion of the first core intermediate results; wherein the portion of the first core intermediate results contains the target code; feeding back the first core intermediate results after deleting the target code to the input of the multimodal model, and determining the second core intermediate results output by the multimodal model includes: feeding back the portion of the first core intermediate results after deleting the target code to the input of the multimodal model, so that the multimodal model outputs multiple subsequent second core intermediate results based on the portion of the first core intermediate results after deleting the target code.

[0057] In this embodiment of the invention, not every first core intermediate result necessarily contains a target code. Therefore, a portion of the first core intermediate results containing the target code is determined from the multiple first core intermediate results, and the target code is deleted from this portion. The portion of the first core intermediate results after removing the target code is then fed back to the input of the multimodal model. This allows the multimodal model to perform inference again based on the portion of the first core intermediate results after removing the target code, obtaining the multimodal model output corresponding to the portion of the first core intermediate results and its subsequent multiple second core intermediate results. For example, the multiple first core intermediate results contain 26 codes, representing the meaning of "the user inputs an image, and the content of the image is the addition operation 1+1=?". If the 13th core intermediate result among the multiple first core intermediate results contains a target code, the target code in the portion of the first core intermediate results is deleted, and then fed back to the input of the multimodal model. This allows the multimodal model to re-infer the 13th and subsequent first core intermediate results based on the portion of the first core intermediate results after removing the target code, and uses the re-inferred core intermediate results as the second core intermediate results. The advantage of this setup is that, while ensuring that the actual modal coding errors generated during the second inference process do not replace the preset placeholder coding, it can further save computing resources and improve the inference efficiency of the multimodal model.

[0058] In some embodiments, before feeding back multiple first core intermediate results to the input of the multimodal model, the method further includes: determining the actual modal encoding generated by the multimodal encoder in the multimodal model during the generation of multiple first core intermediate results, and replacing the preset placeholder encoding in the fusion encoding based on the actual modal encoding; feeding back multiple first core intermediate results to the input of the multimodal model, and determining multiple second core intermediate results output by the multimodal model, including: feeding back multiple first core intermediate results to the input of the multimodal model so that the multimodal model determines the second core intermediate results output by the multimodal model based on multiple first core intermediate results and the fusion encoding after the preset placeholder encoding is replaced by the actual modal encoding.

[0059] In this embodiment of the invention, the actual modal encoding generated by the multimodal encoder in the multimodal model during the generation of multiple first core intermediate results is determined. The actual modal encoding may include modal encodings generated by multimodal encoders such as image encoders, audio encoders, and video encoders in the multimodal model. The preset placeholder encoding in the fusion encoding is replaced based on the actual modal encoding, and the fusion encoding after the preset placeholder encoding is replaced by the actual modal encoding and the multiple first core intermediate results are fed back to the input end of the multimodal model. This allows the multimodal model to perform inference again based on the multiple first core intermediate results and the fusion encoding after the preset placeholder encoding is replaced by the actual modal encoding, and determine the second core intermediate result output by the multimodal model.

[0060] Figure 2 This is a flowchart illustrating a multimodal model training method provided in an embodiment of the present invention. This embodiment is applicable to multimodal model training and can be executed by a multimodal model training device. The multimodal model training device can be implemented in hardware and / or software and can be configured in an electronic device. Figure 2 As shown, the method includes:

[0061] S210. Obtain at least two multimodal sample data groups; wherein each multimodal sample data group contains sample data of at least two modalities.

[0062] In this embodiment of the invention, at least two multimodal sample data sets are included, wherein each multimodal sample data set contains sample data of at least two modalities. For example, the multimodal sample data set may contain sample data of different modalities such as text modality, audio modality, video modality, and image modality. It should be noted that the modal types of the sample data contained in each multimodal sample data set may be the same or different, and this embodiment of the invention does not limit the modal types of the sample data contained in each multimodal sample data set.

[0063] S220. For each of the multimodal sample data groups, using the sample data of any modality in the multimodal sample data group as reference data, determine the reference code corresponding to the reference data.

[0064] In this embodiment of the invention, for each multimodal sample data set, a reference code corresponding to the reference data is determined using sample data of any modality within the multimodal sample data set as reference data. For example, if the multimodal sample data set includes sample data of text modality, audio modality, image modality, and video modality, then the sample data of text modality, audio modality, image modality, or video modality can be used as the reference data. The reference data is input into the encoder, and the encoder obtains the reference code corresponding to the reference data. It should be noted that for different multimodal sample data sets, sample data of different modalities within the multimodal sample data set can be used as reference data; that is, the modality types corresponding to the reference data in each multimodal sample data set can be the same or different.

[0065] S230. Take the sample data other than the reference data in the multimodal sample data group as the data to be processed, determine the second code number generated by encoding the data to be processed for each modality, and determine the sample code corresponding to the data to be processed based on the second code number; wherein, the sample code includes a preset start code, a preset end code, and a preset number of preset placeholder codes located between the preset start code and the preset end code.

[0066] In this embodiment of the invention, for each multimodal sample data group, the sample data of other modalities in the multimodal sample data group, excluding the reference data, are taken as data to be processed. For example, a multimodal sample data group includes sample data of text modality, sample data of audio modality, and sample data of image modality. If the sample data of text modality is taken as the reference data, then the sample data of audio modality and the sample data of image modality are both data to be processed. For each type of data to be processed, the second number of encodings generated by encoding the data to be processed is determined according to the size of the data to be processed. For example, if the data to be processed is a 128*128 image, and each 16*16 pixels corresponds to one code, then without repeated encoding, the number of second codes generated by encoding the data to be processed is 64. For another example, if the data to be processed is 44.1KHz audio, where the audio has 44,100 sampling points per second, and if each 441 sampling points of audio information is compressed to generate one audio code, then without repeated encoding, the number of first codes generated by encoding one second of audio is 100.

[0067] In this embodiment of the invention, a sample code corresponding to the data to be processed is determined based on a second encoding quantity N. The sample code includes a preset start code, a preset end code, and a second encoding quantity of preset placeholder codes located between the preset start code and the preset end code. It can be understood that for each type of data to be processed, a preset start code is added at the start position of the data to be processed, and a preset end code is added at the end position. Based on the second encoding quantity generated for each type of data to be processed, the data to be processed is converted into the second encoding quantity N of preset placeholder codes, thereby generating a sample code corresponding to the data to be processed. For example, if the data to be processed is an image, the preset placeholder codes can be:<img_1> ,<img_feat> Special encodings are used. It should be noted that the preset start code, preset end code, and preset placeholder code in the sample encoding are different for different types of data to be processed.

[0068] S240. The sample code of each modality involved in each multimodal sample data group is fused with the reference code to generate a data group code.

[0069] In this embodiment of the invention, for each multimodal sample data group, the sample code corresponding to each type of data to be processed in the multimodal sample data group is fused with the reference code based on a preset encoding fusion strategy to generate a data group code. It is understood that at least two data group codes can be generated in the above manner, wherein the number of data group codes is the same as the number of multimodal sample data groups.

[0070] S250. A preset large language model is trained based on at least two data group codes to generate a multimodal model; wherein, during the training of the preset large language model based on at least two data group codes, the preset placeholder code between the preset start code and the preset end code in the data group codes is replaced by the actual sample code corresponding to the data to be processed generated by the multimodal encoder in the large language model.

[0071] In this embodiment of the invention, at least two data set codes are input into a preset large language model to train the model based on these codes until a preset convergence condition is met, generating a multimodal model. During the training process, the multimodal encoder within the large language model continuously generates actual sample codes corresponding to the data to be processed. Since the large language model is an autoregressive model, its characteristics dictate that all previous input codes and model inference-generated codes will be used as input codes in subsequent model training steps. Therefore, in this embodiment, the actual sample codes corresponding to the data to be processed generated by the multimodal encoder in the large language model replace the preset placeholder codes between the preset start and preset end codes in the data set codes. This avoids the influence of preset placeholder codes generated during model training in other positions within the data set codes, improving the accuracy of the placeholder code replacement. For example, if the data to be processed includes image data and audio data, then the image modal encoding (i.e., the actual sample encoding) corresponding to the image data generated by the image encoder in the large language model and the audio modal encoding (i.e., the actual sample encoding) corresponding to the audio data generated by the audio encoder in the multimodal model are obtained. Then, based on the image modal encoding, the preset image placeholder encoding between the preset image start encoding and the preset image end encoding corresponding to the image data in the data group encoding is replaced, and based on the audio modal encoding, the preset audio placeholder encoding between the preset audio start encoding and the preset audio end encoding corresponding to the audio data in the data group encoding is replaced.

[0072] The multimodal model training method of this invention acquires at least two multimodal sample data sets; wherein each multimodal sample data set contains sample data of at least two modalities; for each multimodal sample data set, using sample data of any modality in the multimodal sample data set as reference data, a reference code corresponding to the reference data is determined; other sample data in the multimodal sample data set besides the reference data are used as data to be processed, and a second code quantity generated by encoding the data to be processed for each modality is determined, and a sample code corresponding to the data to be processed is determined based on the second code quantity; wherein the sample code includes a preset starting code and a preset... The system includes a termination code and a number of preset placeholder codes located between the preset start code and the preset termination code; the sample codes of each modality involved in each multimodal sample data group are fused with the reference code to generate a data group code; a preset large language model is trained based on at least two data group codes to generate a multimodal model; wherein, during the training of the preset large language model based on at least two data group codes, the preset placeholder codes between the preset start code and the preset termination code in the data group code are replaced by the actual sample codes corresponding to the data to be processed generated by the multimodal encoder in the large language model. The technical solution provided by this invention adds a preset start code and a preset end code corresponding to each type of data to be processed to the data group encoding. During the training of a large language model, the actual sample encoding corresponding to the data to be processed generated by the multimodal encoder in the large language model only replaces the preset placeholder encoding between the preset start code and the preset end code in the data group encoding. This effectively avoids the influence of preset placeholder encodings generated by model training in other positions in the data group encoding, improving the accuracy of encoding alignment of sample data from each modality during multimodal model training. This, in turn, helps improve the accuracy of subsequent inference based on the multimodal model.

[0073] Figure 3 This is a schematic diagram of the structure of a multimodal model inference device provided in an embodiment of the present invention.

[0074] like Figure 3 As shown, the device includes:

[0075] The reference code determination module 310 is used to acquire input data of at least two modes, and determine the reference code corresponding to the reference data using any one of the input data of the at least two modes as reference data.

[0076] The modal encoding determination module 320 is used to take the input data of at least two modalities other than the reference data as the data to be aligned, determine the first number of codes generated by encoding each type of data to be aligned, and determine the modal encoding corresponding to the data to be aligned based on the first number of codes; wherein, the modal encoding includes a preset start code, a preset end code, and the first number of preset placeholder codes located between the preset start code and the preset end code;

[0077] The inference result generation module 330 is used to fuse each modal code with the baseline code to generate a fused code, and input the fused code into a pre-trained multimodal model to obtain the inference result corresponding to the fused code output by the multimodal model; wherein, in the process of obtaining the inference result based on the multimodal model, the actual modal code corresponding to the data to be aligned generated by the multimodal encoder in the multimodal model replaces the preset placeholder code between the preset start code and the preset end code in the fused code.

[0078] Optional, the inference result generation module includes:

[0079] The first core intermediate result determination unit is used to input the fusion encoding into the pre-trained multimodal model and determine multiple first core intermediate results output by the multimodal model.

[0080] The second core intermediate result determination unit is used to feed back multiple first core intermediate results to the input end of the multimodal model and determine multiple second core intermediate results output by the multimodal model.

[0081] The reasoning result acquisition unit is used to acquire the reasoning result corresponding to the fusion encoding output by the multimodal model based on the plurality of second core intermediate results, or at least one first core intermediate result and at least one second core intermediate result.

[0082] Optional, also includes:

[0083] The encoding alignment unit is used to align the multiple first core intermediate results with the reference encoding before feeding them back to the input of the multimodal model.

[0084] The second core intermediate result determination unit is used for:

[0085] The aligned baseline encoding and multiple first core intermediate results are fed back to the input of the multimodal model to determine multiple second core intermediate results output by the multimodal model.

[0086] Optional, also includes:

[0087] A target encoding deletion unit is configured to determine a target encoding in the multiple first core intermediate results and delete the target encoding from the multiple first core intermediate results before feeding back the multiple first core intermediate results to the input of the multimodal model; wherein the target encoding includes start-end encoding and / or placeholder encoding generated by the multimodal model during the determination of the first core intermediate results; the start-end encoding includes the preset start encoding and the preset end encoding;

[0088] The second core intermediate result determination unit includes:

[0089] The second core intermediate result determination subunit is used to feed back the first core intermediate result after deleting the target encoding to the input of the multimodal model, and determine the multiple second core intermediate results output by the multimodal model.

[0090] Optionally, the target encoding deletion unit is used for:

[0091] A subset of first core intermediate results is determined from a plurality of first core intermediate results, and the target code is deleted from the subset of first core intermediate results; wherein the subset of first core intermediate results contains the target code;

[0092] The second core intermediate result determines the subunit, used for:

[0093] The first core intermediate result after deleting the target encoding is fed back to the input of the multimodal model, so that the multimodal model outputs a number of subsequent second core intermediate results based on the first core intermediate result after deleting the target encoding.

[0094] Optional, also includes:

[0095] A preset placeholder code replacement unit is used to determine the actual modal code generated by the multimodal encoder in the multimodal model during the generation of the multiple first core intermediate results before feeding back multiple first core intermediate results to the input end of the multimodal model, and replace the preset placeholder code in the fusion code based on the actual modal code.

[0096] The second core intermediate result determination unit is used for:

[0097] Multiple first core intermediate results are fed back to the input of the multimodal model, so that the multimodal model determines the second core intermediate result output by the multimodal model based on the multiple first core intermediate results and the fusion code after the preset placeholder code is replaced by the actual modal code.

[0098] Optionally, the reference data is text data.

[0099] The multimodal model inference device provided in the embodiments of the present invention can execute the multimodal model inference method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0100] Figure 4 This is a schematic diagram of the structure of a multimodal model inference device provided in an embodiment of the present invention.

[0101] like Figure 4 As shown, the device includes:

[0102] The multimodal sample data set acquisition module 410 is used to acquire at least two multimodal sample data sets; wherein each multimodal sample data set contains sample data of at least two modalities;

[0103] The reference encoding determination module 420 is used to determine the reference encoding corresponding to the reference data for each multimodal sample data group, using the sample data of any modality in the multimodal sample data group as reference data.

[0104] The sample encoding determination module 430 is used to take the sample data other than the reference data in the multimodal sample data group as the data to be processed, determine the second encoding quantity generated by encoding the data to be processed for each modality, and determine the sample encoding corresponding to the data to be processed based on the second encoding quantity; wherein, the sample encoding includes a preset start encoding, a preset end encoding, and a preset number of preset placeholder encodings located between the preset start encoding and the preset end encoding;

[0105] The data group encoding generation module 440 is used to fuse the sample encoding of each modality involved in each of the multimodal sample data groups with the reference encoding to generate data group encoding;

[0106] A multimodal model generation module 450 is used to train a preset large language model based on at least two data group codes to generate a multimodal model; wherein, during the training of the preset large language model based on at least two data group codes, the preset placeholder code between the preset start code and the preset end code in the data group codes is replaced by the actual sample code corresponding to the data to be processed generated by the multimodal encoder in the large language model.

[0107] The multimodal model training device provided in this embodiment of the invention can execute the multimodal model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0108] Figure 5 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0109] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0110] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0111] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as multimodal model inference methods or multimodal model training methods.

[0112] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0113] In some embodiments, the multimodal model inference method or multimodal model training method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multimodal model inference method or multimodal model training method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the multimodal model inference method or multimodal model training method by any other suitable means (e.g., by means of firmware).

[0114] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0119] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0120] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multimodal model inference method, characterized in that, include: In response to a multimodal model inference request, input data of at least two modalities is acquired, wherein the input data of at least two modalities includes text data, as well as image data and / or audio data; And using text data from at least two modal input data as reference data, determine the reference code corresponding to the reference data; At least two modal input data, excluding the reference data, are used as data to be aligned. A first number of codes is determined for each type of data to be aligned, and a modal code corresponding to the data to be aligned is determined based on the first number of codes. Specifically, when the data to be aligned is image data, the first number of codes is determined based on the pixel size of the image data, with one code generated for every preset number of pixels. When the data to be aligned is audio data, the first number of codes is determined based on the number of sampling points of the audio data, with one audio code generated for every preset number of sampling points of audio information. The modal code includes a preset start code, a preset end code, and a preset number of preset placeholder codes located between the preset start code and the preset end code. For different types of data to be aligned, the preset start code, preset end code, and preset placeholder codes in the corresponding modal code are different. Each modal encoding is fused with the baseline encoding to generate a fused encoding, and the fused encoding is input into a pre-trained multimodal model to obtain the inference result corresponding to the fused encoding output by the multimodal model; wherein, the multimodal model is an autoregressive model, and in the process of obtaining the inference result based on the multimodal model, both the fused encoding and the model inference generated encoding are used as input encodings for model inference, and the preset placeholder encoding between the preset start encoding and the preset end encoding in the fused encoding is replaced by the actual modal encoding corresponding to the data to be aligned generated by the multimodal encoder in the multimodal model.

2. The method according to claim 1, characterized in that, The fused encoding is input into a pre-trained multimodal model to obtain the inference result corresponding to the fused encoding output by the multimodal model, including: The fusion encoding is input into a pre-trained multimodal model to determine multiple first core intermediate results output by the multimodal model; Multiple first core intermediate results are fed back to the input of the multimodal model to determine multiple second core intermediate results output by the multimodal model; Based on the plurality of second core intermediate results, or at least one first core intermediate result and at least one second core intermediate result, the inference result corresponding to the fusion encoding output by the multimodal model is obtained.

3. The method according to claim 2, characterized in that, Before feeding back multiple first core intermediate results to the input of the multimodal model, the method further includes: Align multiple first core intermediate results with the baseline encoding; Multiple first core intermediate results are fed back to the input of the multimodal model to determine multiple second core intermediate results output by the multimodal model, including: The aligned baseline encoding and multiple first core intermediate results are fed back to the input of the multimodal model to determine multiple second core intermediate results output by the multimodal model.

4. The method according to claim 2, characterized in that, Before feeding back multiple first core intermediate results to the input of the multimodal model, the method further includes: A target code is determined from a plurality of first core intermediate results, and the target code is deleted from the plurality of first core intermediate results; wherein, the target code includes the start-end code and / or placeholder code generated by the multimodal model during the determination of the first core intermediate results; the start-end code includes the preset start code and the preset end code; Multiple first core intermediate results are fed back to the input of the multimodal model to determine multiple second core intermediate results output by the multimodal model, including: The first core intermediate result after deleting the target encoding is fed back to the input of the multimodal model to determine the multiple second core intermediate results output by the multimodal model.

5. The method according to claim 4, characterized in that, Removing the target encoding from multiple first core intermediate results includes: A subset of first core intermediate results is determined from a plurality of first core intermediate results, and the target code is deleted from the subset of first core intermediate results; wherein the subset of first core intermediate results contains the target code; The first core intermediate result after deleting the target encoding is fed back to the input of the multimodal model to determine the second core intermediate result output by the multimodal model, including: The first core intermediate result after deleting the target encoding is fed back to the input of the multimodal model, so that the multimodal model outputs a number of subsequent second core intermediate results based on the first core intermediate result after deleting the target encoding.

6. The method according to any one of claims 2-5, characterized in that, Before feeding back multiple first core intermediate results to the input of the multimodal model, the method further includes: The actual modal encoding generated by the multimodal encoder in the multimodal model during the generation of multiple first core intermediate results is determined, and the preset placeholder encoding in the fusion encoding is replaced based on the actual modal encoding; Multiple first core intermediate results are fed back to the input of the multimodal model to determine multiple second core intermediate results output by the multimodal model, including: Multiple first core intermediate results are fed back to the input of the multimodal model, so that the multimodal model determines the second core intermediate result output by the multimodal model based on the multiple first core intermediate results and the fusion code after the preset placeholder code is replaced by the actual modal code.

7. A multimodal model training method, characterized in that, include: Acquire at least two sets of multimodal sample data; wherein each set of multimodal sample data contains sample data of at least two modalities; wherein the sample data of at least two modalities includes text data, as well as image data and / or audio data; For each of the multimodal sample data groups, the reference code corresponding to the reference data is determined using the text data in the multimodal sample data group as reference data. The sample data other than the reference data in the multimodal sample data group are taken as the data to be processed. The number of second codes generated by encoding the data to be processed for each modality is determined, and the sample code corresponding to the data to be processed is determined based on the number of second codes. Specifically, when the data to be processed is image data, the number of second codes is determined according to the pixel size of the image data, and one code is generated for every preset number of pixels. When the data to be processed is audio data, the number of second codes is determined according to the number of sampling points of the audio data, and one audio code is generated by compressing the audio information of every preset number of sampling points. The sample code includes a preset start code, a preset end code, and a number of preset placeholder codes of the second code located between the preset start code and the preset end code. For different types of data to be processed, the preset start code, preset end code, and preset placeholder codes in the corresponding modal codes are different. The sample code for each modality involved in each of the multimodal sample data groups is fused with the reference code to generate a data group code; A pre-defined large language model is trained based on at least two data group codes to generate a multimodal model. The pre-defined large language model is an autoregressive model. During the training process, both the data group codes and the model inference generation codes are used as input codes for model training. The pre-defined placeholder codes between the pre-defined start code and the pre-defined end code in the data group codes are replaced by the actual sample codes corresponding to the data to be processed generated by the multimodal encoder in the large language model.

8. A multimodal model inference device, characterized in that, include: A baseline encoding determination module is used to respond to a multimodal model inference request, acquire input data of at least two modalities, wherein the input data of at least two modalities includes text data, image data and / or audio data; and determine the baseline encoding corresponding to the baseline data using the text data in the input data of at least two modalities as baseline data. A modal encoding determination module is used to take input data of at least two modalities, excluding the reference data, as data to be aligned, determine the first number of codes generated by encoding each type of data to be aligned, and determine the modal encoding corresponding to the data to be aligned based on the first number of codes. Specifically, when the data to be aligned is image data, the first number of codes is determined according to the pixel size of the image data, with one code generated for every preset number of pixels; when the data to be aligned is audio data, the first number of codes is determined according to the number of sampling points of the audio data, with one audio code generated for every preset number of sampling points of audio information. The modal encoding includes a preset start code, a preset end code, and the first number of preset placeholder codes located between the preset start code and the preset end code. For different types of data to be aligned, the preset start code, preset end code, and preset placeholder codes in the corresponding modal encoding are different. The inference result generation module is used to fuse each modal encoding with the baseline encoding to generate a fused encoding, and input the fused encoding into a pre-trained multimodal model to obtain the inference result corresponding to the fused encoding output by the multimodal model; wherein, the multimodal model is an autoregressive model, and in the process of obtaining the inference result based on the multimodal model, both the fused encoding and the model inference generation encoding are used as input encodings to perform model inference, and the preset placeholder encoding between the preset start encoding and the preset end encoding in the fused encoding is replaced by the actual modal encoding corresponding to the data to be aligned generated by the multimodal encoder in the multimodal model.

9. A multimodal model training device, characterized in that, include: A multimodal sample data set acquisition module is used to acquire at least two multimodal sample data sets; wherein each multimodal sample data set contains sample data of at least two modalities; wherein the sample data of the at least two modalities includes text data, as well as image data and / or audio data; The reference encoding determination module is used to determine the reference encoding corresponding to the reference data for each multimodal sample data group, using the text data in the multimodal sample data group as reference data. The sample encoding determination module is used to take the sample data other than the reference data in the multimodal sample data group as the data to be processed, determine the number of second codes generated by encoding the data to be processed for each modality, and determine the sample code corresponding to the data to be processed based on the number of second codes; wherein, when the data to be processed is image data, the number of second codes is determined according to the pixel size of the image data, and one code is generated for every preset number of pixels; when the data to be processed is audio data, the number of second codes is determined according to the number of sampling points of the audio data, and one audio code is generated by compressing the audio information of every preset number of sampling points; the sample code includes a preset start code, a preset end code, and a number of preset placeholder codes of the second code located between the preset start code and the preset end code. For different types of data to be processed, the preset start code, preset end code, and preset placeholder codes in the corresponding modality encoding are different; The data group encoding generation module is used to fuse the sample encoding of each modality involved in each multimodal sample data group with the reference encoding to generate data group encoding; A multimodal model generation module is used to train a preset large language model based on at least two data group codes to generate a multimodal model. The preset large language model is an autoregressive model. During the training process based on at least two data group codes, both the data group codes and the model inference generation codes are used as input codes for model training. The preset placeholder code between the preset start code and the preset end code in the data group codes is replaced by the actual sample code corresponding to the data to be processed, generated by the multimodal encoder in the large language model.

Images