Text determination method, multi-modal large model training method and equipment

Abstract

The invention provides a text determination method based on a multi-modal large model, a multi-modal large model training method and equipment, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, natural language processing, large models and the like. According to the specific implementation scheme, a first lexical element sequence is determined through a coding network in a multi-modal large model, a target image and a task description text corresponding to the target image, the first lexical element sequence is input into a large language model in the multi-modal large model, and an output first lexical element feature sequence is obtained, and inputting the last lexical element feature in the first lexical element feature sequence into each decoding network in the multi-modal large model, obtaining a target lexical element output by each decoding network, and obtaining a target text according to the first lexical element sequence and the target lexical element.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the technical fields of computer vision, deep learning, natural language processing, large models, and in particular to a text determination method based on a multimodal large model, a multimodal large model training method, and a device. Background Art

[0002] With the development of artificial intelligence technology, especially the maturity of large model technology, Multimodal Large Language Model (MLLM) has become a research hotspot in recent years. Summary of the invention

[0003] The present disclosure provides a text determination method based on a multimodal large model, a multimodal large model training method and a device.

[0004] According to one aspect of the present disclosure, a text determination method based on a multimodal large model is provided, wherein the multimodal large model includes an encoding network, a large language model, and a plurality of decoding networks, and the method includes: obtaining a first word-gram sequence, wherein the first word-gram sequence is determined according to a target image, a task description text corresponding to the target image, and the encoding network; inputting the first word-gram sequence into the large language model to obtain an output first word-gram feature sequence; inputting the last word-gram feature in the first word-gram feature sequence into each decoding network respectively to obtain a target word-gram output by each decoding network; and determining a target text according to the first word-gram sequence and each of the target words.

[0005] According to another aspect of the present disclosure, a training method for a multimodal large model is provided, the method comprising: acquiring training data and an initial multimodal large model; the training data comprising: a sample image, a sample task description text corresponding to the sample image, and a sample text; the multimodal large model comprising an encoding network, a large language model, and a plurality of decoding networks connected in sequence; inputting the sample image and the sample task description text into the encoding network and the large language model connected in sequence, acquiring a word unit sequence output by the encoding network and a word unit feature sequence output by the large language model; inputting the last word unit feature in the word unit feature sequence into each decoding network, respectively, acquiring a predicted word unit output by each decoding network; updating the word unit sequence according to the predicted word unit until the obtained predicted word unit includes an end word unit; determining the obtained predicted text based on the predicted word unit in the updated word unit sequence and the sample text, training the multimodal large model, and obtaining a trained multimodal large model.

[0006] According to another aspect of the present disclosure, a text determination device based on a multimodal large model is provided, wherein the multimodal large model includes an encoding network, a large language model and multiple decoding networks, and the device includes: a first acquisition module, used to acquire a first word-gram sequence, wherein the first word-gram sequence is determined according to a target image, a task description text corresponding to the target image, and the encoding network; a second acquisition module, used to input the first word-gram sequence into the large language model to acquire an output first word-gram feature sequence; a third acquisition module, used to input the last word-gram feature in the first word-gram feature sequence into each decoding network respectively to acquire a target word-gram output by each decoding network; and a first determination module, used to determine a target text according to the first word-gram sequence and each of the target words.

[0007] According to another aspect of the present disclosure, a training device for a multimodal large model is provided, the device comprising: a seventh acquisition module, used to acquire training data and an initial multimodal large model; the training data comprises: a sample image, a sample task description text corresponding to the sample image, and a sample text; the multimodal large model comprises an encoding network, a large language model, and a plurality of decoding networks connected in sequence; an eighth acquisition module, used to input the sample image and the sample task description text into the encoding network and the large language model connected in sequence, and acquire a word unit sequence output by the encoding network and a word unit feature sequence output by the large language model; a ninth acquisition module, used to input the last word unit feature in the word unit feature sequence into each decoding network, and acquire a predicted word unit output by each decoding network; an updating module, used to update the word unit sequence according to the predicted word unit, until the acquired predicted word unit includes an end word unit; a training module, used to determine the predicted text obtained based on the predicted word unit in the updated word unit sequence and the sample text, and train the multimodal large model to obtain a trained multimodal large model.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the text determination method based on the multimodal large model proposed above in the present disclosure; or, execute the training method of the multimodal large model proposed above in the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the text determination method based on the multimodal large model proposed above in the present disclosure; or, to execute the training method of the multimodal large model proposed above in the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the text determination method based on the multimodal large model proposed above in the present disclosure; or, implements the steps of the training method of the multimodal large model proposed above in the present disclosure.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0014] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0018] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0019] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0020] Figure 8 It is a block diagram of an electronic device used to implement the text determination method based on a multimodal large model or the training method of a multimodal large model of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] With the development of artificial intelligence technology, especially the maturity of large model technology, Multimodal Large Language Model (MLLM) has become a research hotspot in recent years.

[0023] After obtaining the user's task description text for an image, it is very important to combine the multimodal large model to process the image and task description text and quickly obtain the corresponding text.

[0024] In response to the above problems, the present disclosure proposes a text determination method based on a multimodal large model, a multimodal large model training method and a device.

[0025] Figure 1 It is a schematic diagram according to the first embodiment of the present disclosure. It should be noted that the text determination method based on the multimodal large model of the embodiment of the present disclosure can be applied to a text determination device based on the multimodal large model. The device can be an electronic device, or can be configured in an electronic device so that the electronic device can perform a text determination function.

[0026] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0027] The text determination device based on the multimodal large model may also be software in an electronic device, such as interactive dialogue software, etc. It should be noted that in the following embodiments, the execution subject is an electronic device as an example for description.

[0028] It should be noted that the multimodal large model in this embodiment includes an encoding network, a large language model and multiple decoding networks.

[0029] In some embodiments, multiple decoding networks are used to decode a plurality of content in consecutive positions based on the same input.

[0030] For example, after the word-gram feature of a word-gram is input into a plurality of decoding networks, the plurality of decoding networks may output a plurality of word-grams that are located consecutively after the word-gram.

[0031] like Figure 1 As shown, the text determination method based on the multimodal large model may include the following steps:

[0032] Step 101, obtaining a first word-gram sequence, where the first word-gram sequence is determined based on a target image, a task description text corresponding to the target image, and an encoding network.

[0033] The target image can be any image input by the user.

[0034] The task description text refers to the description text written in order to complete the processing task of the target image.

[0035] Among them, the processing tasks may include but are not limited to image interpretation tasks, creative text generation tasks based on the image content of the target image, image recognition tasks, image retrieval tasks, image editing tasks, and image analysis tasks.

[0036] Among them, the image interpretation task refers to the task of interpreting objects, scenes, actions, etc. in the target image.

[0037] Among them, the creative text generation task refers to the task of generating stories, poems, etc. related to the target image.

[0038] The image recognition task refers to the task of identifying a specific object or person in the target image.

[0039] Among them, the image retrieval task refers to the task of finding similar images based on the image content of the target image.

[0040] The image editing task refers to the task of editing the target image by cropping, resizing, adding filters, etc.

[0041] The image analysis task refers to the task of analyzing the emotion, color distribution, etc. in the target image.

[0042] In some embodiments, the first word-gram sequence includes a reference word-gram subsequence, wherein the reference word-gram subsequence is a word-gram subsequence output by the encoding network after the target image and the task description text are input into the encoding network.

[0043] As an example, when the encoding network includes: a visual encoder and a text encoder, a possible implementation method for obtaining a reference word-meta sequence may be: inputting a target image into the visual encoder to obtain an output image word-meta sequence; inputting a task description text into the text encoder to obtain an output text word-meta sequence; and concatenating the image word-meta sequence and the text word-meta sequence to obtain a first word-meta sequence.

[0044] In one embodiment of the present disclosure, a target image is input into a visual encoder, and a process of obtaining an output image word sequence may be, for example, as follows: the target image is input into an encoding layer of the visual encoder to obtain an image block sequence of the target image, and the image block sequence is input into a graph-text mapping layer of the visual encoder to map the image block sequence into a text space through the graph-text mapping layer to obtain an image word sequence.

[0045] Among them, image word-unit refers to the word-unit mapped from the image block to the text space.

[0046] As an example, the text word sequence may be concatenated at the end of the image word sequence to obtain a reference word subsequence.

[0047] As another example, the image word-gram sequence may be concatenated to the end of the text word-gram sequence to obtain a reference word-gram subsequence.

[0048] In this embodiment, the target image and the task description text are processed respectively by the visual encoder and the text encoder, which can better retain and extract the features of each modality. The visual encoder in this embodiment can output image words, map the features in the image space to the text space, and align the features of the target image and the task description text in the text space, so as to facilitate the subsequent large language model to accurately understand the semantics.

[0049] In some embodiments, when the first word-gram sequence is input into the large language model for the first time, the first word-gram sequence may include a starting word-gram in addition to the reference word-gram subsequence.

[0050] As an example, the starting word may be spliced ​​onto the end of the reference word subsequence to obtain a first word sequence.

[0051] Among them, the starting word, which can also be called the starting symbol word, is a special marker, which provides the starting point of the generation process for the multimodal large model. The multimodal large model starts the generation process based on the marker and gradually generates subsequent words.

[0052] In some other embodiments, the first word-gram sequence includes a reference word-gram subsequence and a predicted word-gram subsequence.

[0053] The predicted word-meta subsequence is a word-meta subsequence predicted based on a reference word-meta subsequence, a large language model, and multiple decoding networks.

[0054] Step 102: input the first word-gram sequence into the large language model to obtain an output first word-gram feature sequence.

[0055] In some embodiments, the large language model is a large language model based on a causal attention mask mechanism.

[0056] Among them, the large language model based on the self-attention mask mechanism can ensure that the word feature at the t-th position in the first word feature sequence output by the large language model can include the semantic features of the t-th word in the second word sequence, and also include the semantic features of each word before the t-th word in the second word sequence, thereby facilitating the subsequent accurate determination of other words after the t-th word based on the word feature at the t-th position.

[0057] It should be noted that the first word-gram feature sequence includes word-gram features corresponding to each word-gram in the first word-gram sequence.

[0058] In some embodiments, the first word-gram feature sequence corresponds to the first word-gram sequence one-to-one, that is, each word-gram feature in the first word-gram feature sequence corresponds to the word-gram at the same position in the first word-gram sequence. In other words, the word-gram feature at the t-th position in the first word-gram feature sequence is the word-gram feature of the t-th word-gram in the first word-gram sequence, where t ranges from 1 to N, and N represents the number of words in the first word-gram sequence.

[0059] In some embodiments, the word-gram feature at the t-th position in the first word-gram feature sequence includes both the semantic features of the t-th word-gram in the second word-gram sequence and the semantic features of all the words before it.

[0060] Step 103: input the last word-unit feature in the first word-unit feature sequence into each decoding network respectively, and obtain the word-unit output by each decoding network.

[0061] The last word-unit feature in the first word-unit feature sequence includes the semantic features of each word-unit in the first word-unit sequence.

[0062] Among them, each decoding network in this embodiment is based on the same word list, and decodes the last word-unit feature in the first word-unit feature sequence to obtain the corresponding word-unit.

[0063] It should be noted that the network parameters corresponding to each of the multiple decoding networks in this embodiment are different.

[0064] It can be understood that the last word-gram feature in the first word-gram feature sequence corresponds to the last word-gram in the first word-gram sequence. In other words, the last word-gram feature is the word-gram feature of the last word-gram in the first word-gram sequence.

[0065] For example, the number of decoding networks H is 4. Correspondingly, after the word-gram features of the last word-gram A in the first word-gram sequence are respectively input into the four decoding networks, the four decoding networks respectively perform decoding based on the input content, and four word-grams located in a continuous position after the last word-gram A can be obtained. Assume that the four decoding networks are represented by decoding network 1, decoding network 2, decoding network 3 and decoding network 4 respectively. Correspondingly, decoding network 1 can decode the word-gram features of the last word-gram A input according to the vocabulary to determine the first word-gram after the last word-gram A. Correspondingly, decoding network 2 can decode the word-gram features of the last word-gram A input according to the vocabulary to determine the second word-gram after the last word-gram A. Correspondingly, decoding network 3 can decode the word-gram features of the last word-gram A input according to the vocabulary to determine the third word-gram after the last word-gram A. Correspondingly, decoding network 4 can decode the word-gram features of the input starting word-gram according to the vocabulary to determine the fourth word-gram after the last word-gram A.

[0066] Step 104: Determine a target text according to the first word-gram sequence and each target word-gram.

[0067] In this embodiment, when there is an end word in the target word, it is determined whether the first word sequence includes a predicted word subsequence. When the first word sequence includes a predicted word subsequence, the target text is determined based on the predicted word subsequence and each target word. Thus, the target text is accurately obtained based on the target word and the predicted word subsequence.

[0068] It can be understood that the target text is the text obtained by the multimodal large model based on the target image and the task description text.

[0069] In some embodiments, each target word-gram may be spliced ​​according to the position order of each target word-gram to obtain a word-gram splicing result, and the word-gram splicing result may be spliced ​​at the end of the predicted word-gram subsequence to obtain the target text.

[0070] In some other embodiments, when the first word-gram sequence does not include the predicted word-gram subsequence, the target text is determined according to each target word-gram, thereby accurately determining the target text according to the target word-gram.

[0071] In this embodiment, the target word units may be concatenated according to the position sequence of the target word units to obtain the target text.

[0072] It should be noted that, in this embodiment, a decoding network outputting a word is used as an example for illustration.

[0073] Among them, the end word, also known as the end symbol word, is a special marker that provides the termination point of the generation process for the multimodal large model. After the end word appears, the multimodal large model no longer generates new words.

[0074] The text determination method based on the multimodal large model of the disclosed embodiment determines the first word-unit sequence through the encoding network, the target image, and the task description text corresponding to the target image in the multimodal large model, and inputs the first word-unit sequence into the large language model in the multimodal large model to obtain the output first word-unit feature sequence, and inputs the last word-unit feature in the first word-unit feature sequence into each decoding network in the multimodal large model respectively, obtains the target word-unit output by each decoding network, and obtains the target text based on the first word-unit sequence and the target word-unit. Thus, by decoding the word-unit features output by the large language model in parallel through multiple decoding networks in the multimodal large model, the decoding process can be accelerated and the efficiency of obtaining the target text is improved.

[0075] Combine the following Figure 2 , the text determination method based on the multimodal large model provided by the present invention is further explained.

[0076] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure.

[0077] like Figure 2 As shown, the method may include:

[0078] Step 201, obtaining a first word-gram sequence, where the first word-gram sequence is determined based on a target image, a task description text corresponding to the target image, and an encoding network.

[0079] Step 202: Input the first word-gram sequence into the large language model to obtain an output first word-gram feature sequence.

[0080] Step 203: input the last word-unit feature in the first word-unit feature sequence into each decoding network respectively, and obtain the target word-unit output by each decoding network.

[0081] It should be noted that, for the specific description of steps 201 to 203, reference may be made to the relevant descriptions in other embodiments, which will not be repeated here.

[0082] Step 204: When there is no end word-gram in the target word-grams, each target word-gram is added as a predicted word-gram to the first word-gram sequence to obtain a second word-gram sequence.

[0083] In the disclosed embodiment, the process of the electronic device executing step 204 may be, for example: sorting each target word according to the position sequence of each target word to obtain a predicted word sequence; and splicing the predicted word sequence at the end of the first word sequence to obtain a second word sequence. In this way, the logical coherence of the target text generated subsequently can be ensured, and the quality of the target text obtained can be improved.

[0084] Step 205 , updating the first word-gram sequence according to the second word-gram sequence, so as to re-determine the target word-gram according to the updated first word-gram sequence, until an end word-gram exists in the determined target word-grams.

[0085] In this embodiment, the second word-gram sequence can be used as the first word-gram sequence, and the process jumps to inputting the first word-gram sequence into the large language model until the determined target word-gram has an end word-gram.

[0086] Step 206: Determine the target text according to the predicted word-unit subsequence and each target word-unit.

[0087] It should be noted that, for the specific description of step 206, reference may be made to the relevant descriptions in other embodiments, which will not be repeated here.

[0088] The text determination method based on the multimodal large model of the disclosed embodiment, when there is no end word in the target word, adds each target word as a predicted word to the first word sequence to obtain a second word sequence, and updates the first word sequence according to the second word sequence to re-determine the target word according to the updated first word sequence, until there is an end word in the determined target word, and determines the target text according to the predicted word subsequence and each target word. Thus, by performing multiple rounds of decoding through multiple parallel decoding networks, and according to the target word output by each decoding network in the multiple rounds of decoding, the target text can be accurately determined while improving the efficiency of obtaining the target text.

[0089] Figure 3 2 is a schematic diagram of a third embodiment of the present disclosure. This embodiment further optimizes or refines the above-mentioned embodiment.

[0090] like Figure 3 As shown, the method may include:

[0091] Step 301, obtaining a first word-gram sequence, where the first word-gram sequence is determined based on a target image, a task description text corresponding to the target image, and an encoding network.

[0092] Step 302: Input the first word-gram sequence into the large language model to obtain an output first word-gram feature sequence.

[0093] Step 303: Input the last word-unit feature in the first word-unit feature sequence into each decoding network to obtain the target word-unit output by each decoding network.

[0094] It should be noted that, for the specific description of steps 301 to 303, reference may be made to the relevant descriptions in other embodiments, which will not be repeated here.

[0095] Step 304: When there is no end word in the target word, sort the target word according to the position sequence of each target word to obtain a predicted word sequence.

[0096] Step 305 , concatenate the predicted word-gram sequence to the end of the first word-gram sequence to obtain a second word-gram sequence.

[0097] Step 306: Input the second word-gram sequence into the large language model to obtain an output second word-gram feature sequence.

[0098] Step 307 : extracting a word-unit feature subsequence from the Mth word-unit feature to the last word-unit feature from the second word-unit feature sequence, where M represents the number of word-units in the first word-unit sequence.

[0099] It should be noted that the second word-gram feature sequence corresponds one-to-one to the second word-gram sequence, that is, the i-th word-gram feature in the second word-gram feature sequence is the word-gram feature of the i-th word-gram in the second word-gram sequence, where i is greater than or equal to 1 and less than or equal to L, and L represents the number of words in the second word-gram sequence.

[0100] Step 308: Obtain a word-unit sequence to be compared of the word-unit feature subsequence according to a first decoding network among the multiple decoding networks.

[0101] In this embodiment, the word-unit features in the word-unit feature subsequence are sequentially input into the first decoding network to obtain a word-unit sequence to be compared.

[0102] Each word in the word-gram sequence to be compared is determined based on the word-gram features at the same position in the word-gram feature subsequence.

[0103] The first decoding network refers to a decoding network corresponding to the content at the first position among multiple contents with continuous output positions.

[0104] Step 309 , compare each word in the predicted word-gram sequence with the word at the same position in the word-gram sequence to be compared, and obtain valid word-grams in the predicted word-gram sequence.

[0105] In some embodiments, each word-gram in the predicted word-gram sequence may be compared with the word-gram at the same position in the word-gram sequence to be compared, and the valid word-gram in the predicted word-gram sequence may be determined based on the comparison result.

[0106] When the comparison result indicates that each word-gram in the predicted word-gram sequence is identical to the word-gram at the same position in the to-be-compared word-gram sequence, it can be determined that each word-gram in the predicted word-gram is a valid word-gram.

[0107] In other embodiments, a possible implementation method of comparing each word in the predicted word-unit sequence with the word at the same position in the word-unit sequence to be compared to obtain a valid word in the predicted word-unit sequence is: sequentially comparing each word in the predicted word-unit sequence with the word at the same position in the word-unit sequence to be compared; obtaining the first target word in the predicted word-unit sequence; the first target word is the first word in the predicted word-unit sequence that has not been successfully compared; and determining the word before the first target word in the predicted word-unit sequence as a valid word. Thus, the valid word in the predicted word-unit sequence is accurately determined, which can ensure the quality of the word-unit sequence input into the large language model again, ensure the logical coherence of the target text generated subsequently, and improve the quality of the generated target text.

[0108] In this embodiment, if a word-gram in the predicted word-gram sequence is different from a word-gram at the same position in the word-gram sequence to be compared, the word-gram is referred to as a word-gram that has not been successfully compared.

[0109] Step 310: Filter the ineffective word-grams in the second word-gram sequence.

[0110] In some embodiments, filtering the ineffective word-grams in the second word-gram sequence may be implemented by deleting the ineffective word-grams in the second word-gram sequence.

[0111] Step 311 , updating the first word-gram sequence according to the second word-gram sequence, so as to re-determine the target word-gram according to the updated first word-gram sequence, until an end word-gram exists in the determined target word-grams.

[0112] In some embodiments, before updating the first word sequence according to the second word sequence, it is also possible to: obtain the second target word at the position of the first target word in the predicted word sequence from the word sequence to be compared, and add the second target word to the end of the second word sequence. In this way, the decoding process can be further improved, and the efficiency of obtaining the target text can be improved.

[0113] The position of the second target word in the to-be-compared word sequence is the same as the position of the first target word in the predicted word sequence.

[0114] In this embodiment, the second word-gram sequence can be used as the first word-gram sequence, and the process jumps to inputting the first word-gram sequence into the large language model until the determined target word-gram has an end word-gram.

[0115] Step 312: Determine the target text according to the predicted word-unit subsequence and each target word-unit.

[0116] In this embodiment, when there is no end word in the target word, the target words are sorted according to the position order of each target word to obtain a predicted word sequence, and the predicted word sequence is spliced ​​at the end of the first word sequence to obtain a second word sequence, and the word prediction is performed again according to the large language model and the first decoding network of the multiple decoding networks to obtain a word sequence to be compared, and the predicted word sequence and the word sequence to be compared are compared, and the valid words in the predicted word sequence are determined according to the comparison result, and the invalid words in the second word sequence are filtered, and the first word sequence is updated according to the filtered second word sequence, so as to re-determine the target word according to the updated first word sequence. In this way, the quality of the word sequence input into the large language model again can be guaranteed, the logical coherence of the target text generated subsequently is ensured, and the quality of the generated target text is improved.

[0117] In order to clearly understand the present disclosure, Figure 4 The method of this embodiment is exemplified.

[0118] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure.

[0119] Among them, it should be noted that Figure 4 The structure of the multimodal large model is combined, and the method of this embodiment is exemplarily described by taking the multimodal large model including 4 decoding networks and the task description text input by the user as: Please introduce the animals in the image.

[0120] Among them, it should be noted that Figure 4 The four decoding networks are decoding network 1, decoding network 2, decoding network 3 and decoding network 4 as examples.

[0121] Specifically, the following steps may be included:

[0122] Step 401, input the target image into the visual encoder to obtain the output image word sequence, and input the task description text into the text encoder to obtain the output text word sequence, and concatenate the image word sequence and the text word sequence to obtain a reference word subsequence, and concatenate the starting word after the reference word subsequence to obtain a first word sequence, and input the first word sequence into the large language model.

[0123] Step 402: Obtain a first word-unit feature sequence output by the large language model, and input the last word-unit feature in the first word-unit feature sequence into each decoding network to obtain a target word-unit output by each decoding network.

[0124] The large language model in this embodiment is a large language model based on a causal attention mask mechanism.

[0125] It should be noted that, in this example, the word feature at the t-th position in the first word feature sequence includes both the semantic features of the t-th word in the first word sequence and the semantic features of all words before it.

[0126] It should be noted that the last word-gram feature in the first word-gram feature sequence is the word-gram feature corresponding to the last word-gram in the first word-gram sequence, and the last word-gram in the first word-gram sequence is the starting word-gram (i.e., word-gram No. 0). Therefore, when performing the first round of decoding, the word-gram feature of word-gram No. 0 is input into each decoding network.

[0127] Among them, the decoding network is a mutually independent linear layer (Linear), the number of input channels is equal to the dimensionality of the above semantic features, and the number of output channels is equal to the vocabulary size of the vocabulary used by the decoding network.

[0128] It should be noted that, in this embodiment, each decoding network is based on the same vocabulary to decode the last word feature in the first word feature sequence.

[0129] in, Figure 4 The t0 in represents the 0th word, i.e. the starting word.

[0130] Among them, the last word feature in the first word feature sequence is the word feature of word No. 0. After the word feature of word No. 0 is input into decoding network 1, decoding network 2, decoding network 3 and decoding network 4, correspondingly, decoding network 1 outputs word No. 1, decoding network 2 outputs word No. 2, decoding network 3 outputs word No. 3 and decoding network 4 outputs word No. 4.

[0131] Step 403, when it is determined that the target word-grams output by all decoding networks do not include the ending word-gram, the word-grams output by the four decoding networks can be verified by the verification module, and based on the verification result, the word-gram sequence input to the large language model in the next round of decoding is determined to perform the next round of decoding, until the word-gram output by at least one decoding network in the kth round of decoding is the ending word-gram.

[0132] Among them, the process of the verification module verifying the target word units output by the four decoding networks can be, for example: splicing t1, t2, t3, t4 at the end of the first word unit sequence and inputting it into the large language model to obtain a second word unit feature sequence, and extracting a word unit feature subsequence from the mth word unit feature (where m is determined according to the position of t0 in the first word unit sequence) to the last word unit feature from the second word unit feature sequence, wherein the word unit feature subsequence includes word unit features corresponding to t0, t1, t2, t3, t4 respectively, and inputting the word unit feature subsequence into the decoding network 1 to obtain a word unit sequence to be compared, wherein the word unit sequence to be compared includes the first word unit to be compared s1, the second word unit to be compared s2, the third word unit to be compared s3, the fourth word unit to be compared s4 and the fifth word unit to be compared s5.

[0133] Among them, it should be noted that the first word to be compared s1 is predicted by the decoding network 1 based on the word-meta feature corresponding to t0 in the word-meta feature subsequence. Correspondingly, the second word to be compared s2 is predicted by the decoding network 1 based on the word-meta feature corresponding to t1 in the word-meta feature subsequence. The third word to be compared s3 is predicted by the decoding network 1 based on the word-meta feature corresponding to t2 in the word-meta feature subsequence. The fourth word to be compared s4 is predicted by the decoding network 1 based on the word-meta feature corresponding to t3 in the word-meta feature subsequence. The fifth word to be compared s5 is predicted by the decoding network 1 based on the word-meta feature corresponding to t4 in the word-meta feature subsequence.

[0134] Correspondingly, s1 and t1, s2 and t2, s3 and t3, s4 and t4 can be compared one by one to see if they are the same. If s1 and t1, s2 and t2, s3 and t3, s4 and t4 are all the same, then the verification module can output t1, t2, t3, t4 and s5. Then, t1, t2, t3, t4 and s5 can be concatenated at the end of the first word-gram sequence to obtain a second word-gram sequence, and the second word-gram sequence is used as the updated first word-gram sequence, and in the second round of decoding, the updated first word-gram sequence is input into the large language model.

[0135] It should be noted that, in the second round of decoding, the second word-gram sequence is used as the updated first word-gram sequence, and the updated first word-gram sequence is input into the large language model, and then the last word-gram feature in the word-gram feature sequence output by the large language model (i.e., the word-gram feature corresponding to s5, which includes the semantic feature of s5 and the semantic features of all words before S5) is input into each decoding network. Correspondingly, in the second round of decoding, the target words output by the four decoding networks are t6, t7, t8 and t9 respectively. Correspondingly, the verification module determines that t6, t7, t8 and t9 have all passed the verification. At this time, the verification module outputs t6, t7, t8, t9 and s10. Then, a third round of decoding is performed. t6, t7, t8, t9 and S10 may be spliced ​​onto the back of the first word-gram sequence input into the large language model in the second round of decoding, and input into the large language model. Then, the last word-gram feature in the word-gram feature sequence output by the large language model (i.e., the word-gram feature corresponding to s10, which includes the semantic features of s10 and the semantic features of all words before s10) is input into each decoding network. Correspondingly, in the third round of decoding, the target word-grams output by the four decoding networks...until it is determined that the target word-gram output by at least one decoding network in the kth round of decoding is the end word-gram. At this time, the target word units output by each decoding network in the k-th round of decoding are spliced ​​to obtain a first splicing result, and the target word unit sequence input into the large language model in the k-1-th round of decoding is obtained, and the target word units in the target word unit sequence are located after the starting word unit, and are spliced ​​according to the position of the target word units in the target word unit sequence to obtain a second splicing result, and then, the first splicing result is spliced ​​behind the second splicing result, so that the target text can be obtained.

[0136] Among them, it should be noted that Figure 4 Only the case where all word units output by each decoding network in each decoding round pass the verification is illustrated.

[0137] in, Figure 4 In which t1 indicates the first word, t2 indicates the second word, t3 indicates the third word, t4 indicates the fourth word, s5 indicates the fifth word to be compared, t6 indicates the sixth word, t7 indicates the seventh word, t8 indicates the eighth word, t9 indicates the ninth word, and s10 indicates the tenth word to be compared. In addition, Figure 4 The box with an "e" in it indicates the end of a word.

[0138] in addition, Figure 4 The boxes with horizontal lines in the figure represent image words in the image word sequence, and the boxes with vertical lines represent text words in the text word sequence.

[0139] It should be noted that if s1 and t1, s2 and t2, s3 and t3, s4 and t4 are partially identical, for example, when s1 == t1, s2 == t2, s4 == t4, the verification module retains t1 and t2, s3, and correspondingly, t1 and t2, s3 can be sequentially spliced ​​after the first word sequence to obtain an updated first word sequence, and in the second round of decoding, the updated first word sequence is input into the large language model.

[0140] For another example, when s1==t1, s3==t3, the verification module only retains t1 and s2. Correspondingly, t1 and s2 can be sequentially spliced ​​after the first word sequence to obtain an updated first word sequence, and in the second round of decoding, the updated first word sequence is input into the large language model.

[0141] In this embodiment, combining multiple decoding networks to perform decoding in parallel can improve the decoding speed of the multimodal large model and improve the fluency of the obtained target text.

[0142] Figure 5 It is a schematic diagram according to the fifth embodiment of the present disclosure. It should be noted that the training method of the multimodal large model of the embodiment of the present disclosure can be applied to the training device of the multimodal large model. The device can be an electronic device, or can be configured in an electronic device so that the electronic device can perform the training function of the multimodal large model.

[0143] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0144] The training device for the multimodal large model may also be software in an electronic device, such as training software for the multimodal large model, etc. In the following embodiments, the execution subject is an electronic device as an example for description.

[0145] like Figure 5 As shown, the training method of the multimodal large model may include the following steps:

[0146] Step 501, obtain training data and an initial multimodal large model; the training data includes: sample images, sample task description texts corresponding to the sample images, and sample texts; the multimodal large model includes an encoding network, a large language model, and multiple decoding networks connected in sequence.

[0147] Step 502: Input the sample image and the sample task description text into the encoding network and the large language model connected in sequence, and obtain the word unit sequence output by the encoding network and the word unit feature sequence output by the large language model.

[0148] In this embodiment, the sample image and the sample task description text may be input into the decoding network to obtain the word unit sequence output by the decoding network, and the word unit sequence may be input into the large language model to obtain the word unit feature sequence output by the large language model.

[0149] In one embodiment of the present disclosure, when the encoding network includes a visual encoder and a text encoder, a sample image can be input into the visual encoder to obtain an output first image word-meta sequence; a sample task description text can be input into the text encoder to obtain an output first text word-meta sequence; the first image word-meta sequence and the first text word-meta sequence are concatenated to obtain a reference word-meta subsequence, and the starting word is concatenated at the end of the first reference word-meta subsequence to obtain a word-meta sequence.

[0150] In this embodiment, the sample image and the sample task description text are processed respectively by the visual encoder and the text encoder, which can better retain and extract the features of each modality. The visual encoder in this embodiment can output image words and map the features in the image space to the text space, thereby achieving alignment of the features of the sample image and the sample task description text in the text space, which facilitates the subsequent large language model to accurately understand the semantics.

[0151] Among them, the large language model in this embodiment is a large language model based on the self-attention mask mechanism.

[0152] Step 503: input the last word-unit feature in the word-unit feature sequence into each decoding network respectively, and obtain the predicted word-unit output by each decoding network.

[0153] The last word-unit feature in the word-unit feature sequence includes the semantic features of each word-unit in the word-unit sequence.

[0154] Step 504: update the word-gram sequence according to the predicted word-gram until the acquired predicted word-gram includes the end word-gram.

[0155] In the embodiment of the present disclosure, when it is determined that the predicted word-gram does not include the end word-gram, the predicted word-grams are sorted according to the position order of each predicted word-gram to obtain an intermediate word-gram sequence, and the intermediate word-gram sequence is spliced ​​at the end of the word-gram sequence to obtain an updated word-gram sequence, and the process jumps to inputting the word-gram sequence into the large language model until the obtained predicted word-gram includes the end word-gram.

[0156] Step 505 , based on the predicted word-grams in the updated word-gram sequence, the predicted text and the sample text are determined, and the multimodal large model is trained to obtain a trained multimodal large model.

[0157] In this embodiment, the predicted word-grams in the updated word-gram sequence may be concatenated according to their positions in the updated word-gram sequence to obtain the predicted text.

[0158] It should be noted that the process of obtaining the predicted text is similar to the process of obtaining the target text, which will not be described in detail here.

[0159] In one embodiment, the loss function value of the multimodal large model is determined based on the sample text corresponding to the predicted text and the sample task description text, and the multimodal large model is parameter-adjusted based on the loss function value to obtain the trained multimodal large model. In this way, the training speed and training accuracy of the multimodal large model can be improved.

[0160] The training method of the multimodal large model of the embodiment of the present disclosure processes the sample image and the corresponding sample task description text through the encoding network in the initial multimodal large model to obtain a word unit sequence, and inputs the word unit sequence into the large language model in the initial multimodal large model to obtain a word unit feature sequence, and inputs the last word unit feature in the word unit feature sequence into each decoding network in the initial multimodal large model, obtains the predicted word unit output by each decoding network, and updates the word unit sequence according to the predicted word unit until the obtained predicted word unit includes the end word unit, determines the obtained predicted text and sample text based on the predicted word unit in the updated word unit sequence, trains the multimodal large model, and obtains the trained multimodal large model. In this way, the difficulty and cost of model training can be reduced.

[0161] In order to implement the above embodiments, the present disclosure also provides a text determination device based on a multimodal large model.

[0162] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure.

[0163] Among them, the multimodal large model includes an encoding network, a large language model and multiple decoding networks; the multiple decoding networks are used to obtain multiple contents with continuous positions based on the same input decoding.

[0164] like Figure 6 As shown, the text determination device 60 based on the multimodal large model may include: a first acquisition module 601, a second acquisition module 602, a third acquisition module 603 and a first determination module 604.

[0165] The first acquisition module 601 is used to acquire a first word-gram sequence, and the first word-gram sequence is determined according to a target image, a task description text corresponding to the target image, and an encoding network.

[0166] The second acquisition module 602 is used to input the first word-unit sequence into the large language model to obtain an output first word-unit feature sequence.

[0167] The third acquisition module 603 is used to input the last word-unit feature in the first word-unit feature sequence into each decoding network respectively, and obtain the target word-unit output by each decoding network.

[0168] The first determination module 604 is configured to determine a target text according to the first word-gram sequence and each target word-gram.

[0169] As a possible implementation of the embodiment of the present disclosure, the first word-gram sequence includes a reference word-gram subsequence; or the first word-gram sequence includes a reference word-gram subsequence and a predicted word-gram subsequence;

[0170] The reference word subsequence is the word subsequence output by the encoding network after the target image and the task description text are input into the encoding network;

[0171] The predicted word-meta subsequence is a word-meta subsequence predicted based on a reference word-meta subsequence, a large language model, and multiple decoding networks.

[0172] As a possible implementation of the embodiment of the present disclosure, the first determining module 604 is specifically configured to:

[0173] In the case where there is an end word in the target word, determining whether the first word sequence includes a predicted word subsequence;

[0174] In the case where the first word-gram sequence includes a predicted word-gram subsequence, determining the target text according to the predicted word-gram subsequence and each target word-gram; or,

[0175] In the case that the first word-gram sequence does not include the predicted word-gram subsequence, the target text is determined according to each target word-gram.

[0176] As a possible implementation of the embodiment of the present disclosure, the device further includes:

[0177] A second determination module is used for adding each target word-gram as a predicted word-gram to the first word-gram sequence to obtain a second word-gram sequence when there is no end word-gram in the target word-gram;

[0178] The updating module is used to update the first word-gram sequence according to the second word-gram sequence, so as to re-determine the target word-gram according to the updated first word-gram sequence.

[0179] As a possible implementation of the embodiment of the present disclosure, the second determining module is specifically configured to:

[0180] According to the position order of each target word unit, each target word unit is sorted to obtain a predicted word unit sequence;

[0181] The predicted word sequence is concatenated at the end of the first word sequence to obtain a second word sequence.

[0182] As a possible implementation of the embodiment of the present disclosure, the device further includes:

[0183] A fourth acquisition module, used for inputting the second word-unit sequence into the large language model to obtain an output second word-unit feature sequence;

[0184] A fifth acquisition module, configured to extract a word-unit feature subsequence from the Mth word-unit feature to the last word-unit feature from the second word-unit feature sequence, wherein M represents the number of words in the first word-unit sequence;

[0185] A sixth acquisition module, configured to acquire a to-be-compared word unit sequence of the word unit feature subsequence according to a first decoding network among the multiple decoding networks;

[0186] A third determination module is used to compare each word in the predicted word-unit sequence with the word at the same position in the word-unit sequence to be compared, so as to obtain a valid word in the predicted word-unit sequence;

[0187] The filtering module is used to filter the ineffective words in the second word sequence.

[0188] As a possible implementation manner of the embodiment of the present disclosure, the third determination module is specifically configured to:

[0189] Compare each word in the predicted word-unit sequence with the word at the same position in the word-unit sequence to be compared;

[0190] Obtaining a first target word in the predicted word sequence; the first target word is the first word in the predicted word sequence that has not been successfully matched;

[0191] The word-gram located before the first target word-gram in the predicted word-gram sequence is determined as a valid word-gram.

[0192] As a possible implementation of the embodiment of the present disclosure, the encoding network includes: a visual encoder and a text encoder, and the reference word subsequence is determined as follows:

[0193] Input the target image into the visual encoder and obtain the output image word sequence;

[0194] Input the task description text into the text encoder to obtain the output text word sequence;

[0195] The image word-gram sequence and the text word-gram sequence are concatenated to obtain a reference word-gram subsequence. It should be noted that the above explanation of the text determination method embodiment based on the multimodal large model is also applicable to this embodiment, and will not be repeated here.

[0196] The text determination device based on the multimodal large model of the embodiment of the present disclosure determines the first word-unit sequence through the encoding network, the target image, and the task description text corresponding to the target image in the multimodal large model, and inputs the first word-unit sequence into the large language model in the multimodal large model to obtain the output first word-unit feature sequence, and inputs the last word-unit feature in the first word-unit feature sequence into each decoding network in the multimodal large model respectively, obtains the target word-unit output by each decoding network, and obtains the target text based on the first word-unit sequence and the target word-unit. Thus, by decoding the word-unit features output by the large language model in parallel through multiple decoding networks in the multimodal large model, the decoding process can be accelerated and the efficiency of obtaining the target text is improved.

[0197] In order to implement the above embodiments, the present disclosure also provides a training device for a multimodal large model.

[0198] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure.

[0199] like Figure 7 As shown, the training device 70 of the multimodal large model may include: a seventh acquisition module 701, an eighth acquisition module 702, a ninth acquisition module 703, an update module 704 and a training module 705.

[0200] The seventh acquisition module 701 is used to acquire training data and an initial multimodal large model. The training data includes: sample images, sample task description texts corresponding to the sample images, and sample texts. The multimodal large model includes an encoding network, a large language model, and multiple decoding networks connected in sequence.

[0201] The eighth acquisition module 702 is used to input the sample image and the sample task description text into the encoding network and the large language model connected in sequence, and obtain the word unit sequence output by the encoding network and the word unit feature sequence output by the large language model.

[0202] The ninth acquisition module 703 is used to input the last word-unit feature in the word-unit feature sequence into each decoding network respectively, and obtain the predicted word-unit output by each decoding network.

[0203] The updating module 704 is used to update the word-gram sequence according to the predicted word-gram until the acquired predicted word-gram includes the end word-gram.

[0204] The training module 705 is used to train the multimodal large model based on the predicted text and the sample text determined by the predicted word in the updated word sequence to obtain the trained multimodal large model.

[0205] It should be noted that the aforementioned explanation of the embodiment of the training method for a large multimodal model is also applicable to this embodiment and will not be repeated here.

[0206] The training device of the multimodal large model of the embodiment of the present disclosure processes the sample image and the corresponding sample task description text through the encoding network in the initial multimodal large model to obtain a word unit sequence, and inputs the word unit sequence into the large language model in the initial multimodal large model to obtain a word unit feature sequence, and inputs the last word unit feature in the word unit feature sequence into each decoding network in the initial multimodal large model, obtains the predicted word unit output by each decoding network, and updates the word unit sequence according to the predicted word unit until the obtained predicted word unit includes the end word unit, determines the obtained predicted text and sample text based on the predicted word unit in the updated word unit sequence, trains the multimodal large model, and obtains the trained multimodal large model. In this way, the difficulty and cost of model training can be reduced.

[0207] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0208] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0209] Figure 8 It is a block diagram of an electronic device for implementing a text determination method based on a multimodal large model or a training method for a multimodal large model of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0210] like Figure 8As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0211] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0212] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a text determination method based on a multimodal large model or a training method for a multimodal large model. For example, in some embodiments, a text determination method based on a multimodal large model or a training method for a multimodal large model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the text determination method based on the multimodal large model or the training method of the multimodal large model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the text determination method based on the multimodal large model or the training method of the multimodal large model in any other appropriate manner (e.g., by means of firmware).

[0213] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0214] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0215] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0216] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0217] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0218] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0219] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0220] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A text determination method based on a multimodal large model, wherein the multimodal large model includes an encoding network, a large language model, and multiple decoding networks, and the method includes: Acquire a first word-gram sequence, where the first word-gram sequence is determined according to a target image, a task description text corresponding to the target image, and the encoding network; Inputting the first word-unit sequence into the large language model to obtain an output first word-unit feature sequence; Inputting the last word-unit feature in the first word-unit feature sequence into each decoding network respectively, and obtaining the target word-unit output by each decoding network; A target text is determined according to the first word-gram sequence and each of the target word-grams.

2. The method according to claim 1, wherein: The first word-gram sequence includes a reference word-gram subsequence; or, the first word-gram sequence includes the reference word-gram subsequence and a predicted word-gram subsequence; The reference word-unit subsequence is a word-unit subsequence output by the encoding network after the target image and the task description text are input into the encoding network; The predicted word-meta subsequence is a word-meta subsequence predicted based on the reference word-meta subsequence, the large language model, and the multiple decoding networks.

3. The method according to claim 1 or 2, wherein: Determining the target text according to the first word-unit sequence and each of the target word-units includes: In the case where there is an end word in the target word, determining whether the first word sequence includes a predicted word subsequence; In the case where the first word-gram sequence includes a predicted word-gram subsequence, determining the target text according to the predicted word-gram subsequence and each of the target word-grams; or In a case where the first word-gram sequence does not include the predicted word-gram subsequence, the target text is determined according to each of the target word-grams.

4. The method according to claim 3, wherein: The method further comprises: When there is no end word in the target word, each of the target word is added as a predicted word to the first word sequence to obtain a second word sequence; The first word-unit sequence is updated according to the second word-unit sequence, so as to re-determine the target word-unit according to the updated first word-unit sequence.

5. The method according to claim 4, wherein: The step of adding each of the target words as predicted words to the first word sequence to obtain a second word sequence includes: Sorting the target word units according to the position order of the target word units to obtain a predicted word unit sequence; The predicted word sequence is spliced ​​at the end of the first word sequence to obtain the second word sequence.

6. The method according to claim 5, wherein: Before updating the first word sequence according to the second word sequence, the method further includes: Inputting the second word-unit sequence into the large language model to obtain an output second word-unit feature sequence; Extracting a word-unit feature subsequence from the Mth word-unit feature to the last word-unit feature from the second word-unit feature sequence, where M represents the number of words in the first word-unit sequence; According to a first decoding network among the multiple decoding networks, obtaining a word unit sequence to be compared of the word unit feature subsequence; Compare each word in the predicted word-unit sequence with the word at the same position in the word-unit sequence to be compared, to obtain a valid word in the predicted word-unit sequence; The ineffective word-grams in the second word-gram sequence are filtered.

7. The method according to claim 6, wherein: The step of comparing each word in the predicted word-unit sequence with a word at the same position in the word-unit sequence to be compared to obtain a valid word-unit in the predicted word-unit sequence includes: Comparing each word in the predicted word-word sequence with the word at the same position in the word-word sequence to be compared in sequence; Acquire a first target word in the predicted word sequence; the first target word is the first word in the predicted word sequence that has not been successfully matched; The word-gram located before the first target word-gram in the predicted word-gram sequence is determined as the valid word-gram.

8. The method according to claim 2, wherein: The encoding network includes: a visual encoder and a text encoder, and the reference word subsequence is determined in the following manner: Input the target image into the visual encoder to obtain an output image word sequence; Input the task description text into the text encoder to obtain an output text word sequence; The image word-gram sequence and the text word-gram sequence are concatenated to obtain a reference word-gram subsequence.

9. A method for training a multimodal large model, the method comprising: Obtain training data and an initial multimodal large model; The training data includes: sample images, sample task description texts corresponding to the sample images, and sample texts; the multimodal large model includes an encoding network, a large language model, and a plurality of decoding networks connected in sequence; Input the sample image and the sample task description text into the encoding network and the large language model connected in sequence, and obtain a word unit sequence output by the encoding network and a word unit feature sequence output by the large language model; Inputting the last word-unit feature in the word-unit feature sequence into each decoding network respectively, and obtaining the predicted word-unit output by each decoding network; The word-gram sequence is updated according to the predicted word-gram until the acquired predicted word-gram includes the end word-gram; The multimodal large model is trained based on the predicted text determined by the predicted word-gram in the updated word-gram sequence and the sample text to obtain a trained multimodal large model.

10. A text determination device based on a multimodal large model, wherein the multimodal large model includes an encoding network, a large language model, and a plurality of decoding networks, wherein the device comprises: A first acquisition module is used to acquire a first word-gram sequence, where the first word-gram sequence is determined according to a target image, a task description text corresponding to the target image, and the encoding network; A second acquisition module, configured to input the first word-unit sequence into the large language model and acquire an output first word-unit feature sequence; A third acquisition module is used to input the last word-unit feature in the first word-unit feature sequence into each decoding network respectively, and obtain the target word-unit output by each decoding network; The first determination module is used to determine a target text according to the first word-unit sequence and each of the target word-units.

11. The device according to claim 10, wherein: The first word-gram sequence includes a reference word-gram subsequence; or, the first word-gram sequence includes the reference word-gram subsequence and a predicted word-gram subsequence; The reference word-unit subsequence is a word-unit subsequence output by the encoding network after the target image and the task description text are input into the encoding network; The predicted word-meta subsequence is a word-meta subsequence predicted based on the reference word-meta subsequence, the large language model, and the multiple decoding networks.

12. The device according to claim 10 or 11, wherein: The first determining module is specifically configured to: In the case where there is an end word in the target word, determining whether the first word sequence includes a predicted word subsequence; In the case where the first word-gram sequence includes a predicted word-gram subsequence, determining the target text according to the predicted word-gram subsequence and each of the target word-grams; or In a case where the first word-gram sequence does not include the predicted word-gram subsequence, the target text is determined according to each of the target word-grams.

13. The device according to claim 12, wherein: The device also includes: A second determining module is used for adding each of the target word-grams as a predicted word-gram to the first word-gram sequence to obtain a second word-gram sequence when there is no end word-gram in the target word-grams; The updating module is used to update the first word-gram sequence according to the second word-gram sequence, so as to re-determine the target word-gram according to the updated first word-gram sequence.

14. The device according to claim 13, wherein: The second determining module is specifically used to: Sorting the target word units according to the position order of the target word units to obtain a predicted word unit sequence; The predicted word sequence is spliced ​​at the end of the first word sequence to obtain the second word sequence.

15. The device according to claim 14, wherein: The device also includes: A fourth acquisition module, configured to input the second word-unit sequence into the large language model to acquire an output second word-unit feature sequence; a fifth acquisition module, configured to extract a word-unit feature subsequence from the Mth word-unit feature to the last word-unit feature from the second word-unit feature sequence, wherein M represents the number of words in the first word-unit sequence; A sixth acquisition module, configured to acquire a word-unit sequence to be compared of the word-unit feature subsequence according to a first decoding network among the plurality of decoding networks; A third determination module is used to compare each word in the predicted word-unit sequence with the word at the same position in the word-unit sequence to be compared, so as to obtain a valid word in the predicted word-unit sequence; A filtering module is used to filter ineffective words in the second word sequence.

16. The device according to claim 15, wherein: The third determination module is specifically used to: Comparing each word in the predicted word-word sequence with the word at the same position in the word-word sequence to be compared in sequence; Acquire a first target word in the predicted word sequence; the first target word is the first word in the predicted word sequence that has not been successfully matched; The word-gram located before the first target word-gram in the predicted word-gram sequence is determined as the valid word-gram.

17. The device according to claim 11, wherein: The encoding network includes: a visual encoder and a text encoder, and the reference word subsequence is determined in the following manner: Input the target image into the visual encoder to obtain an output image word sequence; Input the task description text into the text encoder to obtain an output text word sequence; The image word-gram sequence and the text word-gram sequence are concatenated to obtain a reference word-gram subsequence.

18. A training device for a multimodal large model, the device comprising: The seventh acquisition module is used to acquire training data and an initial multimodal large model; The training data includes: sample images, sample task description texts corresponding to the sample images, and sample texts; the multimodal large model includes an encoding network, a large language model, and a plurality of decoding networks connected in sequence; An eighth acquisition module, configured to input the sample image and the sample task description text into the encoding network and the large language model connected in sequence, and acquire a word unit sequence output by the encoding network and a word unit feature sequence output by the large language model; A ninth acquisition module, used for inputting the last word-unit feature in the word-unit feature sequence into each decoding network respectively, and acquiring the predicted word-unit output by each decoding network; An updating module, configured to update the word-gram sequence according to the predicted word-gram until the acquired predicted word-gram includes an end word-gram; The training module is used to train the multimodal large model based on the predicted text determined by the predicted word in the updated word sequence and the sample text to obtain the trained multimodal large model.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8; Alternatively, the method of claim 9 is performed.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8; or, to execute the method according to claim 9.

21. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 8; or implements the method according to claim 9.

Images