Large language model construction training method and structure based on multi-head output

Abstract

The invention provides a large language model construction training method and structure based on multi-head output. The method comprises the steps that input of different modes is fused into a unified expression, codes of all the modes are fused to serve as input of a large language model, input of multi-mode data is supported, and the stability of model training is improved; inputting a multi-modal coding token, and outputting a multi-modal advanced feature through forward reasoning; an output layer is adjusted to be multi-head output, and multi-mode output is supported; and by adding empty units in the word segmentation unit set of each mode, the capability of the large language model is further expanded. The method supports multi-modal input and multi-modal output, is also compatible with types of single-modal input and single-modal output, multi-modal input and single-modal output, single-modal input and multi-modal output and the like, and effectively improves the overall performance of a large language model; the multi-modal encoder results can be fused, and the efficiency of multi-modal data processing and model training is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence large language model technology; in particular, to the field of multimodal large language model human-computer interaction technology; specifically, to a large language model construction training method and structure based on multi-head output. Background Art

[0002] Since the release of ChatGPT, technologies related to big language models have developed rapidly. At present, big language model technology has expanded from text big language models to other fields. For example, in the visual field, many visual big language models such as image-to-text, text-to-image, text-to-video, and video analysis have been released. In the speech field, speech big language models with functions such as speech analysis and text-to-speech have been released.

[0003] In addition, in order to further expand the application scope of the large language model, researchers began to expand the unimodal large language model to a multimodal large language model. The modal types include text, voice, image, video, etc. For example, text and pictures can be provided to the large language model to allow the large language model to understand or edit the pictures. At this time, the input of the large language model includes two modalities: text and picture.

[0004] However, the current structure of the multimodal large language model only supports multimodal input and unimodal output, which limits the application scope of the large language model. For example, in voice interaction, in order to further improve the efficiency and performance of human-computer interaction, OpenAI released the GPT-4o language model, one of the functions of which is to use the large language model to merge the speech recognition module, semantic understanding module, and speech synthesis module in the traditional voice interaction system, and unify them into a large language model, that is, the input is speech and the output is also speech, which eliminates the error accumulation of each module and speeds up the response of the system.

[0005] However, in human-computer interaction, such as in-car voice assistants, when users query real-time information, such as weather, stocks, hot news, etc., if the large language model directly outputs, the output of the large language model will have many illusions because the training data of the large language model does not include the corpus of real-time information. To solve this problem, researchers proposed the concept of intelligent agents, that is, to let the large language model learn to use tools. For example, for real-time information queries, the large language model will first call the corresponding external tool according to the user's intention and context. After the external tool returns the result, the large language model summarizes and provides the user with a more accurate response. However, the end-to-end large language model based on voice input and voice output cannot support the output of external tool call information, because at this time the output of the large language model will include two modes, one is the call information of the external tool, which is the output mode of text, and the other is the information that the large language model replies to the user, which is the output mode of voice. When applications similar to the above scenarios involve multimodal output, the current structure of the multimodal large language model is obviously not supported. Summary of the invention

[0006] In view of this, the purpose of the present invention is to develop a large language model construction training method and structure based on multi-head output, which fuses different input modalities so that the large language model supports multimodal input and increases the stability of model training; multi-head output is adopted, and the model structure includes multiple output layers, which correspond to different output modalities respectively, and a multimodal fusion module is added to the input layer, and an empty unit is added to each modal word segmenter to indicate that the output of the large language model for the modality is empty; the large language model structure supports multimodal input and multimodal output, and is also compatible with single-modal input and single-modal output, multi-modal input and single-modal output, single-modal input and multi-modal output, etc., further expanding the capabilities and application scope of the large language model and improving the overall performance of the large language model.

[0007] The present invention provides a large language model construction training method based on multi-head output, comprising: Fusion of different modal inputs (such as text, images, voice, video, etc.) into a unified representation, fusion of the encodings of each modality as the input of the large language model, so that the large language model supports the input of multimodal data and increases the stability of model training; The main function of multimodal fusion is to fuse inputs of different modalities (such as text, images, voice, video, etc.) into a unified representation, so that large language models can process multimodal data. This fusion not only supports multimodal input, but also increases the stability of model training; Input multimodal encoded tokens into the large language model and output multimodal high-level features through forward reasoning; Adjust the output layer of the large language model to multi-head output so that the large language model supports multimodal output; add empty units to the word segmentation unit set of each modality, such as <eps>, indicating that this mode should output nothing in this output, further expanding the capabilities of the large language model, supporting single-modal input and single-modal output, single-modal input and multi-modal output, multi-modal input and single-modal output, and multi-modal input and multi-modal output. For example, in human-computer interaction, multiple rounds of interaction are involved, and the questions and instructions of the user in each round are different. Sometimes it may involve text and pictures, sometimes it may involve text and videos, and sometimes it may be enough to just reply. Through the method proposed in the present invention, the large language model can select the corresponding mode output according to the user's instructions, combined with context and other information, so that the capabilities and application scope of the large language model are further expanded. After obtaining the word segmentation units of each modality output by the large language model, according to the modality type, the decoder of the corresponding modality is selected to decode the output of the large language model to obtain the output content of the corresponding modality.

[0008] Furthermore, the method for fusing the encodings of each modality comprises the following steps: S21. According to the multimodal input, determine the boundary between different modal inputs to distinguish data of different modalities and ensure that the data of each modality can be correctly processed by the encoder of the corresponding modality during the fusion process; S22, after the boundary is determined, according to the modality type of each mode, the encoder corresponding to the mode is used to encode the data of each mode; Specifically, text may be encoded using BPE, BERT, or GPT models, images may be encoded using CNN or Transformer models, speech may be encoded using acoustic models, and videos may be encoded using visual models; S23, fusing the encoding results of different modes according to the order of the multi-modal original input to obtain the fused encoding of the corresponding multi-modal input; S24. The fused codes are formed into a unified representation, and the unified representation is used as an input of the large language model.

[0009] Specifically, the unified representation can capture the relationships and complementary information between different modalities, enhance the expressiveness and generalization capabilities of the model, and improve the stability of model training.

[0010] Furthermore, the method of fusing the encoding results of different modes according to the order of the multi-modal original input in step S23 includes any one or more combinations of the following: Assign different weights to the encoding results of different modalities, and perform weighted average of the importance of the final task according to the weights of the encoding results of different modalities; Directly concatenate the encoding results of different modes together to form a new long vector; Using a neural network model, the contribution of different modal encodings in the fusion process is dynamically adjusted based on the attention mechanism to form a more complex fusion network.

[0011] Furthermore, the method for determining the boundary between the modes in step S21 includes any one or more combinations of the following methods: Analyze the characteristics of input data and identify different modalities; for example, text data usually contains characters and words, while image data contains pixel values ​​and color information; Based on the specific data format and content type associated with each file type, quickly identify and distinguish different modes and determine the boundaries of each mode before processing; Preferably, files are quickly classified by file extension and appropriate processing methods are selected; For example, here are some common file types and their corresponding modalities: 1. Text: (1) .txt: A plain text file, usually containing characters and words.

[0012] (2).pdf: Portable Document Format, which can contain text, images, and other multimedia content.

[0013] 2. Voice: (1).wav: Waveform audio file format, a lossless audio format commonly used to store raw audio data.

[0014] (2).pcm: Pulse Code Modulation file, usually used to store uncompressed audio data.

[0015] 3. Image: (1) .png: Portable Network Graphics, a format for storing images on the web that supports transparency.

[0016] (2) .jpeg or .jpg: Joint Photographic Experts Group format, a widely used image compression format.

[0017] 4. Video: (1) .mp4: A multimedia container format that can contain video, audio, and subtitle streams.

[0018] (2) .avi: Audio Video Interleave format, a popular video file format.

[0019] 5. Other multimedia: (1).gif: Graphics Interchange Format, an image format that supports animation.

[0020] (2) .webm: an open, web-based video format.

[0021] Exemplarily, when a .mp4 file is received, the .mp4 file is identified as a video file and encoded using a video encoder; In product design based on a multimodal large language model, the input position of each modality is pre-specified, and the boundary of each modality in the multimodal input is determined according to the input position.

[0022] The present invention also provides a large language model structure based on multi-head output, and the large language model construction training method based on multi-head output as described above is applied, including: Multimodal fusion module: used to fuse inputs of different modalities into a unified representation, fuse the encodings of each modality as the input of the large language model, so that the large language model supports the input of multimodal data and increases the stability of model training; The main part of the large language model is used to input multimodal encoding tokens into the large language model and output multimodal high-level features through forward reasoning; Optionally, the main part of the large language model adopts a structure obtained by removing an input layer and an output layer from the structure of the LLaMA model; Multi-head output layer: used to adjust the output layer of the large language model to multi-head output, so that the large language model supports multimodal output; by adding empty units to the word segmentation unit set of each modality, it indicates that this modality should output empty in this output, further expanding the capabilities of the large language model, supporting single-modal input and single-modal output, single-modal input and multi-modal output, multi-modal input and single-modal output, and multi-modal input and multi-modal output.

[0023] Furthermore, the multimodal fusion module includes: Determine the boundary between each modality submodule: It is used to determine the boundary between different modal inputs according to the multi-modal input to distinguish the data of different modalities and ensure that the data of each modality can be correctly processed by the encoder of the corresponding modality during the fusion process; Modal encoding submodule: After the boundary is determined, the data of each modality is encoded using the encoder of the corresponding modality according to the modal type of each modality; Fusion coding submodule: used to fuse the coding results of different modes according to the order of multi-modal original input to obtain the fused coding of the corresponding multi-modal input; Unified representation submodule: used to form a unified representation from the fused codes, and use the unified representation as the input of the large language model.

[0024] Furthermore, the fusion coding submodule includes a combination of any one or more of the following units: Weighted average unit: used to assign different weights to the encoding results of different modes, and perform weighted average of the importance of the final task according to the weights of the encoding results of different modes; Splicing unit: used to directly splice the encoding results of different modes together to form a new long vector; Attention mechanism dynamic adjustment unit: used to use the neural network model to dynamically adjust the contribution of different modal encodings in the fusion process based on the attention mechanism to form a more complex fusion network.

[0025] Furthermore, the submodule for determining the boundary between each mode includes a combination of any one or more of the following units: Data feature recognition unit: used to analyze the features of input data and identify different modes; File type identification unit: used to quickly identify and distinguish different modes and determine the boundaries of each mode before processing based on the specific data format and content type associated with each file type; A predetermined input position distinguishing unit; used to pre-specify the input position of each mode in product design based on a multimodal large language model, and to determine the boundaries of each mode in the multimodal input according to the input position distinction.

[0026] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training method for constructing a large language model based on multi-head output as described above.

[0027] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for constructing a large language model based on multi-head output as described above are implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects: The large language model construction and training method based on multi-head output provided by the present invention fuses different input modalities, and adds a multimodal fusion module to the input layer, so that the large language model supports multimodal input, and at the same time increases the stability of model training; multi-head output is adopted, and the model structure includes multiple output layers, which correspond to different output modalities respectively, and empty units are added in each modal word segmenter to indicate that the output of the large language model for the modality is empty; multimodal input and multimodal output are supported, and at the same time, single-modal input and single-modal output, multi-modal input and single-modal output, single-modal input and multi-modal output, etc. are also compatible, further expanding the capabilities and application scenario scope of the large language model, and effectively improving the overall performance of the large language model; the encoder results of multiple modalities can be fused, which improves the efficiency of multimodal data processing and model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0030] In the attached picture: Figure 1 A schematic diagram of a model architecture of a large language model based on multi-head output in practical applications according to an embodiment of the present invention; Figure 2 This is a flow chart of multimodal coding fusion according to an embodiment of the present invention; Figure 3 This is a diagram of the first forward reasoning method of a large language model based on multi-head outputs according to an embodiment of the present invention; Figure 4 This is a diagram of a second forward reasoning method of a large language model based on multi-head outputs according to an embodiment of the present invention; Figure 5 This is a diagram illustrating the output sequence of a large language model based on multi-head output according to an embodiment of the present invention; Figure 6 This is a flow chart of a method for fusing the encoding of each modality in the present invention; Figure 7 The figure is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of systems and products consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms "a", "the", and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0033] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0034] The embodiments of the present invention are described in further detail below.

[0035] The embodiment of the present invention provides a large language model construction training method based on multi-head output, including: Fusion of different modal inputs (text, image, voice, video) into a unified representation, fusion of the encodings of each modality as the input of the large language model, so that the large language model supports the input of multimodal data and increases the stability of model training; The method for fusing the encodings of each modality comprises the following steps (see Figure 6 shown): S21. According to the multimodal input, determine the boundary between different modal inputs to distinguish data of different modalities and ensure that the data of each modality can be correctly processed by the encoder of the corresponding modality during the fusion process; The method for determining the boundary between the modes includes any one or more combinations of the following methods: Analyze the characteristics of input data and identify different modalities; for example, text data usually contains characters and words, while image data contains pixel values ​​and color information; Based on the specific data format and content type associated with each file type, quickly identify and distinguish different modes and determine the boundaries of each mode before processing; In this embodiment, files are quickly classified by file extension and a suitable processing method is selected; when a .mp4 file is received, the .mp4 file is identified as a video file and encoded using a video encoder; In product design based on a multimodal large language model, the input position of each modality is pre-specified, and the boundary of each modality in the multimodal input is determined according to the input position.

[0036] S22, after the boundary is determined, according to the modality type of each mode, the encoder corresponding to the mode is used to encode the data of each mode; Specifically, text may be encoded using BPE, BERT, or GPT models, images may be encoded using CNN or Transformer models, speech may be encoded using acoustic models, and videos may be encoded using visual models; S23, fusing the encoding results of different modes according to the order of the multi-modal original input to obtain the fused encoding of the corresponding multi-modal input; The method of fusing the encoding results of different modes according to the order of the multi-modal original input includes any one or more combinations of the following: Assign different weights to the encoding results of different modalities, and perform weighted average of the importance of the final task according to the weights of the encoding results of different modalities; Directly concatenate the encoding results of different modes together to form a new long vector; Using a neural network model, the contribution of different modal encodings in the fusion process is dynamically adjusted based on the attention mechanism to form a more complex fusion network.

[0037] S24. The fused codes are formed into a unified representation, and the unified representation is used as an input of the large language model.

[0038] Specifically, the unified representation can capture the relationships and complementary information between different modalities, enhance the expressiveness and generalization capabilities of the model, and improve the stability of model training.

[0039] The flowchart of multimodal fusion is as follows: Figure 2 As shown in the figure, different modal inputs (text, image, speech, video) are fused into a unified representation, allowing large language models to process multimodal data. This not only supports multimodal input, but also increases the stability of model training. Input multimodal encoded tokens into the large language model and output multimodal high-level features through forward reasoning; Adjust the output layer of the large language model to multi-head output so that the large language model supports multimodal output; add empty units to the word segmentation unit set of each modality <eps>, indicating that this modality should output empty in this output, further expanding the capabilities of the large language model, supporting single-modal input and single-modal output, single-modal input and multi-modal output, multi-modal input and single-modal output, and multi-modal input and multi-modal output.

[0040] Through the method proposed in the embodiment of the present invention, in human-computer interaction, the large language model can select the corresponding mode output according to the user's instructions and in combination with context and other information, so that the ability and application scope of the large language model can be further expanded. After obtaining the word segmentation units of each modality output by the large language model, according to the modality type, the decoder of the corresponding modality is selected to decode the output of the large language model to obtain the output content of the corresponding modality.

[0041] The embodiment of the present invention further provides a large language model structure based on multi-head output, and the large language model construction training method based on multi-head output as described above is applied, including: Multimodal fusion module: used to fuse inputs of different modalities into a unified representation, fuse the encodings of each modality as the input of the large language model, so that the large language model supports the input of multimodal data and increases the stability of model training; The main part of the large language model is used to input multimodal encoding tokens into the large language model and output multimodal high-level features through forward reasoning; In this embodiment, the main part of the large language model adopts the structure of the LLaMA (LLM) model after removing the input layer and the output layer (such as Figure 1 shown); Multi-head output layer: used to adjust the output layer of the large language model to multi-head output, so that the large language model supports multimodal output; by adding empty units to the word segmentation unit set of each modality, it indicates that this modality should output empty in this output, further expanding the capabilities of the large language model, supporting single-modal input and single-modal output, single-modal input and multi-modal output, multi-modal input and single-modal output, and multi-modal input and multi-modal output.

[0042] The multimodal fusion module comprises: Determine the boundary between each modality submodule: It is used to determine the boundary between different modal inputs according to the multi-modal input to distinguish the data of different modalities and ensure that the data of each modality can be correctly processed by the encoder of the corresponding modality during the fusion process; Modal encoding submodule: After the boundary is determined, the data of each modality is encoded using the encoder of the corresponding modality according to the modal type of each modality; Fusion coding submodule: used to fuse the coding results of different modes according to the order of multi-modal original input to obtain the fused coding of the corresponding multi-modal input; Unified representation submodule: used to form a unified representation from the fused codes, and use the unified representation as the input of the large language model.

[0043] The fusion coding submodule includes a combination of any one or more of the following units: Weighted average unit: used to assign different weights to the encoding results of different modes, and perform weighted average of the importance of the final task according to the weights of the encoding results of different modes; Splicing unit: used to directly splice the encoding results of different modes together to form a new long vector; Attention mechanism dynamic adjustment unit: used to use the neural network model to dynamically adjust the contribution of different modal encodings in the fusion process based on the attention mechanism to form a more complex fusion network.

[0044] The submodule for determining the boundary between each mode includes a combination of any one or more of the following units: Data feature recognition unit: used to analyze the features of input data and identify different modes; File type identification unit: used to quickly identify and distinguish different modes and determine the boundaries of each mode before processing based on the specific data format and content type associated with each file type; A predetermined input position distinguishing unit; used to pre-specify the input position of each mode in product design based on a multimodal large language model, and to determine the boundaries of each mode in the multimodal input according to the input position distinction.

[0045] In practical applications, the forward reasoning methods of the large language model of the embodiment of the present invention include two methods. The multimodal fusion modules of the two methods are divided into two parts for processing: one part is the original multimodal input part of the user; the other part is the autoregressive output part of the large language model based on multi-head output; The multimodal input part is arranged according to the original order of each modal input, and if the multimodal input involves a time series, the time alignment of the modality must be maintained. For example, when the multimodal input contains voice and video modalities, the frame rates of the two must be kept the same, that is, the length of the feature sequence contained in the unit time must be consistent; The output sequence of the autoregressive output part of the large model is as follows Figure 5 As shown, each mode is output in a frame-aligned manner. If a mode is not included in the frame output, the output corresponding to the mode is an "empty unit" and is represented by " <eps>” symbol.

[0046] The steps of the first forward reasoning method are as follows: Step 01: The output of the autoregressive output part of the large language model of the multimodal fusion module contains tokens of multiple modalities at the same time. These tokens are arranged in sequence according to the output order of the large language model, that is, the output tokens of each modality are staggered, such as Figure 3 As shown; Step 02: The large language model inputs the encoded tokens after multimodal fusion, and the main part of the large language model processes these tokens and outputs multimodal high-level features; Step 03: The output layer of the large language model is adjusted to multi-head output (multi-head output layer processing) to support multimodal output; add empty units (" <eps>”), indicating that the modality should be output as empty in this output, so that the large language model can support input and output forms of multiple modalities; Step 04: According to the modal tokens output by the large language model, select the decoder of the corresponding modality to decode the output and obtain the output content of the corresponding modality; Step 05: If all modal tokens output in step 03 are stop symbols (" <eos>”), then the forward reasoning ends, otherwise return to step 01 and output the multimodal token generated in step 03, such as Figure 3 As shown, they are arranged in an interleaved manner behind the autoregressive output part contained in the large language model input data of the current multi-head output, and the next round of autoregressive generation calculation is performed.

[0047] The steps of the second forward reasoning method are as follows: Step 11: The output of the large model autoregressive output part of the multimodal fusion module contains tokens of multiple modalities at the same time. These tokens are arranged according to the modality type, that is, the output modality tokens are arranged continuously according to the modality, such as Figure 4 As shown; Step 12: The large language model inputs the encoded tokens after multimodal fusion, and the main part of the large language model processes these tokens and outputs multimodal high-level features; Step 13: The output layer of the large language model is adjusted to multi-head output (multi-head output layer processing) to support multimodal output; add empty units to the word segmentation unit set of each modality (" <eps>”), indicating that the mode should be output as empty in this output, so that the large model can support input and output forms of multiple modes; Step 14: According to the modal tokens output by the large language model, select the decoder of the corresponding modality to decode the output to obtain the output content of the corresponding modality; Step 15: If all modal tokens outputted in step 13 are stop symbols, the forward reasoning ends. Otherwise, return to step 11 and output the multimodal tokens generated in step 13, such as Figure 4 As shown, they are classified according to the modality type and placed behind each modality token of the autoregressive output part contained in the input data of the current multi-head output large language model to perform the next round of autoregressive generation calculation.

[0048] An embodiment of the present invention further provides a computer device, Figure 7 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention; see the accompanying drawings Figure 7 As shown, the computer device includes: an input system 23, an output system 24, a memory 22 and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the large language model construction training method based on multi-head output as provided in the above embodiment; wherein the input system 23, the output system 24, the memory 22 and the processor 21 can be connected by a bus or other means, Figure 7 The example of connecting through bus is taken in the following.

[0049] The memory 22 is a readable and writable storage medium of a computing device, which can be used to store software programs and computer executable programs, such as the program instructions corresponding to the large language model construction training method based on multi-head output described in the embodiment of the present invention; the memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the device, etc.; in addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device; in some instances, the memory 22 can further include a memory remotely arranged relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0050] The input system 23 may be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the device; the output system 24 may include display devices such as display screens.

[0051] The processor 21 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 22, that is, implements the above-mentioned large language model construction training method based on multi-head output.

[0052] The computer device provided above can be used to execute the large language model construction training method based on multi-head output provided in the above embodiment, and has corresponding functions and beneficial effects.

[0053] The embodiment of the present invention also provides a storage medium containing computer executable instructions, which are used to execute the large language model construction training method based on multi-head output as provided in the above embodiment when executed by a computer processor. The storage medium is any of various types of memory devices or storage devices, and the storage medium includes: installation media, such as CD-ROM, floppy disk or tape system; computer system memory or random access memory, such as DRAM, DDRRAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc.; the storage medium may also include other types of memory or a combination thereof; in addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, the second computer system is connected to the first computer system via a network (such as the Internet); the second computer system may provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (for example, in different computer systems connected via a network). The storage medium can store program instructions (for example, specifically implemented as a computer program) that can be executed by one or more processors.

[0054] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present invention is not limited to the large language model construction training method based on multi-head output as described in the above embodiment, and can also execute related operations in the large language model construction training method based on multi-head output provided in any embodiment of the present invention. So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / eps> < / eos> < / eps> < / eps> < / eps> < / eps>

Claims

1. A method for constructing a large language model based on multi-head output, characterized in that: include: The inputs of different modalities are integrated into a unified representation. The encodings of each modality are integrated as the input of the large language model, so that the large language model supports the input of multimodal data and increases the stability of model training. Input multimodal encoded tokens into the large language model and output multimodal high-level features through forward reasoning; The output layer of the large language model is adjusted to multi-head output so that the large language model supports multimodal output. By adding empty units to the set of word segmentation units in each modality, it indicates that this modality should output empty in this output, and further expands the capabilities of the large language model to support single-modal input and single-modal output, single-modal input and multi-modal output, multi-modal input and single-modal output, and multi-modal input and multi-modal output.

2. The method for constructing a large language model based on multi-head output according to claim 1, characterized in that: The method for fusing the encodings of each modality comprises the following steps: S21. According to the multimodal input, determine the boundary between different modal inputs to distinguish data of different modalities and ensure that the data of each modality can be correctly processed by the encoder of the corresponding modality during the fusion process; S22, after the boundary is determined, according to the modality type of each mode, the encoder corresponding to the mode is used to encode the data of each mode; S23, fusing the encoding results of different modes according to the order of the multi-modal original input to obtain the fused encoding of the corresponding multi-modal input; S24. The fused codes are formed into a unified representation, and the unified representation is used as an input of the large language model.

3. The method for constructing a large language model based on multi-head output according to claim 2, characterized in that: The method of fusing the encoding results of different modes according to the order of the multi-modal original input in step S23 includes any one or more combinations of the following: Assign different weights to the encoding results of different modalities, and perform weighted average of the importance of the final task according to the weights of the encoding results of different modalities; Directly concatenate the encoding results of different modes together to form a new long vector; Using a neural network model, the contribution of different modal encodings in the fusion process is dynamically adjusted based on the attention mechanism to form a more complex fusion network.

4. The method for constructing a large language model based on multi-head output according to claim 1, characterized in that: The method for determining the boundary between the modes in step S21 includes any one or more combinations of the following methods: Analyze the characteristics of input data and identify different modes; Based on the specific data format and content type associated with each file type, quickly identify and distinguish different modes and determine the boundaries of each mode before processing; In product design based on a multimodal large language model, the input position of each modality is pre-specified, and the boundary of each modality in the multimodal input is determined according to the input position.

5. A large language model structure based on multi-head output, using the large language model construction training method based on multi-head output as described in any one of claims 1 to 4, characterized in that: include: Multimodal fusion module: used to fuse inputs of different modalities into a unified representation, fuse the encodings of each modality as the input of the large language model, so that the large language model supports the input of multimodal data and increases the stability of model training; The main part of the large language model is used to input multimodal encoding tokens into the large language model and output multimodal high-level features through forward reasoning; Multi-head output layer: used to adjust the output layer of the large language model to multi-head output, so that the large language model supports multimodal output; by adding empty units to the word segmentation unit set of each modality, it indicates that this modality should output empty in this output, further expanding the capabilities of the large language model, supporting single-modal input and single-modal output, single-modal input and multi-modal output, multi-modal input and single-modal output, and multi-modal input and multi-modal output.

6. The large language model structure based on multi-head output according to claim 5, characterized in that: The multimodal fusion module comprises: Determine the boundary between each modality submodule: It is used to determine the boundary between different modal inputs according to the multi-modal input to distinguish the data of different modalities and ensure that the data of each modality can be correctly processed by the encoder of the corresponding modality during the fusion process; Modal encoding submodule: After the boundary is determined, the data of each modality is encoded using the encoder of the corresponding modality according to the modal type of each modality; Fusion coding submodule: used to fuse the coding results of different modes according to the order of multi-modal original input to obtain the fused coding of the corresponding multi-modal input; Unified representation submodule: used to form a unified representation from the fused codes, and use the unified representation as the input of the large language model.

7. The large language model structure based on multi-head output according to claim 6 is characterized in that: The fusion coding submodule includes a combination of any one or more of the following units: Weighted average unit: used to assign different weights to the encoding results of different modes, and perform weighted average of the importance of the final task according to the weights of the encoding results of different modes; Splicing unit: used to directly splice the encoding results of different modes together to form a new long vector; Attention mechanism dynamic adjustment unit: used to use the neural network model to dynamically adjust the contribution of different modal encodings in the fusion process based on the attention mechanism to form a more complex fusion network.

8. The large language model structure based on multi-head output according to claim 6, characterized in that: The submodule for determining the boundary between each mode includes a combination of any one or more of the following units: Data feature recognition unit: used to analyze the features of input data and identify different modes; File type identification unit: used to quickly identify and distinguish different modes and determine the boundaries of each mode before processing based on the specific data format and content type associated with each file type; A predetermined input position distinguishing unit; It is used to pre-specify the input position of each mode in product design based on a multimodal large language model, and to distinguish and determine the boundaries of each mode in the multimodal input according to the input position.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for building a large language model based on multi-head output as described in any one of claims 1 to 4 are implemented.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for building a large language model based on multi-head output training are implemented as described in any one of claims 1 to 4.

Images