Large model distributed separate training method and device

Abstract

The disclosure provides a large model distributed separation training method and device, relates to the technical field of artificial intelligence, and in particular to the technical field of multi-modal large models. A specific embodiment of the method comprises: cutting a large language model into a stage sequence, and replicating a video encoder in each stage in the stage sequence; inputting image data to the video encoder in the stage sequence to generate image features; using the image features, training a part of the large language model in the stage sequence, and transmitting the gradient of the image features back to the video encoder in the stage sequence during the training process; and using the image data, training the video encoder in the stage sequence.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular to the technical field of multi-modal large model. BACKGROUND

[0002] It has been a long-term goal to build a general-purpose model that can effectively perceive the world through multi-modal signals. The architecture of the late fusion multi-modal large model is to connect the visual encoder to the large language model. The image data is processed by the video encoder to generate image features, the image features are spliced with the text features, and the large language model is input for autoregressive training.

[0003] The parameter quantity of the multi-modal large model is huge. In order to improve the training efficiency of the model, the PP (Pipeline Parallel, pipeline parallel) distributed architecture is usually used to train the multi-modal large model. Since the parameter quantity of the visual encoder is much smaller than that of the large language model, the visual encoder is usually placed in PP0 together with the embedding layer of the large language model. However, the number of layers of the visual encoder is very deep, which will cause PP0 to become a computational bottleneck. SUMMARY

[0004] Embodiments of the present disclosure provide a large model distributed separation training method, device, equipment, storage medium and program product.

[0005] In a first aspect, the embodiments of the present disclosure provide a large model distributed separation training method, comprising: dividing a large language model into a stage sequence, and replicating a video encoder on each stage in the stage sequence; inputting image data to the video encoder in the stage sequence to generate image features; training a part of the large language model in the stage sequence using the image features, and transmitting the gradient of the image features back to the video encoder in the stage sequence during the training process; and training the video encoder in the stage sequence using the image data and the gradient of the image features.

[0006] In a second aspect, the embodiments of the present disclosure provide a large model distributed separation training device, comprising: a division module configured to divide a large language model into a stage sequence, and replicate a video encoder on each stage in the stage sequence; an encoding module configured to input image data to the video encoder in the stage sequence to generate image features; a first training module configured to train a part of the large language model in the stage sequence using the image features, and transmit the gradient of the image features back to the video encoder in the stage sequence during the training process; and a second training module configured to train the video encoder in the stage sequence using the image data and the gradient of the image features.

[0007] In a third aspect, the embodiments of the present disclosure provide an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0008] In a fourth aspect, the embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause a computer to perform the method described in the first aspect.

[0009] In a fifth aspect, the embodiments of the present disclosure provide a computer program product comprising a computer program, the computer program being executed by a processor to implement the method described in the first aspect.

[0010] The key or important features of the embodiments of the present disclosure are not used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] Other features, objects and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the drawings. The drawings are intended to better understand the present disclosure and do not constitute a limitation thereof. Among them:

[0012] Figure 1 is a flowchart of an embodiment of the distributed separation training method of the large model according to the present disclosure;

[0013] Figure 2 is a flowchart of another embodiment of the distributed separation training method of the large model according to the present disclosure;

[0014] Figure 3 is a flowchart of the distributed separation training of the large model;

[0015] Figure 4 is a structural schematic diagram of an embodiment of the distributed separation training device of the large model according to the present disclosure;

[0016] Figure 5 is a block diagram of an electronic device for implementing the distributed separation training method of the large model according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are cited as illustrative examples. Various modifications and changes can be suggested by those skilled in the art, and it is intended that the present disclosure encompass such modifications and changes as fall within the scope of the appended claims. Likewise, the present disclosure is not to be limited in terms of the description thereof as has been set forth hereinabove, but is to be understood to cover anyone or more aspects, whether or not in that specific combination.

[0018] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0019] Figure 1 Flow 100 of one embodiment of a large model distributed separation training method according to the present disclosure is shown. The large model distributed separation training method includes the following steps:

[0020] Step 101, the large language model is cut into a stage sequence, and the video encoder is replicated on each stage in the stage sequence.

[0021] In this embodiment, the execution subject of the large model distributed separation training method can cut the large language model into a stage sequence, and replicate the video encoder on each stage in the stage sequence.

[0022] A multi-modal large model can be composed of a visual encoder and a large language model, and its architecture can be to connect the visual encoder to the large language model. The multi-modal large language model can process data of two modalities of images and texts. The visual encoder can be used to process images. The large language model is used to process texts.

[0023] The large language model is sequentially split into multiple parts, each part is sequentially replicated into a stage to generate a stage sequence (PP0, PP1, PP2, …, PPn). Unlike the visual encoder in the prior art, which is placed in PP0, the video encoder is replicated on each stage in the stage sequence. That is, a complete copy of the video encoder is replicated on each stage in the stage sequence, and the initial parameters of these copies are the same.

[0024] Step 102, input image data to the video encoder in the stage sequence to generate image features.

[0025] In this embodiment, the execution subject described above can input image data to the video encoder in the stage sequence to generate image features. Wherein the video encoder in each stage can be used to process a part of image data.

[0026] To make the video encoder in each stage of the stage sequence process a part of data, the image data can be split in the stage dimension to generate sub-image data corresponding to each stage. For example, if the stage sequence includes four stages, the image data can be split into four parts, and each stage processes one part. Then, the sub-image data corresponding to each stage is forward propagated in the video encoder in the corresponding stage to generate sub-image features corresponding to each stage, and the sub-image features corresponding to each stage are combined to generate image features. Wherein, the video encoder in each stage independently processes the corresponding sub-image data, but the parameters of the video encoder stop the gradient and do not train.

[0027] Step 103, using the image features, training the large language model part in the stage sequence, and transmitting the gradient of the image features back to the video encoder in the stage sequence during the training process.

[0028] In the embodiment, the above execution subject can train the large language model part in the stage sequence using the image features, and transmit the gradient of the image features back to the video encoder in the stage sequence during the training process.

[0029] Based on the image features, input data of the large language model can be generated. Forward propagation of the input data in the large language model part in the stage sequence can generate the gradient of the large language model. Backward propagation of the gradient of the large language model in the large language model part in the stage sequence can update the parameters of the large language model. And during the training process of the large language model, the gradient of the image features can be transmitted back to the video encoder in the stage sequence.

[0030] During the training process of the large language model, the parameters of the video encoder are set to stop the gradient. That is, the gradient of the large language model will not automatically backward propagate back to the video encoder. The video encoder is only responsible for providing the image features in this forward propagation, and its parameters are temporarily frozen.

[0031] Step 104, using the image data, training the video encoder in the stage sequence.

[0032] In the embodiment, the above execution subject can train the video encoder in the stage sequence using the image data.

[0033] The video encoder in the stage sequence reprocesses the image data. Forward propagation of the image data in the video encoder in the stage sequence generates the image features again. Then, the video encoder in the stage sequence manually performs the backward propagation of the video encoder using the gradient of the image features returned back to update the parameters of the video encoder.

[0034] The embodiment of the present disclosure provides a large model distributed separation training method, which separates the training of a large language model and a video encoder, adopts a differentiated distributed parallel strategy, and optimizes resource allocation. The training of the video encoder no longer blocks the pipeline parallel process of the large language model, and the training of the large language model can maintain an efficient standard pipeline parallel mode, thereby significantly improving the overall training efficiency of the multi-modal large model.

[0035] With reference to Figure 2 which shows a flow 200 of still another embodiment of the large model distributed separation training method according to the present disclosure. The large model distributed separation training method comprises the following steps:

[0036] In step 201, the large language model is divided into a stage sequence by layers using pipeline parallelism, and a video encoder is replicated on each stage in the stage sequence.

[0037] In this embodiment, the execution subject of the large model distributed separation training method can divide the large language model into a stage sequence by layers using pipeline parallelism, and replicate a video encoder on each stage in the stage sequence.

[0038] The multi-modal large model can be composed of a visual encoder and a large language model, and the architecture thereof can be connecting the visual encoder to the large language model. The multi-modal large language model can process data of two modalities of images and texts. The visual encoder can be used to process images. The large language model is used to process texts.

[0039] The large language model can include multiple layers. Using the PP technology, the large language model can be divided into multiple parts (referred to as stages) in layer order to generate a stage sequence (PP0, PP1, PP2, …, PPn). Each stage can include one or more layers of the large language model and be deployed on one GPU (Graphics Processing Unit, graphics processor). Data flows between GPUs in a pipeline manner, each GPU processes one stage of the model, and then the result is passed to the next GPU.

[0040] Unlike the visual encoder in the prior art which is placed in PP0, the video encoder is replicated on each stage in the stage sequence. That is, a complete copy of the video encoder is replicated on each stage in the stage sequence, and the initial parameters of these copies are the same.

[0041] In step 202, the image data is divided in the stage dimension to generate sub-image data corresponding to each stage.

[0042] In this embodiment, the above execution subject can divide the image data in the stage dimension to generate sub-image data corresponding to each stage.

[0043] To make the video encoder in each stage of the stage sequence process a part of data, the image data can be split in the stage dimension to generate sub-image data corresponding to each stage. For example, if the stage sequence includes four stages, the image data can be split into four parts, and each stage processes one part.

[0044] Step 203, forward propagate the sub-image data corresponding to each stage in the video encoder in the corresponding stage to generate sub-image features corresponding to each stage.

[0045] In this embodiment, the execution subject described above can forward propagate the sub-image data corresponding to each stage in the video encoder in the corresponding stage to generate sub-image features corresponding to each stage. Wherein the video encoder in each stage independently processes the corresponding sub-image data, but the parameters of the video encoder stop the gradient and do not train.

[0046] Step 204, aggregate the sub-image features corresponding to each stage to the first stage in the stage sequence to generate image features.

[0047] In this embodiment, the execution subject described above can aggregate the sub-image features corresponding to each stage to the first stage PP0 in the stage sequence (PP0, PP1, PP2, …, PPn) to generate image features.

[0048] Step 205, splice the image features and text features based on the image features to generate input data.

[0049] In this embodiment, the execution subject described above can splice the image features and text features based on the image features to generate input data.

[0050] Generally, the image features and text features are different in dimension, in order to be able to splice the image features and text features, the image features can be input to a resampler for resampling to obtain resampled image features. Wherein the resampler can be used to align the image features and text features. Then, the resampled image features and text features are spliced in the sequence dimension to generate input data. For example, the image features After the resampler, the hidden layer size of the video encoder can be aligned to the hidden layer size of the embedding layer of the large language model , to obtain resampled image features . The resampled image features are spliced with the text features in the sequence dimension to generate input data .

[0051] Step 206, forward propagate the input data in the large language model part of the stage sequence to generate the gradient of the large language model.

[0052] In this embodiment, the execution subject can forward propagate the input data in the large language model part of the stage sequence to generate the gradient of the large language model.

[0053] The input data can flow forward through PP0, PP1, PP2, …, PPn in turn to generate the gradient of the large language model.

[0054] Step 207, backward propagate the gradient of the large language model in the large language model part of the stage sequence to update the parameters of the large language model.

[0055] In this embodiment, the execution subject can backward propagate the gradient of the large language model in the large language model part of the stage sequence to update the parameters of the large language model.

[0056] The gradient of the large language model can flow backward through PPn, …, PP2, PP1, PP0 in turn to update the parameters of the large language model.

[0057] In the training process of the large language model, the parameters of the video encoder are set to stop the gradient. That is, the gradient of the large language model will not be automatically backward propagated back to the video encoder. The video encoder is only responsible for providing image features in this forward propagation, and its parameters are temporarily frozen.

[0058] Step 208, backward propagate the gradient of the large language model to the first stage in the stage sequence to generate the gradient of the image feature.

[0059] In this embodiment, the execution subject can backward propagate the gradient of the large language model to the first stage PP0 in the stage sequence to generate the gradient of the image feature.

[0060] In the training process of the large language model, the gradient of the large language model is backward propagated from PPn to PP0. When the backward propagation reaches PP0, the gradient of the image feature can be obtained.

[0061] Step 209, backward propagate the gradient of the image feature between the video encoders in the stage sequence.

[0062] In this embodiment, the execution subject can backward propagate the gradient of the image feature between the video encoders in the stage sequence.

[0063] The gradient of the image feature on PP0 can be backward broadcasted between PPs to return the gradient of the image feature to the video encoder on each PP.

[0064] Step 210, forward-propagating the image data in the video encoder in the stage sequence, manually back-propagating the gradient of the image feature in the video encoder in the stage sequence, and updating the parameters of the video encoder in the stage sequence.

[0065] In the embodiment, the execution subject can forward-propagate the image data in the video encoder in the stage sequence, manually back-propagate the gradient of the image feature in the video encoder in the stage sequence, and update the parameters of the video encoder in the stage sequence.

[0066] The video encoder in each stage reprocesses the corresponding sub-image data. The sub-image data corresponding to each stage is forward-propagated in the video encoder in the corresponding stage to generate the sub-image feature corresponding to each stage again. Then, the video encoder in each stage manually performs the back-propagation of the video encoder using the gradient of the corresponding sub-image feature returned back. Through the manual back-propagation, the gradient of the video encoder in each stage is calculated, and the parameters of the video encoder in the corresponding stage are independently updated according to the gradient of the video encoder in each stage. Finally, the training of the video encoder is completed through the parameter synchronization of the video encoders between stages.

[0067] The training of the video encoder is essentially a data parallel training manner, and the video encoder in each stage updates the local model copy using the gradient of the corresponding sub-image feature, and completes the training of the video encoder through the parameter synchronization of the video encoders between stages.

[0068] The embodiment of the present disclosure provides a large model distributed separated training method, which separates the training of a large language model from the training of a video encoder, adopts a differentiated distributed parallel strategy, and optimizes resource allocation. The training of the video encoder no longer blocks the pipeline parallel process of the large language model, and the training of the large language model can maintain an efficient standard pipeline parallel mode, thereby significantly improving the overall training efficiency of the multi-modal large language model. The video encoder is trained in parallel on the stage sequence, and uses the computing resources of all devices to process its deeper layers. The video encoder in each stage processes a part of the image data, realizes data parallel processing, and improves the processing efficiency of the image data. Since the computing load of the video encoder is dispersed to all stages for parallel processing, the problem that PP0 becomes a bottleneck due to long video encoder computing time in the traditional scheme is completely eliminated, and the air bubble in the pipeline training process is significantly reduced. In the video encoder training stage, all stages are computing at the same time, which can fill the computing density of the GPU, and greatly improves the resource utilization and training throughput of the overall cluster.

[0069] Figure 3 A flowchart of the large model distributed separated training is shown. As shown in FIG. 1, the large model distributed separated training includes the following steps. Figure 3As shown, the design is heterogeneous distributed strategy, LLM (Large Language Model) part normally carries out PP architecture segmentation. VIT (Vision Transformer) will copy a copy on each PP.

[0070] In the first fwd (forward), the image data is segmented on the PP, and img_fea (image feature) is obtained through VIT. Then, img_fea is gathered to PP0 to obtain Acc img_fea.

[0071] In the Acc fwd, Acc img_fea flows through PP0, PP1, PP2, …, PPn in turn, and generates Acc bwd. Then, Acc bwd flows through PPn, …, PP2, PP1, PP0 in turn, and updates the parameters of LLM. It should be noted that in the training phase of the LLM backbone network, the parameters of VIT are stop_gradient (stop gradient) and are not trained.

[0072] In the LLM backbone network training process, Acc bwd is back-propagated to PP0, and img_fea_grad can be obtained. Then, img_fea_grad is reversely broadcasted between PPs, and img_fea_grad is also given to VIT on each PP.

[0073] In the second fwd, each PP separately re-forwards the image data, and then uses img_fea_grad to manually back-propagate, calculate the gradient of the VIT parameters, and update the VIT parameters, so that the VIT is trained in parallel between PPs.

[0074] Figure 4 Further referring to Figure 1 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a large model distributed separate training device. The device embodiment corresponds to the method embodiment shown in

[0075] As Figure 4As shown, the large model distributed separated training apparatus 400 in this embodiment can include a splitting module 401, an encoding module 402, a first training module 403, and a second training module 404. The splitting module 401 is configured to split the large language model into a stage sequence and replicate the video encoder on each stage in the stage sequence. The encoding module 402 is configured to input the image data to the video encoder in the stage sequence to generate the image feature. The first training module 403 is configured to train the large language model part in the stage sequence by using the image feature, and transmit the gradient of the image feature back to the video encoder in the stage sequence during the training process. The second training module 404 is configured to train the video encoder in the stage sequence by using the image data and the gradient of the image feature.

[0076] In this embodiment, the splitting module 401, the encoding module 402, the first training module 403, and the second training module 404 in the large model distributed separated training apparatus 400 have specific processing and bring about technical effects, which can be respectively referred to Figure 1 The related description of steps 101-104 in the corresponding embodiment will not be repeated here.

[0077] In some optional implementation manners of this embodiment, the splitting module 401 is further configured to split the large language model into the stage sequence by layer by using pipeline parallelism, where each stage includes at least one layer of the large language model.

[0078] In some optional implementation manners of this embodiment, the encoding module 402 is further configured to split the image data in the stage dimension to generate sub-image data corresponding to each stage, forward propagate the sub-image data corresponding to each stage in the video encoder in the corresponding stage to generate sub-image features corresponding to each stage, and merge the sub-image features corresponding to each stage to generate the image feature.

[0079] In some optional implementation manners of this embodiment, the encoding module 402 is further configured to aggregate the sub-image features corresponding to each stage to the first stage in the stage sequence to generate the image feature.

[0080] In some optional implementation manners of this embodiment, the first training module 403 is further configured to generate input data based on the image feature, forward propagate the input data in the large language model part in the stage sequence to generate the gradient of the large language model, and backward propagate the gradient of the large language model in the large language model part in the stage sequence to update the parameters of the large language model.

[0081] In some optional implementations of the embodiment, the first training module 403 is further configured to splice the image feature and the text feature to generate the input data.

[0082] In some optional implementations of the embodiment, the first training module 403 is further configured to input the image feature to a resampler to obtain resampled image feature, where the resampler is used to align the image feature and the text feature; and splice the resampled image feature and the text feature in a sequence dimension to generate the input data.

[0083] In some optional implementations of the embodiment, the first training module 403 is further configured to back-propagate the gradient of the large language model to a first stage in the stage sequence to generate a gradient of the image feature; and back-propagate the gradient of the image feature among the video encoders in the stage sequence.

[0084] In some optional implementations of the embodiment, the second training module 404 is further configured to forward-propagate the image data in the video encoders in the stage sequence, manually back-propagate the gradient of the image feature in the video encoders in the stage sequence, and update the parameters of the video encoders in the stage sequence.

[0085] In the technical solution of the present disclosure, the acquisition, storage and application of user personal information involved comply with relevant laws and regulations and do not violate public order and good customs.

[0086] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0087] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0088] As Figure 5As shown, the device 500 includes a computing unit 501 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0089] A plurality of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, and the like; an output unit 507, such as various types of displays, speakers, and the like; a storage unit 508, such as a magnetic disk, an optical disk, and the like; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0090] The computing unit 501 can be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 501 performs various methods and processes described above, such as the large model distributed detached training method. For example, in some embodiments, the large model distributed detached training method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the large model distributed detached training method described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the large model distributed detached training method by any other appropriate means, such as by means of firmware.

[0091] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0092] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0093] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical conductors, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0095] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0096] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established by computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0097] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without departing from the desired results provided by the technical solutions of the present disclosure, and this is not limited herein.

[0098] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalent replacements, improvements, and the like made within the spirit and principles of the present disclosure should be included within the scope of the present disclosure.

Claims

1. A distributed separate training method for large models, comprising: The large language model is divided into a sequence of stages, and the video encoder is replicated at each stage of the sequence of stages. Image data is input into the video encoder in the stage sequence to generate image features; Using the image features, the large language model part in the stage sequence is trained, and during the training process, the gradient of the image features is passed back to the video encoder in the stage sequence, and the parameters of the video encoder are set to stop gradient. The video encoder in the stage sequence is trained using the image data and the gradient of the image features; The step of inputting image data into the video encoder in the stage sequence to generate image features includes: The image data is segmented along the stage dimension to generate sub-image data corresponding to each stage; The sub-image data corresponding to each stage is forward-propagated in the video encoder of the corresponding stage to generate sub-image features corresponding to each stage. Merge the sub-image features corresponding to each stage to generate the image features; The step of training the large language model portion of the stage sequence using the image features includes: Based on the image features, input data is generated; The input data is forward-propagated in the large language model portion of the stage sequence to generate the gradient of the large language model; The gradient of the large language model is backpropagated in the large language model portion of the stage sequence to update the parameters of the large language model.

2. The method according to claim 1, wherein, The process of dividing the large language model into a sequence of stages includes: By using pipelined parallelism, the large language model is divided into a sequence of stages, wherein each stage includes at least one layer of the large language model.

3. The method according to claim 1, wherein, The process of merging the sub-image features corresponding to each stage to generate the image features includes: The sub-image features corresponding to each stage are aggregated into the first stage in the stage sequence to generate the image features.

4. The method according to claim 1, wherein, The step of generating input data based on the image features includes: The input data is generated by concatenating the image features and text features.

5. The method according to claim 4, wherein, The step of concatenating the image features and text features to generate the input data includes: The image features are input into a resampler for resampling to obtain resampled image features, wherein the resampler is used to align the image features with the text features; The input data is generated by concatenating the resampled image features and the text features along the sequence dimension.

6. The method according to claim 1, wherein, The step of transmitting the gradient of the image features back to the video encoder in the stage sequence during training includes: The gradient of the large language model is backpropagated to the first stage in the stage sequence to generate the gradient of the image features; The gradient of the image features is backpropagated between the video encoders in the stage sequence.

7. The method according to claim 1, wherein, The step of training the video encoder in the stage sequence using the image data and the gradient of the image features includes: The image data is forward-propagated in the video encoder of the stage sequence, and the gradient of the image features is manually backward-propagated in the video encoder of the stage sequence to update the parameters of the video encoder of the stage sequence.

8. A large-scale distributed separable training device, comprising: The segmentation module is configured to segment the large language model into a sequence of stages and replicate the video encoder at each stage in the sequence of stages; The encoding module is configured to input image data into the video encoder in the stage sequence to generate image features; The first training module is configured to train the large language model part in the stage sequence using the image features, and to transmit the gradient of the image features back to the video encoder in the stage sequence during the training process, wherein the parameters of the video encoder are set to stop gradient. The second training module is configured to train the video encoder in the stage sequence using the image data and the gradient of the image features; The encoding module is further configured to: The image data is segmented along the stage dimension to generate sub-image data corresponding to each stage; The sub-image data corresponding to each stage is forward-propagated in the video encoder of the corresponding stage to generate sub-image features corresponding to each stage. Merge the sub-image features corresponding to each stage to generate the image features; The first training module is further configured to: Based on the image features, input data is generated; The input data is forward-propagated in the large language model portion of the stage sequence to generate the gradient of the large language model; The gradient of the large language model is backpropagated in the large language model portion of the stage sequence to update the parameters of the large language model.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images