Model training method and model training device

Abstract

The application provides a model training method and a model training device, and relates to the technical field of computers. The method comprises the following steps: obtaining target data; determining a sampling time in a training iteration, and performing disturbance sampling on the target data based on the sampling time to obtain noisy data; inputting the noisy data and the sampling time into a backbone network of an initial large language model to perform deep feature interaction and fusion, obtaining a hidden vector sequence output by the backbone network, and mapping the hidden vector sequence back to discrete codebook indexes of each modality through a decoding head to obtain predicted data; calculating a loss based on the predicted data and the target data to obtain a target loss; updating model parameters of the initial large language model based on the target loss, and obtaining a large language model that has completed training under the condition that a training termination condition is met. The application solves the problem that the prior art in the related art cannot simultaneously consider multi-modal understanding, generation and retrieval capabilities under a unified architecture.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a model training method and a model training device. Background Technology

[0002] With the continuous development of artificial intelligence technology, large-scale models are gradually expanding from the single text domain to multimodal domains such as images, audio, and video. To adapt to the needs of practical applications for various tasks such as "understanding, generation, retrieval, and multi-turn interaction," the industry has begun to build unified multimodal models that can process multimodal data simultaneously and have cross-modal collaborative capabilities. Such models typically need to support "any-to-any" input and output within a single system and maintain stability and efficient inference capabilities in complex interactive environments.

[0003] Existing solutions often design different encoders, decoders or training paradigms for understanding tasks and generation tasks respectively. Although this alleviates the conflict between different tasks to some extent, it makes it difficult to unify the feature representations of different modalities and tasks, which is not conducive to cross-modal retrieval and general feature reuse.

[0004] As can be seen from the above, the problem that existing technologies cannot simultaneously achieve multimodal understanding, generation, and retrieval capabilities under a unified architecture still needs to be solved. Summary of the Invention

[0005] This application provides a model training method and a model training device, which can solve the problem that existing technologies in related fields cannot simultaneously achieve multimodal understanding, generation, and retrieval capabilities under a unified architecture. The technical solutions are as follows: According to one aspect of this application, a model training method includes: acquiring target data; the target data being data in the form of discrete word sequence; determining a sampling time during training iterations and perturbing the target data based on the sampling time to obtain noisy data; performing deep feature interaction and fusion between the noisy data and the backbone network of an initial large language model input at the sampling time to obtain a latent vector sequence output by the backbone network, and mapping the latent vector sequence back to the discrete codebook index of each modality through a decoding head to obtain prediction data; calculating a loss based on the prediction data and the target data to obtain a target loss; updating the model parameters of the initial large language model based on the target loss, and obtaining a completed large language model when the training termination condition is met.

[0006] According to one aspect of this application, a model training apparatus includes: a data acquisition module for acquiring target data, wherein the target data is data in the form of discrete word sequence; a data noise-adding module for determining a sampling time during training iterations and perturbing the target data based on the sampling time to obtain noisy data; a data prediction module for performing deep feature interaction and fusion between the noisy data and the backbone network of an initial large language model input at the sampling time to obtain a latent vector sequence output by the backbone network, and mapping the latent vector sequence back to the discrete codebook index of each modality through a decoding head to obtain predicted data; a loss calculation module for calculating a loss based on the predicted data and the target data to obtain a target loss; and a model training module for updating the model parameters of the initial large language model based on the target loss, and obtaining a trained large language model when the training termination condition is met.

[0007] According to one aspect of this application, an electronic device includes at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, implements the model training method as described above.

[0008] According to one aspect of this application, a storage medium having a computer program stored thereon, which, when executed by one or more processors, implements the model training method as described above.

[0009] According to one aspect of this application, a computer program product includes a computer program that, when executed by one or more processors, implements the model training method as described above.

[0010] The beneficial effects of the technical solution provided in this application are: In the above technical solution, by uniformly representing the target data as discrete word sequences, a discrete stream matching training mechanism driven by sampling time is introduced, and a unified large language model backbone network is used to complete cross-modal deep fusion and prediction. While ensuring unified semantic representation and cross-modal collaborative capabilities, non-autoregressive parallel sampling and denoising generation are achieved, significantly reducing inference latency. Furthermore, the generation process is controllable and interpretable through sampling time and noise modeling mechanisms. This effectively solves the problem that existing technologies in related fields cannot simultaneously take into account multimodal understanding, generation, and retrieval capabilities under a unified architecture. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram based on the implementation environment involved in this application; Figure 2 This is a hardware structure diagram of an electronic device according to an exemplary embodiment; Figure 3 This is a flowchart illustrating a model training method according to an exemplary embodiment; Figure 4 yes Figure 3 A flowchart of step 310 in one embodiment corresponds to the following example; Figure 5 yes Figure 3 A flowchart of step 330 in one embodiment corresponds to the following example; Figure 6 yes Figure 3 A flowchart of the steps following step 390 in the corresponding embodiment is provided. Figure 7 This is a schematic diagram illustrating a specific implementation of output data in an application scenario; Figure 8 This is a structural block diagram of a model training device according to an exemplary embodiment; Figure 9 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0013] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0014] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0015] As mentioned earlier, it is difficult to unify the feature representations of different modalities and different tasks in the existing technology, which is not conducive to cross-modal retrieval and reuse of general features.

[0016] Existing multimodal large-scale models mainly evolve along two technical paths: First, based on the large language model architecture, multimodal understanding and generation are achieved by discretizing different modal data into a unified word sequence and using an autoregressive prediction paradigm. This type of method has a relatively simple structure and is easy to reuse the language model ecosystem, but it relies on word-by-word decoding, has a slow inference speed, and the requirements for information granularity between understanding and generation tasks conflict, which can easily lead to a decrease in performance after unification.

[0017] Secondly, there is an attempt to break free from autoregressive dependence by constructing a unified modeling framework based on discrete diffusion or discrete flow matching, reducing generation latency through parallel sampling. However, this approach is limited by the capabilities of the language's underlying model and the maturity of engineering optimization; its overall speed stability, multi-turn interaction capabilities, and generalization ability still struggle to meet the demands of large-scale applications.

[0018] In addition, to balance multi-task performance, some solutions adopt a "multi-encoder / multi-decoder" or "expert hybrid (MoE / MoT)" structure to decouple the understanding and generation tasks. While this alleviates the task conflict problem to some extent, it leads to a more complex model structure, increased inference overhead, and higher deployment costs. Furthermore, it weakens the ability to achieve unified representation across modalities and reuse features across tasks, making it difficult to efficiently perform tasks such as retrieval and cross-modal matching.

[0019] At the reasoning and interaction levels, existing multimodal models also have significant limitations in representation space and generation mechanisms. On the one hand, different modalities and tasks typically employ independent encoding methods and feature spaces, making it difficult to construct stable and consistent joint representations, thus affecting cross-modal retrieval performance and semantic alignment. On the other hand, autoregressive generation relies on word-by-word decoding during the reasoning stage, resulting in slow overall reasoning speed. While discrete diffusion-based schemes introduce parallel sampling mechanisms, they are still limited by sampling efficiency and the capabilities of the underlying language model, making it difficult to balance speed and performance in large-scale applications. Furthermore, existing schemes employ relatively crude methods for controlling generation length, and the sampling process generally lacks interpretability and controllability, making it difficult to dynamically adjust generation strategies according to task requirements.

[0020] As can be seen from the above, the existing technologies still have the drawback that they cannot simultaneously take into account the multimodal understanding, generation and retrieval capabilities under a unified architecture.

[0021] Therefore, the model training method provided in this application can effectively improve the accuracy of model training. Accordingly, the model training method is applicable to a model training device, which can be deployed on an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, such as a desktop computer, a laptop computer, a server, etc.

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0023] Figure 1 This is a schematic diagram of an implementation environment involved in a model training method. It should be noted that this implementation environment is merely an example adapted to the present invention and should not be considered as providing any limitation on the scope of the invention.

[0024] The implementation environment includes a data acquisition terminal 110 and a server terminal 130.

[0025] Specifically, the acquisition terminal 110 can also be considered a data acquisition device, including but not limited to electronic devices with multimedia data acquisition functions such as cameras, camcorders, and recorders. For example, the acquisition terminal 110 is a camera.

[0026] Server 130 can be an electronic device such as a desktop computer, laptop computer, or server, or it can be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide backend services, such as, but not limited to, model training services.

[0027] The server 130 and the acquisition terminal 110 establish a network communication connection in advance via wired or wireless means, and data transmission between the server 130 and the acquisition terminal 110 is realized through this network communication connection. The transmitted data includes, but is not limited to, raw data.

[0028] In one application scenario, the acquisition terminal 110 collects raw data and transmits the raw data to the server terminal 130. The raw data includes at least one of image data, video data, and audio data.

[0029] The server 130 acquires the raw data transmitted from the acquisition terminal 110, converts the raw data into a discrete word sequence, and inserts padding words into the discrete word sequence to make the length of the inserted discrete word sequence an integer multiple of the preset block length, thus obtaining the target data. During the training iteration, the sampling time is determined, and the target data is perturbed and sampled based on the sampling time to obtain noisy data. The noisy data and the sampling time are input into the backbone network of the initial large language model for deep feature interaction and fusion to obtain the latent vector sequence output by the backbone network. The latent vector sequence is then mapped back to the discrete codebook index of each modality through the decoding head to obtain the predicted data. The loss is calculated based on the predicted data and the target data to obtain the target loss. The model parameters of the initial large language model are updated based on the target loss, and the trained large language model is obtained when the training termination condition is met.

[0030] Please see Figure 2 , Figure 2 This is a hardware structure diagram of an electronic device according to an exemplary embodiment. This electronic device is suitable for... Figure 1 The server 130 in the implementation environment is shown.

[0031] It should be noted that this electronic device is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Furthermore, this electronic device should not be interpreted as requiring or depending on any specific feature. Figure 2 One or more components of the exemplary electronic device 200 shown.

[0032] The hardware structure of electronic device 200 can vary significantly due to differences in configuration or performance, such as... Figure 2 As shown, the electronic device 200 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0033] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 200.

[0034] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. For example, to perform... Figure 1 The diagram shows the interaction between the acquisition terminal 110 and the server terminal 130 in the implementation environment.

[0035] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 2 As shown, this does not constitute a specific limitation.

[0036] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0037] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 200, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0038] Application 253 is a computer program formed by computer-readable instructions based on operating system 251 to perform at least one specific task, and may include at least one module ( Figure 2 (Not shown), each module can contain corresponding computer-readable instructions. For example, the model training device can be considered as an application 253 deployed on electronic device 200.

[0039] Data 255 can be photos, pictures, etc. stored on a disk, or it can be target data, raw data, etc., stored in memory 250.

[0040] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer programs stored in the memory 250, thereby enabling the computation and processing of massive amounts of data 255 in the memory 250. For example, a model training method may be implemented by the central processing unit 270 reading an application program 253 stored in the memory 250.

[0041] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.

[0042] Please see Figure 3This application provides a model training method applicable to electronic devices, such as electronic devices that can be... Figure 1 The server 130 in the implementation environment is shown. The hardware structure of this electronic device can be as follows: Figure 2 As shown.

[0043] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0044] like Figure 3 As shown, the method may include the following steps: Step 310: Obtain the target data.

[0045] The target data is data in the form of discrete word sequence; discrete word sequence form can refer to the original data (such as image data / video data / audio data / text data) after discretization processing, which is represented as a data form composed of multiple discrete words in sequence.

[0046] In one embodiment, such as Figure 4 As shown, step 310 may also include the following steps: Step 311: Obtain the raw data.

[0047] Step 313: Convert the original data into a discrete word sequence, and insert padding words into the discrete word sequence so that the length of the inserted discrete word sequence is an integer multiple of the preset block length, thus obtaining the target data.

[0048] The raw data includes at least one of the following: text data, image data, video data, and audio data.

[0049] It is understandable that raw data in different modalities (i.e., text, images, video, and audio) carries different dimensions of information. Text data is convenient for expressing instructions, conditions, semantic tags, or contextual cues; image data is used to express static visual content and spatial structure; video data is used to express action processes and spatiotemporal relationships that change over time; and audio data is used to express speech content or environmental acoustic features.

[0050] By allowing the aforementioned multimodal raw data as input, multimodal tasks can be supported within the same training framework. This enables the subsequently trained large language model to handle both text-to-text understanding and generation, as well as cross-modal tasks, thereby expanding the model's applicability and enhancing its comprehensive modeling capabilities for complex scenarios.

[0051] To achieve a unified representation of multimodal raw data for subsequent model training, raw data from different modalities can be uniformly represented as discrete word sequences that can be serialized, enabling them to be processed by the backbone network of large language models using a unified sequence modeling approach.

[0052] Specifically, the discrete word sequence is composed of multiple discrete words in sequence; wherein each discrete word is an index value selected from a preset discrete word list.

[0053] In one embodiment, the discrete lexicon includes a set of discrete codebook indexes of a VQVAE quantizer (Understanding Variational Quantization Autoencoder).

[0054] By converting the original data into discrete word sequences, on the one hand, large language models can output predicted data based on noisy data and sampling time during training iterations, and calculate the cross-entropy loss between the predicted data and the target data to obtain the target loss, thereby updating the model parameters. On the other hand, it enables perturbation sampling of the target data to be implemented by "replacing discrete word indices," and iterative denoising processing can be performed during the inference phase by "updating the current input sequence based on the predicted data," thus obtaining a denoised discrete word sequence, which is further reconstructed through a modal decoder to obtain the output data. Based on the above unified discrete word sequence representation, data from different modalities can be used as target data input into the backbone network for deep feature interaction and fusion, supporting cross-modal training and generation.

[0055] Regarding padding lexical units, these can be pre-defined special discrete lexical units used to pad the length of the discrete lexical sequence without changing its semantic content. The pre-defined block length can be a pre-set number of lexical units, which can be used to limit the number of lexical units contained in each pre-defined block, so that the backbone network can perform calculations and cache management on a block-by-block basis.

[0056] Since backbone networks typically batch process discrete word sequences with fixed block sizes and utilize cache reuse for acceleration, if the lengths of different discrete word sequences do not meet the preset block length alignment, it will lead to inconsistent block numbers, inconsistent batch processing tensor shapes, and complex cache management. This can be addressed by inserting preset padding words (e.g., ...) at the end of the discrete word sequence. <pad>This allows the length of the padded discrete word sequence to become an integer multiple of the preset block length, thereby achieving batch sample alignment, improving matrix operation efficiency, and reducing the difficulty of implementing block inference and cache reuse.

[0057] It should be noted that the raw data can be converted into a discrete word sequence by a modal encoder after the preheating is completed.

[0058] The significance of pre-training lies in ensuring that the module converting raw data into discrete word sequences (i.e., the modal encoder) possesses stable and usable discrete representation capabilities before entering the initial training of the large language model. Through pre-training, the modal encoder and VQVAE quantizer can output information-accurate and semantically aligned discrete word sequences. This allows subsequent processes in the large language model training, such as perturbation sampling, prediction data output, loss supervision, iterative denoising generation, and output reconstruction, to be built on a reliable discrete word space, avoiding training convergence difficulties or degraded generation quality due to discretization instability. Furthermore, the pre-trained discrete word sequences are more easily aligned with the textual description data in the semantic space, making the deep feature interaction and fusion of the backbone network more effective and improving the overall performance of cross-modal training and inference.

[0059] Specifically, in one embodiment, the steps prior to step 313 may include: obtaining text description data corresponding to the original data; constructing a modal encoder for generating discrete word sequences, and introducing an auxiliary VQVAE quantizer and modal decoder; converting the original data into discrete word sequences via the modal encoder and VQVAE quantizer; reconstructing the discrete word sequences using the modal decoder and calculating the reconstruction loss; obtaining a pre-trained auxiliary semantic model, and inputting the text description data and discrete word sequences into the auxiliary semantic model for semantic representation, respectively, to obtain text semantic representation and modal semantic representation; calculating semantic alignment loss based on the text semantic representation and modal semantic representation; updating the parameters of the modal encoder based on the reconstruction loss and semantic alignment loss to obtain the modal encoder after preheating.

[0060] First, it should be noted that text description data is textual information paired with the original data, used to provide semantic constraints and supervision signals for the original data. Text description data can be used to characterize the content summary, keywords and phrases, natural language descriptions, or transcribed text of the original data, thereby guiding the discrete word sequence output by the modality encoder during the warm-up phase to have the characteristics of being interpretable by language semantics.

[0061] In one embodiment, the sources of text description data may include: text description text pre-annotated from a dataset; text description text generated from user input; speech-to-text output by a speech recognition system when the original data is audio data; and titles / labels / descriptions generated by an annotation system when the original data is image / video data.

[0062] Furthermore, the modal encoder is used to map the original data into continuous feature representations, the VQVAE quantizer is used to discretize the continuous feature representations into discrete codebook indices, and the modal decoder is used to restore the discrete codebook indices into the reconstructed original data, thus forming a closed-loop structure of encoding-quantization-decoding.

[0063] In one embodiment, the construction method may include: selecting a suitable network structure as a modal encoder and modal decoder for different modalities of the original data. For example, a convolutional network or visual Transformer structure may be used for image data / video data, and a time-frequency convolutional network or sequence network structure may be used for audio data.

[0064] A VQVAE quantizer is introduced, which has a discrete codebook index set and corresponding codebook vectors. It can match and map the continuous features output by the modal encoder to the discrete codebook index set. The modal decoder then reconstructs approximately the original data based on the discrete codebook indexes and their corresponding codebook vectors.

[0065] Therefore, the modal encoder is characterized by extracting continuous modal feature representations, the VQVAE quantizer is characterized by mapping continuous feature representations to discrete codebook indices, and the modal decoder is characterized by reconstructing the original data from the discrete codebook indices.

[0066] To elaborate further, the purpose of calculating the reconstruction loss is to constrain the information fidelity of the discretization process, preventing the discrete word sequence output by the modal encoder and VQVAE quantizer from losing too much detailed information related to the original data, thus enabling the discrete word sequence to support subsequent reconstructed output data. In other words, the reconstruction loss can be used to ensure that the discrete word sequence is reconstructable.

[0067] In one embodiment, the reconstruction loss is calculated as follows: Among them, among them, To restore the loss, To quantify the loss, This is due to discriminator loss.

[0068] The purpose of calculating semantic alignment loss is to constrain the high-level semantic consistency of discrete word sequences, so that the discrete word sequences can not only reconstruct morphologically similar data, but also match the meaning expressed by the textual description data at the semantic level. This gives the discrete word sequences a semantic foundation that allows them to be aligned and interacted across modalities. In other words, semantic alignment loss is used to ensure that the discrete word sequences are semantically interpretable and alignable.

[0069] The transformation process of the raw data can be divided into two stages: continuous feature extraction and discretization. Specifically, the raw data is first encoded using a modal encoder to obtain continuous feature representations. Then, these continuous feature representations are input into a VQVAE quantizer, which determines matching codebook vectors for the continuous feature representations from a set of discrete codebook indices and outputs the corresponding discrete codebook index sequence. This discrete codebook index sequence constitutes a discrete word sequence. Thus, raw data from different modalities can be uniformly transformed into target data in the form of discrete word sequences, laying the foundation for unified processing in the subsequent large language model backbone network.

[0070] Auxiliary semantic models can be used to map input data into semantic representations. Semantic representations can be understood as semantic features in the form of continuous vectors, used to represent the position of input data in semantic space, thus facilitating the measurement of semantic similarity between different input data.

[0071] In one embodiment, the auxiliary semantic model may include a text encoder and / or a text decoder: for example, when the original data is image data / video data and the text description data is natural language description, the auxiliary semantic model may be a cross-modal model containing a text encoder, used to output text semantic representation and modal semantic representation respectively; when the original data is audio data and the text description data is transcribed text, the auxiliary semantic model may be a model containing a text decoder, used to generate semantic representation based on the discrete word sequence corresponding to the audio data and align it with the text semantic representation.

[0072] Textual semantic representation refers to the vectorized feature representation of the semantic information of the textual description data, output by the pre-trained auxiliary semantic model after inputting textual description data. Modal semantic representation refers to the vectorized feature representation of the semantic information of the original data corresponding to the discrete word sequence, output by the auxiliary semantic model after inputting the discrete word sequence. Both textual semantic representation and modal semantic representation can be feature representations in vector form, which are subsequently used to calculate semantic alignment loss.

[0073] The reconstruction loss and semantic alignment loss are combined to obtain the warm-up target loss, and the parameters of the modal encoder are optimized and updated based on the warm-up target loss. Specifically, in one warm-up iteration, the reconstruction loss and semantic alignment loss are calculated based on the discrete word sequence output by the current modal encoder; then, the two are weighted and summed to obtain the warm-up target loss; finally, the modal encoder parameters are updated by gradient based on the warm-up target loss, so that the modal encoder can reduce reconstruction error and improve semantic consistency when outputting discrete word sequences in subsequent iterations.

[0074] Step 330: Determine the sampling time during the training iteration, and perform perturbation sampling on the target data based on the sampling time to obtain noisy data.

[0075] First, it should be noted that training iterations can be used to indicate the cyclical process of updating the model parameters of the initial large language model. Specifically, in each training iteration, the initial large language model selects a batch of target data, performs noise addition, feature interaction, prediction, and loss calculation on this batch of target data based on the current model parameters of the initial large language model, and updates the model parameters once based on the obtained target loss. After multiple training iterations, the model parameters of the initial large language model gradually converge, and finally, a fully trained large language model is obtained.

[0076] Regarding sampling time, it can be used to characterize the time step or noise level at which a disturbance is applied to the target data. The sampling time can be a discrete time step index or a real value within a continuous interval (e.g., the interval [0,1]), and can be mapped to the corresponding disturbance intensity parameter through a preset noise scheduling function.

[0077] During training iterations, sampling times can be randomly sampled based on a preset distribution, allowing the initial large language model to learn the ability to reconstruct target data from noisy data under different perturbation intensities. Perturbation sampling can be used to indicate the process of applying random perturbations to the discrete word sequence corresponding to the target data.

[0078] Noisy data can refer to a discrete word sequence obtained by applying perturbation sampling to the discrete word sequence corresponding to the target data. Compared with the target data, the discrete codebook indices at some word positions in the noisy data have been replaced by replacement indices obtained by sampling the state transition probability distribution, thus introducing controlled random noise. The noisy data serves as the actual input to the initial large language model, and can be used to train the initial large language model to recover the target data from the noisy state under given sampling time conditions.

[0079] In one embodiment, such as Figure 5 As shown, step 330 may also include: Step 331: Obtain the discrete codebook index set of the VQVAE quantizer and the codebook vector corresponding to each discrete codebook index in the discrete codebook index set.

[0080] First, it should be noted that the discrete codebook index set can be used to indicate the set of indices used in the VQVAE quantizer to represent the discrete representation of the feature space. Specifically, after training (i.e., warm-up training), the VQVAE quantizer pre-stores multiple codebook vectors, each corresponding to a unique discrete codebook index. The discrete codebook index set is composed of these discrete codebook indices. Codebook vectors can be used to represent the distribution center position of specific semantic or content units in the feature space, and are the basis for mapping continuous features to discrete tokens.

[0081] Furthermore, by obtaining the discrete codebook index set of the VQVAE quantizer and the codebook vector corresponding to each discrete codebook index, it is possible to construct a state transition probability space on this finite and stable discrete set. This provides a data foundation for calculating the state transition probability based on cosine distance in the following steps, thereby ensuring that perturbation sampling is carried out in a controlled, finite, and interpretable discrete space.

[0082] Step 332: Calculate the cosine distance between any two discrete codebook indices in the discrete codebook index set based on the codebook vector, and determine the state transition probability distribution between discrete codebook indices based on the cosine distance.

[0083] Cosine distance can be used to measure the similarity between different codebook vectors in the vector space. Specifically, for any two discrete codebook indices in the discrete codebook index set, their corresponding codebook vectors can be obtained, and the cosine distance between them can be calculated based on the codebook vectors. If the cosine distance is small, it indicates that the two have high similarity in the semantic space or feature space; if the cosine distance is large, it indicates that the two have large differences.

[0084] In one embodiment, the state transition probability distribution is as follows: Wherein, the sampling time is t, The noise intensity coefficient corresponding to the sampling time t; Represents the position index in the sequence, such as the position of the i-th word; This represents the state transition probability distribution. The cosine distance is calculated based on their respective codebook vectors; It is the discrete codebook index of the lexical position in the target data. It is the discrete codebook index of the i-th term position in the discrete codebook index set of the VQVAE quantizer.

[0085] Based on the aforementioned cosine distance, a corresponding state transition probability distribution can be constructed for any discrete codebook index. Specifically, the cosine distance between the codebook vector corresponding to the discrete codebook index and other codebook vectors can be normalized to obtain the state transition probabilities from the discrete codebook index to other discrete codebook indices. In this way, discrete codebook indices with similar semantics or close spatial distances are assigned larger transition probabilities, while discrete codebook indices with significant semantic differences are assigned smaller transition probabilities.

[0086] Therefore, by calculating the cosine distance in the codebook vector space and determining the state transition probability distribution, fine control over the direction and amplitude of perturbation sampling can be achieved. This ensures that the perturbation sampling performed based on the sampling time in subsequent steps is not only random but also semantically reasonable, thereby improving the effectiveness and training stability of the noisy data.

[0087] Step 333: Determine the perturbation intensity parameter based on the sampling time, and determine the set of word positions to be perturbed in the discrete word sequence corresponding to the target data according to the perturbation intensity parameter.

[0088] First, it should be noted that the perturbation intensity parameter can be used to characterize the magnitude of noise injected into the target data at the current sampling time. A pre-defined correspondence can exist between the sampling time and the perturbation intensity parameter. For example, a noise scheduling function can map a smaller sampling time to a weaker perturbation intensity parameter, ensuring that only a smaller proportion of word positions are selected as perturbation positions, and that the state transition probability is more biased towards indices semantically similar to the original discrete codebook index. Conversely, a larger sampling time can be mapped to a stronger perturbation intensity parameter, increasing the proportion of perturbed word positions and making the state transition probability closer to a uniform distribution. Thus, different sampling times result in different levels of noise in the data, enabling the model to learn its ability to recover the target data under different noise levels during training iterations.

[0089] Furthermore, the set of word positions in the discrete word sequence corresponding to the target data that need to be injected with noise can be determined based on the perturbation intensity parameter. For example, the perturbation intensity parameter can be mapped to a target proportion of the positions to be perturbed, and then random sampling can be performed at each word position in the discrete word sequence according to the target proportion to determine the set of word positions to be perturbed; or, the number of words to be perturbed can be determined based on the perturbation intensity parameter, and a corresponding number of word positions can be randomly selected in the sequence as the positions to be perturbed.

[0090] By using the above method, the set of word positions to be disturbed can be adaptively determined at different sampling times, thereby achieving fine control over the disturbance range.

[0091] Step 334: For any word position in the word position set, extract the original discrete codebook index at the word position from the discrete word sequence corresponding to the target data.

[0092] In the discrete word sequence corresponding to the target data, each word position stores an original discrete codebook index, which indicates the value of that position in the discrete codebook index set. For the set of word positions to be perturbed determined in the preceding steps, the currently stored discrete codebook index at any word position in the set can be read from the discrete word sequence corresponding to the target data, and this index can be used as the original discrete codebook index.

[0093] Step 335: Determine the replacement discrete codebook index based on the state transition probability distribution corresponding to the original discrete codebook index.

[0094] First, it should be noted that in the preceding steps, the cosine distance between any two discrete codebook indices in the discrete codebook index set has been calculated based on the codebook vector, and the state transition probability distribution between discrete codebook indices has been determined based on the cosine distance. For any original discrete codebook index, the probability distribution of transitioning from the original discrete codebook index to each candidate discrete codebook index can be obtained from this state transition probability distribution.

[0095] Based on this, for the original discrete codebook index, random sampling or maximum probability selection can be performed on the discrete codebook index set of the VQVAE quantizer, using the state transition probability distribution corresponding to the original discrete codebook index as a basis, to obtain the replacement discrete codebook index. In this way, discrete codebook indices that are semantically closer to the original discrete codebook index will receive a higher sampling probability, while discrete codebook indices that differ significantly in semantics will have a lower sampling probability, thus maintaining a certain semantic relevance while introducing random perturbations.

[0096] Step 336: Replace the original discrete codebook index at the word position with the replacement discrete codebook index to generate noisy data.

[0097] First, it should be noted that after obtaining the replacement discrete codebook index, the original discrete codebook index at the position of the word to be perturbed can be replaced by the replacement discrete codebook index. Specifically, for each word position in the set of word positions to be perturbed, steps 334 and 335 are executed sequentially: first, the original discrete codebook index is extracted from the discrete word sequence corresponding to the target data; then, the replacement discrete codebook index is determined according to the state transition probability distribution corresponding to the original discrete codebook index; and the replacement discrete codebook index is written into the word position, thereby completing the noise addition operation at the word position.

[0098] Once the above replacements have been completed for each word position in the set of word positions to be disturbed, a new discrete word sequence can be obtained. The new discrete word sequence has been injected with noise generated according to the state transition probability distribution at some word positions relative to the original target data. This new discrete word sequence is the noisy data.

[0099] By taking the above steps, while keeping the overall structure of the discrete word sequence unchanged, the local discrete words can be perturbed in a controlled manner. This allows the noisy data to contain sufficient randomness to drive discrete stream matching training, while maintaining a certain degree of semantic rationality through state transition probability constraints. This improves the stability of the training process and the robustness of the model to different noise levels.

[0100] Step 350: The noisy data and the sampling time are input into the backbone network of the initial large language model for deep feature interaction and fusion to obtain the latent vector sequence output by the backbone network. The latent vector sequence is then mapped back to the discrete codebook index of each modality through the decoding head to obtain the prediction data.

[0101] First, it should be noted that the initial large language model can include a backbone network and decoders corresponding to different modalities. As the core feature modeling structure, the backbone network can establish long-range dependencies and contextual semantic relationships based on the input sequence, thereby outputting a sequence of latent vectors that can fully represent the semantic and structural features of the input data.

[0102] It should be understood that by inputting noisy data as a discrete word sequence into the backbone network and simultaneously inputting the sampling time as conditional information into the backbone network, the backbone network can adaptively adjust its internal feature parsing method based on the perturbation intensity corresponding to different sampling times when performing sequence modeling. This enables robust modeling and representation learning of noisy data under different noise levels.

[0103] The backbone network, through a Transformer structure, performs deep feature interaction and fusion between noisy data and sampling time, ultimately outputting a sequence of latent vectors. This sequence of latent vectors can be used to characterize the denoised state and potential true distribution of noisy data in the semantic space.

[0104] Subsequently, the latent vector sequence is input into the decoder head corresponding to the modality (i.e., the modality corresponding to the original data). The decoder head can be used to map the latent vector sequence back to the discrete codebook index corresponding to each modality, outputting the predicted data. The predicted data can be used to characterize the prediction results of the initial large language model for each discrete codebook index of the target data under the current sampling time conditions, and can be output in the form of probability distribution or maximum likelihood index.

[0105] Through the above process, the initial large language model is equipped with the ability to perform conditional recovery and reconstruction of noisy data under different noise conditions, thus providing a foundation for subsequent loss calculation and parameter updates.

[0106] Step 370: Calculate the loss based on the predicted data and the target data to obtain the target loss.

[0107] It is understandable that a loss function constructed based on the difference between the predicted data and the target data can be used to measure the prediction accuracy of the initial large language model in the current training iteration. The target data corresponds to the true discrete word sequence, while the predicted data corresponds to the discrete codebook index prediction result output by the model under noisy input conditions. Based on the degree of matching between the two, the prediction error can be calculated, and a supervised training signal can be further constructed.

[0108] In one embodiment, the cross-entropy loss is calculated between the predicted data and the target data, and the reconstruction loss is reused as a regularization term to obtain the target loss.

[0109] Specifically, cross-entropy loss can be calculated for the predicted and target data to characterize the probability difference between the model's predicted distribution and the actual discrete codebook index at each word position. Simultaneously, the reconstruction loss obtained in the aforementioned warm-up stage can be reused as a regularization term to impose additional constraints on the reconstruction quality of the initial large language model output, thereby suppressing model degradation and improving the model's generalization ability in multimodal data recovery tasks.

[0110] In one embodiment, the target loss is as follows: Wherein, the cross-entropy loss is The reconstruction loss is .

[0111] Step 390: Update the model parameters of the initial large language model based on the target loss, and obtain the large language model that has completed training if the training termination condition is met.

[0112] In this embodiment, backpropagation and gradient optimization algorithms (such as SGD, Adam, or other optimization algorithms) can be used to update the backbone network parameters and decoder head parameters of the initial large language model based on the target loss, so that the target loss is gradually reduced.

[0113] During training, steps 330 to 390 can be repeated to form multiple training iterations. As the training iterations continue, the initial large language model gradually learns the noise perturbation patterns and input-output mapping relationships corresponding to different sampling times, thereby gaining the ability to recover target data under different noise levels.

[0114] The training termination condition can be used to stop the training of the initial large language model. In one embodiment, the training termination condition may be reaching a preset number of training rounds, the target loss being lower than a preset threshold, and / or the target loss changing less than a preset convergence threshold in several consecutive training rounds. No specific limitation is made here.

[0115] When the training termination condition is met, the initial large language model can be considered to have completed training, and the trained large language model can be used in the subsequent inference stage to perform discrete word sequence recovery, modality reconstruction and multimedia data generation.

[0116] In one embodiment, step 390 may be followed by the following steps: acquiring noise data; constructing an initial input sequence with a preset block length based on the noise data; inputting the initial input sequence into a large language model for iterative denoising processing, outputting prediction data in each iteration step, and updating the current input sequence based on the prediction data to obtain a denoised discrete word sequence; reconstructing the denoised discrete word sequence through a modal decoder to obtain output data.

[0117] In this context, noisy data can be used to indicate the initial input state of a large language model during the inference phase. Noisy data can be any type of data. By inputting noisy data into the model as a starting point, the trained large language model can gradually generate a target discrete word sequence that conforms to the true semantic structure, based on its ability to "recover real data from noise" learned during training.

[0118] In one embodiment, the noisy data can be provided by the user terminal to indicate to the user the multimedia data to be repaired, enhanced, or reconstructed. For example, the noisy data can be image data, video data, or audio data containing noise, occlusion, compression artifacts, or background noise, or a multimedia segment specified by the user that is partially missing or has multiple breaks. For this type of user-provided noisy data, it can first be converted into a discrete word sequence by a modal encoder and a VQVAE quantizer after preheating, and the converted discrete word sequence can be used as the basis for constructing the initial input sequence, so that the trained large language model can perform iterative denoising and reconstruction under the noise input conditions given by the user.

[0119] Specifically, continuous word segments can be selected from noisy data, or a random sequence with a preset block length (e.g., a fixed number of word lengths) can be directly constructed as the initial input sequence. By limiting the block length, the large language model can process only sequences of finite length in the initial stage of inference, which facilitates the gradual introduction of more semantic information in the subsequent denoising and expansion processes, thus enabling flexible modeling of variable-length sequences.

[0120] The current input sequence is the input sequence obtained by updating the initial input sequence through the iteration steps before the current iteration step.

[0121] In this embodiment, an initial input sequence can be input into a large language model, and iterative denoising processing can be performed. In each iteration step, the large language model outputs corresponding predicted data based on the current input sequence. The predicted data can represent the large language model's estimation result of the true discrete word distribution in the current iteration step.

[0122] Subsequently, the current input sequence can be updated based on the predicted data, gradually converging from a noisy state to a true semantic structure. As the iteration process continues, the large language model continuously uses its internal semantic modeling capabilities to correct and supplement the sequence, ultimately obtaining a denoised discrete word sequence.

[0123] In one embodiment, in the current iteration step, the confidence level corresponding to the terminator is determined based on the predicted data. If the confidence level does not exceed a first preset threshold, a block-sized word length is appended to the end of the current input sequence to obtain the input sequence for the next iteration step. If the confidence level exceeds the first preset threshold, the position corresponding to the terminator is determined, and the current input sequence is truncated at the terminator, outputting the truncated discrete word sequence as the denoised discrete word sequence.

[0124] The terminator can be a special discrete codebook index that is reserved in advance in the discrete codebook index set, used to indicate that the valid content of the current sequence has ended; when the terminator is predicted at a certain word position, it can be considered that the word after that position no longer belongs to meaningful generated content.

[0125] Specifically, in the current iteration step, the large language model outputs prediction data based on the current input sequence. The prediction data may include the discrete codebook index prediction distribution corresponding to each word position in the current input sequence.

[0126] In one embodiment, the probability value of "predicted as end symbol" at each word position can be extracted from the prediction data, and the confidence level corresponding to the end symbol can be calculated based on these probability values. For example, the maximum value of the predicted end symbol probability among all word positions, or the maximum value or weighted average value of the predicted end symbol probability at several positions near the end of the sequence, can be taken as the end symbol confidence level for the current iteration step.

[0127] In one embodiment, a first set threshold is used to determine whether the confidence level of the terminator reaches the trigger truncation condition, and its value can be set according to the statistical results or empirical rules in the initial large language model training process.

[0128] For example, during the model training phase, statistical analysis can be performed on the probability distribution of the predicted terminator of a large amount of target data under different iteration steps to obtain the probability range when the terminator is predicted correctly and the probability range when it is predicted incorrectly. Based on the above statistical distribution, a probability boundary value that can significantly distinguish between the two states of "reliable terminator prediction" and "unreliable terminator prediction" can be selected as the first set threshold.

[0129] If the confidence score of the terminator does not exceed the first preset threshold, it can be considered that the large language model is not yet certain that the current sequence has been successfully generated, and it is still necessary to reserve more lexical positions for subsequent content. In this case, a block-sized lexical length can be appended to the end of the current input sequence to obtain the input sequence for the next iteration. The newly added block can be filled with preset noise lexicals, placeholder lexicals, or initial lexicals generated based on noise data, so that the trained large language model can denoise and complete the content at the newly added lexical positions in subsequent iterations. At the same time, the existing lexical positions in the current input sequence can continue to be updated based on the prediction data, thereby gradually refining the semantics and structure of the entire current input sequence in multiple iterations.

[0130] If the confidence score of the terminator exceeds a first set threshold, it can be considered that the large language model has sufficient confidence in determining the end position of the current input sequence. In this case, the word position corresponding to the terminator can be determined based on the prediction data, that is, the word position with the highest predicted terminator probability in the current input sequence is selected as the end position. Subsequently, the current input sequence is truncated at the position corresponding to the terminator, and the discrete word sequence before the terminator (optionally retaining the terminator itself) is taken as the valid part, and the truncated discrete word sequence is output as the denoised discrete word sequence.

[0131] By performing the above truncation operation, we can effectively remove tail words that may still be in a noisy state or have no semantic meaning after the terminator, avoid generating excessively long and redundant sequences, and thus improve the effectiveness of the generated results and the efficiency of reasoning.

[0132] Therefore, by dynamically evaluating the confidence of the terminator based on the predicted data in each iteration step, and adding the block-level word length when the confidence is insufficient and truncating the current input sequence when the confidence is sufficient, adaptive generation and termination control of variable-length discrete word sequences can be achieved. This allows the trained large language model to flexibly expand the generation length and stop generation at the appropriate time to output stable denoising results.

[0133] Regarding the output data, it can be multimedia data corresponding to the modal decoder, and the multimedia data includes at least one of image data, video data, and audio data.

[0134] It should be noted that the denoised discrete word sequence is a discrete representation consistent with the target data and cannot be directly provided to the user. Therefore, in this embodiment, the denoised discrete word sequence can be input into the modality decoder of the corresponding modality.

[0135] A modal decoder can map discrete word sequences back to continuous modal space, reconstructing the output data. Thus, a trained large language model can not only learn discrete word-level denoising capabilities during training, but also achieve a complete generation process from noisy input to usable real modal data during inference.

[0136] In one embodiment, the iterative denoising process further includes adaptive caching acceleration, such as... Figure 6 As shown, adaptive caching acceleration can include the following steps: Step 610: Based on the word positions of the current input sequence, divide it into a set of word positions for the instruction part and a set of word positions for the response part.

[0137] The instruction part corresponds to prompts or conditional terms, such as system instructions, user instructions, and context prefixes. This part usually remains unchanged or changes little during the iterative denoising process. The response part corresponds to the terms to be generated, which are used to represent the content that the model needs to complete or reconstruct. Its hidden features will change significantly in multiple iterations.

[0138] By dividing the word positions as described above, differentiated caching and reuse strategies can be adopted for different parts in subsequent steps, avoiding repeated calculations of stable and unchanging instruction parts, thereby reducing computational overhead.

[0139] Step 620: In each iteration step, calculate the cosine similarity between the hidden features corresponding to the set of word positions in the response part in the current iteration step and the hidden features cached in the previous iteration step.

[0140] Among them, the hidden layer features are the sequence of hidden vectors output by the backbone network of the large language model at the corresponding iteration step.

[0141] Specifically, the hidden layer features are the sequence of latent vectors output at the word positions in the response part after the backbone network of the trained large language model performs forward computation on the current input sequence in the corresponding iteration step; the cached hidden layer features are the sequence of latent vectors obtained and retained at the same word positions in the previous iteration step. By calculating the cosine similarity between the current hidden layer features and the cached hidden layer features for each word position in the response part, the degree of change in the feature representation at that position in two adjacent iteration steps can be characterized, providing a basis for subsequent determination of whether to reuse the cached hidden layer features or perform feature updates.

[0142] Step 630: For the set of lexical positions belonging to the instruction part, reuse the hidden features cached in the previous iteration step.

[0143] Specifically, for each word position in the instruction part, since the corresponding content remains essentially unchanged throughout the entire iterative denoising process, the changes in its hidden features between adjacent iterations are usually minimal. Therefore, the hidden features cached from the previous iteration can be directly used without re-executing the forward computation of the backbone network. By fully reusing the hidden features of the instruction part, redundant computation in this part can be significantly reduced, improving overall inference efficiency.

[0144] Step 640: For the set of lexical positions belonging to the response part, if the cosine similarity exceeds the second set threshold, the hidden features cached in the previous iteration step are reused.

[0145] The second threshold is used to determine whether the hidden features of the response part have changed sufficiently between two adjacent iterations. Specifically, when the cosine similarity between the latent vector sequences of the current and previous iterations of a certain word position in the response part is higher than the second threshold, it indicates that the feature change at that word position is small. Even if the forward result is recalculated, its output is highly close to the cached hidden features. In this case, the cached hidden features from the previous iteration can be directly reused without performing forward calculation at that position, thereby reducing unnecessary redundant calculations.

[0146] The second threshold can be set based on the statistical results of the model on the validation set. For example, a similarity threshold that can distinguish between "hidden features remain basically unchanged" and "hidden features have changed effectively" can be selected. Alternatively, it can be set empirically based on the trade-off between speed and accuracy in actual applications.

[0147] Step 650: When the cosine similarity does not exceed the second set threshold, perform forward computation on the word position set of the response part to obtain the updated hidden features, and replace the cached hidden features with the updated hidden features.

[0148] Forward computation can be used to instruct that, given the current input sequence and the model parameters of the large language model, the corresponding word position is input into the backbone network of the large language model, and the operations of each layer are executed sequentially according to the preset network structure to obtain the hidden features of the word position in the current iteration step, without updating the model parameters.

[0149] Specifically, for word positions in the response portion where the cosine similarity is below a second set threshold, it can be assumed that their hidden features have changed significantly in the current iteration compared to the previous iteration. Continuing to reuse cached features might affect the denoising quality. In this case, based on the current input sequence, it is necessary to re-perform forward computation on these word positions through the backbone network of the large language model to obtain updated hidden features. Subsequently, the updated hidden features replace the cached hidden features at the corresponding positions, providing a new reference for cache reuse in the next iteration.

[0150] Through the above process, an "on-demand update" strategy can be adopted for the hidden layer features at different locations in the response part: the locations with small changes are cached and the locations with large changes are recalculated, thereby reducing the overall amount of computation while ensuring the denoising effect.

[0151] Step 660: The hidden features cached in the previous step and / or the updated hidden features are input into the decoding head to generate prediction data, and the current input sequence is updated based on the prediction data.

[0152] Specifically, after reusing and updating the hidden features of the instruction and response parts, the hidden features obtained in the current iteration step can be used as the hidden vector sequence of the backbone network's output in this iteration step, and this hidden vector sequence is input into the decoder head. The decoder head generates prediction data based on the hidden vector sequence.

[0153] Subsequently, the current input sequence can be updated based on the predicted data. For example, the index at the corresponding position in the current input sequence can be replaced with a discrete codebook index with a higher confidence level, so that the current input sequence gradually converges from the noisy state to the target data during multiple iterations.

[0154] Under the above embodiments, adaptive caching accelerates the use of the characteristics of relatively stable instruction parts and local changes in response parts. It combines cosine similarity and a second set threshold to selectively reuse hidden layer features, and only performs forward calculation on the parts with obvious changes. This significantly reduces the amount of computation in the iterative denoising process without significantly reducing the denoising quality, thereby improving inference efficiency.

[0155] Through the above process, by uniformly representing the target data as discrete word sequences, a sampling time-driven discrete stream matching training mechanism is introduced. A unified large language model backbone network is used to complete cross-modal deep fusion and prediction. While ensuring unified semantic representation and cross-modal collaborative capabilities, non-autoregressive parallel sampling and denoising generation are achieved, significantly reducing inference latency. Furthermore, the generation process is controllable and interpretable through sampling time and noise modeling mechanisms. This effectively solves the problem that existing technologies in related fields cannot simultaneously take into account multimodal understanding, generation, and retrieval capabilities under a unified architecture.

[0156] The following is a detailed implementation description of a model training method in an application scenario. This application scenario is applicable to... Figure 1 The implementation environment is shown.

[0157] First, the acquisition terminal 110 acquires raw data and its corresponding text description data, and transmits the raw data and its corresponding text description data to the server terminal 130. The raw data includes at least one of image data, video data, and audio data.

[0158] Server 130 acquires the raw data and corresponding text description data transmitted from acquisition terminal 110, constructs a modal encoder for generating discrete word sequences, and introduces an auxiliary VQVAE quantizer and modal decoder; the raw data is converted into discrete word sequences through the modal encoder and VQVAE quantizer; the discrete word sequences are reconstructed through the modal decoder, and the reconstruction loss is calculated; a pre-trained auxiliary semantic model is acquired, and the text description data and discrete word sequences are input into the auxiliary semantic model for semantic representation, respectively, to obtain text semantic representation and modal semantic representation; the semantic alignment loss is calculated based on the text semantic representation and modal semantic representation; the parameters of the modal encoder are updated based on the reconstruction loss and semantic alignment loss, resulting in a modal encoder after preheating.

[0159] Then, the original data can be converted into a discrete word sequence by the modal encoder after the preheating is completed.

[0160] Further, during training iterations, the sampling time is determined, and the set of discrete codebook indices for the VQVAE quantizer and the codebook vector corresponding to each discrete codebook index in the set are obtained. Based on the codebook vectors, the cosine distance between any two discrete codebook indices in the set is calculated, and the state transition probability distribution between the discrete codebook indices is determined according to the cosine distance. The perturbation strength parameter is determined based on the sampling time, and the set of word positions to be perturbed is determined in the discrete word sequence corresponding to the target data according to the perturbation strength parameter. For any word position in the word position set, the original discrete codebook index at that position is extracted from the discrete word sequence corresponding to the target data. Based on the state transition probability distribution corresponding to the original discrete codebook index, the replacement discrete codebook index is determined. The replacement discrete codebook index replaces the original discrete codebook index at the word position, generating noisy data.

[0161] Based on this, the noisy data and the sampling time input to the backbone network of the initial large language model can be used for deep feature interaction and fusion to obtain the latent vector sequence output by the backbone network. The latent vector sequence can then be mapped back to the discrete codebook index of each modality through the decoding head to obtain the prediction data.

[0162] The cross-entropy loss is calculated between the predicted data and the target data, and the reconstruction loss is reused as a regularization term to obtain the target loss. Based on the target loss, the model parameters of the initial large language model are updated, and the large language model that has been trained is obtained when the training termination condition is met.

[0163] In the inference phase of the large language model, noisy data is first acquired, and an initial input sequence of a preset block length is constructed based on the noisy data. The initial input sequence is then input into the large language model for iterative denoising. In each iteration step, predicted data is output, and the current input sequence is updated based on the predicted data to obtain a denoised discrete word sequence. The denoised discrete word sequence is then reconstructed through a modal decoder to obtain the output data.

[0164] Furthermore, in terms of inference efficiency and multi-turn interaction scenarios, a parallel denoising mechanism based on discrete stream matching replaces traditional autoregressive word-by-word inference, and introduces adaptive caching to accelerate the process during iterative denoising: In each iteration step, the cosine similarity between the hidden features corresponding to the word positions in the response part of the current input sequence and the cached hidden features from the previous iteration step is calculated. The cached hidden features are directly reused for the instruction part, and for response positions with small feature changes. For positions with significant feature changes, forward computation is performed and the hidden features are updated only. Predictive data is generated based on the reused and / or updated hidden features, and the current input sequence is updated accordingly. Through this mechanism, the token generation speed in tasks with sequences of equal length, such as video generation, is significantly faster than that of autoregressive models of the same scale, effectively reducing inference latency while ensuring generation quality. Simultaneously, relying on a unified discrete word stream and a unified representation space, it can naturally support multi-turn dialogue interactions interwoven with text, speech, and vision, without the need for additional modal cascading or complex conversion modules, thus achieving both efficient inference capabilities and a good multi-turn interaction experience in real-world application scenarios.

[0165] Figure 7 A schematic diagram illustrating a specific implementation of output data is shown in the figure. It demonstrates that after converting multimodal noisy data (video data, image data, audio data, text data, etc.) into a unified discrete word sequence and inputting it into the large language model for this application scenario, the large language model can simultaneously support three types of tasks—content generation, content understanding, and content retrieval—under a unified architecture. Typical application examples under different modalities are also given.

[0166] In generative application scenarios, such as Figure 7 As shown on the left, high-quality multimedia content can be automatically synthesized based on blurry or partially missing noise. For example, based on an input still image or a sequence of partially noisy images, a complete video clip describing "flowers gradually moving on a table" can be generated, achieving cross-modal generation from image to video. Another example is the generation of natural-sounding, steady-paced speech or music clips with clear rhythmic structure from blurry text prompts or noisy frequency spectrums. Furthermore, even with only a text description or rough sketch, a diverse and detailed collection of images can be generated for creative design, content creation, and other scenarios.

[0167] In understanding application scenarios, such as Figure 7 As shown on the right, the model can perform deep semantic analysis on noisy or low-quality videos, audio, and images. For video input, the model can identify people, actions, and scene changes in the scene after denoising and reconstruction, and generate text descriptions that conform to human language habits. For speech or audio input, the model can combine spectral features and contextual information to reconstruct and summarize dialogue content or event information, achieving semantic understanding and summarization of noisy speech. For image input, the model can analyze the composition, posture, and emotional expression of the scene even with blurring or compression artifacts, and output explanatory text about the plot, emotional atmosphere, etc., supporting high-level semantic understanding and interpretable analysis.

[0168] In search-based application scenarios, such as Figure 7 As shown below, by uniformly mapping noisy data from different modalities to a shared discrete lexical representation space, the model can perform cross-modal retrieval and matching based on denoising and semantic recovery. For example, it can query for video or image clips that match its semantics based on a noisy speech segment, retrieve corresponding high-definition image resources using blurry screenshots or low-definition video frames, or search for relevant audio, video, or image content in a multimodal database based on a brief text description. Through unified representation and discrete stream matching mechanisms, this invention can still achieve stable and reliable cross-modal retrieval and similar content discovery even when faced with fuzzy input.

[0169] In summary, by discretizing and controlling the noise of multimodal data, large language models can still perform multiple tasks such as content generation, semantic understanding, and cross-modal retrieval under the same architecture when faced with fuzzy, noisy, or partially missing data. This significantly improves the quality of unified representation, the adaptability to multimodal tasks, and the practicality in real-world complex scenarios.

[0170] In this application scenario, within a multimodal understanding context, raw data such as text, image, video, and audio data are uniformly converted into target data in the form of discrete word sequences. Deep feature interaction and fusion are then performed within a unified large language model backbone network, enabling semantic alignment and representation learning across different modalities within the same discrete word space. Based on this unified representation, the trained large language model can perform event recognition and long-term temporal understanding on video content, semantic summarization and sentiment analysis on audio content, and fine-grained description and composition analysis on image content. This achieves performance superior to or on par with mainstream autoregressive models on understanding benchmarks such as OmniBench and WorldSense, demonstrating that this invention achieves higher-quality unified representation and stronger semantic alignment capabilities in multimodal understanding tasks.

[0171] In multimodal generation scenarios, the Discrete Stream Matching (DFM) training paradigm is utilized. During training iterations, the discrete word sequence corresponding to the target data is perturbed and sampled based on the sampling time to obtain noisy data. This noisy data, along with the sampling time, is input into the backbone network of a large language model. The decoder then maps the latent vector sequence output by the backbone network back to the discrete codebook index of each modality to obtain the predicted data. By learning an iterative denoising process to recover the target data from the noisy data at different sampling times, excellent performance is achieved in tasks such as image generation, video generation, and audio generation. For example, it outperforms or significantly outperforms existing diffusion models and unified models on benchmarks such as GenEval, VBench, and AudioCaps. This demonstrates that while maintaining semantic consistency, it can generate multimedia data with richer texture and acoustic details, effectively suppressing blurring and "illusion" phenomena, and significantly improving generation quality.

[0172] In cross-modal retrieval scenarios, a modal encoder and a unified discrete lexicon, obtained through pre-training, map raw data from different modalities into a unified discrete codebook index sequence. This unified representation is then deeply fused within the backbone network of a large language model, enabling alignment of multimodal data such as text, images, videos, and audio within the same semantic space. Based on this unified representation, cross-modal matching and retrieval can be performed directly using discrete lexicon sequences without the need for an additional decoupled retrieval encoder, achieving various retrieval methods such as "searching for images by text, searching for videos by audio, and searching for text by images." Experimental results on datasets such as InfoSeek and OVEN demonstrate that the retrieval accuracy is significantly better than decoupled architecture models employing multiple encoders or expert hybrid structures, proving the outstanding technical performance of unified discrete representation and a unified backbone network in retrieval tasks.

[0173] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0174] The following are embodiments of the apparatus described in this application, which can be used to execute the model training method involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the model training method involved in this application.

[0175] Please see Figure 8 This application provides a model training device 900, including but not limited to: a data acquisition module 910, a data noise addition module 930, a data prediction module 950, a loss calculation module 970, and a model training module 990.

[0176] The data acquisition module 910 is used to acquire target data; the target data is data in the form of discrete word sequence.

[0177] The data noise-adding module 930 is used to determine the sampling time during training iterations and to perform perturbation sampling on the target data based on the sampling time to obtain noisy data.

[0178] The data prediction module 950 is used to perform deep feature interaction and fusion on the backbone network of the initial large language model with the noisy data and the sampling time input, to obtain the latent vector sequence output by the backbone network, and to map the latent vector sequence back to the discrete codebook index of each modality through the decoding head to obtain the prediction data.

[0179] The loss calculation module 970 is used to calculate the loss based on the predicted data and the target data to obtain the target loss.

[0180] The model training module 990 is used to update the model parameters of the initial large language model based on the target loss, and to obtain the fully trained large language model when the training termination condition is met.

[0181] It should be noted that the model training device provided in the above embodiments is only illustrated by the division of the above functional modules when performing model training. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the model training device will be divided into different functional modules to complete all or part of the functions described above.

[0182] Furthermore, the model training apparatus and model training method embodiments provided in the above embodiments belong to the same concept, and the specific way in which each module performs operations has been described in detail in the method embodiments, and will not be repeated here.

[0183] Please see Figure 9 This application provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc.

[0184] exist Figure 9 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0185] Data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not indicate that there is only one bus or one type of bus.

[0186] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0187] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0188] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing computer programs having instruction or data structure forms and accessible by electronic device 400, but not limited to these.

[0189] The memory 4003 stores a computer program, and the processor 4001 can read the computer program stored in the memory 4003 through the communication bus 4002.

[0190] The computer program is executed by one or more processors 4001 to implement the model training methods in the above embodiments.

[0191] Furthermore, this application provides a storage medium storing a computer program, which is executed by one or more processors to implement the model training method described above.

[0192] This application provides a computer program product, including a computer program that is executed by one or more processors to implement the model training method described above.

[0193] Compared with related technologies, ... (the beneficial effects that this application can produce are exhaustively described in the disclosure).

[0194] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / pad>

Claims

1. A model training method, characterized in that, include: Obtain the target data; The target data is in the form of discrete word sequence; The sampling time is determined during the training iteration, and the target data is perturbed and sampled based on the sampling time to obtain noisy data; The noisy data is combined with the backbone network of the initial large language model at the sampling time for deep feature interaction and fusion to obtain the latent vector sequence output by the backbone network. The latent vector sequence is then mapped back to the discrete codebook index of each modality through the decoder to obtain the prediction data. The target loss is obtained by calculating the loss based on the predicted data and the target data; The model parameters of the initial large language model are updated based on the target loss, and the large language model that has been trained is obtained when the training termination condition is met.

2. The method as described in claim 1, characterized in that, The acquisition of target data includes: Obtain raw data; the raw data includes at least one of image data, video data, and audio data; The original data is converted into a discrete word sequence, and padding words are inserted into the discrete word sequence to make the length of the inserted discrete word sequence an integer multiple of the preset block length, thus obtaining the target data.

3. The method as described in claim 2, characterized in that, Before converting the original data into a discrete word sequence, the method further includes: Obtain the text description data corresponding to the original data; A modal encoder for generating discrete word sequences is constructed, and an auxiliary VQVAE quantizer and modal decoder are introduced. The raw data is converted into a discrete word sequence using the modal encoder and the VQVAE quantizer; The discrete word sequence is reconstructed using the modality decoder, and the reconstruction loss is calculated. A pre-trained auxiliary semantic model is obtained, and the text description data and the discrete word sequence are respectively input into the auxiliary semantic model for semantic representation to obtain text semantic representation and modal semantic representation. Calculate semantic alignment loss based on the text semantic representation and the modal semantic representation; The parameters of the modal encoder are updated based on the reconstruction loss and the semantic alignment loss to obtain the modal encoder after preheating. The process of converting the original data into a discrete word sequence includes: The preheated modal encoder converts the raw data into a discrete word sequence.

4. The method as described in claim 3, characterized in that, The step of calculating the loss based on the predicted data and the target data to obtain the target loss includes: The cross-entropy loss is calculated between the predicted data and the target data, and the reconstruction loss is reused as a regularization term to obtain the target loss.

5. The method as described in claim 1, characterized in that, The step of determining the sampling time during training iterations and perturbing the target data based on the sampling time to obtain noisy data includes: Obtain the discrete codebook index set of the VQVAE quantizer, and the codebook vector corresponding to each discrete codebook index in the discrete codebook index set; Based on the codebook vector, calculate the cosine distance between any two discrete codebook indices in the discrete codebook index set, and determine the state transition probability distribution between discrete codebook indices according to the cosine distance; The perturbation intensity parameter is determined based on the sampling time, and the set of word positions to be perturbed is determined in the discrete word sequence corresponding to the target data according to the perturbation intensity parameter. For any lexical position in the set of lexical positions, extract the original discrete codebook index at the lexical position from the discrete lexical sequence corresponding to the target data; The replacement discrete codebook index is determined based on the state transition probability distribution corresponding to the original discrete codebook index; The original discrete codebook index at the lexical position is replaced with the replaced discrete codebook index to generate the noisy data.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes updating the model parameters of the initial large language model based on the target loss, and obtaining a fully trained large language model when the training termination condition is met. Acquire noise data; An initial input sequence with a preset block length is constructed based on the noise data; The initial input sequence is input into the large language model for iterative denoising. In each iteration step, predicted data is output, and the current input sequence is updated based on the predicted data to obtain a denoised discrete word sequence. The current input sequence is the input sequence obtained by updating the initial input sequence through the iteration steps before the current iteration step. The denoised discrete word sequence is reconstructed through a modal decoder to obtain output data, wherein the output data is multimedia data corresponding to the modal decoder, and the multimedia data includes at least one of image data, video data, and audio data.

7. The method as described in claim 6, characterized in that, The process involves inputting the initial input sequence into the large language model for iterative denoising. In each iteration, predicted data is output, and the current input sequence is updated based on the predicted data to obtain a denoised discrete word sequence, including: In the current iteration step, the confidence level corresponding to the terminator is determined based on the predicted data; If the confidence level does not exceed the first set threshold, then append a block-sized word length to the end of the current input sequence to obtain the input sequence for the next iteration step; If the confidence level exceeds the first set threshold, the position corresponding to the terminator is determined, and the current input sequence is truncated at the terminator, and the truncated discrete word sequence is output as the denoised discrete word sequence.

8. The method as described in claim 6, characterized in that, The iterative denoising process also includes adaptive caching acceleration, which includes: The word positions in the current input sequence are divided into a set of word positions for the instruction part and a set of word positions for the response part; wherein, the instruction part corresponds to prompt / condition words, and the response part corresponds to words to be generated; In each iteration step, the cosine similarity between the hidden features of the word position set of the response part in the current iteration step and the hidden features cached in the previous iteration step is calculated; wherein, the hidden features are the hidden vector sequence output by the backbone network of the large language model in the corresponding iteration step. For the set of lexical positions belonging to the instruction part, reuse the hidden features cached in the previous step; For the set of lexical positions belonging to the response part, if the cosine similarity exceeds the second set threshold, the hidden features cached in the previous step are reused. When the cosine similarity does not exceed the second set threshold, forward computation is performed on the word position set of the response part to obtain the updated hidden features, and the cached hidden features are replaced with the updated hidden features. The hidden features cached in the previous step and / or the updated hidden features are input into the decoding head to generate the prediction data, and the current input sequence is updated based on the prediction data.

9. A model training device, characterized in that, include: The data acquisition module is used to acquire target data; The target data is in the form of discrete word sequence; The data noise-adding module is used to determine the sampling time during training iterations and to perform perturbation sampling on the target data based on the sampling time to obtain noisy data; The data prediction module is used to perform deep feature interaction and fusion between the noisy data and the backbone network of the initial large language model input at the sampling time, to obtain the latent vector sequence output by the backbone network, and to map the latent vector sequence back to the discrete codebook index of each modality through the decoding head to obtain the prediction data. The loss calculation module is used to calculate the loss based on the predicted data and the target data to obtain the target loss; The model training module is used to update the model parameters of the initial large language model based on the target loss, and to obtain the trained large language model when the training termination condition is met.

10. An electronic device comprising at least one processor and at least one memory, wherein, The memory stores a computer program, characterized in that the computer program, when executed by the processor, implements the model training method as described in any one of claims 1 to 8.

Images