Multi-modal task unified modeling method and processing method based on flow matching

Abstract

The invention discloses a multi-modal task unified modeling method and processing method based on flow matching, and relates to the technical field of task unified frameworks. For all visual tasks of an image mode, after a token and model representation of a sample of each target visual task are obtained, a continuous evolution process from the token to the model representation is simulated by constructing an intermediate state token, and a task unified model is obtained through training by adopting a flow matching method. The method comprises the following steps: performing word segmentation on to-be-processed information to obtain an initialized token, gradually evolving a target model representation of any target task from the initialized token through m times of iteration according to a task unification model, and inputting the target model representation into a decoder corresponding to the target task for decoding to obtain a target task result. Due to the fact that the representation forms of the model representations of the different visual tasks are the same, the task unified model only needs to determine the model representations and then decodes the model representations through decoders of the different visual tasks to obtain the task results, and therefore unified modeling and processing of all the visual tasks are achieved.

Description

Technical Field

[0001] This application relates to the field of unified task framework technology, and in particular to a unified modeling method and processing method for multimodal tasks based on flow matching. Background Technology

[0002] To eliminate the need for generating customized visual models for specific visual tasks, a unified visual model architecture is needed to provide a unified training paradigm for diverse visual tasks, thereby enabling the processing of all visual tasks.

[0003] However, in the first type of scheme, which models visual tasks as image generation, the visual task is transformed into generating a target image given conditions and an input image, thus enabling a single generative model to handle multiple visual tasks uniformly. In the second type of scheme, which uses diffusion objectives to fine-tune large multimodal language models, the model is fine-tuned by introducing the image reconstruction feature objective of the diffusion model, allowing it to learn to reconstruct images and thus handle multiple visual tasks.

[0004] However, because the representation of results differs across visual tasks, the models constructed in the two aforementioned schemes can only handle segmentation tasks, which share the same representation, and cannot handle detection and classification tasks with different representations. Specifically, the results of segmentation tasks and the tasks handled by the models constructed in the aforementioned schemes are both represented as images. However, the results of segmentation tasks are represented as coordinate values, while the results of classification tasks are represented as global vector information of the image. Therefore, the above schemes still cannot establish a unified model that can handle all types of visual tasks. Summary of the Invention

[0005] This application provides a unified modeling and processing method for multimodal tasks based on flow matching, which is used to solve the problem that it is impossible to establish a unified model for processing all types of visual tasks.

[0006] Firstly, a unified modeling method for multimodal tasks based on flow matching is provided. This method includes: obtaining the smallest representation unit (token) and model representation of samples for each of multiple target tasks. The target task is a task of at least one modality, and the model representation is obtained by encoding the samples using the encoder of the corresponding target task. Model representations for different tasks of different modalities have the same representation form. Based on the token and model representation, intermediate tokens are constructed for each time step in multiple time steps. Multiple time steps represent the steps in the evolution process from token to model representation. Based on the token, model representation, target task identification information, each time step, and the corresponding intermediate token, a unified task model is trained using the flow matching method. The same task model is used to predict the prediction speed of each time step of the target task, and the prediction speed represents the difference between the intermediate token and the target token.

[0007] Based on the aforementioned unified modeling method for multimodal tasks, when at least one modality is image-based and multiple target tasks are all visual tasks, after obtaining the token and model representation of samples for each target visual task, the continuous evolution process from token to model representation is simulated by constructing intermediate tokens. After training the token, model representation, target visual task identification information, each time step, and the corresponding intermediate token using a flow matching method, the resulting unified task model can predict the model representation of any target visual task based on the token. Since the model representation can be decoded by the decoder of the target task to obtain the task result, and the model representations of different visual tasks have the same representation form, the trained unified task model only needs to determine the unified representation form of the model representation for different visual tasks, and then decode it by the decoders of different visual tasks to obtain the task result, thus achieving unified modeling for all visual tasks.

[0008] One possible implementation involves acquiring samples from each of multiple target tasks, segmenting the samples to obtain their tokens, and then inputting these tokens into the encoder of the corresponding target task for encoding to obtain a model representation. In this way, after acquiring samples from each target task, by segmenting and encoding the samples, both the token and model representation can be obtained. Unified modeling based on these tokens and model representations enables integrated modeling across multiple modalities and tasks.

[0009] As one possible implementation, at least one target modality is an image modality, and multiple target tasks include all visual tasks of the image modality, with samples being sample images. The sample images are input into a visual foundation model (VFM) for word segmentation to obtain tokens. The samples are then input into the corresponding visual foundation transformer (VFT) for encoding to obtain model representations. Thus, for each visual task's sample image across all visual tasks, word segmentation via VFM and encoding via VFT yield tokens and model representations. This models the visual task encoding process as a diffusion matching from tokens to model representations, unifying the visual understanding process across all visual tasks and achieving unified modeling for all visual tasks, avoiding the problem of inconsistent representations due to different visual task formats.

[0010] One possible implementation involves inputting the target task's identifier information, each time step, and the corresponding intermediate state token into the initial unified task model to obtain the predicted velocity for each time step. The true velocity is then obtained based on the difference between the model representation and the token. The initial unified task model is trained using the predicted and true velocities for each time step until the mean squared error between the predicted and true velocities at each time step is minimized, resulting in a fully trained unified task model. Thus, during flow matching-based training, a continuous velocity field is constructed by determining the predicted velocity at each time step, and this process iterates until the velocity field approximates the true velocity, thereby obtaining the fully trained unified task model and achieving unified modeling for all visual tasks.

[0011] One possible implementation involves determining a preset scale embedding vector corresponding to the target output scale required by the objective task, and determining the updated token adapted to the target output scale as the sum of the original token and the preset scale embedding vector. Based on the updated token, model representation, objective task identifier information, preset scale embedding vector, and intermediate tokens at each time step, a unified task model is trained using a flow matching method. Thus, by introducing a scale embedding vector matching the scale required by the task during training, multi-scale modeling is achieved, improving the unified task model's expressive power and adaptability to tasks relying on multi-scale features.

[0012] One possible implementation involves determining the target angle mapped to the unit circle based on the target index number corresponding to the target task and the total number of tasks. The identifier is then determined as a task embedding vector representing the target dimension expanded from the target angle. The target dimension is the dimension of the token. Thus, by employing a circular task embedding mechanism, discrete task index numbers can be mapped to continuous and periodic task embedding vectors, and these embedding vectors serve as identifiers, enhancing the continuity between tasks.

[0013] As one possible implementation, the unified task model is a velocity field function. Thus, by modeling the velocity field function through a flow-matching-based velocity field learning mechanism, and by modeling the continuous evolution path from token to model representation through the velocity field function, the efficiency of generating model representations is improved.

[0014] Secondly, a unified processing method for multimodal tasks based on stream matching is provided. This method includes: segmenting the target modality's information to be processed to obtain an initial token for the information to be processed. The m iterations of the information to be processed include the nth iteration. During the nth iteration, the single-step field and the intermediate token r generated in the previous iteration are... n-1 Input the identification information corresponding to (n-1) and the target task into the unified task model to obtain the prediction speed for the nth time. The single step size is the reciprocal of m, and r0 is the initialization token when n is 1. Based on the prediction speed for the nth time, the single step size is the intermediate token r. n-1 Determine the intermediate state token r generated after the nth iteration. n The target model representation obtained after m iterations is input into the decoder corresponding to the target task for decoding, thereby obtaining the target task result.

[0015] Based on the above-mentioned unified processing method for multimodal tasks, when the target modality is an image and the target task is any visual task among all visual tasks, the initial token is obtained by segmenting the information to be processed. Then, according to the unified task model, the target model representation of any target task is gradually evolved from the initial token through m iterations. The target model representation is input into the decoder corresponding to the target task for decoding to obtain the target task result, thereby achieving unified processing of all visual tasks.

[0016] As one possible implementation, the product of the single step size and the prediction velocity at the nth iteration is determined, and the intermediate state token r generated after the nth iteration is determined. n For intermediate state token r n-1The sum of the products. Thus, during the inference phase, intermediate tokens for the next iteration are generated based on tokens generated in the previous iteration, ultimately yielding the model representation. Increasing the number of iterations improves the quality of the output model representation, optimizing performance without additional training and enhancing deployment controllability.

[0017] One possible implementation is to use the task embedding vector corresponding to the target task as the identification information. In this way, during the inference phase, the unified task model can identify different tasks using the task embedding vector, thereby improving the continuity of task processing.

[0018] As one possible implementation, the single step size and r n-1 The target task's identification information and scale embedding vector (n-1) are input into the unified task model to obtain the prediction speed for the nth iteration. Thus, when predicting the prediction speed for each iteration during the inference phase, the target task's scale embedding vector is also needed, thereby enabling inference for tasks relying on multi-scale features and improving the adaptability and expressive power of the unified task model.

[0019] Thirdly, a unified modeling apparatus for multimodal tasks based on flow matching is provided, the apparatus comprising modules for executing the unified modeling method for multimodal tasks based on flow matching in the first aspect or any possible implementation thereof.

[0020] The multimodal task unified modeling device based on flow matching described in the third aspect can be a terminal device or a network device, or it can be a chip (system), network card or other component or assembly that can be set in a terminal device or network device, or it can be a device that includes a terminal device or network device. This application does not limit it in this regard.

[0021] Furthermore, the technical effects of the multimodal task unified modeling device based on flow matching described in the third aspect can be referred to the technical effects of the multimodal task unified modeling method based on flow matching described in the first aspect, and will not be repeated here.

[0022] Fourthly, a unified processing apparatus for multimodal tasks based on flow matching is provided, the apparatus comprising modules for executing the unified processing method for multimodal tasks based on flow matching in the second aspect or any possible implementation thereof.

[0023] The multimodal task unified processing device based on flow matching described in the fourth aspect can be a terminal device or a network device, or it can be a chip (system), network card or other component or assembly that can be set in a terminal device or a network device, or it can be a device that includes a terminal device or a network device. This application does not limit it in this regard.

[0024] Furthermore, the technical effects of the multimodal task unified processing device based on flow matching described in the fourth aspect can be referred to the technical effects of the multimodal task unified processing method based on flow matching described in the second aspect, and will not be repeated here.

[0025] Fifthly, an electronic device is provided, comprising a processor and a memory; the processor is coupled to the memory; the memory stores computer instructions, which are loaded and executed by the processor to enable the computing device to perform the operational steps of the methods described in any possible implementation of the first and / or second aspects above.

[0026] A sixth aspect provides a computer-readable storage medium comprising: computer software instructions; which, when executed in an electronic device, cause the electronic device to perform operational steps of the method as described in any possible implementation of the first and / or second aspects.

[0027] In a seventh aspect, embodiments of this application provide a chip system. The chip system includes a memory and at least one processor. The memory stores a set of computer instructions, which, when executed by the processor, perform operational steps of the methods described in any possible implementation of the first and / or second aspects.

[0028] Eighthly, a computer program product is provided that, when run on an electronic device, causes the electronic device to perform the operational steps of the method as described in any possible implementation of the first and / or second aspects.

[0029] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0030] Figure 1 A schematic diagram of the architecture of a unified multimodal task system provided in this application embodiment;

[0031] Figure 2 A schematic diagram of the architecture of a unified visual task system for image modalities provided in this application embodiment;

[0032] Figure 3 A flowchart illustrating a unified modeling method for multimodal tasks based on flow matching, provided in an embodiment of this application;

[0033] Figure 4 A flowchart illustrating a unified processing method for multimodal tasks based on stream matching, provided in an embodiment of this application;

[0034] Figure 5A schematic diagram of a possible multimodal task unified modeling device 500 based on flow matching provided in an embodiment of this application;

[0035] Figure 6 A schematic diagram of a possible multimodal task unified processing device 600 based on stream matching provided in an embodiment of this application;

[0036] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The embodiments of this application can be applied not only to existing vision tasks, 3D point cloud processing tasks, and cross-modal tasks, but also to future vision tasks, 3D point cloud processing tasks, and cross-modal tasks. This application does not impose any limitations on these applications. For ease of understanding, the relevant terminology used in the embodiments of this application will be introduced below.

[0038] (1) Stream matching

[0039] Flow matching is a core method for training continuous normalized flow models. Its goal is to learn a probabilistic path that transforms a simple distribution (such as Gaussian noise) into a complex data distribution (such as images or audio).

[0040] Flow matching establishes a continuous and reversible transformation by having a neural network learn to match a velocity field defined by a simple prior path, thus "flowing" a simple noise distribution into a complex data distribution.

[0041] (2) Velocity field

[0042] The velocity field is a vector field defined at all points in space and at all points in time.

[0043] The velocity field can be understood as a dynamic map filled with invisible arrows. Each arrow corresponds to a spatial location and a point in time. The arrow indicates that a data particle (such as a vector representing a part of the generated image) has arrived at this location at a certain moment, and then it should move in which direction and at what speed in the next instant.

[0044] (3) Velocity field function

[0045] The velocity field function is a neural network that learns to predict particle motion commands.

[0046] The velocity field function takes the current state and time of the data point as input, and its output is a velocity vector that precisely indicates the direction and speed at which the data point should move in the next instant.

[0047] (4) Diffusion model

[0048] The diffusion model is a generative artificial intelligence model whose core principle is to learn data generation through a forward and backward process. First, noise is gradually added to the training data (such as images) through the forward process until it becomes completely random noise. Then, in the backward process, a neural network is trained to learn how to reverse this noise-adding process step by step, that is, to gradually remove noise from pure noise, and finally reconstruct a completely new sample that conforms to the original data distribution.

[0049] To better understand the problem mentioned in the background that it is impossible to establish a unified model for handling all types of visual tasks, the following section explains existing schemes for establishing a unified model for visual tasks.

[0050] One existing unified framework for visual tasks employs image generation models to unify multiple visual tasks through a generative architecture. A generative architecture refers to a design paradigm for a class of artificial intelligence models whose core objective is to learn the distribution of data and create entirely new data samples similar to the original data.

[0051] The above scheme models some visual tasks as image generation tasks to achieve multi-task end-to-end training. The image generation task serves as a bridge connecting various visual understanding tasks, effectively unifying the paradigms of different visual understanding tasks.

[0052] The aforementioned solutions only cover segmentation tasks and cannot cover visual tasks such as classification and detection, where the representation is not an image. Furthermore, while these solutions achieve unified processing of different visual tasks through a diffusion model, they still rely on large amounts of data and massive models, limiting their application in resource-constrained environments. In addition, compared to single-visual-task-specific models, the multi-task framework of generative models exhibits performance degradation when handling specific tasks.

[0053] Another existing unified framework for visual tasks utilizes a diffusion objective to fine-tune a large multimodal language model. By combining the diffusion model with an autoregressive large language model, it can perform image generation or image editing tasks that follow user instructions, but it cannot perform visual perception tasks. Furthermore, the above approach relies on large-scale data and computational resources, making it unsuitable for resource-constrained scenarios.

[0054] To address the problem of not being able to establish a unified model for handling all types of visual tasks, this application provides a unified modeling and processing method for multimodal tasks based on flow matching. Specifically, it provides a method that models the continuous evolution process from token to model representation, enabling joint training and inference of visual tasks with different representations under a unified architecture. When the target modality is an image modality and multiple target tasks are all visual tasks, after obtaining the token and model representation of samples for each target visual task, an intermediate token is constructed to simulate the continuous evolution process from token to model representation. After training the token, model representation, target visual task identifier information, each time step, and the corresponding intermediate token using the flow matching method, the resulting unified task model can predict the model representation of any target visual task based on the token. An initial token is obtained by segmenting the information to be processed. Based on the unified task model, the target model representation of any target task is gradually evolved from the initial token through m iterations. The target model representation is then input into the decoder corresponding to the target task for decoding to obtain the target task result. Since the model representations of different visual tasks have the same representation form, the unified model for the trained task only needs to determine the model representation of the unified representation form of different visual tasks. After being decoded by the decoders of different visual tasks, the task results can be obtained, thus realizing the unified modeling and processing of all visual tasks.

[0055] Next, we will combine the appendix Figure 1 To be continued Figure 4 The implementation methods of the embodiments of this application will be described in detail.

[0056] Figure 1 This is a schematic diagram of the architecture of a unified multimodal task system provided in an embodiment of this application.

[0057] like Figure 1 As shown, the multimodal task unified system 100 includes an execution device 110, a training device 120, a database 130, a terminal device 140, a data storage system 150, and a data acquisition device 160.

[0058] The execution device 110 can be a terminal, such as a mobile phone, tablet computer, laptop computer, virtual reality (VR) device, augmented reality (AR) device, mixed reality (MR) device, extended reality (ER) device, camera, or in-vehicle terminal, etc., or it can be an edge device (e.g., a box carrying a chip with processing capabilities). The execution device 110 can also be a gateway, such as an API gateway or application gateway. The execution device 110 can also be a cloud platform, where the cloud platform can be a public cloud, private cloud, hybrid cloud, etc.

[0059] The training device 120 can be a terminal, or other computing devices such as servers or cloud devices.

[0060] As one possible embodiment, the execution device 110 and the training device 120 are different processors deployed on different physical devices (e.g., servers in a server or cluster). For example, the execution device 110 may be a graphics processing unit (GPU), a central processing unit (CPU), other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The training device 120 may be a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0061] The data acquisition device 160 is used to collect samples of multiple target tasks in multiple modalities for training and store the samples in the database 130. The data acquisition device 160, the execution device 110, and the training device 120 can be the same or different devices. For example, in the image modality, the samples can be sample images, training video frame images, etc.; in the text modality, the samples can be text content such as papers, novels, and news reports; and in the three-dimensional modality, the samples can be three-dimensional point cloud information collected by vehicle radar, etc. This application embodiment does not limit this.

[0062] Training device 120 trains task-unified model 101 using multimodal samples until the loss function in task-unified model 101 converges and the loss function value is less than a certain threshold, at which point task-unified model 101 is considered trained successfully, thus enabling the neural network to achieve a certain level of accuracy. Alternatively, task-unified model 101 is considered trained successfully when it meets the training objective of minimizing the mean squared error between the predicted speed and the actual speed, giving it the ability to uniformly process multimodal tasks. Then, training device 120 configures the trained task-unified model 101 to execution device 110. Execution device 110 is used to implement the function of uniformly processing multimodal tasks based on the trained task-unified model 101.

[0063] In some embodiments, the data acquisition device 160 and the training device 120 are the same computing device. The computing device can train the task unified model 101 based on the acquired samples until the task unified model 101 is trained.

[0064] In some embodiments, the execution device 110 and the training device 120 are the same computing device. The computing device can configure the trained task unified model 101 into itself and use the trained task unified model 101 to realize the function of unified processing of multimodal tasks.

[0065] In other embodiments, the training device 120 can configure the trained unified task model 101 to multiple execution devices 110. Each execution device 110 utilizes the trained unified task model 101 to perform unified processing of multimodal tasks.

[0066] Combined with the multimodal task unified system 100, the multimodal task unified modeling method and multimodal task unified processing method based on flow matching provided in this application embodiment can be applied to scenarios of unified modeling and inference for multimodal tasks. Specifically, the multimodal task unified modeling method and multimodal task unified processing method based on flow matching in this application embodiment can be applied to scenarios of unified modeling and processing of visual tasks, unified modeling and processing of 3D point cloud tasks, unified modeling and processing of image-text matching tasks, etc. The scenario of unified modeling and processing of visual tasks is briefly introduced below.

[0067] For example, in the scenario of unified modeling and processing of visual tasks: training device 120 (e.g., mobile phone, computer, tablet computer) acquires sample images, and training device 120 trains based on the sample images to obtain the trained unified model 101 for the task.

[0068] It should be noted that in practical applications, the samples maintained in database 130 may not all come from data acquisition device 160; they may also be received from other devices. Furthermore, training device 120 may not be entirely based on the samples maintained in database 130 to train the unified model 101 of the task; it may also obtain the samples from the cloud or other sources to train the unified model 101 of the task. The above description should not be construed as limiting the embodiments of this application.

[0069] Furthermore, based on the function performed by the execution device 110, the execution device 110 can be further subdivided into, for example... Figure 1 The architecture shown is as follows: Figure 1 As shown, the execution device 110 is equipped with a computing module 111, an I / O interface 112, and a preprocessing module 113.

[0070] I / O interface 112 is used for data interaction with external devices. Users can input data into I / O interface 112 through terminal device 140. Input data may include information to be processed or target tasks. Alternatively, input data may also come from database 130.

[0071] The preprocessing module 113 is used to preprocess the information to be processed received from the I / O interface 112. In this embodiment, the preprocessing module 113 can be used to extract the initialization token of the information to be processed received from the I / O interface 112.

[0072] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data and instructions obtained from the corresponding processing into the data storage system 150.

[0073] For example, the unified task model 101 stored in the execution device 110 can be applied to the execution device 110. After the execution device 110 obtains the input data, the calculation module 111 inputs the input data into the unified task model 101 to obtain the target model representation, and inputs the target model representation into the decoder corresponding to the target task for decoding to obtain the target task result corresponding to the target task.

[0074] Finally, I / O interface 112 returns the target task result to terminal device 140, which then provides it to the user so that the user can view the target task result. It should be understood that terminal device 140 and execution device 110 can also be the same physical device.

[0075] exist Figure 1 In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the terminal device 140 can automatically send input data to I / O interface 112. If user authorization is required for the terminal device 140 to automatically send input data, the user can set the appropriate permissions in the terminal device 140. The user can view the target task results output by the execution device 110 on the terminal device 140. The terminal device 140 can also act as a data acquisition terminal, collecting data such as... Figure 1 The input data from the input I / O interface 112 shown is used as new sample data and stored in the database 130.

[0076] Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. Figure 1 The positional relationships between the devices, components, modules, etc., shown do not constitute any limitation; for example, in Figure 1 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0077] In the multimodal task unified modeling method based on flow matching, the training device 120 trains the task unified model 101 based on samples of multiple target tasks of at least one modality to obtain the trained task unified model 101.

[0078] As one possible implementation, the training device 120 acquires the token and model representation of samples from each of the multiple target tasks. The training device 120 constructs intermediate state tokens corresponding to each time step in multiple time steps based on the tokens and model representations. Based on the tokens, model representations, task identifiers of the target tasks, each time step, and the corresponding intermediate state tokens, the training device 120 trains a unified task model 101 using a stream matching method.

[0079] In this context, multiple target tasks are tasks of at least one modality. The task of at least one module can be all or part of the tasks encompassing at least one modality. Multiple target tasks can refer to tasks of a single modality, cross-modal tasks, or a set of multiple single-modal tasks. For example, at least one modality can include image modality, audio modality, text modality, 3D point cloud modality, image-text modality, etc., and this application embodiment does not impose such limitations.

[0080] Model representations are obtained by encoding samples through the encoder of the corresponding target task. Model representations for different tasks in different modalities have the same representation form. Model representations are transformed into semantic representations of samples in mathematical form. The model representations are then decoded by the decoder of the target task to obtain the task result. Intermediate tokens refer to the tokens of intermediate states at each time step in the continuous evolution from tokens to model representations.

[0081] Optionally, the training device 120 can construct multiple intermediate tokens corresponding to multiple time steps using linear interpolation based on the tokens and model representations. The number of time steps can be pre-set; for example, the multiple time steps can be any positive integer value such as 10, 74, or 100, or values ​​determined by interpolation between the model representations and the tokens.

[0082] Optionally, the training device 120 can also be an intermediate token corresponding to each time step in the continuous evolution process from token to model representation, constructed using methods such as multinomial fitting, linear regression, and machine learning.

[0083] The identification information of the target task can be information that uniquely identifies the target task, such as the task name, task number, or index number. The unified task model 101 can be a streaming model used to predict the predicted velocity at each time step of the target task. Streaming models are a type of model that learns data distribution and generates new data through reversible transformations. Specifically, streaming models can be diffusion models, velocity field functions, etc. For example, a diffusion model can be a diffusion transformer (DiT) model. The DiT model is a generative artificial intelligence model that combines the core architecture of a diffusion model with the powerful representational capabilities of a transformation (Transformer) architecture.

[0084] In the unified processing method for multimodal tasks based on stream matching, execution device 120 obtains the information to be processed sent by user terminal device 140. Execution device 120 performs word segmentation on the information to be processed to obtain an initialization token. The m iterations of the information to be processed include the nth iteration. During the nth iteration, execution device 120 converts the single-step size of the intermediate token r generated in the previous iteration into an intermediate state token r. n-1 Inputting the identification information corresponding to (n-1) and the target task into the unified task model 101, the predicted speed of the nth iteration is obtained. The single step size is the reciprocal of m. When n is 1, r0 is the initialization token. The execution device 120 calculates the predicted speed of the nth iteration, the single step size, and the intermediate token r. n-1 Determine the intermediate state token r generated after the nth iteration. n The execution device 120 inputs the target model representation generated after m iterations into the decoder corresponding to the target task for decoding, and obtains the target task result corresponding to the target task.

[0085] The target modality can refer to any one of the multiple modalities trained by the unified task model 101. For example, the target modality can be an image model, a text modality, an audio modality, a 3D point cloud modality, etc., and this application embodiment does not limit this. The m iterations of the information to be processed can also be called the total number of sampling steps, so the single step length corresponding to each iteration is (1 / m).

[0086] The above describes one possible architecture for the multimodal task unified system 100. The following section will combine... Figure 2 Another possible architecture for a unified system for multimodal tasks in image mode is described. Figure 2 This is a schematic diagram of the architecture of a unified visual task system for image modalities provided in an embodiment of this application. Figure 2 As shown, the unified visual task system 200 performs training and inference on the unified visual task model 201.

[0087] The unified visual task system 200 includes a visual base model tokenizer (VFM) 210, a visual base model encoder (VFT encoders) 220 for the target visual task, a visual base model decoder (VFT decoders) 230 for the visual base model, and a unified visual task model 201.

[0088] During the model training phase, the unified visual task system 200 inputs sample images into the word segmenter 210 of the visual base model for word segmentation, obtaining tokens for the sample images. These tokens are then upscaled to obtain multi-scale tokens. The unified visual task system 200 then inputs the sample images into the encoder 220 of the visual base task model for encoding, obtaining the model representation of the target task. Based on the multi-scale tokens and model representations, the unified visual task system 200 trains the unified visual task model 201 using a flow matching method, resulting in the trained unified visual task model 201.

[0089] During the model inference phase, the unified visual task system 200 inputs the image to be processed into the VFM Tokenizer 210 for word segmentation to obtain an initial token for the image. This initial token is then upsampled to obtain an initial multi-scale token. The unified visual task system 200 inputs this initial multi-scale token into the trained unified visual task model 201 for multiple iterations to obtain the target model representation. Finally, the unified visual task system 200 inputs this target model representation into the decoder 230 of the visual base task model for the target visual task for decoding to obtain the target task result.

[0090] The above describes the architectural diagrams of two possible unified systems for multimodal tasks. The following section will combine... Figure 3 The flowchart of the multimodal task unified modeling method based on flow matching in this application is described. Figure 3 This is a flowchart illustrating a unified modeling method for multimodal tasks based on flow matching, provided as an embodiment of this application. (About...) Figure 3 The training equipment 120 mentioned above can be referenced. Figure 1 The relevant descriptions in the embodiments of this application will not be repeated here.

[0091] like Figure 3 As shown, the unified modeling method for multimodal tasks based on flow matching may include steps 301 to 303.

[0092] Step 301: The training device 120 acquires the token and model representation of the samples of each of the multiple target tasks.

[0093] The training device 120 can be a token and model representation of a sample sent by the terminal device 140. Alternatively, the training device 120 can acquire samples from each of multiple target tasks, process the samples to obtain a token and model representation.

[0094] As one possible implementation, the training device 120 can acquire samples for each of the multiple target tasks in the training set. Alternatively, the training device 120 can receive samples sent by the terminal device 140.

[0095] The training set includes multiple samples of multiple target tasks in at least one modality. At least one modality includes at least one of image modality, text modality, audio modality, etc. Multiple samples of multiple target tasks refer to samples corresponding to multiple tasks in one or more target modalities. For example, when the target modality is image modality, multiple samples of multiple target tasks in image modality refer to sample images of multiple visual tasks.

[0096] As one possible implementation, the training device 120 performs word segmentation on the samples to obtain the tokens of the samples.

[0097] Optionally, the training device 120 inputs the sample into the word segmenter for word segmentation to obtain the token after word segmentation.

[0098] Specifically, the tokens after word segmentation for text modal samples can be sub-words; the tokens after word segmentation for image modal samples can be image blocks; the tokens after word segmentation for audio modal samples can be audio segments or small blocks in the spectrogram; the tokens after word segmentation for video modal samples can include image block sequences and audio tokens; and the tokens after word segmentation for 3D point cloud modal samples can be point cloud information in voxels.

[0099] When the target modality is an image modality, the training device 120 inputs the sample image into the VFM Tokenizer for word segmentation to obtain the token of the sample image.

[0100] For example, in multiple visual tasks, including object detection and semantic segmentation, the sample image is a red-green-blue (RGB) image of size H×W×3. When VFM is the second version of the self-supervised learning data-intensive network for object detection (DINOv2), the training device 120 inputs the sample image into the DINOv2 tokenizer to obtain the token r0, r0 = DINOv2_Tokenizer(x), where x is the sample image. r0∈R P·d That is, the dimension of tokenr0 belongs to the range of the real number set formed by P multiplied by d, where P refers to the number of tokens in the sample image and d refers to the dimension of the image.

[0101] Optionally, since the target task depends on multi-scale features, the token of the sample can be a multi-scale token. Specifically, the training device 120 upsamples the tokens after word segmentation to generate multi-scale tokens.

[0102] For example, training device 120 performs an upsampling operation on tokenr0 to generate multi-scale... uosample indicates that an upsampling operation is performed.

[0103] As one possible implementation, the training device 120 inputs the samples into the encoder of the corresponding target task for encoding, thereby obtaining the model representation of the target task.

[0104] After acquiring a sample, the training device 120 determines the encoder of the target task corresponding to the sample, inputs the sample into the encoder for encoding, and obtains the model representation of the target task.

[0105] The encoders differ for different target tasks, while the number of encoders for a unified target task can be one or more. The model representation is then decoded by the target task's decoder to obtain the task result. The model representations for different tasks across different modalities have the same representational form. That is, only one unified model is needed to determine the model representations for different tasks across different modalities; subsequent decoding yields the task result. Furthermore, because the model representations have the same representational form, the problem of being unable to construct a unified model due to different representational forms is avoided.

[0106] When the target modality is an image modality, the training device 120 inputs the sample images into the corresponding VFT encoders of the target visual task for encoding, thereby obtaining the model representation r of the target visual task. t The model representation of the target visual task can also be called visual representation. Visual representation refers to the transformation of the pixels of a sample image into a mathematical form rich in semantic information. For example, visual representation can be feature vectors, feature maps, etc. Visual representation can capture information such as objects, shapes, textures, and spatial relationships in an image.

[0107] For example, multiple visual tasks in image modalities include classification and detection tasks. The training device 120 inputs sample images into a dedicated encoder for classification tasks to obtain a model representation for the classification task. The encoder for the classification task can be an encoder in a detailed image transformer (DeiT) model, and the encoder for the detection task can be an encoder in a transformation-based object detection model (DETR).

[0108] Step 302: The training device 120 constructs intermediate tokens corresponding to each time step in multiple time steps based on the tokens and model representations.

[0109] The training device 120 constructs intermediate tokens for each time step across multiple time steps using various methods based on the tokens and model representations. Specifically, the training device 120 can employ methods such as linear interpolation, polynomial fitting, linear regression, and machine learning to construct intermediate tokens for each time step in the continuous evolution from token to model representation.

[0110] As one possible implementation, the training device 120 constructs multiple intermediate tokens corresponding to multiple time steps using linear interpolation based on the tokens and model representations. The number y corresponding to the multiple time steps can be pre-set; for example, y can be any positive integer value such as 10, 74, or 100, or it can be a value determined by interpolation between the model representations and the tokens.

[0111] Specifically, the multiple time steps include a k-th time step. The training device 120 determines the first weight of the token and the second weight of the model representation based on the index of the k-th time step and the total number of time steps. The training device 120 determines the first product of the first weight and the token, and the second product of the second weight and the model representation. The training device 120 determines the intermediate state token corresponding to the k-th time step as the sum of the first and second products.

[0112] In this system, the first weight decreases as the time step number increases, while the second weight increases as the time step number increases. For example, the second weight can be the ratio of k to K, and the first weight can be the difference between 1 and the second weight.

[0113] For example, when the target modality is an image modality, the training device 120 trains according to multiple scales. Model representation r t The intermediate state token r corresponding to the k-th time step is constructed using linear interpolation. k .in, That is, the range of k is from 0 to y.

[0114] Step 303: The training device 120 trains the unified task model using the sampling stream matching method based on the token, model representation, target task identification information, each time step and the corresponding intermediate state token.

[0115] Among them, the unified task model is used to predict the prediction speed at each time step of the target task, and the prediction speed represents the difference between the intermediate token and the token.

[0116] The training device 120 inputs the token, model representation, target task identification information, each time step and the corresponding intermediate state token into the initial task unified model, and uses the flow matching method to train it, thus obtaining the trained task unified model.

[0117] As one possible implementation, the unified task model is a flow model. The training device 120 inputs the target task's identifier information, each time step, and the corresponding intermediate state token into the initial flow model to obtain the predicted velocity for each time step. The training device 120 obtains the true velocity based on the difference between the model representation and the token. The training device 120 trains the initial flow model based on the predicted velocity and the true velocity for each time step until the mean squared error between the predicted velocity and the true velocity is minimized, thus obtaining the trained flow model.

[0118] Optionally, the flow model is a velocity field function. The training device 120 inputs the target task's identifier information, each time step, and the corresponding intermediate state token into the initial velocity field function to obtain the predicted velocity for each time step. The training device 120 obtains the true velocity based on the difference between the model representation and the token. The training device 120 trains the initial velocity field function based on the predicted velocity and the true velocity for each time step until the mean square error between the predicted velocity and the true velocity is minimized, thus obtaining the trained velocity field function.

[0119] For example, when the target mode is an image mode, the velocity field function is fθ(·), and the identification information of the target visual task is the target index number t of the target visual task. The training device 120 sets the target index number t of the target visual task, the k-th time step, and the corresponding intermediate state token r. k Input the initial velocity field function to obtain the predicted velocity at the k-th time step. Training device 120 based on model representation r t Multiscale The difference yields the true speed. Training device 120 predicts the velocity corresponding to each time step. Real speed Train the initial velocity field function until the trained velocity field function satisfies the training objective.

[0120] The training objective is to minimize the prediction speed. With real speed The mean square error between them, i.e. Where x refers to the sample image, w refers to the set of sample images for multiple visual tasks in the training set, and z is the set of multiple visual tasks.

[0121] Optionally, to enable the unified task model to distinguish different scales, the training device 120, before training the model, also needs to determine the preset scale embedding vector corresponding to the target output scale required by the target task, and determine the updated token adapted to the target output scale as the sum of the token and the preset scale embedding vector. Based on the updated token, model representation, target task identification information, preset scale embedding vector, each time step, and the corresponding intermediate state token, the training device 120 uses a flow matching method to train the unified task model.

[0122] Specifically, the training device 120 pre-stores corresponding scale embedding vectors s for multiple scales, including low resolution, medium resolution, and high resolution. Based on the output scale required by the target task, the training device 120 determines the corresponding pre-defined scale embedding vector s and adds it to the token to obtain an updated token. The training device 120 inputs the updated token, model representation, target task identification information, pre-defined scale embedding vector s, and the tokens at each time step and their corresponding intermediate states into the initial task unified model, and uses a flow matching method to obtain the task unified model. Thus, by adding scale embedding vectors, it can adapt to the modeling needs of different scales within a unified structure.

[0123] For example, when the target modality is an image modality, the training device 120 trains according to the intermediate state token r. k The preset scale embedding vector s corresponding to the output scale required for the target visual task is used to determine the updated intermediate state token r'. k , where r' k =r k +s. Training device 120 will generate the k-th time step and the updated intermediate state token r' k The target index t and the preset scale embedding vector s of the target vision task are input into the initial velocity field function to obtain the predicted velocity at the k-th time step. Training device 120 predicts the velocity corresponding to each time step. Real speed Train the initial velocity field function until the trained velocity field function satisfies the training objective.

[0124] Optionally, to avoid the discontinuity problem caused by traditional text prefix-based task embedding, this embodiment of the application employs a circular task embedding mechanism when determining the identification information of the target task. The training device 120 determines the target angle mapped from the target index number to the unit circle based on the target index number corresponding to the target task and the total number of tasks. The training device 120 determines the identification information of the target task as a task embedding vector of the target dimension expanded from the target angle.

[0125] The target dimension is the same as the token dimension, meaning the task embedding vector has the same dimension as the token.

[0126] Training device 120 determines the target angle θ mapped from the target index number to the unit circle based on the target index number t corresponding to the target task and the total number of tasks z. t ,in Training equipment 120 according to the target angle θ t Expanding using sine and cosine functions yields a d-dimensional task embedding vector e. t The training device 120 uses the task embedding vector as the identification information of the target task.

[0127] in

[0128] For example, when the target modality is an image modality, the training device 120 will use the k-th time step and the intermediate state token r k The task embedding vector e for the target visual task t The preset scale embedding vector s is input into the initial velocity field function to obtain the predicted velocity at the k-th time step. Alternatively, training device 120 can use the k-th time step and the updated intermediate state token r' k , preset scale embedding vector s and target visual task task embedding vector e t Input the initial velocity field function to obtain the predicted velocity at the k-th time step.

[0129] Training device 120 predicts the velocity corresponding to each time step. Real speed Train the initial velocity field function until the trained velocity field function satisfies the training objective.

[0130] Based on the descriptions of steps 301 to 303 above, when the target modality is an image and the target task is a visual task, the training device 120, after extracting the tokens of the samples and the model representation of the target task, simulates the continuous evolution process from tokens to model representations by constructing intermediate tokens. After training the tokens, model representations, target task identification information, each time step, and the corresponding intermediate tokens using a flow matching method, the resulting unified task model can predict the model representation of any visual task based on the tokens. Since the model representation can be decoded by the decoder of the target task to obtain the task result, and the model representations of different visual tasks have the same representation form, the unified task model, once trained, only needs to determine the unified representation form of the model representation for different visual tasks. After decoding by the decoders of different visual tasks, the task result can be obtained, thus achieving unified modeling for all visual tasks.

[0131] Furthermore, the training device 120 enhances the continuity between tasks by introducing a circular task embedding mechanism, which maps discrete task indexes into continuous and periodic embedding vectors. The training device 120 supports multi-scale modeling by introducing learnable scale embedding vectors, thus effectively adapting to tasks that rely on multi-scale features, such as object detection and semantic segmentation, improving the model's adaptability and expressive power. In addition, the training device 120 reduces its dependence on training data by reusing existing visual priors from VFM and VFT.

[0132] The above describes the complete process of a unified modeling method for multimodal tasks. The following section will combine... Figure 4 The process of a unified processing method for multimodal tasks is described. Figure 4 This is a flowchart illustrating a unified processing method for multimodal tasks based on stream matching, provided in an embodiment of this application. Figure 4 The training equipment 110 mentioned above can be referenced. Figure 1 The relevant descriptions in the embodiments of this application will not be repeated here.

[0133] like Figure 4 As shown, the unified processing method for multimodal tasks may include steps 401 to 404.

[0134] Step 401: The execution device 110 performs word segmentation on the information to be processed in the target modality to obtain the initialization token of the information to be processed.

[0135] The execution device 110 receives the information to be processed from the target modality. The modality to which the information to be processed belongs is the target modality that the trained unified task model can process. The execution device 110 inputs the information to be processed into a word segmenter for word segmentation to obtain an initial token after word segmentation of the information to be processed.

[0136] When the target modality is an image modality, the execution device 110 inputs the image to be processed into the VFM Tokenizer for word segmentation to obtain the initial tokenr0 of the image to be processed.

[0137] When the target modality is audio, the execution device 110 inputs the audio to be processed into the word segmenter for word segmentation to obtain the initialization token of the audio to be processed.

[0138] As one possible implementation, execution device 110 upsamples the initialization token to generate a multi-scale initialization.

[0139] Step 402: During the nth iteration, the execution device 110 inputs the single step size, the intermediate state token generated in the previous iteration, (n-1), and the identification information corresponding to the target task into the unified task model to obtain the predicted speed of the nth iteration.

[0140] The execution device 110 determines the single step length Δk for each iteration based on the preset total number of steps m. The total number of steps, m, can be configured by the user or by the administrator; this embodiment does not impose any restrictions on this. The total number of steps, m, represents the total number of iterations for the information to be processed.

[0141] The target task can be user-specified or any one of multiple tasks in a target modality. The execution device 110 determines the identification information of the target task.

[0142] As one possible implementation, when the target task's identification information is the target index number t, the m iterations of the information to be processed include the nth iteration. During the nth iteration, the execution device 110 will use the single step size Δk and the intermediate state token r generated in the previous iteration of the nth iteration. n-1 The inputs (n-1) and the target index t corresponding to the target task are used to input the unified model of the task, and the prediction speed of the nth iteration is fθ(r). n-1 ,(n-1),t).

[0143] For example, in the first iteration, the execution device 110 inputs the single step size Δk, the initialization tokens r0 and 0, and the target index number t of the target task into the unified task model to obtain the predicted velocity fθ(r0,0,t) for the first iteration. In the fifth iteration, the execution device 110 inputs the single step size Δk, the intermediate state tokens r4 and 4 generated in the fourth iteration, and the target index number t of the target task into the unified task model to obtain the predicted velocity fθ(r4,4,t) for the fifth iteration.

[0144] As one possible implementation, the identification information of the target task is the task embedding vector e of the target task. t During the nth iteration, execution device 110 combines the single step size Δk with the intermediate state token r generated in the previous iteration. n-1 (n-1) and the task embedding vector e corresponding to the target task t Input the unified model for the task, and obtain the prediction speed for the nth iteration as fθ(r). n-1 ,(n-1),e t ).

[0145] As one possible implementation, the execution device 110 also selects a preset scale embedding vector s corresponding to the output scale required by the target task, and embeds the single step size Δk and the intermediate state token r generated in the (n-1)th iteration. n-1 (n-1) The task embedding vector e corresponding to the target task t The input model is a unified model for the task, with a preset-scale embedding vector s. The prediction speed for the nth iteration is fθ(r). n-1 ,(n-1),e t ,s).

[0146] Step 403: The execution device 110 determines the intermediate state token generated after the nth iteration based on the predicted speed of the nth iteration, the single step size, and the intermediate state token generated in the previous iteration.

[0147] The execution device 110 predicts the velocity fθ(r) based on the nth iteration. n-1 ,(n-1),t), single step size Δk, intermediate state token r generated in the (n-1)th iteration n-1 The intermediate state token r generated after the nth iteration is determined by numerical integration. n .

[0148] As one possible implementation, the execution device 110 can determine the product of the single step size and the prediction velocity of the nth iteration, and determine the intermediate state token r generated after the nth iteration. n For intermediate state token rn-1 The sum of the products. That is, r. n =r n-1 +Δk·fθ(r n-1 ,(n-1),t).

[0149] As one possible implementation, the execution device 110 can also use machine learning to predict the intermediate token r. n .

[0150] Step 404: The execution device 110 inputs the target model representation obtained after m iterations into the decoder corresponding to the target task for decoding, and obtains the target task result corresponding to the target task.

[0151] After m iterations, the execution device 110 generates the intermediate state token r. m This is the target model representation. The execution device 110 inputs the target model representation into the decoder corresponding to the target task for decoding, and obtains the target task result corresponding to the target task.

[0152] When the target modality is an image modality, the execution device 110 decodes the target model representation into the VFT decoders of the target visual task to obtain the target task result corresponding to the target visual task.

[0153] For example, target vision tasks may include classification tasks, detection tasks, and semantic segmentation tasks. The execution device 110 inputs the target model representation into the classification head, detection head, and segmentation head respectively to obtain the category label for the classification task, the bounding box for the detection task, and the pixel segmentation map for the semantic segmentation task.

[0154] The classification head is a fundamental visual task model for classification tasks, specifically used to map the learned feature representations to the category prediction part. The classification head is usually located at the end of the model and contains a small number of layers, such as fully connected layers and activation functions.

[0155] The detector head is the last link in the visual foundation task model for detection tasks. The detector head typically includes multiple convolutional layers, pooling layers, and fully connected layers for classification and regression.

[0156] The segmentation head is the part of a visual model specifically designed to perform image segmentation tasks, which are fundamental to semantic segmentation. It is typically located at the end of the model and is responsible for converting feature maps into similarity-based classification predictions. The segmentation head usually includes convolutional layers, upsampling layers, and activation functions.

[0157] Based on the description of steps 401 to 404 above, when the target modality is an image and the target task is a visual task, the execution device 110 obtains an initial token by segmenting the information to be processed, and gradually evolves the target model representation of any target task from the initial token through m iterations according to the unified task model. The execution device 110 inputs the target model representation into the decoder corresponding to the target task for decoding, and obtains the target task result corresponding to the target task, thereby realizing the unified processing of all visual tasks.

[0158] Furthermore, the execution device 110 uses numerical integration to progressively evolve the initialized tokens to the target task representation. This allows for improved zero-shot performance by increasing the number of sampling steps, optimizing output quality without additional training and enhancing deployment flexibility. Moreover, the execution device 110 can simulate larger models with smaller ones, achieving a "small but powerful" knowledge transfer effect and providing a new paradigm for building lightweight unified models for visual tasks.

[0159] The preceding text described a unified modeling and processing method for multimodal tasks. The following text, in conjunction with specific embodiments, describes a unified modeling and processing method for multi-visual tasks where the target modality is image modality.

[0160] When the target visual task includes classification and detection tasks, the base visual model is a pre-trained DINOv2 model, whose parameters are locked during training and not updated. The base visual task models include a classification model and a detection model. The classification model is a DeiT model trained on the ImageNet visualization database. The detection model is a DETR model trained on the Common Objects in Context (COCO) dataset.

[0161] During the model training phase, the unified visual task system 200 acquires sample images from the training set. These images are then input into the DINOv2 model for word segmentation to obtain tokens. The tokens are upsampled and interpolated to the output scales corresponding to the classification and detection tasks, respectively. The unified visual task system 200 inputs the sample images into the DeiT model to extract model representations for the classification task and into the DETR model to extract model representations for the detection task. The unified visual task system 200 uses a flow-matching diffusion strategy to train the diffusion model, resulting in the trained unified visual task model 201. The diffusion model can be a randomly initialized DiT model, with the number of time steps y set to 10.

[0162] During the model inference phase, the unified visual task system 200 inputs the image to be processed into the DINOv2 model for word segmentation to obtain an initial token. Based on the target sequence number t of the target visual task and the initial image, it inputs the token into the trained unified visual task model 201, using numerical integration to evolve the initial image into the target model representation. The integration step size for numerical integration is set to 1. The unified visual task system 200 then inputs the target model representation into the classification head of the DeiT model and the detection head of the DETR model, respectively, obtaining the output target task results, including class labels and bounding boxes.

[0163] To verify the performance of the unified vision task model described above in real-world vision tasks, this application evaluates image classification and object detection on the ImageNet and COCO datasets, respectively. To verify the adaptability and stability of the method provided in this application under different model sizes, this application evaluates the model using two different sized models: the larger model DiT-B and the smaller model DiT-S.

[0164] Table 1 shows the evaluation results of different sized models provided in the embodiments of this application.

[0165] Table 1

[0166]

[0167] As shown in Table 1, using the method provided in this application, the accuracy of the large-scale DiT-B model trained for image classification on the ImageNet dataset is 82.32 and 81.45 in the untuned zero-shot learning setting, and 83.13 and 82.12 in the linear probing setting. This accuracy is similar to the 83.4 and 81.8 accuracies of the basic vision task model DeiT-B.

[0168] The trained, smaller-scale DiT-S model for image classification achieves an accuracy of 77.3 on the ImageNet dataset in the untuned zero-shot learning setting and 80.14 in the linear probing setting. This accuracy is comparable to the 79.8 accuracy of the base vision task model, DeiT-S.

[0169] The mean accuracy of the trained, large-scale DiT-B model performing object detection on the COCO dataset is 39.2 in the untuned zero-shot learning setting and 42.8 in the encoder-decoder-head setting. This mean accuracy is similar to the mean accuracy of 41.9 of the base vision task model DETR-BK.

[0170] Evaluation results show that, under a zero-sample setting without fine-tuning, the performance of the aforementioned unified visual task model on the two tasks is already close to that of its corresponding basic visual task models (such as DeiT and DETR). Furthermore, after fine-tuning with a small number of task-related decoders, the performance of the unified visual task model is further improved, surpassing the performance of the basic visual task models. The evaluation results demonstrate that the unified visual task model trained using the method provided in this application not only possesses powerful representation learning capabilities but can also quickly adapt to specific visual tasks under limited supervision signals.

[0171] The trained, smaller-scale DiT-S model and larger-scale DiT-B model for image classification both achieved an accuracy of around 80% on the ImageNet dataset, showing similar accuracies. Evaluation results indicate that regardless of model size, training using the method described in this application maintains good task performance and exhibits good consistency across different scales. This demonstrates that the unified vision task model trained using the method described in this application possesses excellent scaling ability, enabling flexible deployment in resource-constrained or demand-changing scenarios while maintaining stable performance.

[0172] To further verify the differences and advancements of the embodiments of this application compared to existing methods, we conducted comparative experiments with the traditional single-step knowledge distillation (KD) method. To further explore the potential of the method in model compression and knowledge transfer, we also conducted cross-scale knowledge transfer experiments. These experiments involved fitting the task representation of the larger model DiT-B to a smaller model, DiT-S.

[0173] Table 2 shows the results of the comparative experiment and knowledge transfer experiment provided in the embodiments of this application.

[0174] Table 2

[0175]

[0176] As shown in Table 2, using the method provided in this application, the accuracy of the large-scale DiT-B model trained for image classification on the ImageNet dataset is 81.45 in the untuned zero-shot learning setting and 82.24 in the single-step knowledge distillation setting. The accuracy of the basic vision task model DeiT-B on the ImageNet dataset is 81.8.

[0177] The trained, smaller-scale DiT-S model for image classification achieves an accuracy of 79.46 on the ImageNet dataset in the untuned zero-shot learning setting and 81.78 in the linear probing setting. The base vision task model, DeiT-B, achieves an accuracy of 79.8 on the ImageNet dataset.

[0178] Experimental results show that, under the premise that all experimental settings (including model structure, training data, optimizer parameters, etc.) remain consistent, the accuracy of DiT-B trained using the method of this application embodiment is superior to that trained using the single-step knowledge distillation method. This result highlights the advantages of the generative flow matching architecture adopted in this application embodiment compared to the traditional discriminative distillation method. Specifically, the flow matching mechanism can not only capture the dynamic evolution path from the base model to the task representation, but also has stronger modeling and generalization capabilities, thereby obtaining high-quality task output under the same conditions.

[0179] As shown in Table 2, under the zero-shot setting, the accuracy of the larger-scale DiT-B model for image classification, which is trained, is 81.45 on the ImageNet dataset, which is similar to the accuracy of the smaller-scale DiT-S model, which is 79.46.

[0180] Experimental results show that although DiT-S is much smaller than DiT-B in terms of parameter count and expressive power, it can still effectively approximate the task output of large models through the flow matching mechanism of this invention. This finding indicates that the generative unified modeling architecture proposed in this invention has powerful knowledge transfer and representation learning capabilities, achieving a "small but powerful" effect, and providing new ideas and directions for building efficient and compact visual unified models.

[0181] During the inference phase, embodiments of this application employ numerical integration to progressively evolve and initialize the token to the target model representation of the target task.

[0182] Table 3 shows the results of the impact of different sampling steps on the performance of the flow model provided in the embodiments of this application.

[0183] Table 3

[0184]

[0185] As shown in Table 3, the accuracy of the smaller-scale DiT-S model trained for image classification on the ImageNet dataset is 77.3 with 1 sampling step, 77.78 with 2 sampling steps, and 77.93 with 10 sampling steps. The accuracy of the basic vision task model DeiT-S on the ImageNet dataset is 79.8.

[0186] The results show that as the number of sampling steps increases, the performance of the DiT-S model continuously improves without fine-tuning, gradually approaching the level of the basic visual task model, DeiT-S. This phenomenon indicates that the generation process under the flow matching mechanism has higher path modeling accuracy, and that the model can better approximate the optimal task representation path as the number of sampling steps increases. This also means that the present invention can improve performance by adjusting the inference strategy without additional training, enhancing its practical value and flexibility.

[0187] The foregoing described the unified modeling and processing method for multimodal tasks based on flow matching, along with specific examples, provided in the embodiments of this application. The following will combine... Figure 5 and Figure 6 This describes the unified modeling and processing apparatus for multimodal tasks based on flow matching provided in this embodiment.

[0188] Figure 5 This is a schematic diagram of a possible multimodal task unified modeling apparatus 500 based on flow matching, provided in an embodiment of this application. The multimodal task unified modeling apparatus 500 can be used to implement the model training function of the training device 120 in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the multimodal task unified modeling apparatus 500 can be... Figure 1 The training device 120 used to implement the model training function can also be a module (such as a chip) applied to the training device 120.

[0189] like Figure 5 As shown, the multimodal task unified modeling device 500 includes: an acquisition module 501, a construction module 502, and a training module 503.

[0190] The acquisition module 501 is used to acquire the smallest representation unit token and model representation of samples for each of the multiple target tasks. The multiple target tasks are tasks of at least one modality, and the model representation is obtained by encoding the samples using the encoder of the corresponding target task. Model representations of different tasks in different modalities have the same representation form. For example, the acquisition module 501 is used to execute... Figure 3 Step 301 in the process.

[0191] Construction module 502 is used to construct intermediate tokens corresponding to each of multiple time steps based on the token and model representation; the multiple time steps are the steps in the evolution process from the token to the model representation. For example, construction module 502 is used to execute... Figure 3 Step 302 in the process.

[0192] Training module 503 is used to train a unified task model using a stream matching method based on the token, model representation, target task identifier information, each time step, and the corresponding intermediate token. The unified task model is used to predict the prediction speed of the target task at each time step, where the prediction speed represents the difference between the intermediate token and the target token. For example, training module 503 is used to execute... Figure 3 Step 303 in the process.

[0193] As one possible implementation, module 501 is specifically used to acquire samples from each of the multiple target tasks; to segment the samples to obtain their tokens; and to input the samples into the encoder of the corresponding target task for encoding to obtain the model representation.

[0194] As one possible implementation, at least one target modality is an image modality, and multiple target tasks include all visual tasks of the image modality, with the sample being a sample image; the acquisition module 501 is specifically used to input the sample image into the visual foundation model VFM for word segmentation to obtain the token of the sample image; and input the sample image into the visual foundation task model VFT of the corresponding target visual task for encoding to obtain the model representation.

[0195] As one possible implementation, the training module 503 is specifically used to input the target task's identification information, each time step, and the corresponding intermediate state token into the initial task unified model to obtain the predicted speed for each time step; obtain the true speed based on the difference between the model representation and the token; train the initial task unified model based on the predicted speed and the true speed for each time step until the mean square error between the predicted speed and the true speed for each time step is minimized, thus obtaining the trained task unified model.

[0196] As one possible implementation, the multimodal task unified modeling apparatus 500 also includes:

[0197] The determination module 504 is used to determine the preset scale embedding vector corresponding to the target output scale required by the target task; determine that the updated token is the sum of the token and the preset scale embedding vector; and determine that the updated token is a token adapted to the target output scale.

[0198] Training module 503 is specifically used to train a unified task model using a flow matching method based on the updated token, model representation, target task identification information, preset scale embedding vector, each time step and the corresponding intermediate token.

[0199] As one possible implementation, the multimodal task unified modeling apparatus 500 also includes:

[0200] The determination module 504 is used to determine the target angle mapped to the unit circle based on the target index number corresponding to the target task and the total number of tasks; determine the task embedding vector of the target dimension as the identification information expanded from the target angle; and determine the target dimension as the dimension of the token.

[0201] As one possible implementation, the unified task model is a velocity field function.

[0202] It should be understood that the multimodal task unified modeling device 500 in this application embodiment can be implemented using a GPU, a neural network processing unit (NPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Software implementation is also possible. Figure 3 When using the multimodal task unified modeling method shown, the multimodal task unified modeling device 500 and its various modules can also be software modules.

[0203] Figure 6 This is a schematic diagram of a possible multimodal task unified processing device 600 based on flow matching, provided in an embodiment of this application. The multimodal task unified processing device 600 can be used to implement the model training function of the execution device 110 in the above method embodiments, and therefore can also achieve the beneficial effects of the above method embodiments. In this embodiment, the multimodal task unified processing device 600 can be... Figure 1 The execution device 110 used to implement the model training function can also be a module (such as a chip) applied to the execution device 110.

[0204] like Figure 6 As shown, the multimodal task unified processing device 600 includes: a word segmentation module 601 and an inference module 602.

[0205] The word segmentation module 601 is used to segment the information to be processed in the target modality to obtain the initial token of the information to be processed; the m iterations of the information to be processed include the nth iteration. For example, the word segmentation module 601 is used to perform... Figure 4 Step 401 in the process.

[0206] Inference module 602 is used to process the intermediate state token r generated in the previous iteration with a single step size during the nth iteration. n-1 Input the identification information corresponding to (n-1) and the target task into the unified task model to obtain the prediction speed of the nth iteration; the single step size is the reciprocal of m, and when n is 1, r0 is the initialization token; based on the prediction speed of the nth iteration, the single step size, and the intermediate token r n-1 Determine the intermediate state token r generated after the nth iteration. n The target model representation obtained after m iterations is input into the decoder corresponding to the target task for decoding, thereby obtaining the target task result. For example, the inference module 602 is used to execute... Figure 4 Steps 402 to 404 in the process.

[0207] As one possible implementation, the inference module 602 is specifically used to determine the product of the single step size and the prediction velocity of the nth iteration; and to determine the intermediate state token r generated after the nth iteration. n For intermediate state token r n-1 The sum of the products.

[0208] As one possible implementation, the identification information is the task embedding vector corresponding to the target task.

[0209] As one possible implementation, the inference module 602 is specifically used to convert the single step size and r... n-1 The identification information and target scale embedding vector are input into the unified model of the task to obtain the prediction speed of the nth iteration; the target scale embedding vector is the preset scale embedding vector corresponding to the target output scale required by the target task.

[0210] It should be understood that the multimodal task unified processing device 600 of this application embodiment can be implemented by a GPU, a neural network processing unit (NPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. Software implementation is also possible. Figure 4In the multimodal task unified processing method shown, the multimodal task unified processing device 600 and its various modules can also be software modules.

[0211] This application also provides an electronic device, please refer to... Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 700 includes a memory 701, a processor 702, a communication interface 703, and a bus 704. The memory 701, processor 702, and communication interface 703 are interconnected via the bus 704.

[0212] The memory 701 can be a read-only memory, a static storage device, a dynamic storage device, or a random access memory. The memory 701 can store computer instructions. When the computer instructions stored in the memory 701 are executed by the processor 702, the processor 702 and the communication interface 703 are used to execute the steps in the unified modeling and processing method for multimodal tasks based on flow matching. For example, the processor 702 is used to execute the aforementioned... Figure 3 The steps in the unified modeling method for multimodal tasks based on flow matching shown below. Figure 4 The steps in the unified processing method for multimodal tasks based on stream matching, as shown above, are as follows: Figure 5 The multimodal task unified modeling device 500 and Figure 6 The functions of the multimodal task unified processing device 600.

[0213] Processor 702 may be a general-purpose CPU, an application-specific integrated circuit (ASIC), a GPU, or any combination thereof. Processor 702 may include one or more chips.

[0214] The communication interface 703 uses a transceiver module, such as, but not limited to, a transceiver, to enable communication between the electronic device 700 and other devices or communication networks.

[0215] Bus 704 may include a pathway for transmitting information between various components of electronic device 700 (e.g., memory 701, processor 702, communication interface 703).

[0216] Electronic device 700 can be a computer (e.g., a server) in a cloud data center, a computer in an edge data center, or a terminal. For example, electronic device 700 is... Figure 1 The execution device 110 in the middle.

[0217] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on an electronic device or stored on any usable medium. When the computer program product runs on an electronic device, it causes the electronic device to execute a unified modeling method for multimodal tasks based on flow matching, or a unified processing method for multimodal tasks based on flow matching.

[0218] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that an electronic device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the electronic device to execute a unified modeling method for multimodal tasks based on flow matching, or a unified processing method for multimodal tasks based on flow matching.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A unified modeling method for multimodal tasks based on flow matching, characterized in that, The method includes: Obtain the smallest representation unit token and model representation of the samples of each of the multiple target tasks; the multiple target tasks are tasks of at least one modality, and the model representation is obtained by encoding the samples through the encoder of the corresponding target task. The model representations of different tasks of different modalities have the same representation form. Based on the token and the model representation, an intermediate token is constructed for each of the multiple time steps; the multiple time steps are the steps in the evolution process from the token to the model representation; Based on the token, the model representation, the identification information of the target task, each time step and the corresponding intermediate token, a unified task model is trained using a flow matching method. The unified task model is used to predict the prediction speed of each time step of the target task, and the prediction speed represents the difference between the intermediate token and the token.

2. The method according to claim 1, characterized in that, The process of obtaining the smallest representation unit token and model representation of samples from each of the multiple target tasks includes: Obtain the sample for each of the plurality of target tasks; The sample is segmented into words to obtain the token of the sample; The sample is input into the encoder of the corresponding target task for encoding to obtain the model representation.

3. The method according to claim 2, characterized in that, The at least one target modality is an image modality, the plurality of target tasks include all visual tasks of the image modality, and the sample is a sample image; the step of segmenting the sample to obtain the token of the sample includes: The sample image is input into the Visual Foundation Model (VFM) for word segmentation to obtain the token of the sample image; The step of inputting the sample into the encoder corresponding to the target task for encoding to obtain the model representation includes: The sample image is input into the Visual Basic Task Model (VFT) of the corresponding target visual task for encoding to obtain the model representation.

4. The method according to any one of claims 1-3, characterized in that, The step of training a unified task model using a stream matching method based on the token, the model representation, the identifier information of the target task, each time step and its corresponding intermediate token includes: The identification information of the target task, each time step and the corresponding intermediate state token are input into the initial task unified model to obtain the prediction speed corresponding to each time step; The actual speed is obtained based on the difference between the model representation and the token; The initial unified task model is trained based on the predicted speed and the actual speed at each time step until the mean square error between the predicted speed and the actual speed at each time step is minimized, thus obtaining the trained unified task model.

5. The method according to any one of claims 1-4, characterized in that, Before training a unified task model using a stream matching method based on the token, the model representation, the target task's identifier information, each time step, and the corresponding intermediate state token, the method further includes: Determine the preset scale embedding vector corresponding to the target output scale required for the target task; The updated token is determined to be the sum of the token and the preset scale embedding vector; the updated token is a token adapted to the target output scale; The step of training a unified task model using a stream matching method based on the token, the model representation, the identifier information of the target task, each time step and its corresponding intermediate token includes: The unified task model is trained using a stream matching method based on the updated token, the model representation, the target task identification information, the preset scale embedding vector, each time step and the corresponding intermediate token.

6. The method according to any one of claims 1-5, characterized in that, Before training a unified task model using a stream matching method based on the token, the model representation, the target task's identifier information, each time step, and the corresponding intermediate state token, the method further includes: Based on the target index number corresponding to the target task and the total number of tasks, determine the target angle that the target index number maps to the unit circle; The identification information is determined to be a task embedding vector of the target dimension expanded from the target angle; the target dimension is the dimension of the token.

7. The method according to any one of claims 1-6, characterized in that, The unified model for the task is a velocity field function.

8. A unified processing method for multimodal tasks based on flow matching, characterized in that, The method includes: The information to be processed in the target modality is segmented to obtain the initialization token of the information to be processed; the m iterations of the information to be processed include the nth iteration; During the nth iteration, the single step size and the intermediate state tokenr generated in the previous iteration are used. n-1 The identification information corresponding to (n-1) and the target task is input into the unified task model to obtain the prediction speed of the nth iteration; the single step size is the reciprocal of m, and when n is 1, r0 is the initialization token; Based on the prediction speed of the nth iteration, the single step size, and the intermediate state token r n-1 Determine the intermediate state token r generated after the nth iteration. n ; The target model representation obtained after m iterations is input into the decoder corresponding to the target task for decoding, thereby obtaining the target task result corresponding to the target task.

9. The method according to claim 8, characterized in that, The prediction speed based on the nth prediction, the single step size, and the intermediate state token r n-1 Determine the intermediate state token r generated after the nth iteration. n ,include: Determine the product of the single step size and the prediction speed of the nth iteration; Determine the intermediate state token r generated after the nth iteration. n For the intermediate state token r n-1 The sum of the products.

10. The method according to claim 8 or 9, characterized in that, The identification information is the task embedding vector corresponding to the target task.

11. The method according to any one of claims 8-10, characterized in that, The single step size, the intermediate state token r generated by the previous iteration of the nth iteration, is... n-1 The identification information corresponding to (n-1) and the target task is input into the unified task model to obtain the prediction speed of the nth iteration, including: The single step size, the r n-1 The (n-1)th step, the identification information, and the target scale embedding vector are input into the unified task model to obtain the prediction speed of the nth iteration; the target scale embedding vector is a preset scale embedding vector corresponding to the target output scale required by the target task.

12. A unified modeling device for multimodal tasks based on flow matching, characterized in that, The device includes: The acquisition module is used to acquire the smallest representation unit token and model representation of the samples of each target task in multiple target tasks; the multiple target tasks are tasks of at least one modality, and the model representation is obtained by encoding the samples through the encoder of the corresponding target task. The model representations of different tasks of different modalities have the same representation form. A construction module is used to construct an intermediate token corresponding to each of multiple time steps based on the token and the model representation; the multiple time steps are the steps in the evolution process from the token to the model representation; The training module is used to train a unified task model using a stream matching method based on the token, the model representation, the identification information of the target task, each time step and the corresponding intermediate token. The unified task model is used to predict the prediction speed of each time step of the target task, and the prediction speed represents the difference between the intermediate token and the token.

13. A unified processing device for multimodal tasks based on stream matching, characterized in that, The device includes: The word segmentation module is used to segment the information to be processed in the target modality to obtain the initialization token of the information to be processed; the m iterations of the information to be processed include the nth iteration; The inference module is used to, during the nth iteration, convert the single-step size of the intermediate state token r generated in the previous iteration of the nth iteration into a single-step size. n-1 The identification information corresponding to (n-1) and the target task is input into the unified task model to obtain the prediction speed of the nth iteration; the single step size is the reciprocal of m, and when n is 1, r0 is the initialization token; based on the prediction speed of the nth iteration, the single step size, and the intermediate token r n-1 Determine the intermediate state token r generated after the nth iteration. n The target model representation obtained after m iterations is input into the decoder corresponding to the target task for decoding to obtain the target task result.

14. An electronic device, characterized in that, The electronic device includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the electronic device to perform the method as claimed in any one of claims 1-7 or 8-11.

15. A computer program product containing instructions, characterized in that, When the instructions are executed by an electronic device or a cluster of computing devices, the electronic device causes the electronic device to perform the method as described in any one of claims 1-7 or 8-11.

16. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by an electronic device, perform the method as claimed in any one of claims 1-7 or 8-11.

17. A chip system, characterized in that, The chip system includes a memory and a processor. The memory is used to store a set of computer instructions. When the processor executes the set of computer instructions, it performs the method as described in any one of claims 1-7 or 8-11.

Images