Bimodal data prediction method, hybrid expert model training method and device
By employing a hybrid expert model training method, selecting important data, and performing modality separation and semantic alignment, the problem of high computational cost in large bimodal image-text models is solved, achieving efficient bimodal data prediction and training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA NAT PETROLEUM CORP
- Filing Date
- 2024-12-04
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, bimodal large image-text models consume a lot of computation when integrating information across modalities and generating intermodal content, and their training efficiency and modality perception sparsity are poor.
A hybrid expert model training method is adopted. The initial gating network and the initial expert network are trained by selecting bimodal sample data with importance above the threshold. The data selection unit and the self-attention unit are combined to perform modality separation and semantic alignment, thereby constructing a hybrid expert model, reducing the computation of unimportant data and improving the pre-training efficiency.
It significantly reduces computational load, improves the model's resource efficiency and capability in bimodal data prediction, solves the problem of modality-aware sparsity, and enhances pre-training efficiency.
Smart Images
Figure CN122153546A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of artificial intelligence technology, and in particular to bimodal data prediction methods, hybrid expert model training methods and devices. Background Technology
[0002] With the continuous development of artificial intelligence technology, numerous AI systems have emerged, such as the construction and application of bimodal large models for image and text data types. A large model refers to a machine learning model with a large number of parameters and complex computational structures, typically built from deep neural networks. The purpose of large models is to improve their expressive power and predictive performance, enabling them to handle more complex tasks and data.
[0003] Taking a bimodal image-text model as an example, one approach is to use a hybrid modal base model (such as the Gemini hybrid modal base model), which processes hybrid modal inputs and generates hybrid modal outputs by fusing modality-specific encoders or decoders. Hybrid modal base models are typically designed with an architecture that fuses two modality-specific encoders or decoders. This approach limits the model's ability to integrate information across modalities and generate intermodal content.
[0004] Another approach is to employ the Chameleon bimodal large model, which uses a single Transformer architecture and a next-to-the-token prediction objective to simulate mixed-modal sequences composed of discrete images and text tokens, allowing for seamless inference and generation across modalities. The drawback of the Chameleon bimodal model is that extending the model fusion can lead to significant computational overhead with a large number of parameters. Summary of the Invention
[0005] To address the problems of the prior art, embodiments of this specification provide a dual-modal data prediction method, a hybrid expert model training method, and an apparatus.
[0006] This specification provides a bimodal data prediction method, the method comprising: acquiring bimodal data to be predicted; inputting the bimodal data to be predicted into a bimodal expert fusion system to obtain a sentiment classification result output by the bimodal expert fusion system, wherein the sentiment classification result corresponds to the bimodal data to be predicted, and the bimodal expert fusion system includes a hybrid expert model, the hybrid expert model being obtained by training an initial gating network and an initial expert network using bimodal sample data with a central importance higher than a preset threshold in the training sample dataset.
[0007] According to one aspect of an embodiment of this specification, the bimodal expert fusion system further includes: a data filtering unit connected before the hybrid expert model, the data filtering unit outputting data as input data of the hybrid expert model, the data filtering unit including a projection transformation layer for filtering out bimodal data with an importance higher than a preset threshold from the bimodal data to be predicted.
[0008] This specification provides a hybrid expert model training method, the method comprising: acquiring a training sample dataset; inputting the training sample dataset into an initial gating network and an initial expert network respectively, to obtain the initial weights assigned by the initial gating network to each expert in the initial expert network, and the initial feature vector output by the initial expert network; weighted summing the initial weights and the initial feature vectors to obtain an initial image feature vector and an initial text feature vector; semantically aligning the initial image feature vector and the initial text feature vector to obtain a semantically aligned vector; constructing a loss function based on the semantically aligned vector, and using the loss function to update and iterate the parameters of the initial gating network and the initial expert network to construct a hybrid expert model.
[0009] According to one aspect of an embodiment of this specification, the training sample dataset is determined as follows: raw bimodal sample data is input to a data filtering unit to obtain the weight of each token in the raw bimodal sample data; a first bimodal sample data is filtered from the raw bimodal sample data according to the weight; the first bimodal sample data is input to a self-attention unit to obtain a third bimodal sample data; the third bimodal sample data is modally separated to obtain separated image modal sample data and text modal sample data; the image modal sample data and text modal sample data constitute the training sample dataset.
[0010] According to one aspect of the embodiments of this specification, image modal sample data is input to a first initial gating network and a first initial expert network to obtain a first initial weight output by the first initial gating network and a first initial feature vector output by the first initial expert network; text modal sample data is input to a second initial gating network and a second initial expert network to obtain a second initial weight output by the second initial gating network and a second initial feature vector output by the second initial expert network; the first initial weight and the first initial feature vector are weighted and summed to obtain an image initial feature vector; the second initial weight and the second initial feature vector are weighted and summed to obtain a text initial feature vector.
[0011] According to one aspect of an embodiment of this specification, the method further includes: filtering second bimodal sample data from the original bimodal sample data according to the weights, inputting the first bimodal sample data into a self-attention unit to obtain weights between the tokens of the first bimodal sample data;
[0012] The loss function and the hierarchical hybrid expert model are determined as follows: the semantically aligned vector is concatenated with the second bimodal sample data to obtain the fourth bimodal sample data; the loss function is determined based on the difference between the sentiment classification result corresponding to the original bimodal sample data and the sentiment classification result corresponding to the fourth bimodal sample data; the weights between the tokens of the first bimodal sample data are iteratively updated by backpropagation based on the loss function to obtain the constructed hybrid expert model.
[0013] According to one aspect of the embodiments of this specification, the sentiment classification result corresponding to the fourth bimodal sample data is obtained by inputting the fourth bimodal sample data into a sentiment classifier, and obtaining the sentiment classification result corresponding to the fourth bimodal sample data.
[0014] This specification provides a bimodal data prediction device, comprising: a data acquisition unit for acquiring bimodal data to be predicted; and an output unit for inputting the bimodal data to be predicted into a bimodal expert fusion system to obtain a sentiment classification result output by the bimodal expert fusion system, wherein the sentiment classification result corresponds to the bimodal data to be predicted, and the bimodal expert fusion system includes a hybrid expert model, which is obtained by training an initial gating network and an initial expert network using bimodal sample data with a central importance higher than a preset threshold in the training sample dataset.
[0015] This specification provides a hybrid expert model training device, comprising: a training sample dataset acquisition unit for acquiring a training sample dataset; an acquisition unit for inputting the training sample dataset into an initial gating network and an initial expert network respectively, to obtain the initial weights assigned by the initial gating network to each expert in the initial expert network, and the initial feature vectors output by the initial expert network; an initial feature vector acquisition unit for weighted summation of the initial weights and the initial feature vectors to obtain initial image feature vectors and initial text feature vectors; a semantic alignment unit for semantically aligning the initial image feature vectors and initial text feature vectors to obtain semantically aligned vectors; and a construction unit for constructing a loss function based on the semantically aligned vectors, and using the loss function to update and iterate the parameters of the initial gating network and the initial expert network to construct a hybrid expert model.
[0016] This specification also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the bimodal data prediction method and the hybrid expert model training method.
[0017] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the bimodal data prediction method and the hybrid expert model training method.
[0018] This solution enables the slicing and distribution of multimodal data, reducing the amount of data computation; the hybrid expert model enables the model to better capture the features of each modality, solving the problem of modality-aware sparsity. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 The diagram shown is a flowchart of a dual-modal data prediction method according to an embodiment of this specification.
[0021] Figure 2 The diagram shown is a flowchart of a method for training a hybrid expert model according to an embodiment of this specification.
[0022] Figure 3 The diagram shown is a flowchart of a method for determining a training sample dataset according to an embodiment of this specification.
[0023] Figure 4 The diagram shown is a flowchart of a method for determining initial feature vectors of an image and initial feature vectors of text according to an embodiment of this specification.
[0024] Figure 5 The diagram shown is a flowchart of a method for determining a loss function and a hierarchical hybrid expert model according to an embodiment of this specification.
[0025] Figure 6 The diagram shown is a schematic diagram of an image-text dual-modal expert fusion system according to an embodiment of this specification.
[0026] Figure 7 The diagram shown is a structural schematic of a dual-modal data prediction device according to an embodiment of this specification.
[0027] Figure 8The diagram shown is a structural schematic of a hybrid expert model training device according to an embodiment of this specification.
[0028] Figure 9 The diagram shown is a structural schematic of a computer device according to an embodiment of this specification.
[0029] Explanation of symbols in the attached drawings:
[0030] 701. Data Acquisition Unit;
[0031] 702. Output Unit;
[0032] 801. Training Sample Dataset Acquisition Unit;
[0033] 802. Acquisition Unit;
[0034] 803. Initial Feature Vector Acquisition Unit;
[0035] 804, Semantic Alignment Unit;
[0036] 805. Building Unit;
[0037] 902. Computer equipment;
[0038] 904, Processor;
[0039] 906. Memory;
[0040] 908. Drive mechanism;
[0041] 910. Input / Output Module;
[0042] 912. Input devices;
[0043] 914. Output devices;
[0044] 916. Presentation equipment;
[0045] 918. Graphical User Interface;
[0046] 920. Network interface;
[0047] 922. Communication link;
[0048] 924. Communication bus. Detailed Implementation
[0049] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.
[0050] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0051] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.
[0052] It should be noted that the bimodal data prediction method, hybrid expert model training method and device described in this specification can be used in the field of artificial intelligence technology, but this specification does not limit the application areas of the bimodal data prediction method, hybrid expert model training method and device.
[0053] Figure 1 The diagram shown is a flowchart of a dual-modal data prediction method according to an embodiment of this specification, which specifically includes the following steps:
[0054] Step 101: Obtain the bimodal data to be predicted. In this specification, "modality" refers to the data modality, which represents the type or form of data and describes how the data is perceived, represented, and processed. Specifically, modalities can include: numerical, text, image, audio, video, and other similar modalities. Bimodal data can refer to data in both text and image modalities, or it can be other modalities, such as text and audio data, text and video data, video and audio data, image and audio data, image and video data, or audio and image data, or any combination thereof.
[0055] Step 102: Input the bimodal data to be predicted into the bimodal expert fusion system to obtain the sentiment classification result output by the bimodal expert fusion system. The sentiment classification result corresponds to the bimodal data to be predicted. The bimodal expert fusion system includes a hybrid expert model. The hybrid expert model is obtained by training an initial gating network and an initial expert network with bimodal sample data whose central importance is higher than a preset threshold in the training sample dataset.
[0056] The bimodal expert fusion system described in this specification employs a sparse architecture, where the hybrid expert model utilizes a novel hybrid modality, early-stage fusion language model pre-training architecture. A hybrid expert model is a machine learning model that decomposes a large AI model into multiple specialized subnetworks (or "experts"), each focusing on processing a specific subset of the input data to collectively complete a task. Hybrid expert models significantly reduce computational costs during pre-training and inference.
[0057] In this specification, the hybrid expert model further includes a gating network and an expert network. The expert network is divided into multiple expert groups belonging to different modalities. Each expert is a specialized sub-model responsible for handling different tasks or subsets of data. Each expert is typically an independent neural network, processing tokens in image and text data respectively. Within each group, learned routes are used to maintain semantic information adaptability. The hybrid expert model in this application enables the model to better capture features of each modality, addressing the modality-aware sparsity problem.
[0058] In the embodiments of this specification, the bimodal expert fusion system further includes a data filtering unit, which includes a deep feature residual fusion component. The data filtering unit is connected before the hybrid expert model, and its output data is the input data of the hybrid expert model. The data filtering unit further includes a projection transformation layer for filtering out bimodal data with an importance higher than a preset threshold from the bimodal data to be predicted. In this application, the bimodal expert fusion system performs triage and filtering on the bimodal data to be predicted, reducing the computation process for unimportant data, thereby reducing the overall data computation volume.
[0059] The core advantage of this application lies in its modality-specific parameter allocation, which significantly improves pre-training efficiency. Experimental data shows that with a training budget of 1 billion tokens, a 1.4B parameter model with 4 text experts and 4 image experts saves a significant number of floating-point operations (FLOPs) compared to a computationally equivalent dense baseline. Furthermore, combining the hybrid expert model with the deep feature residual fusion component in the data filtering unit further improves pre-training FLOPs savings to an overall 4.0x (3.2x for text and 4.8x for images). In summary, this application demonstrates the enormous potential to improve resource efficiency and capabilities in bimodal AI systems and provides a scalable framework for future development in training hybrid modality-based models.
[0060] The bimodal data prediction method and hybrid expert model training method described in this specification can solve the problems of poor training efficiency and poor modality perception sparsity of existing bimodal large models.
[0061] Figure 2 The diagram shown is a flowchart of a method for training a hybrid expert model according to an embodiment of this specification, which specifically includes the following steps:
[0062] Step 201: Obtain the training sample dataset. In this specification, the training sample dataset consists of two types of unimodal sample data obtained from the original bimodal sample data through a series of processing and transformations. For details of the specific processing and transformation procedures, please refer to... Figure 3 Partial description. The two types of unimodal sample data are text modal sample data and image modal sample data.
[0063] Step 202: Input the training sample dataset into the initial gating network and the initial expert network respectively to obtain the initial weights assigned to each expert in the initial expert network by the initial gating network, and the initial feature vector output by the initial expert network. Specifically, the initial gating network predicts the weights of each expert in the initial expert network for the corresponding modality based on the sample data of the corresponding modality in the training sample dataset; the initial expert network predicts the feature vector for the corresponding modality based on the sample data of the corresponding modality in the training sample dataset.
[0064] Step 203: The initial weights and the initial feature vectors are weighted and summed to obtain the initial feature vectors for the image and the text. In this step, the initial weights and the initial feature vectors are weighted and added to the relevant modality expert group to form high-dimensional feature representations for two modalities, including: the initial feature vector for the image modality and the initial feature vector for the text modality.
[0065] Step 204: Semantically align the initial feature vector of the image and the initial feature vector of the text to obtain a semantically aligned vector.
[0066] In this specification, the hybrid expert model also includes a modality mixing projection layer, which is connected after the expert network. In this step, the high-dimensional features of the initial image feature vector and the initial text feature vector output by the expert network are input into the modality mixing projection layer for feature alignment (see...). Figure 6 It learns the association between tokens in text modal data and tokens in image modal data at the semantic level, thus achieving semantic alignment.
[0067] Step 205: Construct a loss function based on the semantically aligned vectors, and use the loss function to update and iterate the parameters of the initial gating network and the initial expert network to construct a hybrid expert model.
[0068] In this step, the semantically aligned vectors are concatenated with bimodal data whose importance is below a preset threshold selected from the original bimodal sample data. The sentiment classification result of the concatenated vector is determined, and its difference from the sentiment classification result (labeled data, pre-acquired before training) corresponding to the original text-image bimodal data is calculated to construct a loss function. This loss function is then used for extensive backpropagation iterations of the initial gating network and the initial expert network until the initial hybrid expert model composed of the initial gating network and the initial expert network converges, thus constructing the hybrid expert model.
[0069] Figure 3 The diagram shown is a flowchart of a method for determining a training sample dataset according to an embodiment of this specification, which specifically includes the following steps:
[0070] Step 301: Input the original bimodal sample data into the data filtering unit to obtain the weight of each token in the original bimodal sample data.
[0071] In this specification, the data filtering unit is actually a deep feature residual fusion component. The data filtering unit is positioned before the self-attention unit in the Transformer architecture, and calculates the weight of each token in the original bimodal data by performing a projection transformation on the original bimodal data.
[0072] The original bimodal sample data in this step is in the form of a dataset, which can be represented by X, X = {x1, x2, ..., x...} n}(x iThis represents the token for the i-th original bimodal sample data. The original bimodal sample data first enters the deep feature residual fusion component, which calculates the token weight for each original bimodal sample data. Each token is then split according to its corresponding weight value. Tokens with larger weight values are considered important and are input into the self-attention layer of the Transformer structure in the next step. Tokens with smaller weight values are considered unimportant and are filtered out, preventing them from proceeding to the next step. This process distinguishes the original bimodal sample data and reduces the computational load of the self-attention layer in the Transformer structure. The self-attention layer of the Transformer structure is connected after the data filtering unit and before the initial hybrid expert model. It should be noted that the importance of the tokens in the multimodal data represents the correlation between the original bimodal sample data and other original bimodal sample data; strong correlation indicates that the token is important, and weak correlation indicates that the token is unimportant.
[0073] In the hybrid expert model construction stage, the data screening unit of this application performs diversion and screening of the original bimodal sample data, reducing the calculation of unimportant data, thereby reducing the overall data calculation volume.
[0074] Step 302: Based on the weights, filter the original bimodal sample data to obtain the first bimodal sample data. Specifically: bimodal data with weights greater than or equal to the preset weights are used as the first bimodal sample data; bimodal data with weights less than the preset weights are used as the second bimodal sample data.
[0075] As described in step 301, the first bimodal sample data can be data with a higher token weight value, and the second bimodal sample data can be data with a lower token weight value. The first bimodal sample data continues to participate in the calculation as input for the next step, while the second bimodal sample data is no longer involved. Figure 3 Subsequent calculations. Specifically, based on the preset weights and the weights calculated in step 301, the first bimodal sample data and the second bimodal sample data are distinguished. For example, assuming the preset weight value is 0.5, if the weight of each token in the original bimodal sample data calculated in step 301 is greater than or equal to 0.5, then the token is sent to the self-attention layer for the next calculation; if the weight is less than 0.5, then the calculated result {x} is not processed. j ,...,x k In subsequent step 501, the tokens are multiplied with the semantically aligned vector output by the modality mixing projection layer. In this specification, the tokens of the first bimodal sample data are represented by the dataset {x}. p ,....,x qThe tokens of the second bimodal sample data are presented in the form of the dataset {x}. j ,...,x k Presented in the form of}.
[0076] Step 301, combined with step 302, calculates the weight of each token in the original bimodal image-text data using the following expression, and then filters the data based on the weight values to obtain the first and second bimodal sample data. The formula is as follows:
[0077] {x j ,...,x k},{x p ,....,x q} = softmax(X·W%) A ); where X represents the original bimodal sample data, W% A W% represents the projection transformation matrix. A It is a two-layer MLP, where softmax represents the normalization operation, {x j ,...,x k} represents the token for the second bimodal sample data, {x p ,....,x q} represents the token of the first bimodal sample data. After performing a projection transformation on the original bimodal sample data, softmax normalization is then performed to compare the weight values with the preset weight values, thereby filtering out important or useful tokens.
[0078] Step 303: Input the first bimodal sample data into the self-attention unit to obtain the third bimodal sample data.
[0079] Self-attention mechanisms allow models to analyze the relationships between different parts of the input data. A self-attention unit assigns a weight to each word in the input sequence to reflect its relevance to the current context. This allows the model to focus its attention on important words while downplaying the importance of less relevant words.
[0080] In this step, obtaining the third bimodal sample data includes weighting the first bimodal sample data to obtain the weights between the tokens in the first bimodal sample data. Specifically, the first image-text bimodal data is input into the self-attention layer of the Transformer architecture for weight calculation, and the attention score is used as the weight between the first image-text data tokens to characterize the importance between the first image-text data tokens.
[0081] Please refer to the following steps for details:
[0082] (1) Calculate the query vector Q according to the following expression. p Key value vector Kp Sum vector V i :
[0083] Q p =x p W Q +b Q ;
[0084] K p =x p W K +b K ;
[0085] V p =x p W V +b V , where x p Q represents the token for the p-th first bimodal sample data. p x represents p The corresponding query vector, K p x represents p The corresponding key-value vector, V p x represents p The corresponding value vector; W Q b represents the weight matrix corresponding to the query vector; Q W represents the bias vector corresponding to the query vector. K b represents the weight matrix corresponding to the key-value vector. K W represents the bias vector corresponding to the key-value vector. V b represents the weight matrix corresponding to the value vector. V This represents the bias vector corresponding to the value vector.
[0086] (2) According to the expression Calculate x p The corresponding attention score. Where Q... p Let K represent the p-th query vector. p Let d represent the p-th key-value vector. k The dimension of the key-value vectors of the Q and K matrices is used for normalization operations.
[0087] (3) According to the expression Output = score·V p Where Vp represents x p The corresponding value vector. Output represents the output result of the self-attention unit performing weight calculations.
[0088] (4) According to the expression headoutput = Output·W O +b O Perform a linear transformation, where Output represents the output of the self-attention unit after weight calculation, WO Let b represent a matrix. O This represents the bias value of the matrix. Together, they are used to convert the Output into a new vector. Furthermore, the dimensions of the headoutput are the same as the Output dimension; there is no change in dimension during this vector conversion, only a change in numerical value.
[0089] The normalized result of the head output vector is used as the input for the next step, i.e., the third bimodal sample data. The tokens in the third bimodal data all have pre-calculated weights.
[0090] Step 304 involves modal separation of the third bimodal sample data to obtain separated image modal sample data and text modal sample data. In this step, the third bimodal sample data is input into a modality router, which performs modal separation to obtain image modal sample data and text modal sample data. In subsequent steps, the image modal sample data is input into the initial gating network corresponding to the image modality, and the text modal sample data is input into the initial gating network corresponding to the text modality. Expert weights are calculated for each modality, and the weights are weighted to form high-dimensional feature representations for both modalities.
[0091] Step 305: Use the image modal sample data and text modal sample data as the training sample dataset.
[0092] Image and text modality sample data can be used as training sample datasets in step 201, respectively, to train the initial gating network and initial expert network for each modality. In this specification, it is also necessary to pre-obtain the sentiment classification results corresponding to the original bimodal sample data, which, together with the training sample dataset, participate in the training of the hybrid expert model.
[0093] Figure 4 The diagram shown is a flowchart of a method for determining initial feature vectors of an image and initial feature vectors of text according to an embodiment of this specification, which specifically includes the following steps:
[0094] Step 401: Input the image modal sample data into the first initial gating network and the first initial expert network to obtain the first initial weights output by the first initial gating network and the first initial feature vector output by the first initial expert network.
[0095] In this specification, the initial gating network includes an initial text gating network and an initial image gating network, such as... Figure 6As shown, the initial image gating network is the first initial gating network, and the initial text gating network is the second initial gating network. The first initial gating network is used to predict the weight of each expert in the first initial expert network based on the input image modality sample data. The first initial gating network and the first initial expert network constitute the image processing part of the initial hybrid expert model. In this specification, the image modality sample data is input to the first initial gating network and the first initial expert network for prediction. In the embodiments of this specification, the token of the image modality sample data is connected to the first initial gating network W. i Calculations are performed to obtain the corresponding weights α. i .
[0096] Furthermore, each expert in the first initial expert network processes the tokens of the input image modality sample data to obtain the expert output result res. i The output of the first initial expert network is the feature vector of the image modality sample data, as shown in the following formula:
[0097] res i =expert(h·W i ); where W i Let be the learnable parameters of the i-th expert, i.e., the formal representation of the learnable parameters of each expert, where h is the image modality sample data, and res is the receptive parameter. i The output of each expert in the first initial expert network is represented by the first initial feature vector.
[0098] Step 402: Input the text modality sample data into the second initial gating network and the second initial expert network to obtain the second initial weights output by the second initial gating network and the second initial feature vector output by the second initial expert network. In this step, the second initial gating network and the second initial expert network constitute the part of the initial hybrid expert model responsible for text processing. In this specification, the text modality sample data is input into the second initial gating network and the second initial expert network for prediction.
[0099] In this specification, text sample data is input into a second initial gating network and a second initial expert network for prediction. In the embodiments of this specification, the tokens of the text sample data are calculated with the second initial gating network to obtain corresponding weights.
[0100] Furthermore, each expert in the second initial expert network processes the tokens in the input text sample data to obtain the expert output result res. i The feature vector of the text modality sample data output by the second initial expert network is calculated using the formula in step 401. This step will not be elaborated here.
[0101] Step 403: The first initial weights and the first initial feature vector are weighted and summed to obtain the initial image feature vector. In this step, the output of the first initial expert network is weighted and summed with the first initial weights predicted by the first initial gating network to obtain the output initial image feature vector. Where, α i Represents the first initial weight, res i This represents the first initial eigenvector.
[0102] Step 404: The second initial weights and the second initial feature vector are weighted and summed to obtain the initial text feature vector. This step is similar to step 403; the output of the second initial expert network is weighted and summed with the second initial weights predicted by the second initial gating network to obtain the output initial text feature vector. Where, α i Represents the second initial weight, res i This represents the second initial eigenvector.
[0103] Figure 5 The diagram shown is a flowchart of a method for determining a loss function and a hierarchical hybrid expert model according to an embodiment of this specification, which specifically includes the following steps:
[0104] Step 501: Concatenate the semantically aligned vector with the second bimodal sample data to obtain the fourth bimodal sample data. In this step, the semantically aligned vector is a bimodal vector, and the second bimodal sample data consists of less important data from the original bimodal sample data. At this stage, concatenating this data with the semantically aligned vector and the second bimodal sample data yields new bimodal data, referred to as the fourth bimodal sample data. The fourth bimodal sample data is also in vector form. In this specification, the fourth bimodal sample data has the same length as the original bimodal sample data, but its content has changed. The changed data represents the more important part of the original bimodal sample data.
[0105] Step 502: Determine the loss function based on the difference between the sentiment classification result corresponding to the original bimodal sample data and the sentiment classification result corresponding to the fourth bimodal sample data.
[0106] Before performing this step, this specification also includes: inputting the fourth bimodal sample data into the sentiment classifier, and obtaining the sentiment classification result corresponding to the fourth bimodal sample data. Then, a loss function is calculated, which is the difference between the sentiment classification result corresponding to the original bimodal sample data and the sentiment classification result corresponding to the fourth bimodal sample data.
[0107] Step 503: Backpropagate according to the loss function and iteratively update the parameters in the initial gating network and the initial expert network respectively to obtain the constructed hybrid expert model.
[0108] This step iteratively updates the initial gating networks, including the first and second initial gating networks, and the initial expert networks, including the first and second initial expert networks, based on the loss function. Specifically, the weight values between each first image / text data token are updated according to the value of the loss function until the initial hybrid expert model composed of the initial gating networks and the initial expert networks converges. Specifically, when the weight values between each first image / text data token are updated, the third bimodal sample data and the training sample dataset are updated accordingly, thereby achieving the training of the initial gating networks and the initial expert networks.
[0109] After each iteration, the loss function is recalculated until it converges to a preset range; or the number of iterations reaches a preset standard, at which point the initial hybrid expert model training is considered complete, and the hybrid expert model is constructed. The hybrid expert model in this application enables the model to better capture the features of each modality, solving the problem of modality-aware sparsity.
[0110] Figure 6 The diagram shown is a schematic representation of an image-text dual-modal expert fusion system according to an embodiment of this specification. Figure 6 As shown, a residual deep feature mixer is designed before the self-attention layer of the Transformer architecture (see [reference]). Figure 6 In addition to the data filtering unit, a hybrid expert model integrating a hierarchical routing mechanism was also designed. The hybrid expert model includes a first initial expert network and a second initial expert network, used to handle text modality and image modality, respectively. It is important to emphasize that the architecture provided in this application is specifically designed for pre-trained hybrid modality, early fusion language models.
[0111] like Figure 7 The diagram shown is a structural schematic of a dual-modal data prediction device according to an embodiment of this specification. The basic structure of the dual-modal data prediction device is illustrated in this diagram. The functional units and modules can be implemented in software, or dual-modal data prediction can be implemented using general-purpose chips or specific chips. The device specifically includes:
[0112] The data acquisition unit 701 is used to acquire the dual-modal data to be predicted.
[0113] Output unit 702 is used to input the bimodal data to be predicted into the bimodal expert fusion system to obtain the sentiment classification result output by the bimodal expert fusion system. The sentiment classification result corresponds to the bimodal data to be predicted. The bimodal expert fusion system includes a hybrid expert model. The hybrid expert model is obtained by training an initial gating network and an initial expert network with bimodal sample data whose central importance is higher than a preset threshold in the training sample dataset.
[0114] like Figure 8 The diagram shown is a structural schematic of a hybrid expert model training device according to an embodiment of this specification. The basic structure of the hybrid expert model training device is illustrated in this diagram. The functional units and modules can be implemented in software, or the hybrid expert model training can be implemented using general-purpose chips or specific chips. The device specifically includes:
[0115] Training sample dataset acquisition unit 801 is used to acquire training sample dataset;
[0116] The acquisition unit 802 is used to input the training sample dataset into the initial gating network and the initial expert network respectively, to obtain the initial weights assigned by the initial gating network to each expert in the initial expert network, and the initial feature vector output by the initial expert network.
[0117] The initial feature vector acquisition unit 803 is used to perform a weighted summation of the initial weight and the initial feature vector to obtain the initial feature vector of the image and the initial feature vector of the text.
[0118] Semantic alignment unit 804 is used to semantically align the initial feature vector of the image and the initial feature vector of the text to obtain a semantically aligned vector.
[0119] The construction unit 805 is used to construct a loss function based on the semantically aligned vectors, and to use the loss function to update and iterate the parameters of the initial gating network and the initial expert network to construct a hybrid expert model.
[0120] like Figure 9The diagram illustrates a computer device provided in an embodiment of this specification. The bimodal data prediction method and hybrid expert model training method described in this application can be applied to the computer device. The computer device 902 may include one or more processors 904, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 902 may also include any memory 906 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 906 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 902. In one case, when the processor 904 executes associated instructions stored in any memory or combination of memories, the computer device 902 can perform any operation of the associated instructions. The computer device 902 also includes one or more drive mechanisms 908 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.
[0121] Computer device 902 may also include an input / output module 910 (I / O) for receiving various inputs (via input device 912) and providing various outputs (via output device 914). A specific output mechanism may include a presentation device 916 and an associated graphical user interface (GUI) 918. In other embodiments, the input / output module 910 (I / O), input device 912, and output device 914 may be omitted, and the device may function solely as a computer device within a network. Computer device 902 may also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the components described above together.
[0122] Communication link 922 can be implemented in any way, such as via a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 922 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.
[0123] Corresponding to Figures 1 to 5 In addition to the methods described above, embodiments of this specification also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the methods described above.
[0124] This specification also provides computer-readable instructions, wherein when a processor executes the instructions, the program therein causes the processor to perform the following... Figures 1 to 5 The method shown.
[0125] It should be understood that in the various embodiments of this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.
[0126] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this specification generally indicates that the preceding and following related objects have an "or" relationship.
[0127] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.
[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0129] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.
[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described in this specification, depending on actual needs.
[0131] Furthermore, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0133] This specification uses specific embodiments to illustrate the principles and implementation methods of this specification. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this specification. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this specification. Therefore, the content of this specification should not be construed as a limitation of this specification.
Claims
1. A dual-modal data prediction method, characterized in that, The method includes: Acquire the bimodal data to be predicted; The bimodal data to be predicted is input into a bimodal expert fusion system to obtain the sentiment classification result output by the bimodal expert fusion system. The sentiment classification result corresponds to the bimodal data to be predicted. The bimodal expert fusion system includes a hybrid expert model. The hybrid expert model is obtained by training an initial gating network and an initial expert network with bimodal sample data whose central importance is higher than a preset threshold in the training sample dataset.
2. The method according to claim 1, characterized in that, The bimodal expert fusion system further includes: a data filtering unit connected before the hybrid expert model, the output data of the data filtering unit being the input data of the hybrid expert model, the data filtering unit including a projection transformation layer for filtering out bimodal data with an importance higher than a preset threshold from the bimodal data to be predicted.
3. A hybrid expert model training method, characterized in that, The method is applied to the hybrid expert model in the dual-modal data prediction method according to any one of claims 1-2, including: Obtain the training sample dataset; The training sample dataset is input into the initial gating network and the initial expert network respectively to obtain the initial weights assigned to each expert in the initial expert network by the initial gating network, and the initial feature vector output by the initial expert network. The initial weights and the initial feature vectors are weighted and summed to obtain the initial feature vectors of the image and the text. The initial feature vectors of the image and the initial feature vectors of the text are semantically aligned to obtain semantically aligned vectors. A loss function is constructed based on the semantically aligned vectors, and the parameters of the initial gating network and the initial expert network are updated and iterated using the loss function to construct a hybrid expert model.
4. The method according to claim 3, characterized in that, The training sample dataset is determined in the following manner: The original bimodal sample data is input into the data filtering unit to obtain the weight of each token in the original bimodal sample data; Based on the weights, the first bimodal sample data is selected from the original bimodal sample data; The first bimodal sample data is input into the self-attention unit to obtain the third bimodal sample data; The third bimodal sample data is modally separated to obtain separated image modal sample data and text modal sample data; The image modal sample data and text modal sample data are used as the training sample dataset.
5. The method according to claim 4, characterized in that, The method further includes: Image modal sample data are input into a first initial gating network and a first initial expert network to obtain the first initial weights output by the first initial gating network and the first initial feature vector output by the first initial expert network. Text modality sample data is input into a second initial gating network and a second initial expert network to obtain the second initial weights output by the second initial gating network and the second initial feature vector output by the second initial expert network. The first initial weight and the first initial feature vector are weighted and summed to obtain the initial feature vector of the image. The initial feature vector of the text is obtained by weighted summing of the second initial weight and the second initial feature vector.
6. The method according to claim 4, characterized in that, The method further includes: selecting second bimodal sample data from the original bimodal sample data according to the weights. The first bimodal sample data is input into the self-attention unit to obtain the weights between the tokens of the first bimodal sample data; The loss function and the hierarchical hybrid expert model are determined as follows: The semantically aligned vector is concatenated with the second bimodal sample data to obtain the fourth bimodal sample data; The loss function is determined based on the difference between the sentiment classification results corresponding to the original bimodal sample data and the sentiment classification results corresponding to the fourth bimodal sample data. Backpropagation is performed based on the loss function to iteratively update the weights between the first bimodal sample data tokens, thus obtaining the constructed hybrid expert model.
7. The method according to claim 6, characterized in that, The sentiment classification results corresponding to the fourth bimodal sample data are obtained in the following manner: The fourth bimodal sample data is input into the sentiment classifier, and the sentiment classification result corresponding to the fourth bimodal sample data is obtained.
8. A dual-modal data prediction device, characterized in that, The device includes: The data acquisition unit is used to acquire the dual-modal data to be predicted. The output unit is used to input the bimodal data to be predicted into the bimodal expert fusion system to obtain the sentiment classification result output by the bimodal expert fusion system. The sentiment classification result corresponds to the bimodal data to be predicted. The bimodal expert fusion system includes a hybrid expert model. The hybrid expert model is obtained by training an initial gating network and an initial expert network with bimodal sample data whose central importance is higher than a preset threshold in the training sample dataset.
9. A hybrid expert model training device, characterized in that, The device includes: The training sample dataset acquisition unit is used to acquire the training sample dataset; The acquisition unit is used to input the training sample dataset into the initial gating network and the initial expert network respectively, to obtain the initial weights assigned by the initial gating network to each expert in the initial expert network, and the initial feature vector output by the initial expert network; An initial feature vector acquisition unit is used to perform a weighted summation of the initial weights and the initial feature vectors to obtain an initial feature vector for the image and an initial feature vector for the text. A semantic alignment unit is used to semantically align the initial feature vector of the image and the initial feature vector of the text to obtain a semantically aligned vector. The construction unit is used to construct a loss function based on the semantically aligned vectors, and to use the loss function to update and iterate the parameters of the initial gating network and the initial expert network to construct a hybrid expert model.
10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.