Multi-modal mixed data processing method and device, equipment, medium and product

Abstract

The invention relates to a multi-mode mixed data processing method and device, equipment, a medium and a product. The method comprises the following steps: receiving a target problem containing multi-modal mixed information; inputting the target question into a trained multi-modal hybrid field expert large model for processing to obtain an answer corresponding to the target question; the trained multi-modal hybrid field expert large model comprises an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer and an output layer. Therefore, user questions containing multi-modal data such as images and texts can be understood more deeply through unified analysis and processing of the multi-modal mixed information, the question and answer response accuracy is improved, and more detailed answers can be fed back to users; and the model can be applied to professional knowledge questions and answers in a multi-field range, consultation requirements of the user in different fields are met, and the interaction experience is better.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multimodal mixed data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of artificial intelligence technology, government management and customer service technologies that assist in question and answer have emerged. They are used to help consultants provide answers to questions online, thereby guiding users to complete a series of more complex operations and improving interaction efficiency.

[0003] In traditional technologies, most question-answering models target single-modal data. For example, they rely on predefined rules and keywords to filter and extract text data input by users, and then generate answers to questions based on the extracted target text.

[0004] However, this question-and-answer model can only process single-modal data and is limited to a specific service or management field. It is unable to uniformly process multi-modal mixed data, the accuracy of question-and-answer responses is not high, and the scope of application is narrow. Summary of the invention

[0005] Based on this, it is necessary to provide a data processing method, device, computer equipment, computer-readable storage medium and computer program product that can process multimodal data, improve the accuracy of question and answer responses, and be applicable to multimodal hybrids in multiple fields in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a multimodal mixed data processing method, the method comprising:

[0007] Receiving a target question containing multimodal mixed information;

[0008] Input the target question into a trained multimodal hybrid domain expert model for processing to obtain an answer corresponding to the target question;

[0009] Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem.

[0010] In one embodiment, before inputting the target problem into the trained multimodal hybrid domain expert large model for processing, the method further includes:

[0011] Constructing an initial multimodal hybrid domain expert large model; the initial multimodal hybrid domain expert large model comprises: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer;

[0012] Constructing a first data set, the first data set comprising: pre-paired image features and text features;

[0013] The visual-language connector of the initial multimodal hybrid domain expert large model is trained using the first dataset.

[0014] In one of the embodiments, the expert system in the hybrid expert layer of the initial multi-modal hybrid domain expert large model includes: government affairs experts and operation and maintenance experts;

[0015] After constructing the initial multi-modal hybrid domain expert large model, the method further includes:

[0016] Constructing a second data set and a third data set; the second data set includes: text features related to government affairs; the third data set includes: text features related to operation and maintenance;

[0017] The government affairs expert is trained by the second data set, and the operation and maintenance expert is trained by the third data set.

[0018] In one embodiment, after constructing the initial multi-modal hybrid domain expert large model, the method further includes:

[0019] constructing a fourth data set, the fourth data set comprising: mixed multimodal instruction data;

[0020] The mixed multimodal instruction data is used as the input of the initial multimodal mixed domain expert big model, and the self-attention layer and the mixed expert layer in the initial multimodal mixed domain expert big model are trained as a whole to obtain a trained multimodal mixed domain expert big model.

[0021] In one embodiment, before inputting the target problem into the trained multimodal hybrid domain expert large model for processing, the method further includes:

[0022] The trained multimodal hybrid domain expert big model is optimized to obtain an optimized trained multimodal hybrid domain expert big model.

[0023] In one embodiment, the optimizing the trained multimodal hybrid domain expert big model to obtain the optimized trained multimodal hybrid domain expert big model includes:

[0024] Pruning the number of attention heads of the self-attention layer in the trained multimodal hybrid domain expert large model;

[0025] The trained multimodal mixed domain expert big model before pruning is used as the teacher model, and the trained multimodal mixed domain expert big model after pruning is used as the student model. The student model learns the output probability distribution of the teacher model to obtain the optimized trained multimodal mixed domain expert big model.

[0026] In a second aspect, the present application further provides a multi-modal hybrid data processing device, the device comprising:

[0027] A receiving module, used for receiving a target question containing multimodal mixed information;

[0028] A processing module, used to input the target question into a trained multimodal hybrid domain expert large model for processing to obtain an answer corresponding to the target question;

[0029] Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem.

[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] receiving a target question containing multimodal mixed information;

[0032] Input the target question into a trained multimodal hybrid domain expert model for processing to obtain an answer corresponding to the target question;

[0033] Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0035] receiving a target question containing multimodal mixed information;

[0036] Input the target question into a trained multimodal hybrid domain expert model for processing to obtain an answer corresponding to the target question;

[0037] Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem.

[0038] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0039] receiving a target question containing multimodal mixed information;

[0040] Input the target question into a trained multimodal hybrid domain expert model for processing to obtain an answer corresponding to the target question;

[0041] Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem.

[0042] The above-mentioned multimodal mixed data processing method, device, computer equipment, computer-readable storage medium and computer program product receive a target question containing multimodal mixed information; input the target question into a trained multimodal mixed field expert model for processing, and obtain an answer corresponding to the target question; thereby, multimodal data can be processed to generate professional answers in multiple fields, with a wider range of applications. Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem. This will enable a deeper understanding of user questions that contain multimodal data such as images and text, improve the accuracy of question and answer responses, and provide more detailed answers to users. It can also apply professional knowledge questions and answers in multiple fields to meet users' consulting needs in different fields and provide a better interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 An application environment diagram of a multi-modal hybrid data processing method in one embodiment;

[0045] Figure 2 A schematic diagram of a flow chart of a multi-modal mixed data processing method in one embodiment;

[0046] Figure 3 It is a schematic diagram of the structure of a multi-modal hybrid domain expert large model in one embodiment;

[0047] Figure 4 A schematic flow chart of a multi-modal mixed data processing method in another embodiment;

[0048] Figure 5 It is a flowchart of a multi-modal mixed data processing method in another embodiment;

[0049] Figure 6 A schematic diagram of the principle of model distillation in one embodiment;

[0050] Figure 7 is a structural block diagram of a multi-modal hybrid data processing device in one embodiment;

[0051] Figure 8 is a structural block diagram of a multi-modal mixed data processing device in another embodiment;

[0052] Fig. 9 is a structural block diagram of a multi-modal hybrid data processing device in yet another embodiment;

[0053] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] The multi-modal mixed data processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The trained multimodal hybrid domain expert model can be deployed on the terminal 102 or on the server 104. When the trained multimodal hybrid domain expert model is deployed on the terminal 102, it can be applied to offline scenarios, and the terminal 102 receives the target problem containing multimodal mixed information; then the target problem is input into the trained multimodal hybrid domain expert model for processing, and the answer corresponding to the target problem is obtained. When the trained multimodal hybrid domain expert model is deployed on the server 104, the terminal 102 receives the target problem containing multimodal mixed information, and transmits it to the server 104 through the network or the like, and the server 104 inputs the target problem into the trained multimodal hybrid domain expert model for processing, and after obtaining the answer corresponding to the target problem, the answer is fed back to the terminal 102. The trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target question and obtain the image and text contained in the target question; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, which is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, which is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target questions in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target question. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0056] In an exemplary embodiment, Figure 2 As shown, a multi-modal mixed data processing method is provided, which is applied to Figure 1 The terminal in is taken as an example to illustrate, including the following steps 201 to 202. Among them:

[0057] Step 201, receiving a target question containing multimodal mixed information.

[0058] In this embodiment, multimodal mixed information may include text, images, audio, video and other data. The traditional question-answering model responds to text content. The application scenario of this single modal data is relatively narrow and cannot adapt to the increasingly complex data processing needs. The target question composed of multimodal mixed information relaxes the restrictions on input content and can perform in-depth analysis and understanding based on information in multiple modalities, thereby better interpreting the target question and facilitating the subsequent generation of more accurate and detailed answers.

[0059] Step 202: Input the target question into the trained multimodal hybrid domain expert model for processing to obtain an answer corresponding to the target question.

[0060] In this embodiment, a trained multi-modal hybrid domain expert model is used to deeply analyze and understand the input target question with multiple information, and the hybrid domain expert system is used to generate the answer. This enables professional knowledge questions and answers across multiple fields to meet the more diverse needs of customers.

[0061] Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem.

[0062] For example, Figure 3 As shown in the figure, a structural diagram of a multimodal hybrid domain expert model is provided. First, the image and text are divided into two input branches. The first branch uses the visual embedding encoder to encode the input image, and then the encoded visual embedding information is further input into the visual-language connector for feature conversion to obtain the text features corresponding to the image. The second branch uses the text embedding encoder to encode the input text to obtain the text features corresponding to the text. After the text features of the two branches are strengthened by the self-attention layer, they are further transmitted to the hybrid expert layer (MOE) for processing. Optionally, taking the two expert systems including operation and maintenance experts and government affairs experts as an example, the instruction data (including multiple modal information) output by the self-attention layer is distributed to the corresponding experts through the routing mechanism, and the corresponding experts generate answers.

[0063] It should be understood that in addition to the operation and maintenance and government affairs fields mentioned above, the multimodal hybrid domain expert big model can also show great application potential in many fields such as customer service, education and training, and medical health.

[0064] It should be understood that the Mixed Experts (MOE) is a relatively new model design strategy that combines multiple models (also known as "experts") directly together, with different experts specializing in different problems, to achieve better prediction performance. Generally speaking, the mixed expert architecture of large models contains a gating mechanism and an output merging mechanism. The gating mechanism is responsible for dynamically selecting different experts based on the characteristics of the input data, while the output merging mechanism is used to effectively integrate and balance the prediction results of these experts to generate the final prediction output. This can reduce the amount of computation and enable the model to select the most appropriate expert for different inputs.

[0065] In an optional implementation, the routing mechanism can be a linear layer to help determine which expert the token should be sent to. The size of the linear layer is M x E, where E represents the number of experts and M represents the vector dimension of the hidden layer. Optionally, E = 2. After the input data S x M passes through the routing linear layer (S represents the number of instructions), the probability data S x E is obtained, which means the probability of each instruction going to each expert. In this embodiment, the expert with the highest probability is selected as the target expert, for example, to determine whether to route the instruction to a government expert or a customer service expert.

[0066] In the above-mentioned multimodal mixed data processing method, a target question containing multimodal mixed information is received; the target question is input into a trained multimodal mixed domain expert model for processing to obtain an answer corresponding to the target question; thereby, a deeper understanding of user questions containing multimodal data such as images and texts can be achieved, the accuracy of question and answer responses can be improved, and more detailed answers can be fed back to users; professional knowledge questions and answers in multiple fields can also be applied to meet users' consulting needs in different fields, and the interactive experience is better.

[0067] In another exemplary embodiment, Figure 4 As shown, a multi-modal mixed data processing method is provided, which is applied to Figure 1 The terminal in is taken as an example to illustrate, including the following steps 401 to 407. Among them:

[0068] Step 401, receiving a target question containing multimodal mixed information.

[0069] The specific implementation process and technical effects of step 401 in this embodiment can be found in Figure 2 The description of step 201 in the illustrated method embodiment will not be repeated here.

[0070] Step 402, construct an initial multi-modal hybrid domain expert large model.

[0071] The initial multimodal hybrid domain expert model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer.

[0072] Step 403: construct a first data set, and use the first data set to train the visual-language connector of the initial multimodal hybrid domain expert large model.

[0073] The first data set includes: pre-paired image features and text features.

[0074] In this embodiment, the visual-language connector is a functional component for converting visual embedding information into text embedding information so that the two types of information can be processed simultaneously in the subsequent process. Optionally, a large number of pre-paired image features and text features can be used to train the visual-language connector to convert image features into text features, that is, the ability to convert visual embedding information into the first text embedding information mentioned in the above embodiment.

[0075] It should be understood that the purpose of training the visual-language connector is to achieve a unified representation of different modalities in the language space of a large language model (LLM), so that the model can understand and process data from different sources.

[0076] It should be understood that image features are a high-dimensional feature representation, often also called visual embedding. Image features help the model understand and process image content. Text features are also a high-dimensional feature representation that helps the model understand and process text content.

[0077] Step 404, construct a second data set and a third data set, and train government affairs experts through the second data set, and train operation and maintenance experts through the third data set.

[0078] Among them, the second data set includes: text features related to government affairs; and the third data set includes: text features related to operation and maintenance.

[0079] In this embodiment, government affairs experts and operation and maintenance experts are composed of multiple feedforward neural networks, which are used to process the data features of government affairs and operation and maintenance types respectively. By using cross-modal multi-domain (government affairs / operation and maintenance) data sets to train experts in specific fields respectively, the performance of government affairs experts and operation and maintenance experts in their professional fields can be significantly improved.

[0080] Step 405: construct a fourth data set, wherein the fourth data set includes: mixed multimodal instruction data.

[0081] Among them, the mixed multimodal instruction data is a data in the form of an instruction (token), which can contain multimodal information. For example, if the picture depicts a yellow puppy, the corresponding text content is "yellow puppy", and the mixed multimodal instruction data is a text content describing the picture.

[0082] Step 406, using the mixed multimodal instruction data as the input of the initial multimodal mixed domain expert big model, training the initial multimodal mixed domain expert big model as a whole, and obtaining a trained multimodal mixed domain expert big model.

[0083] In this embodiment, the mixed multimodal instruction data is used as the input of the initial multimodal mixed domain expert big model, and the self-attention layer and the mixed expert layer in the multimodal mixed domain expert big model are trained as a whole to obtain a trained multimodal mixed domain expert big model.

[0084] Optionally, the trained experts are integrated into the hybrid expert layer of the LLM. The entire model framework is trained using hybrid multimodal instruction data to ensure that the model can comprehensively process information from multiple modalities and both the government affairs and operation and maintenance fields.

[0085] It should be noted that in order to reduce the training cost, the pre-trained expert and self-attention layers are fine-tuned using the Low-Rank Adaptation (LoRA) technique, which introduces a low-rank matrix to adjust the weights, thereby reducing the need for computational resources while maintaining model performance.

[0086] Step 407, input the target question into the trained multimodal hybrid domain expert large model for processing to obtain the answer corresponding to the target question.

[0087] The specific implementation process and technical effects of step 407 in this embodiment can be found in Figure 2 The description of step 202 in the illustrated method embodiment will not be repeated here.

[0088] This embodiment can process multimodal inputs including images and text, understand and accurately respond to complex user needs, and thus significantly improve the quality and accuracy of answers. By integrating experts in the government affairs and operation and maintenance fields, accurate responses in both fields are achieved, effectively reducing the cost of model construction and expanding the scope of application.

[0089] In yet another exemplary embodiment, Figure 5 As shown, a multi-modal mixed data processing method is provided, which is applied to Figure 1 The terminal in is taken as an example to illustrate, including the following steps 501 to 508. Among them:

[0090] Step 501, receiving a target question containing multimodal mixed information.

[0091] The specific implementation process and technical effects of step 501 in this embodiment can be found in Figure 2 The description of step 201 in the illustrated method embodiment will not be repeated here.

[0092] Step 502, construct an initial multi-modal hybrid domain expert large model.

[0093] Step 503: construct a first data set, and use the first data set to train the visual-language connector of the initial multimodal hybrid domain expert large model.

[0094] Step 504, construct a second data set and a third data set, and train government affairs experts through the second data set, and train operation and maintenance experts through the third data set.

[0095] Step 505: construct a fourth data set, wherein the fourth data set includes: mixed multimodal instruction data.

[0096] Step 506, using the mixed multimodal instruction data as the input of the initial multimodal mixed domain expert big model, training the initial multimodal mixed domain expert big model as a whole, and obtaining a trained multimodal mixed domain expert big model.

[0097] In this embodiment, the specific implementation process and technical effects of steps 502 to 506 can be found in Figure 4 The descriptions of steps 402 to 406 in the illustrated method embodiment are not repeated here.

[0098] Step 507, optimizing the trained multimodal hybrid domain expert large model to obtain an optimized trained multimodal hybrid domain expert large model.

[0099] In this embodiment, the size of the trained multimodal hybrid domain expert model is relatively large. Therefore, the scale of the model can be reduced through a pruning and distillation strategy, which not only reduces the consumption of computing resources, but also makes the multimodal question-answering model more lightweight, facilitating rapid deployment in resource-constrained computing environments, further expanding the scope of application of the model.

[0100] Exemplarily, the number of attention heads of the self-attention layer in the trained multimodal mixed domain expert big model is pruned; the trained multimodal mixed domain expert big model before pruning is used as the teacher model, and the trained multimodal mixed domain expert big model after pruning is used as the student model, and the student model learns the output probability distribution of the teacher model to obtain the optimized trained multimodal mixed domain expert big model.

[0101] Optionally, in the multi-head self-attention mechanism, the adjustment of the width dimension mainly involves optimizing the number of attention heads, for which the importance of the attention heads needs to be evaluated. For example, a small calibration dataset D can be used to calculate the importance score of the attention head based on the activation value, which is calculated as follows:

[0102]

[0103] in, refers to the importance score of the attention head in row i based on the activation value, refers to aggregation along the batch and sequence dimensions, refers to the attention function, refers to the weight matrix of the query corresponding to the attention head in the i-th row, refers to the weight matrix of the key corresponding to the attention head in the i-th row, refers to the weight matrix of the values ​​corresponding to the attention head in the i-th row, refers to the input matrix, Refers to the L2 norm of the vector.

[0104] Optionally, an iterative alternating method of importance estimation and pruning is used. Specifically, given the number of original attention heads L and the target number of attention heads K, the present invention iteratively calculates the importance of each attention head, and then the number of attention heads is reduced by 1 in each iteration, and a total of LK iterations are performed. It is worth noting that when pruning the attention heads, we add the residual information of the pruned heads back to the remaining heads in order to retain the relevant knowledge of the pruned heads. Formally, given L original attention heads is pruned to K heads, each new head will have the following form (for the i-th head): ,in i ∈ [K -(LK),K] .

[0105] Alternatively, if Figure 6As shown, a schematic diagram of the principle of model distillation is provided. Knowledge distillation (KD) involves transferring knowledge from a larger or more complex model (called a teacher) to a smaller, simpler model (called a student). Knowledge transfer is achieved by having the student model imitate the output and / or intermediate state of the teacher model. In the present invention, the uncompressed model and the pruned model correspond to the teacher and the student, respectively.

[0106] For example, for a given token , the output probability distribution of LLM is calculated as follows:

[0107]

[0108] in: is the temperature coefficient, is the vocabulary size. Knowledge distillation loss function based on probability distribution (logit) Calculated as:

[0109]

[0110] in: and Representing teachers and students in The probability distribution of tokens, l represents the sequence length.

[0111] The knowledge distillation loss function of the intermediate state of the sequence based on the Transformer specific hidden state is expressed as:

[0112]

[0113] in: and Respectively represent The teacher and student of the token hidden states, l represents the sequence length; H is the set of selected intermediate states. The mismatch between the student and teacher hidden states is handled by learning a shared linear transformation during learning to map the student hidden states to the dimensions of the teacher hidden states. The hidden states used are always after layer normalization.

[0114] Total loss Calculated as:

[0115]

[0116] in: is the cross entropy loss of the student over the true labels, is a weight coefficient used to balance and Significant size differences.

[0117] In this embodiment, through model distillation, a student model with performance comparable to that of the uncompressed model can be obtained, and its size is much smaller than the original uncompressed model.

[0118] Step 508: Input the target question into the optimized and trained multimodal hybrid domain expert large model for processing to obtain an answer corresponding to the target question.

[0119] The specific implementation process and technical effects of step 508 in this embodiment can be found in Figure 2 The description of step 202 in the illustrated method embodiment will not be repeated here.

[0120] This embodiment achieves efficient processing of text and images, as well as inputs from different fields such as government affairs and operation and maintenance, by building a multimodal hybrid domain expert model that integrates visual embedding encoders, text embedding encoders, visual-language connectors, self-attention layers, routing mechanisms, and expert systems in government affairs and operation and maintenance fields. Among them, the model can use the self-attention mechanism to strengthen feature interaction, and intelligently assign tasks to corresponding experts through the routing mechanism, and finally generate accurate prediction results through the output layer to meet the complex consulting needs in government affairs and operation and maintenance scenarios.

[0121] This embodiment also uses a small calibration dataset to calculate the importance score of each attention head; then, the number of attention heads is gradually reduced through an iterative process, while the information of the pruned heads is integrated into the remaining heads to retain key knowledge. This method effectively maintains model performance while reducing model complexity.

[0122] In this embodiment, the uncompressed model is used as the teacher and the pruned model is used as the student. Knowledge transfer is achieved by imitating the teacher's output and intermediate states, and the loss function based on probability distribution and hidden state is used to guide the learning of the student model. The student model trained by this method not only reduces the consumption of computing resources, but also maintains the performance of the model, making the model more lightweight and easy to deploy.

[0123] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0124] Through the same inventive concept, the embodiment of the present application also provides a multimodal mixed data processing device for implementing the multimodal mixed data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more multimodal mixed data processing device embodiments provided below can refer to the limitations of the multimodal mixed data processing method above, and will not be repeated here.

[0125] In an exemplary embodiment, Figure 7 As shown, a multi-modal hybrid data processing device is provided, including: a receiving module 701 and a processing module 702, wherein:

[0126] A receiving module 701 is used to receive a target question containing multimodal mixed information;

[0127] Processing module 702, used for inputting the target question into the trained multi-modal hybrid domain expert large model for processing to obtain an answer corresponding to the target question;

[0128] Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem.

[0129] In another exemplary embodiment, Figure 8 As shown, a multi-modal mixed data processing device is provided. Figure 7 Based on the device shown, it can also include:

[0130] The model construction module 703 is used to construct an initial multimodal hybrid domain expert large model; the initial multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer;

[0131] The data set construction module 704 is used to construct at least one of a first data set, a second data set, and a third data set; wherein the first data set includes: pre-paired image features and text features; the second data set includes: government affairs-related text features; the third data set includes: operation and maintenance-related text features; the fourth data set includes: mixed multimodal instruction data;

[0132] The first training module 705 is used to train the visual-language connector of the initial multimodal hybrid domain expert large model through the first data set.

[0133] Exemplarily, the expert system in the hybrid expert layer of the initial multi-modal hybrid domain expert model includes: government affairs experts and operation and maintenance experts.

[0134] Exemplarily, the above-mentioned apparatus may further include: a second training module 706, configured to train the government affairs expert through the second data set, and to train the operation and maintenance expert through the third data set.

[0135] Exemplarily, the above-mentioned device may also include: a third training module 707, which is used to use the mixed multimodal instruction data as the input of the initial multimodal mixed domain expert big model, and overall train the self-attention layer and the mixed expert layer in the initial multimodal mixed domain expert big model to obtain a trained multimodal mixed domain expert big model.

[0136] In yet another exemplary embodiment, Fig. 9 As shown, a multi-modal mixed data processing device is provided. Figure 7 Based on the device shown, it can also include:

[0137] The model optimization module 708 is used to optimize the trained multimodal hybrid domain expert big model to obtain an optimized trained multimodal hybrid domain expert big model.

[0138] Exemplarily, the model optimization module 708 is specifically used to prune the number of attention heads of the self-attention layer in the trained multimodal mixed domain expert big model; using the trained multimodal mixed domain expert big model before pruning as the teacher model, and using the trained multimodal mixed domain expert big model after pruning as the student model, and having the student model learn the output probability distribution of the teacher model to obtain the optimized trained multimodal mixed domain expert big model.

[0139] Each module in the above multimodal hybrid data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0140] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.10As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a multi-modal mixed data processing method is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0141] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0143] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0144] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0146] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases through blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit through quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0148] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-modal mixed data processing method, characterized in that: The method comprises: receiving a target question containing multimodal mixed information; Input the target question into a trained multimodal hybrid domain expert model for processing to obtain an answer corresponding to the target question; Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem.

2. The method according to claim 1, characterized in that Before inputting the target problem into the trained multimodal hybrid domain expert large model for processing, the method further includes: Constructing an initial multimodal hybrid domain expert large model; the initial multimodal hybrid domain expert large model comprises: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; Constructing a first data set, the first data set comprising: pre-paired image features and text features; The visual-language connector of the initial multimodal hybrid domain expert large model is trained using the first dataset.

3. The method according to claim 2, characterized in that The expert system in the hybrid expert layer of the initial multi-modal hybrid domain expert model includes: government affairs experts and operation and maintenance experts; After constructing the initial multi-modal hybrid domain expert large model, the method further includes: Constructing a second data set and a third data set; the second data set includes: text features related to government affairs; the third data set includes: text features related to operation and maintenance; The government affairs expert is trained by the second data set, and the operation and maintenance expert is trained by the third data set.

4. The method according to claim 3, characterized in that After constructing the initial multi-modal hybrid domain expert large model, the method further includes: constructing a fourth data set, the fourth data set comprising: mixed multimodal instruction data; The mixed multimodal instruction data is used as the input of the initial multimodal mixed domain expert big model, and the self-attention layer and the mixed expert layer in the initial multimodal mixed domain expert big model are trained as a whole to obtain a trained multimodal mixed domain expert big model.

5. The method according to any one of claims 1 to 4, characterized in that: Before inputting the target problem into the trained multimodal hybrid domain expert large model for processing, the method further includes: The trained multimodal hybrid domain expert big model is optimized to obtain an optimized trained multimodal hybrid domain expert big model.

6. The method according to claim 5, characterized in that The step of optimizing the trained multimodal hybrid domain expert big model to obtain an optimized trained multimodal hybrid domain expert big model includes: Pruning the number of attention heads of the self-attention layer in the trained multimodal hybrid domain expert large model; The trained multimodal mixed domain expert big model before pruning is used as the teacher model, and the trained multimodal mixed domain expert big model after pruning is used as the student model. The student model learns the output probability distribution of the teacher model to obtain the optimized trained multimodal mixed domain expert big model.

7. A multi-modal hybrid data processing device, characterized in that: The device comprises: A receiving module, used for receiving a target question containing multimodal mixed information; A processing module, used for inputting the target question into a trained multimodal hybrid domain expert large model for processing to obtain an answer corresponding to the target question; Among them, the trained multimodal hybrid domain expert large model includes: an input layer, a visual embedding encoder, a text embedding encoder, a visual-language connector, a self-attention layer, a hybrid expert layer, and an output layer; the input layer is used to parse the target problem and obtain the image and text contained in the target problem; the visual embedding encoder is used to encode the image and obtain the corresponding visual embedding information; the visual-language connector is used to convert the visual embedding information into first text embedding information, and the first text embedding information is the text feature corresponding to the image; the text embedding encoder is used to encode the text and obtain second text embedding information, and the second text embedding information is the text feature corresponding to the text; the self-attention layer is used to transmit the relationship between the first text embedding information and the second text embedding information to the expert system in the hybrid expert layer after strengthening the relationship between the first text embedding information and the second text embedding information; the hybrid expert layer includes at least two expert systems for answering target problems in different fields; the output layer is used to integrate the answers of the expert systems and output the answer corresponding to the target problem.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images