A multimodal data analysis method based on training of a large hybrid expert structure model

Abstract

The present invention discloses a multimodal data analysis method based on hybrid expert structure large model training, the method comprising: obtaining historical multimodal data, and preprocessing the historical multimodal data to obtain multiple groups of preprocessed multimodal data; obtaining a preset large-scale training model, determining a training parallel mode according to the number of expert networks in the preset large-scale training model, and segmenting multiple groups of preprocessed multimodal data according to the training parallel mode to construct a multimodal data training set; performing model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data analysis model; obtaining current multimodal data, and inputting the current multimodal data into the multimodal data analysis model to obtain a multimodal data analysis result. The present invention can support multimodal data parallel processing and parallel training of multiple expert networks by constructing a multimodal data analysis model, thereby realizing efficient analysis of multimodal data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal data analysis method, system, terminal and computer-readable storage medium based on hybrid expert structure large model training. Background Art

[0002] In recent years, with the development of computer technology and artificial intelligence, people's demand for machine perception and reasoning has increased day by day. Visual analysis has become a hot topic in image research, and visual analysis is particularly important in the field of image understanding. In the existing technology, image parsing is an important task that can extract useful information and features from images for applications such as image classification, object detection, and scene understanding. However, traditional image parsing methods mainly rely on single-modal information, such as image pixel values ​​or image-based feature extraction. This single-modal method may have limitations in some cases and cannot fully utilize the richness of multimodal information.

[0003] In view of the above situation, with the widespread application of multimodal data in the prior art, image parsing methods based on multimodal information have become a hot topic of research. Multimodal information includes data of different modalities such as images, text, voice, and video. These information are interrelated and can provide more comprehensive and accurate image understanding and analysis. By comprehensively utilizing multimodal information, higher-level image parsing tasks can be achieved, such as image description generation, multimodal retrieval, etc. However, in the prior art, multimodal data cannot be parsed in parallel, resulting in slow parsing of multimodal data, poor parsing effect, and failure to meet user needs.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a multimodal data analysis method, system, terminal and computer-readable storage medium based on hybrid expert structure large model training, aiming to solve the problem that multimodal data cannot be analyzed in parallel in the prior art, resulting in slow multimodal data analysis, poor analysis effect and failure to meet user needs.

[0006] To achieve the above object, the present invention provides a multimodal data analysis method based on mixed expert structure large model training, the multimodal data analysis method based on mixed expert structure large model training comprises the following steps:

[0007] Acquiring historical multimodal data, and preprocessing the historical multimodal data to obtain multiple groups of preprocessed multimodal data;

[0008] Obtaining a preset large-scale training model, determining a training parallel mode according to the number of expert networks in the preset large-scale training model, and performing segmentation processing on multiple groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set;

[0009] Performing model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data parsing model;

[0010] The current multimodal data is acquired, and the current multimodal data is input into the multimodal data analysis model to obtain a multimodal data analysis result.

[0011] Optionally, in the multimodal data analysis method based on hybrid expert structure large model training, the preprocessing includes: encoding processing, binary identification processing, and modal data division processing;

[0012] The acquiring of historical multimodal data and preprocessing of the historical multimodal data to obtain multiple groups of preprocessed multimodal data specifically includes:

[0013] Acquire historical multimodal data, and use encoders of different modalities to encode the historical multimodal data to obtain multiple modal sequences;

[0014] Performing binary identification processing on the multimodal data in the plurality of modal sequences according to the modality type to obtain a plurality of groups of identified modal data;

[0015] Modal data division processing is performed on the multiple groups of identified modal data to obtain multiple groups of pre-processed multi-modal data.

[0016] Optionally, in the multimodal data analysis method based on hybrid expert structure large model training, the training parallel mode includes a data parallel mode and a dual parallel mode of expert parallel and data parallel;

[0017] The step of obtaining a preset large-scale training model, determining a training parallel mode according to the number of expert networks in the preset large-scale training model, and segmenting a plurality of groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set specifically includes:

[0018] Obtaining the number of expert networks in the preset large-scale training model, and determining whether the number of expert networks is less than a preset number;

[0019] If yes, execute the data parallel mode, and perform a first segmentation process on the plurality of groups of the pre-processed multimodal data according to the data parallel mode to obtain a first multimodal data training set;

[0020] If not, a dual-parallel mode of expert parallelism and data parallelism is executed, and a second segmentation process is performed on the plurality of groups of the pre-processed multimodal data according to the dual-parallel mode of expert parallelism and data parallelism to obtain a second multimodal data training set.

[0021] Optionally, the multimodal data analysis method based on hybrid expert structure large model training, wherein the first segmentation processing is performed on multiple groups of the pre-processed multimodal data according to the data parallel mode to obtain a first multimodal data training set, specifically includes:

[0022] According to the data parallel mode, each group of the pre-processed multimodal data is randomly shuffled, and the pre-processed multimodal data after the random shuffle is first segmented according to a preset size to obtain a plurality of first segmentation groups of equal size;

[0023] Perform sampling processing in each group of first cut groups of equal size to obtain multiple sampling groups;

[0024] The multiple sampling groups in each group of the pre-processed multimodal data are mixed and randomly shuffled to obtain the first multimodal data training set.

[0025] Optionally, the multimodal data analysis method based on hybrid expert structure large model training, wherein the second segmentation processing is performed on multiple groups of the pre-processed multimodal data according to the expert parallel and data parallel dual parallel modes to obtain a second multimodal data training set, specifically includes:

[0026] According to the expert parallel and data parallel dual parallel modes, each group of the pre-processed multimodal data is randomly shuffled, and the pre-processed multimodal data after the random shuffle is sorted according to the preset sorting requirements to obtain a sorting result;

[0027] According to the sorting result, the pre-processed multimodal data after random shuffling is subjected to a second segmentation process according to a preset size to obtain a plurality of second segmentation groups of equal size;

[0028] A plurality of second cut groups of equal sizes in each group of the preprocessed multimodal data are mixed and randomly shuffled to obtain the second multimodal data training set.

[0029] Optionally, the multimodal data analysis method based on hybrid expert structure large model training, wherein the model training is performed on the preset large-scale training model according to the multimodal data training set to obtain the multimodal data analysis model, specifically comprising:

[0030] Performing encoding processing and semantic space projection processing on the first multimodal data training set or the second multimodal data training set through a corresponding modality encoder to obtain a plurality of projection result data sets, and inputting the plurality of projection result data sets into the preset large-scale training model;

[0031] The plurality of projection result data sets are transmitted to the corresponding expert network for analysis and processing through the shared model part in the preset large-scale training model, so as to obtain a trained multimodal data analysis model.

[0032] Optionally, the multimodal data analysis method based on hybrid expert structure large model training, wherein the obtaining of current multimodal data and inputting the current multimodal data into the multimodal data analysis model to obtain the multimodal data analysis result, specifically includes:

[0033] Acquire current multimodal data, and input the current multimodal data into the multimodal data parsing model;

[0034] The current multimodal data is subjected to data analysis processing according to the multimodal data analysis model to obtain the multimodal data analysis result.

[0035] In addition, to achieve the above-mentioned purpose, the present invention also provides a multimodal data analysis system based on mixed expert structure large model training, wherein the multimodal data analysis system based on mixed expert structure large model training includes:

[0036] A data preprocessing module, used to obtain historical multimodal data and preprocess the historical multimodal data to obtain multiple groups of preprocessed multimodal data;

[0037] A multimodal data segmentation module is used to obtain a preset large-scale training model, determine a training parallel mode according to the number of expert networks in the preset large-scale training model, and segment multiple groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set;

[0038] A parsing model building module, used to perform model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data parsing model;

[0039] The multimodal data analysis module is used to obtain current multimodal data and input the current multimodal data into the multimodal data analysis model to obtain a multimodal data analysis result.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multimodal data analysis program based on hybrid expert structure large model training stored in the memory and executable on the processor, wherein the multimodal data analysis program based on hybrid expert structure large model training implements the steps of the multimodal data analysis method based on hybrid expert structure large model training as described above when the multimodal data analysis program based on hybrid expert structure large model training is executed by the processor.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal data analysis program based on hybrid expert structure large model training, and when the multimodal data analysis program based on hybrid expert structure large model training is executed by a processor, the steps of the multimodal data analysis method based on hybrid expert structure large model training as described above are implemented.

[0042] In the present invention, historical multimodal data is obtained, and the historical multimodal data is preprocessed to obtain multiple groups of preprocessed multimodal data; a preset large-scale training model is obtained, and the training parallel mode is determined according to the number of expert networks in the preset large-scale training model, and multiple groups of the preprocessed multimodal data are segmented and processed according to the training parallel mode to construct a multimodal data training set; the preset large-scale training model is model trained according to the multimodal data training set to obtain a multimodal data analysis model; current multimodal data is obtained, and the current multimodal data is input into the multimodal data analysis model to obtain a multimodal data analysis result. The present invention can ensure that the modal data obtained by each process is similar and the total length is similar by segmenting and sampling the acquired historical multimodal data, thereby minimizing the problem that Deepspeed is easily stuck in large-scale multimodal data training. At the same time, the data parallel mode or the expert parallel and data parallel dual parallel mode is determined according to the number of expert networks in the large-scale training model, so as to construct a multimodal data training set to train the large-scale training model and obtain a multimodal data parsing model; the multimodal data parsing model can achieve efficient parsing of the current multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a preferred embodiment of the multimodal data analysis method based on mixed expert structure large model training of the present invention;

[0044] Figure 2 It is a flowchart of the overall structure implementation of a preferred embodiment of the multimodal data analysis method based on mixed expert structure large model training of the present invention;

[0045] Figure 3It is a flow chart of the data sampling process of a preferred embodiment of the multimodal data analysis method based on mixed expert structure large model training of the present invention;

[0046] Figure 4 It is a schematic diagram of the data distribution and propagation process of model training of a preferred embodiment of the multimodal data analysis method based on hybrid expert structure large model training of the present invention;

[0047] Figure 5 It is a structural diagram of a preferred embodiment of a multimodal data analysis system based on mixed expert structure large model training of the present invention;

[0048] Figure 6 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] In recent years, with the development of computer technology and artificial intelligence, people's demand for machine perception and reasoning has increased day by day. Visual analysis has become a hot topic in image research, and visual analysis is particularly important in the field of image understanding. In the existing technology, image parsing is an important task that can extract useful information and features from images for applications such as image classification, object detection, and scene understanding. However, traditional image parsing methods mainly rely on single-modal information, such as image pixel values ​​or image-based feature extraction. This single-modal method may have limitations in some cases and cannot fully utilize the richness of multimodal information.

[0051] In view of the above situation, with the widespread application of multimodal data in the prior art, image parsing methods based on multimodal information have become a hot topic of research. Multimodal information includes data of different modalities such as images, text, voice, and video. These information are interrelated and can provide more comprehensive and accurate image understanding and analysis. By comprehensively utilizing multimodal information, higher-level image parsing tasks can be achieved, such as image description generation, multimodal retrieval, etc. However, in the prior art, multimodal data cannot be parsed in parallel, resulting in slow parsing of multimodal data, poor parsing effect, and failure to meet user needs.

[0052] With the rise of large language models, their outstanding performance in semantic understanding and general knowledge has attracted widespread attention. In previous technologies, in order to enable large language models to understand multimodal information such as vision and speech, encoders of each modality are often used to encode this information and align it to the semantic space of the language model. At the same time, the mixture of experts (MoE) model, as a key technology, has shown great potential in the field of large models. The MoE model trains multiple expert networks simultaneously in a single framework. Each expert network is responsible for processing different aspects of the data, and then the outputs of these expert networks are fused through a gating mechanism to obtain better performance.

[0053] Existing MoE models can integrate the different capabilities of expert networks and use sparse activation mechanisms (combining multiple simple models (called "experts") together, and using a "gating mechanism" (Gate) to determine which experts should be activated for a specific input and participate in the final decision) to achieve excellent performance with less computation. However, these models are usually limited to the capabilities of a single modality and are mainly trained on plain text data, showing strong adaptability to text modalities. Some open source MoE model implementations support multimodality, but their methods are only applicable to the training of a few expert networks. They do not take into account the possibility of expert network parallelism, nor do they fully consider the related issues of multimodal data input in parallel training. On the one hand, when the number of expert networks is expanded, a single card (a GPU card) will not be able to accommodate the entire model, and model parallelism or expert network parallelism must be implemented. At this time, there is a problem of simultaneous parallelism of data parallelism and model parallelism. On the other hand, the sequence length of multimodal data input varies greatly, and can only support a smaller batch_size (the amount of data input to the model at each step during training) to prevent video memory overflow. A smaller batch_size can easily cause Sampler (sampling from training data) imbalance, and the length of input data from different GPUs varies greatly, resulting in a series of problems when communicating with multiple cards.

[0054] In response to the above problems, the present invention proposes a multimodal unified large model construction and training method based on a hybrid expert network structure, which can well support large-scale parallel training of multimodal data on multi-expert networks, thereby providing strong support for the construction of a unified multimodal large model of the hybrid expert network, thereby achieving efficient analysis of multimodal data.

[0055] In response to the problem of existing multimodal data training on multi-expert networks, the present invention proposes a multimodal unified large model construction and training method based on a hybrid expert structure to meet the challenges in the prior art. In response to the problem of data parallel training, the present invention proposes two solutions: one is to support the use of data parallel strategies when training with a small number of expert networks, but not to perform expert network parallelism. The present invention fully supports modal division by constructing a Sampler suitable for the Deepspeed (large model training tool) parallel framework, ensuring that the total length of multimodal data obtained by each computing unit during each training is close, thereby solving the training problem of large length variations in data parallel training.

[0056] Second, when a large number of expert networks are used for simultaneous training, a dual parallel strategy of expert network parallelism and data parallelism is supported. The present invention constructs a Sampler suitable for the Deepspeed MoE parallel framework, and comprehensively considers the sequence length encoded by multimodal data, thereby supporting different GPUs (this refers to the GPU graphics card of the computing service, usually a server contains multiple GPU graphics cards, all numbered, for example, four GPU graphics cards can be: 0-3) to obtain data with similar lengths, and ensures that different communications across GPUs are aligned to the same length, which strongly supports parallel training of multiple expert networks. At the same time, the method reduces the additional communication overhead caused by alignment to a certain extent, while ensuring that the training data sampling still has a certain degree of randomness, and further improves the training efficiency.

[0057] The combined effect of these solutions enables the present invention to support large-scale parallel training, significantly improving the training effect and performance of multimodal large models. Therefore, the present invention not only provides an effective solution for multimodal data analysis, but also brings new ideas and methods for the construction and training of multimodal large models.

[0058] The multimodal data analysis method based on mixed expert structure large model training described in the preferred embodiment of the present invention is as follows: Figure 1 He Ru Figure 2 As shown, the multimodal data analysis method based on hybrid expert structure large model training includes the following steps:

[0059] Step S10: Acquire historical multimodal data, and preprocess the historical multimodal data to obtain multiple groups of preprocessed multimodal data, wherein the preprocessing includes: encoding processing, binary identification processing, and modal data division processing.

[0060] Specifically, historical multimodal data is acquired, and encoders of different modalities are used to encode the historical multimodal data to obtain a plurality of modal sequences; the multimodal data in the plurality of modal sequences are binary identified according to the modality type to obtain a plurality of groups of identified modal data; modal data division is performed on the plurality of groups of identified modal data to obtain a plurality of groups of preprocessed multimodal data.

[0061] Acquire multimodal data, including images, text, audio, and video data, where different modal data have certain semantic logical connections. Preprocess the data through encoders of different modalities and encode them into corresponding token sequences. Record the lengths of each sequence as limg, ltext, and laud, and record the total length of the token sequence l = limg + ltext + laud in the data for subsequent use.

[0062] When loading data, read the total length l of the multimodal data and the corresponding token sequence of each item in turn. At the same time, use 3-bit binary identification according to different modal data, and the three bits are encoded as video (100) / audio (010) / image (001), so as to effectively identify the data of different combined modalities for modal division in subsequent processes. In this process, the plain text model will be identified as 000, the combination of video and audio will be identified as 110, the combination of audio and image will be identified as 011, and so on.

[0063] For the data sampling (Sampler) process: according to the above preprocessing results, distributed sampling is performed, the number of processes on each physical node is set to nproc_per_node, the number of nodes used is nnodes, the amount of data sampled once on each process is batch_size, the world_size is nproc_per_node*nnodes, and the sampler_size is nproc_per_node*nnodes*batch_size. When sampling data, ensure that all data within the sampler_size size comes from the same modal combination and has a similar length, such as Figure 3 As shown, the present invention first divides the data with different binary identifiers into different groups according to the corresponding binary identifiers. middle.

[0064] Step S20, obtaining a preset large-scale training model, determining a training parallel mode according to the number of expert networks in the preset large-scale training model, and performing segmentation processing on multiple groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set. The training parallel mode includes a data parallel mode and a dual parallel mode of expert parallel and data parallel.

[0065] Specifically, obtaining the number of expert networks in the preset large-scale training model, and determining whether the number of expert networks is less than a preset number;

[0066] If so, execute the data parallel mode, and randomly shuffle each group of the preprocessed multimodal data according to the data parallel mode, and perform a first segmentation process on the randomly shuffled preprocessed multimodal data according to a preset size to obtain multiple groups of equal-sized first segmentation groups; perform sampling process on each group of equal-sized first segmentation groups to obtain multiple sampling groups; perform mixing process and randomly shuffle process on the multiple sampling groups in each group of the preprocessed multimodal data to obtain the first multimodal data training set.

[0067] The present invention processes the data of each group as follows: for a small number of experts in parallel mode, the data in the same group is randomly shuffled and divided according to the size of sampler_size, and the data is divided into groups g of the same size as sampler_size. 1，2，3...n , where the ones that are not long enough are discarded. Then in the same cutting group g k Sampling is performed to ensure that the total length of each batch_size data is close. So far, the format of each segmentation group should be a two-dimensional form of [[0, 1, ..., batch_size-1], [0, 1, ..., batch_size-1], ..., [0, 1, ..., batch_size-1]], which is expanded in sequence into a one-dimensional form [0, 1, 2...world_size-1] (g k ), where the total length of batch_size pieces of data taken in sequence is close. Similar operations are performed on other split groups, and finally each The data in can be displayed as follows [[0, 1, 2…world_size-1](g0), [0, 1, 2…world_size-1](g1)…[0, 1, 2…world_size-1](g n )]. Finally, all The data in the data are mixed according to each complete split group. Then the split groups are randomly shuffled (finally expanded into a one-dimensional array in order according to the sampler requirements). At this time, the results of a sampling are all the same type of modal data, and the batch_size length of each process sampling is similar.

[0068] If not, execute the dual parallel mode of expert parallelism and data parallelism, and randomly shuffle each group of the preprocessed multimodal data according to the dual parallel mode of expert parallelism and data parallelism, and sort the preprocessed multimodal data after random shuffling according to preset sorting requirements to obtain a sorting result; according to the sorting result, perform a second segmentation process on the preprocessed multimodal data after random shuffling according to a preset size to obtain multiple groups of second segmentation groups of equal sizes; mix and randomly shuffle the multiple groups of second segmentation groups of equal sizes in each group of the preprocessed multimodal data to obtain the second multimodal data training set.

[0069] For the dual parallel modes of multi-expert parallelism and data parallelism, the data in the same group are randomly shuffled and then directly sorted. Finally, they are directly divided into groups of size sampler_size according to the sorting results. 1，2，3...n , among which, the ones that are not long enough are discarded. The data in the data are mixed according to each complete split group. Then the split groups are randomly shuffled (finally expanded into a one-dimensional array in order according to the sampler requirements). At this time, the results of a sampling are all the same type of modal data, and the batch_size length of each process sampling is similar.

[0070] After sampling, Deepspeed MoE is used to distribute the data of each batch_size to different processes. At this time, each process obtains different data, but at the same time, the data of all processes are of the same modal combination, and the total length of data processed by each process is close.

[0071] In addition, in the stage of modal division and statistics, using binary division and pre-coding statistics is the most efficient way. Binary can represent all data combinations without redundancy and is easy to expand, while pre-coding statistics are beneficial to the subsequent sorting and division of data. The readability and simplicity of using specific text discrimination and digital combination discrimination are obviously not as good as binary division representation. On-site coding and division of data is not conducive to data statistics and reuse.

[0072] In addition, the mode division used in the sampling stage is followed by the same-length sampling method to ensure that the same mode data is obtained in the same group of data. Using the same-mode data can minimize the time fluctuation overhead when processing different processes and minimize the waiting time of parallel processes. At the same time, for the method that does not use advance coding to count the length, after using mode division, since the data types of the same mode combination are similar, simple length statistics can also achieve a similar division effect. Any solution that does not perform mode division does not have the efficiency and scalability of this solution.

[0073] Step S30: performing model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data parsing model.

[0074] Specifically, the first multimodal data training set or the second multimodal data training set is encoded and semantically projected into space by a corresponding modal encoder to obtain a plurality of projection result data sets, and the plurality of projection result data sets are input into the preset large-scale training model; the plurality of projection result data sets are transmitted to the corresponding expert network for parsing by a shared model part in the preset large-scale training model to obtain a trained multimodal data parsing model.

[0075] Training strategy: In the overall model architecture design and training process, after each process obtains its own data, it is first encoded through the encoders of all modes, projected to the semantic space of the language model through the projectors of each mode, and finally the projection results of the tokens of all modes are connected together to enter the large language model. Then, through the shared model part (Self-Attention), the gate (gating value) corresponding to each token is obtained token by token, and the logits (mapping probability distribution of experts) of TopK (after sorting the values, select the K experts with the largest values, TopK) are distributed and transmitted to the corresponding experts. After obtaining the output of the experts, the output of each expert is weighted and synthesized according to the logits. Repeat this process to obtain the final output result.

[0076] The training strategy uses Deepspeed ZeRO2 to train the corresponding MoE model and sets the corresponding parameters in a targeted manner.

[0077] For the parallel mode with a small number of experts, it is allowed to use a combination of custom MoE implementation and Deepspeed ZeRO2 (allowing any TopK expert mode) or Deepspeed MoE+ZeRO2 implementation (only TopK, K<=2) In any implementation, each process will directly load a complete model and perform parallel optimization at the optimizer and gradient descent levels.

[0078] For the dual parallel modes of multi-expert parallelism and data parallelism, only the implementation of Deepspeed MoE+ZeRO2 is supported (only TopK is allowed, K<=2). At this time, each process needs to have a complete model at the non-expert level and local_expert_num=expert_num / ep_size experts at the expert level. Each process is reflected as experts 0, 1, 2...local_expert_num-1 when the model architecture is initialized. According to the arrangement of the expert sequence and the rank (process number) of the process, specific loading is performed. For a process with rank r, r*local_expert_num-(r+1)*local_expert_num experts are loaded from the complete model (or expert list). And open drop_token to pad a small amount of data that may not be padded to the same length across processes. The data sampling phase is divided in a way that can minimize the amount of padding and reduce additional communication overhead. At the same time, it ensures that the capacity of all experts can get the maximum value of all process groups, thereby ensuring that all processes have equal transmission volume in the expert parallel AlltoAll communication phase. The complete process is as follows: Figure 4 shown.

[0079] By calculating the gate, only the corresponding TopK experts will be activated. Therefore, when calculating the corresponding forward (forward propagation calculation), only this part of the experts will perform calculations, thereby ensuring sparse calculations and reducing training costs.

[0080] In addition, if allowed, we should consider further improving the TopK module under Deepspeed MoE to support the condition of K>2, so as to better support the situation where more experts are needed. At the same time, in terms of the randomness of data input, we can intersperse different modal data with similar lengths in different processes to achieve the same number of modal combinations in each process, which is conducive to the auxiliary Loss (loss function) design of the MoE model.

[0081] Step S40: acquiring current multimodal data, and inputting the current multimodal data into the multimodal data analysis model to obtain a multimodal data analysis result.

[0082] Specifically, current multimodal data is acquired, and the current multimodal data is input into the multimodal data parsing model; data parsing processing is performed on the current multimodal data according to the multimodal data parsing model to obtain the multimodal data parsing result.

[0083] The present invention can simultaneously input multimodal data, such as a certain video or an image or a piece of audio, into the multimodal data parsing model that has been trained by the present invention. Then, the multimodal data parsing model can directly realize parallel processing and efficient parsing of the input video or image or audio content, thereby quickly obtaining the parsing results of the input video or image or audio content.

[0084] The present invention focuses on the construction and training method of a unified multimodal large model, and for the first time realizes the reasonable segmentation and sampling of multimodal data, thereby ensuring that the modal data obtained by each process is similar and the total length is similar, thereby minimizing the problem of Deepspeed being easily stuck in large-scale multimodal data training.

[0085] The present invention provides rich options by implementing different parallel schemes through different parallel strategies. Two implementation methods are provided for the case of a small number of experts, and for the case where the same GPU cannot accommodate all experts, expert parallelism and small amount of data alignment are performed to achieve multi-node parallelism, minimizing additional communication overhead.

[0086] Furthermore, if Figure 5 As shown, based on the above-mentioned multimodal data analysis method based on hybrid expert structure large model training, the present invention also provides a multimodal data analysis system based on hybrid expert structure large model training, wherein the multimodal data analysis system based on hybrid expert structure large model training includes:

[0087] A data preprocessing module 51 is used to obtain historical multimodal data and preprocess the historical multimodal data to obtain multiple groups of preprocessed multimodal data;

[0088] The multimodal data segmentation module 52 is used to obtain a preset large-scale training model, determine a training parallel mode according to the number of expert networks in the preset large-scale training model, and segment multiple groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set;

[0089] The parsing model building module 53 is used to perform model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data parsing model;

[0090] The multimodal data analysis module 54 is used to obtain current multimodal data and input the current multimodal data into the multimodal data analysis model to obtain a multimodal data analysis result.

[0091] Furthermore, if Figure 6 As shown, based on the above-mentioned multimodal data analysis method and system based on hybrid expert structure large model training, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0092] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a multimodal data parsing program 40 based on hybrid expert structure large model training is stored on the memory 20, and the multimodal data parsing program 40 based on hybrid expert structure large model training can be executed by the processor 10, thereby realizing the multimodal data parsing method based on hybrid expert structure large model training in the present application.

[0093] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the multimodal data analysis method based on the training of the hybrid expert structure large model.

[0094] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0095] In one embodiment, when the processor 10 executes the multimodal data parsing program 40 based on the training of the hybrid expert structure large model in the memory 20, the following steps are implemented:

[0096] Acquiring historical multimodal data, and preprocessing the historical multimodal data to obtain multiple groups of preprocessed multimodal data;

[0097] Obtaining a preset large-scale training model, determining a training parallel mode according to the number of expert networks in the preset large-scale training model, and segmenting the plurality of groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set;

[0098] Performing model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data parsing model;

[0099] The current multimodal data is acquired, and the current multimodal data is input into the multimodal data analysis model to obtain a multimodal data analysis result.

[0100] Wherein, the preprocessing includes: encoding processing, binary identification processing, and modal data division processing;

[0101] The acquiring of historical multimodal data and preprocessing of the historical multimodal data to obtain multiple groups of preprocessed multimodal data specifically includes:

[0102] Acquire historical multimodal data, and use encoders of different modalities to encode the historical multimodal data to obtain multiple modal sequences;

[0103] Performing binary identification processing on the multimodal data in the plurality of modal sequences according to the modality type to obtain a plurality of groups of identified modal data;

[0104] Modal data division processing is performed on the multiple groups of identified modal data to obtain multiple groups of pre-processed multi-modal data.

[0105] The training parallel mode includes a data parallel mode and an expert parallel and data parallel dual parallel mode;

[0106] The step of obtaining a preset large-scale training model, determining a training parallel mode according to the number of expert networks in the preset large-scale training model, and segmenting a plurality of groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set specifically includes:

[0107] Obtaining the number of expert networks in the preset large-scale training model, and determining whether the number of expert networks is less than a preset number;

[0108] If yes, execute the data parallel mode, and perform a first segmentation process on the plurality of groups of the pre-processed multimodal data according to the data parallel mode to obtain a first multimodal data training set;

[0109] If not, a dual-parallel mode of expert parallelism and data parallelism is executed, and a second segmentation process is performed on the plurality of groups of the pre-processed multimodal data according to the dual-parallel mode of expert parallelism and data parallelism to obtain a second multimodal data training set.

[0110] The step of performing a first segmentation process on the plurality of groups of the pre-processed multimodal data according to the data parallel mode to obtain a first multimodal data training set specifically includes:

[0111] According to the data parallel mode, each group of the pre-processed multimodal data is randomly shuffled, and the pre-processed multimodal data after the random shuffle is first segmented according to a preset size to obtain a plurality of first segmentation groups of equal size;

[0112] Perform sampling processing in each group of first cut groups of equal size to obtain multiple sampling groups;

[0113] The multiple sampling groups in each group of the pre-processed multimodal data are mixed and randomly shuffled to obtain the first multimodal data training set.

[0114] The step of performing a second segmentation process on the plurality of groups of the pre-processed multimodal data according to the expert parallel and data parallel dual-parallel modes to obtain a second multimodal data training set specifically includes:

[0115] According to the expert parallel and data parallel dual parallel modes, each group of the pre-processed multimodal data is randomly shuffled, and the pre-processed multimodal data after the random shuffle is sorted according to the preset sorting requirements to obtain a sorting result;

[0116] According to the sorting result, the pre-processed multimodal data after random shuffling is subjected to a second segmentation process according to a preset size to obtain a plurality of second segmentation groups of equal size;

[0117] A plurality of second cut groups of equal sizes in each group of the preprocessed multimodal data are mixed and randomly shuffled to obtain the second multimodal data training set.

[0118] The step of performing model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data parsing model specifically includes:

[0119] Performing encoding processing and semantic space projection processing on the first multimodal data training set or the second multimodal data training set through a corresponding modality encoder to obtain a plurality of projection result data sets, and inputting the plurality of projection result data sets into the preset large-scale training model;

[0120] The plurality of projection result data sets are transmitted to the corresponding expert network for analysis and processing through the shared model part in the preset large-scale training model, so as to obtain a trained multimodal data analysis model.

[0121] The step of acquiring the current multimodal data and inputting the current multimodal data into the multimodal data analysis model to obtain the multimodal data analysis result specifically includes:

[0122] Acquire current multimodal data, and input the current multimodal data into the multimodal data parsing model;

[0123] The current multimodal data is subjected to data analysis processing according to the multimodal data analysis model to obtain the multimodal data analysis result.

[0124] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal data analysis program based on hybrid expert structure large model training, and when the multimodal data analysis program based on hybrid expert structure large model training is executed by a processor, the steps of the multimodal data analysis method based on hybrid expert structure large model training as described above are implemented.

[0125] In summary, the present invention provides a multimodal data analysis method and related equipment based on hybrid expert structure large model training, the method comprising: obtaining historical multimodal data, and preprocessing the historical multimodal data to obtain multiple groups of preprocessed multimodal data; obtaining a preset large-scale training model, determining the training parallel mode according to the number of expert networks in the preset large-scale training model, and segmenting multiple groups of the preprocessed multimodal data according to the training parallel mode to construct a multimodal data training set; performing model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data analysis model; obtaining current multimodal data, and inputting the current multimodal data into the multimodal data analysis model to obtain a multimodal data analysis result. The present invention can ensure that the modal data obtained by each process is similar and the total length is similar by segmenting and sampling the acquired historical multimodal data, thereby minimizing the problem that Deepspeed is easily stuck in large-scale multimodal data training. At the same time, the data parallel mode or the expert parallel and data parallel dual parallel mode is determined according to the number of expert networks in the large-scale training model, so as to construct a multimodal data training set to train the large-scale training model and obtain a multimodal data parsing model; the multimodal data parsing model can achieve efficient parsing of the current multimodal data.

[0126] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0127] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0128] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal data analysis method based on mixed expert structure large model training, characterized in that: The multimodal data analysis method based on mixed expert structure large model training includes: Acquiring historical multimodal data, and preprocessing the historical multimodal data to obtain multiple groups of preprocessed multimodal data; Obtaining a preset large-scale training model, determining a training parallel mode according to the number of expert networks in the preset large-scale training model, and segmenting the plurality of groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set; The training parallel mode includes a data parallel mode and an expert parallel and data parallel dual parallel mode; The step of obtaining a preset large-scale training model, determining a training parallel mode according to the number of expert networks in the preset large-scale training model, and segmenting a plurality of groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set specifically includes: Obtaining the number of expert networks in the preset large-scale training model, and determining whether the number of expert networks is less than a preset number; If yes, execute the data parallel mode, and perform a first segmentation process on the plurality of groups of the pre-processed multimodal data according to the data parallel mode to obtain a first multimodal data training set; If not, executing the expert parallel and data parallel dual-parallel mode, and performing a second segmentation process on the plurality of groups of the pre-processed multimodal data according to the expert parallel and data parallel dual-parallel mode to obtain a second multimodal data training set; The step of performing a first segmentation process on the plurality of groups of the pre-processed multimodal data according to the data parallel mode to obtain a first multimodal data training set specifically includes: According to the data parallel mode, each group of the pre-processed multimodal data is randomly shuffled, and the pre-processed multimodal data after the random shuffle is first segmented according to a preset size to obtain a plurality of first segmentation groups of equal size; Perform sampling processing in each group of first cut groups of equal size to obtain multiple sampling groups; Mixing and randomly shuffling the multiple sampling groups in each group of the pre-processed multimodal data to obtain the first multimodal data training set; The step of performing a second segmentation process on the plurality of groups of the pre-processed multimodal data according to the expert parallel and data parallel dual-parallel modes to obtain a second multimodal data training set specifically includes: According to the expert parallel and data parallel dual parallel modes, each group of the pre-processed multimodal data is randomly shuffled, and the pre-processed multimodal data after the random shuffle is sorted according to the preset sorting requirements to obtain a sorting result; According to the sorting result, the pre-processed multimodal data after random shuffling is subjected to a second segmentation process according to a preset size to obtain a plurality of second segmentation groups of equal size; Mixing and randomly shuffling a plurality of second cut groups of equal size in each group of the preprocessed multimodal data to obtain the second multimodal data training set; Performing model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data parsing model; The current multimodal data is acquired, and the current multimodal data is input into the multimodal data analysis model to obtain a multimodal data analysis result.

2. The multimodal data analysis method based on mixed expert structure large model training according to claim 1 is characterized in that: The preprocessing includes: encoding processing, binary identification processing, and modal data division processing; The acquiring of historical multimodal data and preprocessing of the historical multimodal data to obtain multiple groups of preprocessed multimodal data specifically includes: Acquire historical multimodal data, and use encoders of different modalities to encode the historical multimodal data to obtain multiple modal sequences; Performing binary identification processing on the multimodal data in the plurality of modal sequences according to the modality type to obtain a plurality of groups of identified modal data; Modal data division processing is performed on the multiple groups of identified modal data to obtain multiple groups of pre-processed multi-modal data.

3. The multimodal data analysis method based on mixed expert structure large model training according to claim 1 is characterized in that: The step of performing model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data parsing model specifically includes: Performing encoding processing and semantic space projection processing on the first multimodal data training set or the second multimodal data training set through a corresponding modality encoder to obtain a plurality of projection result data sets, and inputting the plurality of projection result data sets into the preset large-scale training model; The plurality of projection result data sets are transmitted to the corresponding expert network for analysis and processing through the shared model part in the preset large-scale training model, so as to obtain a trained multimodal data analysis model.

4. The multimodal data analysis method based on mixed expert structure large model training according to claim 1 is characterized in that: The acquiring of current multimodal data and inputting the current multimodal data into the multimodal data analysis model to obtain a multimodal data analysis result specifically includes: Acquire current multimodal data, and input the current multimodal data into the multimodal data parsing model; The current multimodal data is subjected to data analysis processing according to the multimodal data analysis model to obtain the multimodal data analysis result.

5. A multimodal data analysis system based on hybrid expert structure large model training, characterized in that: The multimodal data analysis system based on hybrid expert structure large model training includes: A data preprocessing module, used to obtain historical multimodal data and preprocess the historical multimodal data to obtain multiple groups of preprocessed multimodal data; A multimodal data segmentation module is used to obtain a preset large-scale training model, determine a training parallel mode according to the number of expert networks in the preset large-scale training model, and segment multiple groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set; The training parallel mode includes a data parallel mode and an expert parallel and data parallel dual parallel mode; The step of obtaining a preset large-scale training model, determining a training parallel mode according to the number of expert networks in the preset large-scale training model, and segmenting a plurality of groups of the pre-processed multimodal data according to the training parallel mode to construct a multimodal data training set specifically includes: Obtaining the number of expert networks in the preset large-scale training model, and determining whether the number of expert networks is less than a preset number; If yes, execute the data parallel mode, and perform a first segmentation process on the plurality of groups of the pre-processed multimodal data according to the data parallel mode to obtain a first multimodal data training set; If not, executing the expert parallel and data parallel dual-parallel mode, and performing a second segmentation process on the plurality of groups of the pre-processed multimodal data according to the expert parallel and data parallel dual-parallel mode to obtain a second multimodal data training set; The step of performing a first segmentation process on the plurality of groups of the pre-processed multimodal data according to the data parallel mode to obtain a first multimodal data training set specifically includes: According to the data parallel mode, each group of the pre-processed multimodal data is randomly shuffled, and the pre-processed multimodal data after the random shuffle is first segmented according to a preset size to obtain a plurality of first segmentation groups of equal size; Perform sampling processing in each group of first cut groups of equal size to obtain multiple sampling groups; Mixing and randomly shuffling the multiple sampling groups in each group of the pre-processed multimodal data to obtain the first multimodal data training set; The step of performing a second segmentation process on the plurality of groups of the pre-processed multimodal data according to the expert parallel and data parallel dual-parallel modes to obtain a second multimodal data training set specifically includes: According to the expert parallel and data parallel dual parallel modes, each group of the pre-processed multimodal data is randomly shuffled, and the pre-processed multimodal data after the random shuffle is sorted according to the preset sorting requirements to obtain a sorting result; According to the sorting result, the pre-processed multimodal data after random shuffling is subjected to a second segmentation process according to a preset size to obtain a plurality of second segmentation groups of equal size; Mixing and randomly shuffling a plurality of second cut groups of equal size in each group of the preprocessed multimodal data to obtain the second multimodal data training set; A parsing model building module, used to perform model training on the preset large-scale training model according to the multimodal data training set to obtain a multimodal data parsing model; The multimodal data analysis module is used to obtain current multimodal data and input the current multimodal data into the multimodal data analysis model to obtain a multimodal data analysis result.

6. A terminal, characterized in that: The terminal includes: a memory, a processor, and a multimodal data analysis program based on hybrid expert structure large model training stored in the memory and executable on the processor. When the multimodal data analysis program based on hybrid expert structure large model training is executed by the processor, the steps of the multimodal data analysis method based on hybrid expert structure large model training are implemented as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal data analysis program based on training of a hybrid expert structure large model. When the multimodal data analysis program based on training of a hybrid expert structure large model is executed by a processor, the steps of the multimodal data analysis method based on training of a hybrid expert structure large model are implemented as described in any one of claims 1 to 4.

Images