Dynamic token allocation method and device based on multi-modal fusion and storage medium

Abstract

The application discloses a multi-modal fusion dynamic token allocation method and device and a storage medium, relates to the technical field of data fusion, and mainly aims to improve the effect of token allocation on multi-modal data, so that a subsequent large model pays more attention to key information in the multi-modal data when solving a problem, calculation resources are saved, and the ability of the large model to solve a problem based on multi-modal data is improved. The application comprises the following steps: acquiring multi-modal data to be fused required for processing a target problem, wherein the multi-modal data have a complementary relationship in the processing target problem scene; predicting the individual contribution degree of the multi-modal data, and respectively allocating a weight coefficient to the multi-modal data based on the individual contribution degree; and respectively performing dynamic token allocation on the multi-modal data based on the weight coefficient. The application is mainly suitable for a token allocation scene of multi-modal data.

Description

Technical Field

[0001] The present invention relates to the field of data fusion technology, and in particular to a multimodal fusion dynamic token allocation method, device and storage medium. Background Art

[0002] Currently, large model technology is experiencing explosive growth. Large language models such as GPT-4, Claude, Qwen, and LLaMA, trained with hundreds to trillions of parameters and massive amounts of data, have pushed the boundaries of natural language understanding, logical reasoning, and code generation. They are widely used in fields such as intelligent customer service, educational tutoring, scientific research literature analysis, and legal document generation. Multimodal large models further integrate multimodal inputs such as vision, language, and audio, achieving a unified semantic representation across multiple modalities through cross-modal alignment techniques, significantly improving cognitive accuracy and adaptability for complex tasks. Representative models such as GPT-4V, InternVL, Qwen VL, Gemini, and LLaVA, integrate multimodal data such as vision, language, and audio, and achieve cross-modal joint modeling based on the Transformer architecture. Through large-scale pre-training and instruction fine-tuning, these models significantly improve cross-modal semantic understanding, generation, and reasoning capabilities. They have been widely used in fields such as image captioning, visual question answering, cross-modal retrieval, and intelligent education. Based on this, in order to improve the problem processing accuracy of large models, it is necessary to assign tokens to multimodal data.

[0003] Currently, tokens are typically assigned to all multimodal data using a uniform, artificially determined ratio. However, different modal data contain different information, and this uniform allocation can lead to suboptimal token allocation. This can result in excessive allocations of tokens to data that doesn't need them, leading to wasted resources and the introduction of additional noise. Furthermore, a small number of tokens can be assigned to data that needs more, resulting in insufficient information and hindering the problem-solving capabilities of subsequent large-scale models. Summary of the Invention

[0004] The present invention provides a dynamic token allocation method, device and storage medium for multimodal fusion, which mainly aims to improve the effect of token allocation for multimodal data, save computing resources, and thus enhance the ability of large models to solve problems based on multimodal data.

[0005] According to a first aspect of the present invention, a multimodal fusion dynamic token allocation method is provided, comprising:

[0006] Acquire multimodal data to be fused required for processing a target problem, wherein there is a complementary relationship between the multimodal data in a scenario of processing the target problem;

[0007] Predicting individual contributions of the multimodal data, and assigning weight coefficients to the multimodal data based on the individual contributions;

[0008] Based on the weight coefficients, tokens are dynamically allocated for the multimodal data.

[0009] Optionally, the multimodal data includes text modal data;

[0010] The method for predicting the monomer contribution of the text modal data includes:

[0011] Inputting the text modal data multiple times into the text problem processing model to process the target problem, thereby obtaining multiple text processing results;

[0012] Performing similarity matching on each text processing result and a text processing reference result corresponding to the target question, and determining the text processing accuracy of each text processing result based on the text similarity matching results;

[0013] Determining the number of precise text processing results of the text processing results having a text processing accuracy greater than a first preset threshold, and taking the ratio of the number of precise text processing results to the total number of text processing results as the precise result hit rate of the text modality data;

[0014] Based on the precise result hit rate, the individual contribution of the text modality data is determined.

[0015] Optionally, before performing similarity matching on each text processing result and a text processing reference result corresponding to the target question, the method further includes:

[0016] The result of the target problem is predicted through a multimodal large model to obtain the text processing reference result.

[0017] Optionally, the multimodal data further includes non-text modal data;

[0018] The method for predicting the individual contribution of the non-text modal data includes:

[0019] Using the conversion evaluation model to judge multiple times whether the non-text modal data can be generated based on the text modal data, and determining the generation ratio based on the multiple judgment results;

[0020] Determining a hit rate of accurate processing results when processing the target question based solely on the non-text modal data, and determining an importance score of the non-text modal data based on the hit rate of accurate processing results;

[0021] Based on the generable determination ratio and the importance score, the individual contribution of the non-text modality data is determined.

[0022] Optionally, the conversion evaluation model is used to determine multiple times whether the non-text modal data can be generated based on the text modal data, including:

[0023] Using the transformation evaluation model, respectively extracting text feature vectors from the text modal data and non-text feature vectors from the non-text modal data;

[0024] Calculate the cosine similarity between the text feature vector and the non-text feature vector, and determine whether the cosine similarity is greater than a preset similarity threshold; if so, determine that the non-text modal data can be generated based on the text modal data; otherwise, determine that the non-text modal data cannot be generated based on the text modal data.

[0025] Optionally, determining a hit rate of accurate processing results when processing the target question based solely on the non-text modal data includes:

[0026] Inputting the non-text modal data multiple times into the non-text question processing large model to process the target question, thereby obtaining multiple non-text processing results;

[0027] Performing similarity matching on each non-text processing result and a non-text processing reference result corresponding to the target question, and determining the non-text processing accuracy of each non-text processing result based on the non-text similarity matching results;

[0028] The number of accurate non-text processing results of the non-text processing results whose non-text processing accuracy is greater than a second preset threshold is determined, and the ratio of the number of accurate non-text processing results to the total number of non-text processing results is used as the processing accuracy result hit rate.

[0029] Optionally, determining the individual contribution of the non-text modality data based on the generable determination ratio and the importance score includes:

[0030] Get the proportion of the generated judgment The corresponding first adjustable parameter and the importance score The corresponding second adjustable parameter , based on the determination ratio that can be generated , the first adjustable parameter , the importance score The second adjustable parameter , determine the monomer contribution of the non-text modal data ,in, .

[0031] Optionally, based on the individual contribution, weight coefficients are assigned to the multimodal data respectively, including:

[0032] If the multimodal data includes image data, determining the image complexity of the image data;

[0033] Determining a weight coefficient of the image data based on the image complexity, the monomer contribution corresponding to the image data, and a first preset normalization coefficient;

[0034] A weight coefficient of the residual modal data is determined based on a single contribution of the residual modal data and a second preset normalization coefficient, wherein the multimodal data after removing the image data is used as the residual modal data.

[0035] Optionally, determining the image complexity of the image data includes:

[0036] Determining an image feature vector corresponding to the image data, and determining a feature dimension of the image feature vector, a dimension value of each feature dimension, and a dimension weight of each feature dimension;

[0037] The image complexity of the image data is determined based on the feature dimension, the dimension value, and the dimension weight.

[0038] Optionally, dynamically allocating tokens for the multimodal data based on the weight coefficients includes:

[0039] Divide the weight coefficient corresponding to each multimodal data by the sum of the weight coefficients of the multimodal data to obtain the proportion of the to-be-allocated tokens of the multimodal data in the overall allocated tokens, and use the ratio of each proportion as the token allocation ratio between the multimodal data;

[0040] Based on the token allocation ratio, tokens are dynamically allocated for the multimodal data.

[0041] Optionally, dynamically allocating tokens for the multimodal data based on the weight coefficients includes:

[0042] Determine the problem complexity of the target problem and the total token limit ;

[0043] a weight coefficient of a first modality data in the multi-modal data a problem complexity a total token limit number a total number of categories of the multi-modal data a token quota of the first modality data is dynamically determined wherein, wherein, , is a decay factor of the problem complexity .

[0044] According to a second aspect of the present application, there is provided a multi-modal fusion dynamic token allocation device, comprising:

[0045] an acquisition unit configured to acquire multi-modal data to be fused required for processing a target problem, wherein there is a complementary relationship between the multi-modal data in the processing of the target problem scenario;

[0046] a prediction unit configured to predict a single contribution degree of the multi-modal data, and respectively allocate a weight coefficient to the multi-modal data based on the single contribution degree;

[0047] a dynamic allocation unit configured to respectively perform dynamic allocation of tokens to the multi-modal data based on the weight coefficient.

[0048] According to a third aspect of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above multi-modal fusion dynamic token allocation method.

[0049] According to a fourth aspect of the present application, there is provided a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the above multi-modal fusion dynamic token allocation method when executing the program.

[0050] ​According to the multi-modal fusion dynamic token allocation method, device and storage medium provided by the application, compared with the current token allocation method for all multi-modal data according to the unified proportion determined by human, the application predicts the single contribution degree of multi-modal data, and respectively allocates weight coefficients to the multi-modal data based on the single contribution degree, and finally dynamically allocates tokens to the multi-modal data based on the weight coefficients, so that the application dynamically allocates tokens to the multi-modal data according to the contribution degree of the multi-modal data to the problem solving, can reasonably allocate tokens to the multi-modal data according to the value of the information in the data, avoids excessive processing of low contribution modes in the fixed allocation mode, reduces invalid calculation, at the same time, by allocating more tokens to the modes with high contribution degree, the sufficiency of the amount of information required for problem solving can be ensured, and the subsequent large model pays more attention to the key information in the multi-modal data when solving the problem, so as to improve the problem solving ability of the subsequent large model, at the same time, the application dynamically allocates tokens according to the single contribution degree of the multi-modal data, can flexibly adjust the token allocation proportion according to the task demand, so as to significantly improve the model performance, resource utilization efficiency and application adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application, and do not limit the application. In the drawings:

[0052] Figure 1 A multi-modal fusion dynamic token allocation method flowchart provided by an embodiment of the application is shown;

[0053] Figure 2 Another multi-modal fusion dynamic token allocation method flowchart provided by an embodiment of the application is shown;

[0054] Figure 3 A structural schematic diagram of a multi-modal fusion dynamic token allocation device provided by an embodiment of the application is shown;

[0055] Figure 4 Another structural schematic diagram of a multi-modal fusion dynamic token allocation device provided by an embodiment of the application is shown;

[0056] Figure 5 An entity structure schematic diagram of a computer device provided by an embodiment of the application is shown. DETAILED DESCRIPTION

[0057] Hereinafter, the present application will be described in detail with reference to the accompanying drawings and embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0058] At present, the way of assigning tokens to all multi-modal data according to a unified proportion determined by human beings may cause the problem of assigning too many tokens to data that does not need to be assigned too many tokens, thereby causing resource waste and introducing additional noise, and the problem of assigning a small number of tokens to data that needs to be assigned a large number of tokens, thereby causing insufficient information, and further affecting the problem solving ability of the subsequent large model.

[0059] To solve the above problems, the embodiment of the present application provides a multi-modal fusion dynamic token assignment method, as shown in Figure 1 The method comprises the following steps.

[0060] 101. Obtain multi-modal data to be fused required for processing a target problem, wherein in the processing of the target problem scene, there is a complementary relationship between the multi-modal data.

[0061] The target problem can be a problem in any scene such as intelligent customer service, education guidance, literature analysis, such as a question answering problem in a test paper, a question answering problem in an intelligent customer service scene, a visual question answering problem, etc. The multi-modal data includes text modal data and non-text modal data, such as image data, video data, audio data, voice data, etc. Different modal data complement and synergistically enhance each other by providing different angles or different types of information, thereby more comprehensively and accurately describing or solving the target problem.

[0062] For the embodiment of the present application, if a certain question in a test paper is to be answered, and the question includes text information and picture information, first, the multi-modal data such as the text information and the picture information in the question are obtained, then the text information and the picture information are dynamically assigned tokens according to the contribution degree of the text information and the picture information in answering the question, and finally the question is answered according to the token assignment result. Token is the smallest processing unit of data, which is used to decompose continuous and complex data into structured and operable segments.

[0063] 102. Predict the single contribution degree of the multi-modal data, and assign a weight coefficient to each multi-modal data based on the single contribution degree.

[0064] The single contribution degree refers to the contribution degree of each modal data to solving the target problem when the multi-modal data is used to solve the target problem cooperatively.

[0065] For an embodiment of the present invention, if the multimodal data includes text modal data and non-text modal data, the contribution of each modal data to solving the target problem when the text modal data and non-text modal data work together to solve the target problem is determined separately. Then, based on the contribution, weight coefficients are assigned to the text modal data and the non-text modal data. That is, the greater the contribution, the more important the modal data is to solving the target problem, and a larger weight should be assigned to it. The smaller the contribution, the less important the modal data is to solving the target problem, and a smaller weight should be assigned to it. In one embodiment of the present invention, a method for assigning weight coefficients to multimodal data includes: if the multimodal data includes image data, determining the image complexity of the image data; determining the weight coefficient of the image data based on the image complexity, the individual contribution corresponding to the image data, and a first preset normalization coefficient; and determining the weight coefficient of the remaining modal data based on the individual contribution of the remaining modal data and a second preset normalization coefficient, wherein the multimodal data after removing the image data is used as the remaining modal data. Among them, the method for determining the image complexity of image data includes: determining the image feature vector corresponding to the image data, and determining the feature dimension of the image feature vector, the dimension value of each feature dimension, and the dimension weight of each feature dimension; based on the feature dimension, the dimension value, and the dimension weight, determining the image complexity of the image data.

[0066] Specifically, in the case where multimodal data includes text modal data and non-text modal data, if the non-text modal data is image data, then in order to determine the distribution weight coefficients corresponding to the text modal data and the image data respectively, it is necessary to first determine the image feature vector of the image data, such as using a pre-built image feature extraction model to extract the image feature vector in the image data, and then determine the length of the image feature vector, that is, the total number of feature dimensions , and directly obtain each feature dimension according to the generation method of the image feature vector Dimension value of If the image feature vector in the embodiment of the present invention is extracted based on the model, the dimension value of the image feature vector is determined by the model architecture, that is, the dimension value can be directly output from the image feature extraction process of the model. At the same time, according to the importance of each feature dimension to the image complexity calculation, Assigning dimension weights , and then determine the image complexity of the image data according to the following formula :

[0067]

[0068] Furthermore, the weight coefficient of the image data is determined according to the following formula: :

[0069]

[0070] in, is the individual contribution of image data, The first preset normalization coefficient is set according to actual needs. The embodiment of the present invention determines its weight coefficient by comprehensively analyzing the complexity of image data, monomer contribution and other information. It can assign weights to the information according to the value of the information contained in the image, ensuring the accuracy of weight allocation, and thus ensuring the rationality and accuracy of token allocation. At the same time, the weight coefficient of any remaining modal data, such as text modal data, is determined according to the following formula: :

[0071]

[0072] in, is the monomer contribution of any remaining modal data, is a second preset normalization coefficient set according to actual needs. This embodiment of the present invention assigns weights based on the contribution of multimodal data to problem solving, and then allocates tokens based on the weights. This allows for reasonable allocation of tokens to multimodal data based on the value of the information contained in the data, avoiding over-processing of low-contribution modalities under a fixed allocation method, reducing ineffective computations, and thus improving problem-solving efficiency.

[0073] 103. Based on the weight coefficient, tokens are dynamically allocated for multimodal data.

[0074] For an embodiment of the present invention, after determining the weight coefficient of the multimodal data, it is necessary to dynamically allocate tokens for the multimodal data based on the weight coefficient. Based on this, step 103 includes: dividing the weight coefficient corresponding to the multimodal data individually by the sum of the weight coefficients of the multimodal data, and obtaining the proportion of the to-be-allocated tokens of the multimodal data in the overall allocated tokens, and taking the ratio of each proportion as the token allocation ratio between the multimodal data; based on the token allocation ratio, dynamically allocate tokens for the multimodal data respectively.

[0075] Specifically, the proportion of tokens to be allocated for multimodal data in the total allocated tokens is calculated according to the following formula: :

[0076]

[0077] in, is the weight coefficient of the i-th modal data, is the total number of modalities of multimodal data. Finally, the ratio of the proportion of each modal data is used as the token allocation ratio between multimodal data. For example, in multimodal data, the token of text modal data can be set according to actual needs. The token allocation ratio and the token allocation number of text modal data can be used to determine the token allocation number of other modal data. The embodiment of the present invention dynamically allocates tokens to multimodal data according to the degree of contribution to problem solving, and can reasonably allocate tokens to multimodal data, avoiding excessive processing of low-contribution modalities under a fixed allocation method, reducing invalid calculations, and at the same time, by allocating more tokens to modalities with high contribution, it can ensure the sufficiency of the amount of information required for problem solving, thereby improving the problem-solving ability of subsequent large models. At the same time, the present invention dynamically allocates tokens according to the individual contribution of multimodal data, and can flexibly adjust the token allocation ratio according to task requirements, thereby significantly improving model performance, resource utilization efficiency and application adaptability.

[0078] According to a dynamic token allocation method for multimodal fusion provided by the present invention, compared with the current method of allocating tokens to all multimodal data based on a uniform ratio determined artificially, the present invention predicts the individual contribution of multimodal data and assigns weight coefficients to the multimodal data based on the individual contribution. Finally, based on the weight coefficients, tokens are dynamically allocated to the multimodal data. As a result, the present invention dynamically allocates tokens to the multimodal data according to its contribution to problem solving, can reasonably allocate tokens to the multimodal data, avoids excessive processing of low-contribution modalities under a fixed allocation method, and reduces invalid calculations. At the same time, by allocating more tokens to modalities with high contribution, it can ensure the sufficiency of the amount of information required for problem solving, and enable subsequent large models to pay more attention to key information in the multimodal data when solving problems, thereby improving the problem-solving ability of subsequent large models. At the same time, the present invention dynamically allocates tokens based on the individual contribution of multimodal data, and can flexibly adjust the token allocation ratio according to task requirements, thereby significantly improving model performance, resource utilization efficiency, and application adaptability.

[0079] Furthermore, in order to better illustrate the above process of dynamically allocating tokens, as a refinement and extension of the above embodiment, the embodiment of the present invention provides another multimodal fusion dynamic token allocation method, such as Figure 2 As shown, the method includes:

[0080] 201. Obtain multimodal data to be fused required for processing a target problem, wherein, in the scenario of processing the target problem, there is a complementary relationship between the multimodal data, and the multimodal data includes text modal data and non-text modal data.

[0081] 202. Predicting the single contribution degree of the text modal data.

[0082] For the embodiment of the present application, in order to allocate tokens to the text modal data, it is necessary to first determine the single contribution degree of the text modal data. Based on this, step 202 specifically includes: inputting the text modal data into a text question processing large model multiple times to process the target question, obtaining multiple text processing results; performing similarity matching between each text processing result and a text processing reference result corresponding to the target question, and determining the text processing accuracy of each text processing result based on the text similarity matching result; determining the number of accurate text processing results of the text processing results whose text processing accuracy is greater than a first preset threshold, and taking the ratio of the number of accurate text processing results to the total number of text processing results as the accurate result hit rate of the text modal data; and determining the single contribution degree of the text modal data based on the accurate result hit rate. The method for determining the text processing reference result corresponding to the target question includes: predicting the result of the target question through a multi-modal large model to obtain the text processing reference result.

[0083] Wherein, the first preset threshold is set according to actual needs; the text processing large model can be GPT, Qwen, etc. series of large models. Specifically, using the text question processing large model to require the large model to solve the target question under the premise of only providing the text modal data, for example, if the multi-modal data is the description text data and the picture data of a certain question in the test paper, using the text question processing large model to require the large model to answer the question multiple times under the premise of only providing the description text data, and to determine whether the answer result is the same as the text processing reference result, and then to determine the accurate result hit rate according to the following formula :

[0084]

[0085] Wherein, is the text processing result, is the text processing reference result, is the number of accurate text processing results, is the total number of text processing results. Further, the single contribution degree of the text modal data is determined according to the following formula :

[0086]

[0087] Wherein, is an adjustable parameter set according to actual needs.

[0088] The embodiment of the present invention solves the problem multiple times through a large model and calculates the accuracy rate through a large amount of experimental data, which can more accurately reflect the contribution of the text content to the problem solving.

[0089] 203. Predicting individual contributions of non-text modal data.

[0090] For the embodiment of the present invention, in order to assign tokens to non-text modal data, it is first necessary to determine the individual contribution of the non-text modal data. Based on this, step 203 specifically includes: using the conversion evaluation large model to judge multiple times whether the non-text modal data can be generated based on the text modal data, and determining the proportion of judgments that can be generated based on the results of multiple judgments; determining the hit rate of accurate processing results when the target problem is processed based on the non-text modal data alone, and determining the importance score of the non-text modal data based on the accurate processing result hit rate; determining the individual contribution of the non-text modal data based on the proportion of judgments that can be generated and the importance score.

[0091] Among them, the method of using a conversion evaluation large model to repeatedly determine whether non-text modal data can be generated based on text modal data includes: using the conversion evaluation large model to respectively extract text feature vectors in the text modal data and non-text feature vectors in the non-text modal data; calculating the cosine similarity between the text feature vector and the non-text feature vector, and determining whether the cosine similarity is greater than a preset similarity threshold; if so, determining that the non-text modal data can be generated based on the text modal data; otherwise, determining that the non-text modal data cannot be generated based on the text modal data.

[0092] Among them, the preset similarity threshold is set according to actual needs. Specifically, if it is determined that the text modal data and the non-text modal data are not similar based on the text feature vector and the non-text feature vector, then it is determined that the non-text modal data cannot be generated based on the text modal data; if it is determined that the similarity between the text modal data and the non-text modal data is high, then it is determined that the non-text modal data can be generated based on the text modal data. Further, the determination ratio can be generated according to the following formula :

[0093]

[0094] in, To determine the number of times non-text modal data can be generated based on text modal data in multiple judgment results, The total number of judgments for the conversion evaluation model.

[0095] The method for determining the processing accuracy result hit rate when the target problem is processed according to the non-text modal data alone comprises: inputting the non-text modal data into a non-text problem processing large model multiple times to process the target problem, and obtaining multiple non-text processing results; performing similarity matching between each non-text processing result and a non-text processing reference result corresponding to the target problem, and determining the non-text processing accuracy of each non-text processing result based on the non-text similarity matching result; determining the number of accurate non-text processing results of the non-text processing result whose non-text processing accuracy is greater than a second preset threshold, and taking the ratio of the number of accurate non-text processing results to the total number of non-text processing results as the processing accuracy result hit rate.

[0096] The second preset threshold is set according to actual needs, and the non-text processing reference result can be obtained by predicting the result of the target problem by the multi-modal large model. Specifically, the non-text problem processing large model is required to solve the target problem under the premise of providing only non-text modal data, for example, if the multi-modal data is description text data and image data of a certain question in a test paper, the non-text problem processing large model is required to answer the question multiple times under the premise of providing only image data, and it is determined whether the answer result is the same as the non-text processing reference result, and then the processing accuracy result hit rate is determined according to the following formula :

[0097]

[0098] wherein, is the non-text processing result, is the non-text processing reference result, is the number of accurate non-text processing results, is the total number of non-text processing results. Further, the importance score of the non-text modal data is determined according to the following formula :

[0099]

[0100] wherein, is an adjustable parameter set according to actual needs. Further, after determining the generatable judgment proportion and the importance score, the single contribution degree of the non-text modal data needs to be determined according to the generatable judgment proportion and the importance score, based on which the method comprises: obtaining the generatable judgment proportion corresponding to the first adjustable parameter and the importance score corresponding to the second adjustable parameter , based on the generatable judgment proportion , the first adjustable parameter , the importance score The second adjustable parameter , determine the monomer contribution of the non-text modal data ,in, .

[0101] Among them, the first adjustable parameter , the second adjustable parameter They are constant values ​​set according to actual needs.

[0102] The embodiment of the present invention determines the contribution of non-text modal data by comprehensively considering the ability of text modal data to generate non-text modal data and the importance score of non-text modal data when solving the target problem alone. By increasing the analysis dimension, the comprehensiveness of the problem analysis is ensured, thereby improving the accuracy of determining the contribution of individual non-text modal data.

[0103] 204. Based on the individual contribution, weight coefficients are assigned to the multimodal data.

[0104] Specifically, if the contribution of a single entity is greater, the weight coefficient of the corresponding modal data is greater; if the contribution of a single entity is smaller, the weight coefficient of the corresponding modal data is smaller.

[0105] 205. Based on the weight coefficient, tokens are dynamically allocated for multimodal data.

[0106] For the embodiment of the present invention, after determining the weight coefficients corresponding to the multimodal data, it is necessary to dynamically allocate tokens for the multimodal data according to the weight coefficients. Based on this, step 205 specifically includes: determining the problem complexity of the target problem and the total token limit Based on the multimodal data Weight coefficient of modal data , the complexity of the problem 、Total limit of tokens , the total number of types of multimodal data , dynamically determine the Token quota for modal data ,in, , Problem complexity attenuation factor.

[0107] Specifically, first determine the text features, mathematical features, and logical features of the target problem. Text features include question stem length, number of keywords, density of professional terms, and nesting levels of conditional statements; mathematical features include the number of variables involved, the number of equations or inequalities, and the relationship between unknowns and equations; logical features include the number of logical branches, the number of implicit conditions, and the dependency relationship of multi-step reasoning. Then, assign corresponding weights to different features, and perform feature weighted scoring based on the weights. The complexity of the problem is determined based on the scoring results. The total token limit is determined based on the pre-set value according to actual needs. The token allocation in the embodiment of the present invention not only takes into account the modality weight but also performs dynamic adjustments based on the problem complexity and attenuation factor, ensuring that token resources are efficiently utilized and avoiding excessive consumption on simple problems or low-weight modalities.

[0108] According to another dynamic token allocation method for multimodal fusion provided by the present invention, compared with the current method of allocating tokens to all multimodal data based on a uniform ratio determined artificially, the present invention predicts the individual contribution of multimodal data and assigns weight coefficients to the multimodal data based on the individual contribution. Finally, based on the weight coefficients, tokens are dynamically allocated to the multimodal data. As a result, the present invention dynamically allocates tokens to the multimodal data according to its contribution to problem solving, and can reasonably allocate tokens to the multimodal data, avoiding excessive processing of low-contribution modes under a fixed allocation method and reducing invalid calculations. At the same time, by allocating more tokens to modes with high contribution, the sufficiency of the information required for problem solving can be ensured, and subsequent large models can pay more attention to key information in the multimodal data when solving problems, thereby improving the problem-solving ability of subsequent large models. At the same time, the present invention dynamically allocates tokens based on the individual contribution of multimodal data, and can flexibly adjust the token allocation ratio according to task requirements, thereby significantly improving model performance, resource utilization efficiency and application adaptability.

[0109] Further, as Figure 1 The specific implementation of the present invention provides a multi-modal fusion dynamic token allocation device, such as Figure 3 As shown, the device includes: an acquisition unit 31, a prediction unit 32, and a dynamic allocation unit 33.

[0110] The acquisition unit 31 may be configured to acquire multimodal data to be fused that is required for processing a target problem, wherein, in a scenario of processing the target problem, there is a complementary relationship between the multimodal data.

[0111] The prediction unit 32 can be configured to predict individual contribution degrees of the multi-modal data, and respectively assign weight coefficients to the multi-modal data based on the individual contribution degrees.

[0112] The dynamic allocation unit 33 can be configured to respectively perform dynamic allocation of tokens to the multi-modal data based on the weight coefficients.

[0113] In a specific application scenario, the multi-modal data includes text modal data; in order to predict the individual contribution degree of the text modal data, the prediction unit 32 can be configured to respectively input the text modal data into a text question processing large model multiple times to process the target question, and obtain multiple text processing results. Figure 4 As shown in the figure, the prediction unit 32 includes a processing module 321, a matching module 322, and a first determination module 323.

[0114] The processing module 321 can be configured to respectively input the text modal data into a text question processing large model multiple times to process the target question, and obtain multiple text processing results.

[0115] The matching module 322 can be configured to respectively perform similarity matching between each text processing result and a text processing reference result corresponding to the target question, and determine a text processing accuracy of each text processing result based on the text similarity matching result.

[0116] The first determination module 323 can be configured to determine a number of accurate text processing results of the text processing results whose text processing accuracy is greater than a first preset threshold, and take a ratio of the number of accurate text processing results to a total number of the text processing results as an accurate result hit rate of the text modal data.

[0117] The first determination module 323 can also be configured to determine the individual contribution degree of the text modal data based on the accurate result hit rate.

[0118] In a specific application scenario, in order to determine the text processing reference result, the first determination module 323 can also be configured to perform result prediction on the target question by a multi-modal large model to obtain the text processing reference result.

[0119] In a specific application scenario, the multi-modal data also includes non-text modal data; in order to predict the individual contribution degree of the non-text modal data, the prediction unit 32 further includes a judgment module 324.

[0120] The judgment module 324 can be configured to respectively determine whether the non-text modal data can be generated according to the text modal data multiple times by using a conversion evaluation large model, and determine a generation determination proportion according to multiple determination results.

[0121] The first determination module 323 can also be used to determine a processing accuracy result hit rate when the target problem is processed according to the non-text modality data alone, and determine an importance score of the non-text modality data based on the processing accuracy result hit rate.

[0122] The first determination module 323 can also be used to determine a single contribution degree of the non-text modality data based on the generatable judgment proportion and the importance score.

[0123] In a specific application scenario, in order to determine whether the non-text modality data can be generated according to the text modality data, the judgment module 324 can specifically be used to extract a text feature vector in the text modality data and a non-text feature vector in the non-text modality data by using the conversion evaluation large model; calculate a cosine similarity between the text feature vector and the non-text feature vector, and determine whether the cosine similarity is greater than a preset similarity threshold, if yes, it is determined that the non-text modality data can be generated according to the text modality data, otherwise, it is determined that the non-text modality data cannot be generated according to the text modality data.

[0124] In a specific application scenario, in order to determine the processing accuracy result hit rate when the target problem is processed according to the non-text modality data alone, the first determination module 323 can specifically be used to input the non-text modality data into a non-text problem processing large model multiple times to process the target problem, and obtain multiple non-text processing results; perform similarity matching between each non-text processing result and a non-text processing reference result corresponding to the target problem, and determine a non-text processing accuracy of each non-text processing result based on the non-text similarity matching result; determine a number of accurate non-text processing results of the non-text processing results whose non-text processing accuracy is greater than a second preset threshold, and take a ratio of the number of accurate non-text processing results to a total number of the non-text processing results as the processing accuracy result hit rate.

[0125] In a specific application scenario, in order to determine the single contribution degree of the non-text modality data, the first determination module 323 can specifically be used to obtain the generatable judgment proportion a corresponding first adjustable parameter and the importance score a corresponding second adjustable parameter , determine the single contribution degree of the non-text modality data based on the generatable judgment proportion , the first adjustable parameter , the importance score , the second adjustable parameter . ​.

[0126] In a specific application scenario, in order to assign a weight coefficient to multimodal data, the first determination module 323 can also be used to determine the image complexity of the image data if the multimodal data includes image data; determine the weight coefficient of the image data based on the image complexity, the individual contribution corresponding to the image data, and a first preset normalization coefficient; determine the weight coefficient of the remaining modal data based on the individual contribution of the remaining modal data and a second preset normalization coefficient, wherein the multimodal data after removing the image data is used as the remaining modal data.

[0127] In a specific application scenario, in order to determine the image complexity of image data, the first determination module 323 can be specifically used to determine the image feature vector corresponding to the image data, and determine the feature dimension of the image feature vector, the dimension value of each feature dimension, and the dimension weight of each feature dimension; based on the feature dimension, the dimension value, and the dimension weight, the image complexity of the image data is determined.

[0128] In a specific application scenario, in order to dynamically allocate tokens for multimodal data, the dynamic allocation unit 33 includes a division module 331 and a dynamic allocation module 332 .

[0129] The division module 331 can be used to divide the weight coefficient corresponding to each multimodal data by the sum of the weight coefficients of the multimodal data, and obtain the proportion of the to-be-allocated token of the multimodal data in the overall allocated token, and use the ratio of each proportion as the token allocation ratio between the multimodal data.

[0130] The dynamic allocation module 332 may be configured to dynamically allocate tokens to the multimodal data based on the token allocation ratio.

[0131] In a specific application scenario, in order to dynamically allocate tokens for multimodal data, the dynamic allocation unit 33 further includes a second determination module 333 .

[0132] The second determination module 333 can be used to determine the problem complexity of the target problem. and the total token limit Based on the multimodal data Weight coefficient of modal data , the complexity of the problem 、Total limit of tokens , the total number of types of multimodal data , dynamically determine the Token quota of modal data wherein, , is a problem complexity decay factor.

[0133] It should be noted that other corresponding descriptions of the various functional modules involved in the multi-modal fusion dynamic token allocation device provided by the embodiments of the present application can be referred to the corresponding descriptions of the method shown in Figure 1 , which will not be repeated here.

[0134] Based on the above method as shown in Figure 1 , accordingly, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program, the program is executed by a processor to realize the following steps: obtaining the multi-modal data to be fused required for processing a target problem, wherein there is a complementary relationship between the multi-modal data in processing the target problem scenario; predicting the single contribution degree of the multi-modal data, and based on the single contribution degree, respectively allocating a weight coefficient to the multi-modal data; based on the weight coefficient, respectively performing dynamic allocation of tokens for the multi-modal data.

[0135] Based on the above method as shown in Figure 1 and the device as shown in Figure 3 embodiments, the embodiments of the present application also provide an entity structure diagram of a computer device, as shown in Figure 5 , the computer device comprises a processor 41, a memory 42, and a computer program stored on the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43, and the processor 41 executes the program to realize the following steps: obtaining the multi-modal data to be fused required for processing a target problem, wherein there is a complementary relationship between the multi-modal data in processing the target problem scenario; predicting the single contribution degree of the multi-modal data, and based on the single contribution degree, respectively allocating a weight coefficient to the multi-modal data; based on the weight coefficient, respectively performing dynamic allocation of tokens for the multi-modal data.

[0136] Through the technical scheme of the present application, the present application predicts the single contribution degree of the multi-modal data, and respectively allocates a weight coefficient to the multi-modal data based on the single contribution degree, and finally respectively allocates tokens to the multi-modal data based on the weight coefficient, thereby the present application dynamically allocates tokens to the multi-modal data according to the contribution degree of the multi-modal data to problem solving, can reasonably allocate tokens to the multi-modal data, avoids excessive processing of low contribution modes under a fixed allocation mode, reduces invalid calculation, at the same time, by allocating more tokens to modes with high contribution degree, the sufficiency of the amount of information required for problem solving can be ensured, and the subsequent large model pays more attention to the key information in the multi-modal data when solving the problem, thereby improving the problem solving capability of the subsequent large model, at the same time, the present application dynamically allocates tokens according to the single contribution degree of the multi-modal data, can flexibly adjust the token allocation ratio according to the task demand, thereby significantly improving the model performance, resource utilization efficiency and application adaptability.

[0137] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by a general computing device, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by the computing device, so that they can be stored in the storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0138] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A dynamic token allocation method for multimodal fusion, characterized in that: include: Acquire multimodal data to be fused required for processing the target problem, wherein there is a complementary relationship between the multimodal data in the scenario of processing the target problem; Predicting individual contributions of the multimodal data, and assigning weight coefficients to the multimodal data based on the individual contributions; Based on the weight coefficients, dynamically allocate tokens for the multimodal data respectively; The multimodal data includes text modal data and non-text modal data, and the method for predicting the individual contribution of the text modal data includes: The text modality data is input into the text question processing large model multiple times to process the target question, and multiple text processing results are obtained; each text processing result is matched with the text processing reference result corresponding to the target question for similarity, and the text processing accuracy of each text processing result is determined based on the text similarity matching result; the number of precise text processing results of the text processing results whose text processing accuracy is greater than a first preset threshold is determined, and the ratio of the number of precise text processing results to the total number of text processing results is used as the precise result hit rate of the text modality data; based on the precise result hit rate, the monomer contribution of the text modality data is determined; The method for predicting the individual contribution of the non-text modal data includes: The conversion evaluation large model is used to judge multiple times whether the non-text modal data can be generated based on the text modal data, and the proportion of generable judgments is determined based on the multiple judgment results; the processing accuracy result hit rate when the target problem is processed based on the non-text modal data alone is determined, and the importance score of the non-text modal data is determined based on the processing accuracy result hit rate; based on the proportion of generable judgments and the importance score, the individual contribution of the non-text modal data is determined.

2. The method according to claim 1, characterized in that Before performing similarity matching on each text processing result and the text processing reference result corresponding to the target question, the method further includes: The result of the target problem is predicted through a multimodal large model to obtain the text processing reference result.

3. The method according to claim 1, characterized in that The conversion evaluation model is used to determine multiple times whether the non-text modal data can be generated based on the text modal data, including: Using the transformation evaluation model, respectively extracting text feature vectors from the text modal data and non-text feature vectors from the non-text modal data; Calculate the cosine similarity between the text feature vector and the non-text feature vector, and determine whether the cosine similarity is greater than a preset similarity threshold; if so, determine that the non-text modal data can be generated based on the text modal data; otherwise, determine that the non-text modal data cannot be generated based on the text modal data.

4. The method according to claim 1, wherein Determining a hit rate of accurate processing results when processing the target question based solely on the non-text modal data includes: Inputting the non-text modal data multiple times into the non-text question processing large model to process the target question, thereby obtaining multiple non-text processing results; Performing similarity matching on each non-text processing result and a non-text processing reference result corresponding to the target question, and determining the non-text processing accuracy of each non-text processing result based on the non-text similarity matching results; The number of accurate non-text processing results of the non-text processing results whose non-text processing accuracy is greater than a second preset threshold is determined, and the ratio of the number of accurate non-text processing results to the total number of non-text processing results is used as the processing accuracy result hit rate.

5. The method according to claim 1, wherein Determining the individual contribution of the non-text modality data based on the generable determination ratio and the importance score includes: Get the proportion of the generated judgment The corresponding first adjustable parameter and the importance score The corresponding second adjustable parameter , based on the determination ratio that can be generated , the first adjustable parameter , the importance score The second adjustable parameter , determine the monomer contribution of the non-text modal data ,in, .

6. The method according to claim 1, characterized in that Based on the individual contribution, weight coefficients are assigned to the multimodal data respectively, including: If the multimodal data includes image data, determining the image complexity of the image data; Determining a weight coefficient of the image data based on the image complexity, the monomer contribution corresponding to the image data, and a first preset normalization coefficient; A weight coefficient of the residual modal data is determined based on a single contribution of the residual modal data and a second preset normalization coefficient, wherein the multimodal data after removing the image data is used as the residual modal data.

7. The method according to claim 6, characterized in that Determining the image complexity of the image data includes: Determining an image feature vector corresponding to the image data, and determining a feature dimension of the image feature vector, a dimension value of each feature dimension, and a dimension weight of each feature dimension; The image complexity of the image data is determined based on the feature dimension, the dimension value, and the dimension weight.

8. The method according to claim 1, characterized in that Based on the weight coefficients, dynamically allocating tokens for the multimodal data includes: Divide the weight coefficient corresponding to each multimodal data by the sum of the weight coefficients of the multimodal data to obtain the proportion of the to-be-allocated tokens of the multimodal data in the overall allocated tokens, and use the ratio of each proportion as the token allocation ratio between the multimodal data; Based on the token allocation ratio, tokens are dynamically allocated for the multimodal data.

9. The method according to claim 1, characterized in that Based on the weight coefficients, dynamically allocating tokens for the multimodal data includes: Determine the problem complexity of the target problem and the total token limit ; Based on the multimodal data Weight coefficient of modal data , the complexity of the problem 、Total limit of tokens , the total number of types of multimodal data , dynamically determine the Token quota for modal data ,in, , Problem complexity The attenuation factor.

10. A multimodal fusion dynamic token allocation device, characterized in that: include: An acquisition unit, configured to acquire multimodal data to be fused required for processing a target problem, wherein, in a scenario of processing the target problem, the multimodal data have a complementary relationship; A prediction unit is used to predict the individual contribution of the multimodal data, and based on the individual contribution, respectively assign weight coefficients to the multimodal data; wherein, the multimodal data includes text modal data and non-text modal data, and the method for predicting the individual contribution of the text modal data includes: inputting the text modal data into the text problem processing large model for multiple times to process the target problem, and obtaining multiple text processing results; performing similarity matching on each text processing result and the text processing reference result corresponding to the target problem, and determining the text processing accuracy of each text processing result based on the text similarity matching result; determining the number of accurate text processing results of the text processing results whose text processing accuracy is greater than a first preset threshold, and assigning the accurate text The ratio of the number of processing results to the total number of text processing results is used as the accurate result hit rate of the text modal data; based on the accurate result hit rate, the individual contribution of the text modal data is determined; the method for predicting the individual contribution of the non-text modal data includes: using the conversion evaluation large model to judge multiple times whether the non-text modal data can be generated based on the text modal data, and determining the proportion of generable judgments based on the multiple judgment results; determining the accurate result hit rate of processing when the target problem is processed based on the non-text modal data alone, and determining the importance score of the non-text modal data based on the accurate result hit rate; determining the individual contribution of the non-text modal data based on the proportion of generable judgments and the importance score; A dynamic allocation unit is used to dynamically allocate tokens to the multimodal data based on the weight coefficients.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images