Data decision-making method and system based on multi-modal large model analysis

Abstract

The invention provides a data decision-making method and system based on multi-modal large model analysis, and the method comprises the steps: collecting multi-modal data, carrying out the preprocessing, and carrying out the multi-modal feature extraction and fusion; performing feature splicing and weighted combination according to the multi-modal features to generate initial situation features, and performing feature optimization on the initial situation features to obtain final situation features representing dynamic situation information under the current task and scene; performing specialized optimization on each modal feature by using the final situation feature to obtain optimized modal features, and splicing the optimized modal features into an optimized joint feature vector; and aligning the final modal specialization optimization features to generate a fused global feature. When specific modal analysis enters, global features are optimized according to specific tasks, and a decision result is generated and explained. According to the method, the higher context sensing capability, the more efficient modal fusion method and the more accurate task decision-making capability can be realized in the multi-modal data decision-making.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal large models, and in particular relates to a data decision method and system based on multimodal large model analysis. Background Art

[0002] With the rapid development of artificial intelligence and big data technology, multimodal data analysis has gradually become an important direction to promote the development of intelligent decision-making systems. Multimodal data comes from different data modalities, such as text, images, voice, time series, etc., and is characterized by rich information and diversity. By integrating multimodal data, complex problems can be understood more comprehensively, thereby improving the scientificity and accuracy of decision-making. Especially in the field of machine learning integrated models and classification decision-making, the use of multimodal data can significantly improve the prediction performance and classification accuracy of the model. However, the current multimodal data decision-making methods still face many challenges: Current multimodal data often contains some redundant information (such as duplication of text descriptions and content presented in images), and even data conflicts between different modalities (such as inconsistencies between sensor data and image information). Existing methods usually use simple splicing or alignment when fusion data, which fails to effectively filter or correct these redundant and conflicting information, resulting in limited generalization ability of decision models.

[0003] At the same time, existing multimodal decision models have a weak ability to understand contextual situations, and it is difficult to dynamically adjust modal weights according to specific scenarios or tasks. For example, in medical diagnosis, different diseases may require different modal features to be focused on (e.g., heart disease focuses more on time series modalities, while lung diseases rely more on imaging modalities). In existing technologies, large models often treat all modalities equally and fail to fully utilize contextual information to improve the targeting of decisions. In particular, in machine learning integrated models and classification decision tasks, there is a lack of dynamic adjustment capabilities for specific scenarios.

[0004] Many large multimodal models are designed for general scenarios and are difficult to adapt to task requirements in specific fields. This general design leads to insufficient decision-making efficiency and accuracy of the model in specific tasks. For example, when faced with complex tasks, ensemble classifiers may not be able to effectively fuse information from different modalities, resulting in reduced classification performance. In addition, semantic alignment between different modalities often relies on a unified feature space, ignoring the impact of the semantic characteristics of specific tasks on decision-making.

[0005] Current multimodal large models usually need to process a large amount of data features, and the current technology lacks a dynamic clipping mechanism for non-important information during the modal fusion process, resulting in high system computational complexity and inability to meet real-time decision-making needs. For example, in autonomous driving scenarios, massive amounts of data from lidar, cameras, and sensors need to be analyzed in real time, and the efficiency of existing technologies is not enough to meet this requirement. Especially in integrated learning models, data from different modalities may bring too much redundant information, affecting the efficiency of real-time decision-making.

[0006] Based on the above problems, the current multimodal decision-making technology has obvious performance bottlenecks when facing complex tasks. In order to overcome these shortcomings, an innovative system and method is needed to achieve efficient analysis and optimization of multimodal data, so as to give full play to the potential of large models in machine learning integration and classification decision-making. Summary of the invention

[0007] The purpose of the present invention is to propose a data decision-making method and system based on multimodal large model analysis, which can achieve higher context perception ability, more efficient modal fusion method and more accurate task decision-making ability in multimodal data decision-making, thereby breaking through the bottleneck of existing technology and being widely applicable to complex scenarios such as medical care, transportation, and agriculture.

[0008] In order to achieve the above object, a first aspect of the present invention provides a data decision method based on multimodal large model analysis, the method comprising: S1, collecting multimodal data and preprocessing it, extracting and fusing modal features of the preprocessed data to obtain multimodal features fused by splicing operations; wherein the multimodal features include text features, image features and time series features; S2, performing feature concatenation and weighted combination on the multimodal features to generate initial context features, and performing feature optimization on the initial context features to obtain final context features, representing dynamic context information under the current task and scenario; S3. Using the final context features, each modality feature is optimized separately, redundant information is removed and task-related characteristics are strengthened, and the final modality-specific optimization features are generated. The modality-specific optimization features include optimized specialized text features, optimized specialized image features, and optimized specialized time series features. The specialized optimization includes: Generate dynamic weights of modal specialization based on the final situational characteristics; Use the modality-specific dynamic weights to perform specialized optimization on the multimodal features respectively, and generate optimized modality features, including optimized text features, image features, and time series features; The modal interaction enhancement module is introduced to guide the high-order information interaction between modalities through the final context features to obtain the interaction features; The interactive features are respectively fused with the optimized text features, image features and time series features to generate the final modality-specific optimized features, including optimized specialized text features, optimized specialized image features and optimized specialized time series features; S4, aligning the final modality-specific optimization features to generate fused global features; S5. When a specific task of multimodal analysis comes in, optimize the global features according to the specific task, generate decision results and provide explanations, including: According to the needs of the current task, the global features are dynamically weighted to strengthen the task-related information while suppressing redundant features; If the task has a multi-objective scenario, a collaborative optimization module is designed to model the sharing relationship between subtasks through the task collaboration matrix of the collaborative optimization module, and combine the specialized features of each subtask into the final task specialized features; The task-specific features after task optimization are used to generate the final output decision result through the task-specific decision network.

[0009] Furthermore, multimodal data is represented as , where each data point Contains three modes: text mode , Image Modality and time series modality ; The pre-processing comprises: For text modal, set the text modal Converted into word vector representation matrix ,in is the number of words in the sentence, is the embedding dimension of each word; Designing a dynamic semantic aggregation module based on contextual attention , weight the word vectors and generate text feature vectors : ; in, is a task-related semantic query vector used to guide the attention mechanism. The attention weight is calculated by the following formula: ; in represents a semantic query, Indicates word embeddings, It is The attention weight of each word; The dynamic semantic aggregation module By dynamically adjusting the task-related semantic query vector Capture the most relevant task-related information in context, is the final text feature, is the embedding dimension of the final text feature; For the image modality, a layered convolutional network is used Extracting image features , add the frequency domain disturbance suppression term to the last layer of the convolutional network to suppress high-frequency noise by analyzing the frequency domain information of the image: ; in, is an image The frequency domain representation of is the regularization coefficient, which is used to balance the influence of frequency domain suppression on features and enhance the stability and effectiveness of image features by reducing high-frequency noise. is the final image feature, is the embedding dimension of the final image feature; For time series modality, design a multi-scale trend capture module , the multi-scale trend capture module First, decompose the time series into long-term trends and short-term fluctuations: ; in, Represents the long-term trend part, Represents the short-period fluctuation part; Use smooth convolution to extract Global features and multi-scale window convolution extraction The local features of the two parts are weighted fused to generate the final time series feature vector : ; in, is the weight of the trend feature, which is dynamically learned by minimizing the squared error of the local residual, , is the embedding dimension of the final time series features.

[0010] Furthermore, the joint feature vector is constructed as follows: The three modal features extracted , and Normalize them so that they have consistent distribution in the same feature space: ; in, and Mode The mean and standard deviation of The normalized features are fused into a joint feature vector through concatenation. : ; in, represents vector concatenation, , .

[0011] Furthermore, the S2 specifically includes: Design a task-aware self-attention mechanism, generate an attention weight vector, and dynamically adjust the modal weight, expressed as: ; in is a trainable parameter, is the attention weight vector, which represents the dynamic weights of text, image and time series modalities respectively; The features of each modality are weighted and combined according to the generated attention weights to obtain the initial context features, which are expressed as: ; in, is the initial situation feature, are the dynamic weights for text, image, and time series modalities, respectively; According to the correlation between modalities, a modal interaction enhancement module is introduced to model the high-order relationship between modalities through tensorization to obtain enhanced situational features. The enhanced situational features are The norm constraint is used to optimize feature sparsity, remove dimensions of low importance, and generate the final contextual features.

[0012] Furthermore, the modal interaction enhancement module is calculated as follows: First, calculate the modal interaction tensor , capturing the two-way interaction between different modalities: ; in, represents the outer product operation, is a high-dimensional tensor of modal interaction information; High-dimensional tensors using modality interaction information Initial situation characteristics Enhance and add regularization terms To suppress excessive interaction complexity: ; in, is the linear mapping matrix, is the regularization coefficient, Used to introduce nonlinearity; The final situation feature is expressed as: ; in, is the sparsification constraint strength, It is the final situational feature.

[0013] Furthermore, the modality-specific dynamic weight , calculated as follows: ; in, is modal The linear mapping matrix is ​​used to convert context features into modality-specific weights. It is a nonlinear activation function used to suppress the influence of negative values. It is a sparse smoothing term, which ensures smooth adjustment of the weight matrix when the task correlation is low; The optimized modal characteristics , expressed as: ; in, is the modal feature; The generation formula of the interaction feature is: ; in, is modal The interactive enhancement feature is generated by the weighted sum of contextual features and optimized modality features, capturing the implicit association between modalities. The final modal specialization optimization feature is expressed as: ; in, It is the balance coefficient of interactive enhancement, which is used to control the degree of fusion between optimized features and interactive features.

[0014] Furthermore, the S4 specifically includes: The final modality-specific optimized features are projected into the unified semantic space to obtain the aligned features, which are expressed as: ; in, is modal The projection matrix, is the bias term, represents the aligned features; The modal complementarity regularization term is designed to encourage the complementarity between modes, which can be expressed as: ; The regularization term By constraining the feature point product between modalities, redundancy is reduced to ensure the global features after fusion It not only retains the unique information of the modality, but also effectively combines the modal synergy; represents element-wise product, is the Frobenius norm.

[0015] Furthermore, the S4 also includes introducing a context-weighted fusion strategy to dynamically adapt to current tasks and context requirements, which is expressed as: ; in, is a global feature, is modal The weight of control.

[0016] Furthermore, the global features are dynamically weighted, and the weights are determined by the final context features. Guided generation, weight calculation is directly related to global features Related, expressed as: ; in, is the final situational feature, Guaranteed weight The normalization of Specialized task optimization features By element-wise weighted calculation: ; in represents element-wise multiplication; The task-specific features The calculation is as follows: ; in, It is Specialized features of each subtask; synergy matrix It is learnable and is used to capture the implicit correlation between subtasks while ensuring the synergy of specialized features between tasks; In the process of specialized optimization, in order to avoid the concentration of features in a few dimensions, a distribution balance regularization term is designed , expressed as: ; in, is a vector of all 1s, yes Dimensions; Finally, the task-specific features after task optimization are used , through a task-specific decision network , generating the final output decision : ; in, The structure varies according to the task type.

[0017] In another aspect of the present invention, a data decision system based on multimodal large model analysis is provided, the system comprising: A multimodal data acquisition unit, used to acquire multimodal data and perform preprocessing, extract modal features from the preprocessed data and fuse them to obtain multimodal features fused by splicing operations; wherein the multimodal features include text features, image features and time series features; A multimodal feature analysis unit, used to perform feature concatenation and weighted combination on the multimodal features to generate initial context features, and perform feature optimization on the initial context features to obtain final context features, representing dynamic context information under the current task and scenario; The multimodal feature optimization unit is used to perform specialized optimization on each modal feature using the final contextual feature, remove redundant information and strengthen task-related characteristics, and generate the final modal specialized optimization features, wherein the modal specialized optimization features include optimized specialized text features, optimized specialized image features, and optimized specialized time series features; the specialized optimization includes: Generate dynamic weights of modal specialization based on the final situational characteristics; Use the modality-specific dynamic weights to perform specialized optimization on the multimodal features respectively, and generate optimized modality features, including optimized text features, image features, and time series features; The modal interaction enhancement module is introduced to guide the high-order information interaction between modalities through the final context features to obtain the interaction features; The interactive features are respectively fused with the optimized text features, image features and time series features to generate the final modality-specific optimized features, including optimized specialized text features, optimized specialized image features and optimized specialized time series features; Multimodal feature alignment unit, used to align the final modality-specific optimization features and generate fused global features; The decision generation unit is used to optimize the global features according to the specific task when a specific task of multimodal analysis comes in, generate decision results and provide explanations, including: According to the needs of the current task, the global features are dynamically weighted to strengthen the task-related information while suppressing redundant features; If the task has a multi-objective scenario, a collaborative optimization module is designed to model the sharing relationship between subtasks through the task collaboration matrix of the collaborative optimization module, and combine the specialized features of each subtask into the final task specialized features; The task-specific features after task optimization are used to generate the final output decision result through the task-specific decision network.

[0018] The beneficial technical effects of the present invention are at least as follows: (1) The present invention dynamically extracts contextual information of multimodal data through a situational modeling module to guide subsequent modal optimization and decision-making. This mechanism can adjust the modal weights according to specific scenarios, solving the problem of insufficient contextual awareness in the prior art. For example, in a traffic management scenario, the system can identify the "rush hour" scenario and give priority to real-time sensor data rather than historical data; in medical diagnosis, the situational model can automatically adjust the weights of image data and time series data according to the characteristics of different diseases to improve the accuracy of classification decisions. This mechanism is particularly suitable for machine learning integrated models and classification decision-making tasks, and improves the flexibility and pertinence of the model through dynamic perception of different scenarios.

[0019] (2) The present invention targets the characteristics of different modalities (such as the spatial characteristics of images and the semantic information of texts). The present invention uses a specialized optimization module to trim and enhance modal features, and uses a modal conflict detection mechanism to eliminate inconsistencies between modalities, thereby achieving more accurate feature fusion. This design overcomes the problem of decision failure caused by information redundancy and conflict in the prior art. In particular, in the integrated model, through efficient feature fusion and conflict processing, it avoids excessive reliance on irrelevant information, thereby improving the stability and accuracy of the model.

[0020] (3) The present invention constructs a three-layer decision framework of "modal independent reasoning-context-aware fusion-task-driven optimization", realizes the preliminary extraction of modal features in the modal independent reasoning stage, completes dynamic feature weighting in the context-aware fusion stage, and further strengthens the semantic alignment related to specific tasks in the task-driven optimization stage. This framework not only improves the task-specificity of the model, but also significantly reduces the computational complexity by dynamically cutting redundant information, thus solving the problem of insufficient computational efficiency in the prior art. Especially in ensemble learning and classification decision-making tasks, this framework can flexibly process multimodal data, enhance task relevance, optimize the model's computational process, and improve the efficiency of real-time decision-making.

[0021] (4) The present invention can achieve higher context awareness, more efficient modal fusion methods and more accurate task decision-making capabilities in multimodal data decision-making, thereby breaking through the bottleneck of existing technologies and being widely applicable to complex scenarios such as medical care, transportation, and agriculture.

[0022] In these scenarios, the innovative decision-making mechanism of the present invention can achieve accurate and efficient intelligent decision-making through machine learning integrated models and classification decision-making methods, combining the characteristics of different modal data, and provide strong technical support for practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0024] Figure 1 The present invention is a flowchart of a data decision method based on multimodal large model analysis according to an embodiment of the present invention.

[0025] Figure 2 This is a framework diagram of a data decision system based on multimodal large model analysis according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0027] like Figure 1 As shown, an embodiment of the present invention provides a data decision method based on multimodal large model analysis, the method comprising: S1. Collect multimodal data and perform preprocessing, extract and fuse modal features of the preprocessed data to obtain multimodal features fused by splicing operations; wherein the multimodal features include text features, image features and time series features.

[0028] Specifically, input multimodal dataset , where each data point Contains three modes: text mode , Image Modality and time series modality There are obvious distribution differences and heterogeneous characteristics between multimodal data. For example, the text modality has semantic dependence, the image modality has spatial characteristics, and the time series modality has temporal dependence. Therefore, feature extraction and standardization must be performed separately for each modality.

[0029] Among them, text modality feature extraction: Enter text Converted into word vector representation matrix ,in is the number of words in the sentence, is the embedding dimension of each word.

[0030] Designing a dynamic semantic aggregation module based on contextual attention , weight the word vectors and generate text feature vectors : ; in is the task-related semantic query vector used to guide the attention mechanism. The attention weight is calculated by the following formula: ; in represents a semantic query, Indicates word embeddings, It is The attention weight of each word.

[0031] By dynamically adjusting Capture the most task-relevant information in context. The final text feature.

[0032] Among them, image modality feature extraction: Using layered convolutional networks Extracting image modality features .

[0033] A frequency domain disturbance suppression term is added to the last layer of the convolutional network to suppress high-frequency noise by analyzing the frequency domain information of the image: ; in is an image The frequency domain representation of is a regularization coefficient used to balance the effect of frequency domain suppression on features. This design enhances the stability and effectiveness of image features by reducing high-frequency noise. .

[0034] Among them, time series modal feature extraction: For the input time series Designing a multi-scale trend capture module The module first decomposes the time series into long-term trends and short-term fluctuations: ; in Represents the long-term trend part, Represents the short-term fluctuation part.

[0035] Use smooth convolution to extract Global features and multi-scale window convolution extraction The two parts of the features are weighted fused to generate the final time series feature vector : ; in is the weight of the trend feature, which is dynamically learned by minimizing the squared error of the local residual. .

[0036] Furthermore, the three modal features extracted , and Normalize them so that they have consistent distribution in the same feature space: ; in and Mode The mean and standard deviation of .

[0037] The normalized features are fused into a joint feature vector through concatenation. : ; in represents vector concatenation, , .

[0038] Understandably, output joint features , which contains standardized feature representations of multimodal data as input to the subsequent situation modeling module. Through specific processing of each modality, the generated features not only have high-quality expression capabilities, but also effectively reduce noise interference and unify the feature scales of different modalities.

[0039] S2. Perform feature concatenation and weighted combination on the multimodal features to generate initial context features, and perform feature optimization on the initial context features to obtain final context features, which represent dynamic context information under the current task and scenario.

[0040] Specific, multimodal features The information distribution in the image may vary significantly depending on the task or scenario. For example, in traffic dispatch, the visual mode in the daytime Possibly better than sensor time series modality To this end, a task-aware self-attention mechanism is designed to dynamically adjust the modal weights.

[0041] First, the multimodal features are concatenated and input into the attention weight generation module: ; in is a trainable parameter, are attention weight vectors, representing the dynamic weights of text, image, and time series modalities respectively.

[0042] Furthermore, according to the generated attention weights Weighted combination of each modal feature: ; in It is a preliminary generated context feature that has combined the dynamic importance of multimodal features.

[0043] Furthermore, considering that there may be strong correlations between multimodal features (for example, the description in the text modality may be closely related to the image modality information), a modality interaction enhancement module is introduced to model the high-order relationship between modalities through tensorization: First, calculate the modal interaction tensor , capturing the two-way interaction between different modalities: ; in represents the outer product operation, is a high-dimensional tensor of modal interaction information.

[0044] Further, using Preliminary situational characteristics Enhance and add regularization terms (Frobenius norm) to suppress excessive interaction complexity: ; in is the linear mapping matrix, is the regularization coefficient, Used to introduce nonlinearity.

[0045] Furthermore, in the generated context features There may still be some noise features or information redundancy in . Therefore, a feature sparsification mechanism is designed to remove low-importance dimensions. The norm constraint is used to optimize feature sparsity, and the final situational feature is expressed as: ; in is the sparsification constraint strength, It is the optimized situational feature.

[0046] Understandable, output situational features , which represents the dynamic context information of the current task and scene, integrates the dynamic importance of multimodal features, the interaction information between modalities, and removes redundant features. This feature will be used as the input for the next step of modal specialization optimization to guide the subsequent steps to adjust the modal features more accurately.

[0047] S3. Use the final contextual features to perform specialized optimization on each modal feature, remove redundant information and strengthen task-related characteristics, and generate the final modal specialized optimization features. The modal specialized optimization features include optimized specialized text features, optimized specialized image features, and optimized specialized time series features.

[0048] Specific, according to the characteristics of the situation , generating modality-specialized dynamic weights ( ). The weight is calculated by the following formula: ; in is modal The linear mapping matrix is ​​used to convert context features into modality-specific weights. It is a nonlinear activation function used to suppress the influence of negative values. is a sparse smoothing term that ensures smooth adjustment of the weight matrix when the task correlation is low. The size and modal characteristics of The dimensions are consistent.

[0049] Use dynamic weights Modal characteristics Perform specialized optimization to generate optimized modal features : ; in Represents an element-by-element weighted operation. The optimized modal features It retains task-related features and suppresses redundant information that is irrelevant to the current task.

[0050] Furthermore, in order to enhance the synergy between modalities, a modal interaction enhancement module is introduced to Guide the high-order information interaction between modalities. The generation formula of the interaction feature is: ; in is modal The interaction-enhanced features are generated by the weighted sum of contextual features and optimized modal features, capturing the implicit associations between modalities.

[0051] The interactive features With optimization features Fusion to generate the final modal specialization optimization features : ; in It is the balance coefficient of interactive enhancement, which is used to control the degree of fusion between optimized features and interactive features.

[0052] Furthermore, the final optimized modal characteristics are output , and , and concatenate them into joint features . Joint Features It contains the modal information after removing redundancy and combines the interaction enhancement between modalities to provide high-quality input for subsequent cross-modal alignment and fusion steps.

[0053] By situational features The guided modality-specific optimization and inter-modality interaction enhancement not only improve the expressiveness of unimodal features, but also capture the collaborative information between modalities, ensuring that the final features can better serve multimodal tasks.

[0054] S4. Align the final modality-specific optimization features to generate fused global features.

[0055] Specifically, the features of different modalities may be distributed in heterogeneous semantic spaces, so they need to be aligned to a shared feature space. To this end, a context-guided feature alignment module is designed to align the modal features.

[0056] Projection into a unified semantic space: ; in, is modal The projection matrix, is the bias term, Represents the aligned features. This step achieves the semantic alignment of modal features, enabling them to have consistent expressiveness in subsequent fusion.

[0057] Furthermore, the fused global features need to dynamically adapt to the current task and context requirements. To this end, a context-weighted fusion strategy is introduced: in is modal The weight of Control. For example, Can be Calculate, where is modal The weights dynamically adjust the importance of the modality, for example, increasing the contribution of the time series modality in time-sensitive scenarios.

[0058] Furthermore, in feature fusion, there may be redundant information between modalities or the risk of over-reliance on a single modality. To this end, a modality complementarity regularization term is designed to encourage complementarity between modalities: ; This regularization term reduces redundancy by constraining the feature dot product between modalities, ensuring that the fused global features It not only retains the unique information of the modality, but also effectively combines the modal synergy. represents element-wise product, is the Frobenius norm.

[0059] Understandably, output global features It integrates the shared information of text, image and time series modalities, and achieves a balance between modalities through dynamic weights and regularization control. As the input for subsequent task-driven optimization decisions, it supports decision-making needs in complex scenarios.

[0060] S5. When a specific modal analysis is entered, the global features are optimized according to the specific task, and the decision results are generated and explained.

[0061] Specifically, according to the requirements of the current task, the global features Dynamically weighted, strengthen task-related information, and suppress redundant features. Design a task relevance optimization module with core weights By situational characteristics Guided generation, the weight calculation can be directly compared with The associated form is as follows: ; in, It is a situational feature. Guaranteed weight Here, the reliance on complex matrix mapping is reduced, and the dynamic relationship between tasks and global features is directly captured through context features.

[0062] Furthermore, specialized task optimization features It can be calculated by element-by-element weighting: ; in represents element-wise multiplication. The features generated in this way Task-related features are retained and unnecessary noise is excluded.

[0063] Furthermore, if the task is a multi-objective scenario (for example, classification and prediction tasks coexist), it is necessary to further coordinate the specialized features of different tasks. Design a collaborative optimization module, through the task collaboration matrix Model the shared relationship between subtasks and combine the specialized features of each subtask into the final task-specific features : ; in It is The specialized features of each subtask. It is learnable and is used to capture the implicit correlation between subtasks while ensuring the synergy of specialized features between tasks.

[0064] Furthermore, in the process of specialized optimization, in order to avoid the features being concentrated in a few dimensions, a distribution balancing regularization term is designed : ; in is a vector of all 1s, yes The regularization term ensures that the task weights The uniform distribution of features can avoid excessive reliance on a single dimension and improve the robustness and generalization ability of features.

[0065] Furthermore, using the specialized features after task optimization , through a task-specific decision network , generating the final output : ; in The structure of can vary according to the task type, for example, a fully connected layer is used in a classification task, and a regression network is used in a prediction task.

[0066] like Figure 2 As shown, in another embodiment of the present invention, a data decision system based on multimodal large model analysis is provided, and the system includes: The multimodal data acquisition unit 101 is used to collect and preprocess multimodal data, extract and fuse modal features of the preprocessed data, and obtain multimodal features fused by splicing operations; wherein the multimodal features include text features, image features, and time series features; The multimodal feature analysis unit 102 is used to perform feature concatenation and weighted combination on the multimodal features to generate initial context features, and perform feature optimization on the initial context features to obtain final context features, which represent dynamic context information under the current task and scenario; The multimodal feature optimization unit 103 is used to perform specialized optimization on each modal feature using the final context feature, remove redundant information and strengthen task-related characteristics, and generate the final modal specialized optimization feature, wherein the modal specialized optimization feature includes optimized specialized text features, optimized specialized image features, and optimized specialized time series features; the specialized optimization includes: Generate dynamic weights of modal specialization based on the final situational characteristics; Use the modality-specific dynamic weights to perform specialized optimization on the multimodal features respectively, and generate optimized modality features, including optimized text features, image features, and time series features; The modal interaction enhancement module is introduced to guide the high-order information interaction between modalities through the final context features to obtain the interaction features; The interactive features are respectively fused with the optimized text features, image features and time series features to generate the final modality-specific optimized features, including optimized specialized text features, optimized specialized image features and optimized specialized time series features; A multi-modal feature alignment unit 104, used to align the final modality-specific optimization features to generate fused global features; The decision generation unit 105 is used to optimize the global features according to the specific task when a specific task of multimodal analysis is entered, generate a decision result and provide an explanation, including: According to the needs of the current task, the global features are dynamically weighted to strengthen the task-related information while suppressing redundant features; If the task has a multi-objective scenario, a collaborative optimization module is designed to model the sharing relationship between subtasks through the task collaboration matrix of the collaborative optimization module, and combine the specialized features of each subtask into the final task specialized features; The task-specific features after task optimization are used to generate the final output decision result through the task-specific decision network.

[0067] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0068] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the parameter operation method provided in any embodiment of the present invention, and will not be repeated here.

[0069] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0070] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0071] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0072] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data decision method based on multimodal large model analysis, characterized in that: The method comprises: S1, collecting multimodal data and preprocessing it, extracting and fusing modal features of the preprocessed data to obtain multimodal features fused by splicing operations; wherein the multimodal features include text features, image features and time series features; S2, performing feature concatenation and weighted combination on the multimodal features to generate initial context features, and performing feature optimization on the initial context features to obtain final context features, representing dynamic context information under the current task and scenario; S3. Using the final context features, each modality feature is optimized separately, redundant information is removed and task-related characteristics are strengthened, and the final modality-specific optimization features are generated. The modality-specific optimization features include optimized specialized text features, optimized specialized image features, and optimized specialized time series features. The specialized optimization includes: Generate dynamic weights of modal specialization based on the final situational characteristics; Use the modality-specific dynamic weights to perform specialized optimization on the multimodal features respectively, and generate optimized modality features, including optimized text features, image features, and time series features; The modal interaction enhancement module is introduced to guide the high-order information interaction between modalities through the final context features to obtain the interaction features; The interactive features are respectively fused with the optimized text features, image features and time series features to generate the final modality-specific optimized features, including optimized specialized text features, optimized specialized image features and optimized specialized time series features; S4, aligning the final modality-specific optimization features to generate fused global features; S5. When a specific task of multimodal analysis comes in, optimize the global features according to the specific task, generate decision results and provide explanations, including: According to the needs of the current task, the global features are dynamically weighted to strengthen the task-related information while suppressing redundant features; If the task has a multi-objective scenario, a collaborative optimization module is designed to model the sharing relationship between subtasks through the task collaboration matrix of the collaborative optimization module, and combine the specialized features of each subtask into the final task specialized features; The task-specific features after task optimization are used to generate the final output decision result through the task-specific decision network.

2. The data decision method based on multimodal large model analysis according to claim 1 is characterized in that: Multimodal data is represented as , where each data point Contains three modes: text mode , Image Modality and time series modality ; The pre-processing comprises: For text mode, the text mode of the i-th data point Converted into word vector representation matrix ,in is the number of words in the sentence, is the embedding dimension of each word, and R is the real number space; Designing a dynamic semantic aggregation module based on contextual attention , weight the word vectors and generate text feature vectors : ； in, is the word vector representation matrix, is a task-related semantic query vector used to guide the attention mechanism. The attention weight is calculated by the following formula: ； in represents a semantic query, Indicates word embedding, T is the text modality, It is The attention weight of each word, is the normalization function, is the transpose of the semantic query vector; The dynamic semantic aggregation module By dynamically adjusting the task-related semantic query vector Capture the most relevant task-related information in context, is the final text feature, is the embedding dimension of the final text feature; For the image modality, a layered convolutional network is used Extracting image features , add the frequency domain disturbance suppression term to the last layer of the convolutional network to suppress high-frequency noise by analyzing the frequency domain information of the image: ； in, is an image The frequency domain representation of is the regularization coefficient, which is used to balance the influence of frequency domain suppression on features and enhance the stability and effectiveness of image features by reducing high-frequency noise. is the final image feature, is the embedding dimension of the final image feature; For time series modality, design a multi-scale trend capture module , the multi-scale trend capture module First, decompose the time series into long-term trends and short-term fluctuations: ； in, is the ith time series data point, Represents the long-term trend part, Represents the short-period fluctuation part; Use smooth convolution to extract Global features and multi-scale window convolution extraction The local features of the two parts are weighted fused to generate the final time series feature vector : ； in, is the weight of the trend feature, which is dynamically learned by minimizing the squared error of the local residual, , is the embedding dimension of the final time series feature, is the global eigenvector of the long-term trend part, It is the local eigenvector of the short-period fluctuation part.

3. The data decision method based on multimodal large model analysis according to claim 2 is characterized in that: The multimodal features are constructed as follows: Extracted text features , Image features and time series features Normalize them so that they have consistent distribution in the same feature space: ； in, is the normalized eigenvector of mode m, is the eigenvector of the original mode m, and The modal The mean and standard deviation of The normalized features are fused into multimodal features through splicing operations : ； in, represents vector concatenation, , , is the dimension of the final multimodal feature, is the embedding dimension of the time series feature vector.

4. The data decision method based on multimodal large model analysis according to claim 3 is characterized in that: The S2 specifically includes: Design a task-aware self-attention mechanism, generate an attention weight vector, and dynamically adjust the modal weight, expressed as: ； in, is a trainable parameter, is the attention weight vector, which represents the dynamic weights of text, image and time series modalities respectively, is the normalization function; The features of each modality are weighted and combined according to the generated attention weights to obtain the initial context features, which are expressed as: ； in, is the initial situation feature, are the dynamic weights for text, image, and time series modalities, respectively; According to the correlation between modalities, a modal interaction enhancement module is introduced to model the high-order relationship between modalities through tensorization to obtain enhanced situational features. The enhanced situational features are The norm constraint is used to optimize feature sparsity, remove dimensions of low importance, and generate the final contextual features.

5. The data decision method based on multimodal large model analysis according to claim 4 is characterized in that: The modal interaction enhancement module is calculated as follows: First, calculate the modal interaction tensor , capturing the two-way interaction between different modalities: ； in, represents the outer product operation, is a high-dimensional tensor of modal interaction information; High-dimensional tensors using modality interaction information Initial situation characteristics Enhance and add regularization terms To suppress excessive interaction complexity: ； in, is the enhanced situational feature, is the linear mapping matrix, is the regularization coefficient; Represents a nonlinear activation function, which is used to introduce nonlinearity; The final situation feature is expressed as: ； in, is the sparsification constraint strength, is the final situational feature, is a coefficient that adjusts the strength of the interaction between modalities and is used to control the difference between the initial context features and the interaction tensor.

6. The data decision method based on multimodal large model analysis according to claim 5 is characterized in that: Dynamic weights of the modality specialization , calculated as follows: ； in, is modal The linear mapping matrix is ​​used to convert context features into modality-specific weights. It is a nonlinear activation function used to suppress the influence of negative values. It is a sparse smoothing term, which ensures smooth adjustment of the weight matrix when the task correlation is low; The optimized modal characteristics , expressed as: ； in, is the modal feature; The generation formula of the interaction feature is: ； in, is modal The interactive enhancement feature is generated by the weighted sum of the contextual features and the optimized modal features, capturing the implicit association between the modalities. is the optimized text modality feature, is the optimized image modality feature, is the optimized time series modal feature; The final modal-specific optimization feature , expressed as: ； in, It is the balance coefficient of interactive enhancement, which is used to control the degree of fusion between optimized features and interactive features.

7. The data decision method based on multimodal large model analysis according to claim 6 is characterized in that: The S4 specifically includes: The final modality-specific optimized features are projected into the unified semantic space to obtain the aligned features, which are expressed as: ； in, is modal The projection matrix, is the bias term, represents the aligned features; Designing the modal complementarity regularization term , encourages complementarity between modes, expressed as: ； Among them, m is mode m, n is mode n, and the regularization term By constraining the feature point product between modalities, redundancy is reduced to ensure the global features after fusion It not only retains the unique information of the modality, but also effectively combines the modal synergy; represents element-wise product, is the Frobenius norm.

8. The data decision method based on multimodal large model analysis according to claim 7 is characterized in that: The S4 also includes introducing a context-weighted fusion strategy to dynamically adapt to current tasks and context requirements, which is expressed as: ； in, is a global feature, is modal The weight of control.

9. The data decision method based on multimodal large model analysis according to claim 7 is characterized in that: The global features are dynamically weighted, and the weights are determined by the final situational features. Guided generation, weight calculation is directly related to global features Related, expressed as: ； in, is the final situational feature, is the weight of dynamic weighting, Guaranteed weight The normalization of Specialized task optimization features By element-wise weighted calculation: ； in represents element-wise multiplication; The task-specific features The calculation is as follows: ； in, It is The specialized features of each subtask, For the The specialized features of each subtask, For the The specialized features of each subtask, For the Specialized features of each subtask; synergy matrix It is learnable and is used to capture the implicit correlation between subtasks while ensuring the synergy of specialized features between tasks; In the process of specialized optimization, in order to avoid the concentration of features in a few dimensions, a distribution balance regularization term is designed , expressed as: ； in, is a vector of all 1s, yes Dimensions; Finally, the task-specific features after task optimization are used , through a task-specific decision network , generating the final output decision : ； in, The structure varies according to the task type.

10. A data decision system based on multimodal large model analysis, characterized in that: The system comprises: A multimodal data acquisition unit, used to acquire multimodal data and perform preprocessing, extract modal features from the preprocessed data and fuse them to obtain multimodal features fused by splicing operations; wherein the multimodal features include text features, image features and time series features; A multimodal feature analysis unit, used to perform feature concatenation and weighted combination on the multimodal features to generate initial context features, and perform feature optimization on the initial context features to obtain final context features, representing dynamic context information under the current task and scenario; The multimodal feature optimization unit is used to perform specialized optimization on each modal feature using the final contextual feature, remove redundant information and strengthen task-related characteristics, and generate the final modal specialized optimization features, wherein the modal specialized optimization features include optimized specialized text features, optimized specialized image features, and optimized specialized time series features; the specialized optimization includes: Generate dynamic weights of modal specialization based on the final situational characteristics; Use the modality-specific dynamic weights to perform specialized optimization on the multimodal features respectively, and generate optimized modality features, including optimized text features, image features, and time series features; The modal interaction enhancement module is introduced to guide the high-order information interaction between modalities through the final context features to obtain the interaction features; The interactive features are respectively fused with the optimized text features, image features and time series features to generate the final modality-specific optimized features, including optimized specialized text features, optimized specialized image features and optimized specialized time series features; Multimodal feature alignment unit, used to align the final modality-specific optimization features and generate fused global features; The decision generation unit is used to optimize the global features according to the specific task when a specific task of multimodal analysis comes in, generate decision results and provide explanations, including: According to the needs of the current task, the global features are dynamically weighted to strengthen the task-related information while suppressing redundant features; If the task has a multi-objective scenario, a collaborative optimization module is designed to model the sharing relationship between subtasks through the task collaboration matrix of the collaborative optimization module, and combine the specialized features of each subtask into the final task specialized features; The task-specific features after task optimization are used to generate the final output decision result through the task-specific decision network.

Images