A multi-modal context reasoning method and device for complex task scenarios

Abstract

The application discloses a kind of multi-modal context reasoning methods and devices for complex task scene, it is related to artificial intelligence technical field, the method includes: obtaining the original data of target business scene containing multiple modes, and the original data is preprocessed;The original data obtained after preprocessing is obtained, and the data of various modes is uniformly mapped to the preset semantic vector space, and the semantic association between the data of various modes is established;Acquire target task, and determine the data weight of the data of each mode and at least one candidate preset reasoning path according to the task context semantics of target task;According to target task, the data of various modes is inferred using preset reasoning path, to obtain reasoning result, and evaluate reasoning result according to the consistency evaluation model constructed, determine the final reasoning output result of target task from reasoning result.The application significantly improves the inference robustness and accuracy in complex, variable scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to a multimodal contextual reasoning method and apparatus for complex task scenarios. Background Technology

[0002] With the acceleration of digitalization, massive amounts of heterogeneous data have been generated across various industries. In complex business scenarios, such as clause matching, financial risk control auditing, medical diagnostic assistance, and industrial fault analysis, key information often does not exist in a single modality, but rather in a complex mix of multiple forms, including text, tables, images, charts, and structured database records. For example, a "corporate product innovation proposal" may contain descriptive text information (text modality), a table specifying funding amounts and application conditions (table modality), a diagram illustrating the application process (image modality), and structured data about the applicant company, such as its finances, R&D, and geographic location (structured data modality).

[0003] Currently, artificial intelligence technology, especially deep learning-based large language models, has made significant progress in understanding and generating single modalities (especially text). However, when faced with the complex multimodal task scenarios mentioned above, single-modal large language models struggle to achieve comprehensive understanding.

[0004] Therefore, there is an urgent need for a multimodal context-adaptive reasoning method for complex scenarios to improve the accuracy, interpretability, and context-adaptive capability of intelligent understanding and matching of multi-source information. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a multimodal contextual reasoning method and apparatus for complex task scenarios, thereby solving the problem that the adaptive reasoning capability of a single-modal large language model for multimodal contextual scenarios cannot meet business needs.

[0006] According to a first aspect, embodiments of the present invention provide a multimodal contextual reasoning method for complex task scenarios, the method comprising:

[0007] Acquire raw data containing multiple modalities from the target business scenario and preprocess the raw data;

[0008] The process involves acquiring the preprocessed raw data and mapping the data from various modalities to a predefined semantic vector space, establishing semantic relationships between the data from different modalities, and using a trained cross-modal semantic coding model for unified mapping. This model also establishes semantic relationships between the data from different modalities within the semantic vector space. The cross-modal semantic coding model includes multiple parallel modal data encoders and a cross-modal fusion encoder that is connected to all the modal data encoders.

[0009] Obtain the target task, and determine the data weight of each modality and at least one candidate preset inference path based on the task context semantics of the target task;

[0010] Based on the target task, the candidate preset reasoning paths are used to reason about data of various modalities to obtain the reasoning result of each preset reasoning path. The reasoning result is evaluated according to the constructed consistency evaluation model, and the final reasoning output result of the target task is determined from the reasoning result. Each reasoning result includes the reasoning result and the original data referenced by the reasoning conclusion. The consistency evaluation model is used to evaluate the reasoning result from the perspectives of condition satisfaction, data consistency, and the rationality of reasoning logic.

[0011] In conjunction with the first aspect, in the first embodiment of the first aspect, the cross-modal semantic coding model includes:

[0012] Text data encoder, tabular data encoder, image data encoder, structured data encoder, and cross-modal fusion encoder;

[0013] The text data encoder uses a large language model as its backbone network.

[0014] The table data encoder includes a content embedding layer, a row embedding layer, a column embedding layer, a vector fusion layer, and a Transformer encoder. The content embedding layer projects the content of each cell in the preprocessed table data to obtain a content feature vector. The row embedding layer projects the content of each row in the preprocessed table data to obtain a row feature vector. The column embedding layer projects the content of each column in the preprocessed table data to obtain a column feature vector. The vector fusion layer fuses the content feature vector, row feature vector, and column feature vector to obtain a table feature vector.

[0015] The image data encoder uses an image processing network model as its backbone network.

[0016] The structured data encoder is a multilayer perceptron composed of fully connected layers;

[0017] The cross-modal fusion encoder includes a cross-modal attention layer and a Transformer module formed by multiple stacked layers.

[0018] In conjunction with the first embodiment of the first aspect, in the second embodiment of the first aspect, the step of acquiring the preprocessed raw data, uniformly mapping the data of various modalities to a preset semantic vector space, and establishing semantic relationships between the data of various modalities specifically includes:

[0019] The preprocessed text data is input into the text data encoder to obtain the text feature vector and the context vector of the text feature vector output by the text data encoder.

[0020] The preprocessed tabular data is input into the tabular data encoder to obtain the overall feature vector output by the tabular data encoder;

[0021] The preprocessed image data is input into the image data encoder to obtain the local feature vector output by the image data encoder and the global feature vector concatenated from all the local feature vectors.

[0022] The preprocessed structured business data is input into the structured data encoder to obtain the data feature vector output by the structured data encoder.

[0023] Project text feature vectors, context vectors, global feature vectors, local feature vectors, global feature vectors, and data feature vectors into the semantic vector space;

[0024] The projected feature vectors of various modalities are input into the cross-modal fusion encoder, and the cross-modal fusion encoder is used to perform semantic alignment of the feature vectors of various modalities, thereby establishing semantic associations between the feature vectors of various modalities.

[0025] In conjunction with the first aspect, in the third embodiment of the first aspect, the step of acquiring the target task and determining the data weights of each modality and at least one candidate preset inference path based on the task context semantics of the target task specifically includes:

[0026] Obtain historical tasks and extract historical task semantics from them;

[0027] Based on the semantic content of the historical task semantics, the historical task semantics are classified to obtain historical classification results, and an output head is trained for each historical classification result to obtain at least one preset reasoning path corresponding to each historical classification result.

[0028] Obtain the target task and extract the task context semantics from the target task. Based on the semantic similarity between the context semantics and the historical classification results, match the preset reasoning path corresponding to the historical classification result with the highest semantic similarity for the target task, and obtain at least one candidate preset reasoning path.

[0029] Feature analysis is performed on the task context vector to obtain the data weights for each modality;

[0030] The multimodal data is weighted and fused according to the data weights to obtain context fusion features.

[0031] In conjunction with the third embodiment of the first aspect, in the fourth embodiment of the first aspect, the step of reasoning about data of various modalities using candidate preset inference paths according to the target task, obtaining the inference result of each preset inference path, evaluating the inference result according to the constructed consistency evaluation model, and determining the final inference output result of the target task from the inference results specifically includes:

[0032] The context fusion features are input into each candidate preset inference path to obtain the inference result output by each preset inference path;

[0033] The context fusion features and each inference result are input into the consistency evaluation model to obtain the consistency score of each inference result output by the consistency evaluation model.

[0034] The inference result with the highest consistency score is used as the final inference output.

[0035] In conjunction with the first aspect, in the fifth embodiment of the first aspect, the step of acquiring raw data containing multiple modalities of the target business scenario and preprocessing the raw data specifically includes:

[0036] Identify at least one target business scenario and obtain the raw data for each target business scenario;

[0037] For each modality in the original data, a data processing template is matched, and the original data of the corresponding modality is processed using the matching data processing template to obtain the preprocessed original data.

[0038] In conjunction with the fifth embodiment of the first aspect, in the sixth embodiment of the first aspect, the step of matching a data processing template to the data of each modality in the original data, and processing the original data of the corresponding modality using the matching data processing template to obtain preprocessed original data, specifically includes:

[0039] Match the first processing template to the text data in the original data, and process the text data using the first processing template;

[0040] Match the second processing template to the tabular data in the original data, and use the second processing template to process the tabular data;

[0041] A third processing template is matched to the image data in the original data, and the image data is processed using the third processing template;

[0042] The fourth processing template is matched to the structured business data in the original data, and the structured business data is processed using the fourth processing template.

[0043] According to a second aspect, embodiments of the present invention also provide a multimodal context reasoning device for complex task scenarios, the device comprising:

[0044] The data acquisition module is used to acquire raw data containing multiple modalities from the target business scenario and to preprocess the raw data.

[0045] The semantic association module is used to acquire the preprocessed raw data and map the data of various modalities to a preset semantic vector space, and establish semantic associations between the data of various modalities. The module uses a trained cross-modal semantic coding model to perform unified mapping and establish semantic associations between the data of various modalities in this semantic vector space. The cross-modal semantic coding model includes multiple parallel modal data encoders and a cross-modal fusion encoder that is connected to all modal data encoders.

[0046] The path selection module is used to obtain the target task and determine the data weight of each modality and at least one candidate preset inference path based on the task context semantics of the target task.

[0047] The data reasoning module is used to reason about data of various modalities based on the target task and using candidate preset reasoning paths to obtain the reasoning result of each preset reasoning path. The reasoning result is evaluated according to the constructed consistency evaluation model, and the final reasoning output result of the target task is determined from the reasoning result. Each reasoning result includes the reasoning result and the original data referenced by the reasoning conclusion. The consistency evaluation model is used to evaluate the reasoning result from the perspectives of condition satisfaction, data consistency, and the rationality of reasoning logic.

[0048] According to a third aspect, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal context reasoning method for complex task scenarios as described above.

[0049] According to a fourth aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal context reasoning method for complex task scenarios as described above.

[0050] This invention presents a multimodal contextual reasoning method and apparatus for complex task scenarios. It maps feature information from different modalities to a shared, high-dimensional semantic vector space using a cross-modal semantic encoding model based on the Transformer architecture. Within this semantic vector space, precise semantic relationships are established between them, achieving deep fusion and precise alignment of multimodal data such as text, tables, images, and structured business data within a unified semantic space, thus enabling better reasoning in complex scenarios. By flexibly adjusting the preset reasoning paths and data weights of candidates based on the task context semantics of the target task, the reasoning process is no longer static and fixed, but rather dynamically adjusts information utilization strategies based on the specific target task, significantly improving the robustness and accuracy of reasoning in complex and variable scenarios, thereby enhancing contextual reasoning capabilities. Through the generation and consistency evaluation of multiple candidate reasoning paths, not only is the optimal solution found, but clear problem-solving steps are also provided, making the model's decision-making process transparent and easy for users to understand and verify.

[0051] This invention provides a complete and systematic context-adaptive reasoning solution for complex scenarios, from data preprocessing to final output and optimization. It forms a closed loop, is easy to integrate into actual business systems, and has high practical value and broad prospects for industrial application. Attached Figure Description

[0052] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0053] Figure 1 This paper illustrates one of the flowcharts of the multimodal context reasoning method for complex task scenarios provided by the present invention.

[0054] Figure 2 The second flowchart of the multimodal context reasoning method for complex task scenarios provided by the present invention is shown.

[0055] Figure 3 A schematic diagram of the structure of the multimodal context reasoning device for complex task scenarios provided by the present invention is shown.

[0056] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] With the acceleration of digitalization, massive amounts of heterogeneous data have been generated across various industries. In complex business scenarios, such as clause matching, financial risk control auditing, medical diagnostic assistance, and industrial fault analysis, key information often does not exist in a single modality, but rather in a complex mix of multiple forms, including text, tables, images, charts, and structured database records. For example, a "corporate product innovation proposal" may contain descriptive text information (text modality), a table specifying funding amounts and application conditions (table modality), a diagram illustrating the application process (image modality), and structured data about the applicant company, such as its finances, R&D, and geographic location (structured data modality).

[0059] Currently, artificial intelligence technology, especially deep learning-based large language models, has made significant progress in understanding and generating single modalities (especially text). However, when faced with the complex multimodal task scenarios mentioned above, single-modal large language models struggle to achieve comprehensive understanding, exhibiting the following limitations:

[0060] Weak context dependence: Existing single-modal large language models, including some multimodal large language models, usually focus on shallow semantic associations. They lack the ability to effectively model the deep, implicit logical relationships between text information and external tables and image data. The models have difficulty understanding the implicit logic between text information and external data, resulting in information fragmentation.

[0061] Insufficient modality fusion: Common multimodal fusion methods such as early fusion (feature splicing) or late fusion (merging decision results) are relatively simple and crude, failing to fully consider the differences in importance of different modalities in different contexts, resulting in multimodal data not being effectively fused;

[0062] Unexplainable reasoning: Most deep learning models are "black boxes" with a lack of transparency in their reasoning process. In some high-risk decision-making scenarios, users not only need to know "what" the result is, but also "why" that the result is. Existing single-modal large language models are unable to provide clear and traceable reasoning paths, making it difficult to trust and verify the decision results.

[0063] Insufficient domain adaptability: When models are transferred to specific domains, performance often degrades due to differences in domain terminology and business logic. Existing single-modal large language models typically rely on large amounts of domain data for retraining or fine-tuning, which is costly and lacks flexibility. In complex contexts, the models cannot dynamically adjust their inference strategies based on the context, nor can they make rapid and adaptive adjustments based on contextual feedback during task execution.

[0064] In summary, there is an urgent need for a multimodal context-adaptive reasoning method for complex scenarios to improve the accuracy, interpretability, and context-adaptive capability of intelligent understanding and matching of multi-source information.

[0065] Due to the aforementioned technical problems, this invention provides a multimodal contextual reasoning method for complex task scenarios. This method aims to dynamically adjust information utilization strategies, significantly improving the robustness and accuracy of reasoning in complex and variable scenarios. This multimodal contextual reasoning method for complex task scenarios can be used in electronic devices, including but not limited to computers and mobile terminals. Figure 1 This is a flowchart illustrating a multimodal contextual reasoning method for complex task scenarios according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method may include the following steps:

[0066] S101. Obtain raw data containing multiple modalities of the target business scenario, thereby obtaining multimodal data, and preprocess the raw data. The target business scenario refers to a business scenario highly relevant to the user's current business scenario, a business scenario the user expects to apply, or a business scenario with potential applications. The target business scenario can characterize the complex business scenarios required by the user. The raw data containing multiple modalities includes at least: text data such as specification documents, product manuals, announcements, implementation details, etc.; tabular data such as indicator tables, classification tables, etc.; image data such as flowcharts, system architecture diagrams, regional indicator maps, etc.; and structured business data such as enterprise information, enterprise profiles, financial indicators, R&D investment, and regional information.

[0067] It should be noted that, depending on the user's needs, the specific business scenario corresponding to the target business scenario can be configured by the user. At the same time, the target business scenario can include at least one business scenario, that is, there can be multiple target business scenarios.

[0068] Raw data containing multiple modalities. Multimodal data can be pre-stored in electronic devices or acquired by electronic devices from external sources. For example, electronic devices may acquire data from various external channels.

[0069] There are no restrictions on the specific form in which the raw data is acquired, as long as the electronic device can acquire the raw data.

[0070] In this embodiment of the invention, the purpose of preprocessing is to transform the acquired raw data into standardized, structured features that the model can process.

[0071] S102. Obtain the preprocessed raw data, map the data of various modalities to the preset semantic vector space, and establish semantic relationships between the data of various modalities.

[0072] In this embodiment of the invention, a trained Transformer-based cross-modal semantic coding model is used to map feature information from different modalities into a shared, high-dimensional semantic vector space. Precise semantic relationships are then established between these modalities within this semantic vector space, thereby achieving deep fusion of multimodal data. The cross-modal semantic coding model comprises multiple parallel modal data encoders and a cross-modal fusion encoder connected to all of these modal data encoders.

[0073] The input of the cross-modal fusion encoder is connected to the output of all modal data encoders and serves as a shared encoder for these modal data encoders. The modal data encoders will have corresponding modal data encoders for each modal data, ensuring that the corresponding feature information can be obtained for each modal data.

[0074] Taking raw data containing four modalities—text data, tabular data, image data, and structured business data—as an example, the cross-modal semantic coding model includes a text data encoder, a tabular data encoder, an image data encoder, a structured data encoder, and a cross-modal fusion encoder. More specifically:

[0075] Text Encoder: This uses a large language model pre-trained on a large corpus of text as the backbone network, such as BERT, RoBERTa, or Ernie. To adapt to specific domains, such as sensitive areas like law and finance, the large language model can be further fine-tuned using domain-specific text data during training. The pre-processed text data is input into the text encoder, which outputs sequence-level text feature vectors. and text feature vectors The context vector.

[0076] The tabular data encoder comprises a content embedding layer, a row embedding layer, a column embedding layer, a vector fusion layer, and a Transformer encoder. The content embedding layer projects the content of each cell in the preprocessed tabular data to obtain a content feature vector. The row embedding layer projects the content of each row in the preprocessed tabular data to obtain a row feature vector. The column embedding layer projects the content of each column in the preprocessed tabular data to obtain a column feature vector. The vector fusion layer fuses the content feature vector, row feature vector, and column feature vector to obtain a table feature vector. This table feature vector is then input into the Transformer encoder, which ultimately outputs the overall feature vector of the tabular data. .

[0077] In this embodiment of the invention, by setting row embedding layer and column embedding layer, the embedding vectors such as row feature vectors and column feature vectors can be fused with cell feature vectors to provide structural information of the table, and the entire table can be regarded as a sequence (expanded by row or by column).

[0078] Image data encoder: Employing mature image processing network models such as ResNet and Vision Transformer as the backbone, the preprocessed image data is input into the image data encoder, yielding local feature vectors for each region of interest in the output image data, as well as a global feature vector concatenated from all local feature vectors. Each region of interest can be set to multiple, thus obtaining multiple local feature vectors from the image data.

[0079] Structured data encoder: Since structured business data is usually a fixed-length vector, it can be directly encoded into data feature vectors using a multilayer perceptron composed of fully connected layers. .

[0080] S103. Obtain the target task, and determine the data weight of each modality and at least one candidate preset inference path based on the task context semantics of the target task. Each preset inference path is used to infer the target task based on the multimodal data of the target business scenario and obtain the inference result.

[0081] In this embodiment of the invention, the candidate preset inference paths and data weights are flexibly adjusted by the task context semantics of the target task, thereby significantly improving the inference accuracy under complex semantics and enhancing contextual inference capabilities. Each modality of data has a corresponding data weight. The input of the preset inference path is the context fusion feature obtained by fusing data weights and multimodal data. By guiding a more suitable preset inference path to the context fusion feature, adaptive, task-related inference is achieved. During the fusion process, different modal features are weighted and fused by task context semantics, and the information flow is determined according to the task context semantics. This allows for context adaptation based on different types of task or data features.

[0082] For example, when determining whether a company meets a certain condition, the text data might state "the company's R&D investment ratio ≥ 5%", while also including information such as company type and industry. By extracting the task context semantics, different importance weights are assigned to each modality of data: if the emphasis is on the amount of funds, the financial figures in the table data become more important; if the emphasis is on the nature of the company or its R&D capabilities, the text data has a higher weight. In this way, when determining whether a company meets a certain condition, key information such as the R&D investment ratio and company tags will be automatically amplified, ensuring more accurate reasoning.

[0083] Even after semantic alignment, the importance of each modality may differ in different task contexts. To address this issue, this invention employs context adaptation to achieve dynamic recalibration of multimodal data and intelligent selection of preset inference paths.

[0084] S104. Based on the target task, infer the data of various modalities using candidate preset inference paths to obtain the inference result of each preset inference path. Evaluate the inference results according to the constructed consistency evaluation model, and determine the final inference output of the target task from the inference results. Each inference result includes both the inference result itself and the original data referenced by the inference conclusion. That is, the inference result not only contains the inference conclusion itself but also carries the source of the original data, thus forming an interpretable inference chain. Users can trace the data source corresponding to each paragraph in the inference conclusion.

[0085] The consistency evaluation model is used to evaluate the reasoning results from multiple dimensions such as condition satisfaction, data consistency, and the rationality of reasoning logic, thereby obtaining a score for each reasoning result, and taking the reasoning result with the highest score as the final reasoning output of the target task.

[0086] For example, the final inference output might be: "Based on the fact that Company A's R&D investment ratio exceeds the minimum requirement specified in document P, and its industry is within the relevant directory, it is deemed to meet the requirements, and a suggested credit limit of 500,000 yuan is recommended. The source of the aforementioned R&D investment ratio could be tabular data T1, the source of the minimum requirement could be text data L2, the source of the industry could be structured business data S3, the source of the directory could be text data L4, and the source of the credit limit requirement could be the credit limit calculation formula in document data L3." This final inference result can be accompanied by an interpretable inference chain, which greatly enhances the transparency and credibility of the final inference output.

[0087] In this embodiment of the invention, all candidate reasoning paths are used to perform multiple inferences, and the interpretable optimal solution is selected from them. This provides clear analytical steps for the reasoning conclusion, making the model's decision-making process transparent, easy for users to understand and verify, which is crucial for deploying AI systems in high-risk areas.

[0088] This invention presents a multimodal contextual reasoning method for complex task scenarios. It maps feature information from different modalities to a shared, high-dimensional semantic vector space using a cross-modal semantic encoding model based on the Transformer architecture. Within this semantic vector space, precise semantic relationships are established between them, achieving deep fusion and accurate alignment of multimodal data such as text, tables, images, and structured business data within a unified semantic space, thus enabling better reasoning in complex scenarios. By flexibly adjusting the preset reasoning paths and data weights of candidates based on the task context semantics of the target task, the reasoning process is no longer static and fixed, but rather dynamically adjusts information utilization strategies based on the specific target task, significantly improving the robustness and accuracy of reasoning in complex and variable scenarios, thereby enhancing contextual reasoning capabilities. Through the generation and consistency evaluation of multiple candidate reasoning paths, not only is the optimal solution found, but clear problem-solving steps are also provided, making the model's decision-making process transparent and easy for users to understand and verify.

[0089] This invention provides a complete and systematic context-adaptive reasoning solution for complex scenarios, from data preprocessing to final output and optimization. It forms a closed loop, is easy to integrate into actual business systems, and has high practical value and broad prospects for industrial application.

[0090] In this embodiment of the invention, step S101 specifically includes:

[0091] S1011. Identify at least one target business scenario and obtain the raw data for each target business scenario.

[0092] In this embodiment of the invention, multimodal data in the target business scenario can be obtained through system interfaces, web crawlers, database queries, file uploads, etc., and may specifically include:

[0093] Text data: such as unstructured texts like specification documents, product manuals, announcements, implementation rules, user comments, news announcements, and contract terms;

[0094] Tabular data: such as indicator tables, classification tables, financial statements, parameter configuration tables, transcripts, and other semi-structured tables embedded in documents or web pages;

[0095] Image data: such as flowcharts, system architecture diagrams, regional indicator maps, statistical charts, physical photographs, scanned documents, etc.

[0096] Structured business data: derived from database enterprise information, enterprise profiles, financial indicators, R&D investment, regional information, equipment operating parameters, etc.

[0097] Electronic devices can acquire multimodal data from the outside world through the methods described above.

[0098] S1012. Match a data processing template to the data of each modality in the original data, and use the matching data processing template to process the original data of the corresponding modality to obtain the preprocessed original data.

[0099] In this embodiment of the invention, for the data characteristics of different modalities, a corresponding processing method is determined for each modal of data, such as feature extraction and standardization, and a preset data processing template is formed according to the corresponding processing method, so that when the corresponding modal data is obtained, the data processing template can be directly matched and called to perform the set processing method.

[0100] The data processing templates include four templates: the first template is used for processing text data such as sentence segmentation, word segmentation, entity recognition, and dependency parsing; entity recognition can include file names, numbers, company names, etc.; the second template is used for processing tabular data such as recognizing row and column semantics, table type, and standardizing table cell content; the third template is used for processing image data such as optical character recognition (ORC) and image-text region registration; and the fourth template is used for processing structured business data such as data normalization, classification coding, and time-series processing.

[0101] Understandably, if other modalities of data need to be processed, corresponding processing methods can be determined for the data of those modalities, forming new data processing templates. Therefore, depending on the number of modalities to be processed, a one-to-one corresponding data processing template can be set for each modality of data. The data processing template contains multiple preset data processing methods to ensure that the data of each modality can be preprocessed quickly.

[0102] After the preprocessing in step S101, the multimodal data contained in the original data can be transformed into a set of well-defined, uniformly formatted, and semantically rich feature information, which is convenient for subsequent deep semantic understanding.

[0103] More specifically, step S1012 includes the following steps:

[0104] S10121. Match the first processing template to the text data in the original data, and process the text data using the first processing template. The first processing template is used to segment and word the text data, dividing it into sentences and word sequences. For example, tools such as HanLP, spaCy, or NLTK are used to cut continuous text into independent sentences and word sequences. Then, target entities in the sentences and word sequences are identified and labeled. For example, trained NER models such as BERT-CRF are used to identify and label target entities that users consider key in the sentences and word sequences, such as names of people, organizations, locations, times, amounts, and clause numbers. After that, the grammatical dependencies between sentences and word sequences are analyzed, the constituent parts of the sentences are extracted, and a dependency syntax tree is constructed based on the constituent parts to understand the subject, predicate, object, and other components in the sentences and word sequences, thereby capturing more complex semantic structures. For example, analyzing the sentence "enterprises with registered capital of not less than 50 million yuan", it is clear that "enterprise" is the subject, "application" is the predicate, and "registered capital of not less than 50 million yuan" is a conditional modifier of "enterprise".

[0105] S10122. Match the second processing template to the tabular data in the original data, and process the tabular data using the second processing template. The second processing template is used to identify cells and cell content in the tabular data. For example, it uses a deep learning-based table recognition model such as TabNet, TableTransformer, or a table parser to identify the boundaries, rows, columns, and cell merging in the table, thereby identifying the cells and cell content of various table types. Based on the cells and cell content, it determines the type of the table data and identifies the semantics of the header row, such as "Project Name", "Amount in 2023", "Amount in 2024", and "Unit". Then, it cleans and normalizes the table content. For example, it converts "1 million", "1,000,000 yuan", and "One Million Yuan" into the numerical value "1,000,000"; and it converts category information such as "Yes / No" and "Qualified / Unqualified" into numerical values ​​or one-hot encoding.

[0106] The types include quantitative tables and qualitative tables. Quantitative tables mainly include those containing numerical values, such as financial statements, while qualitative tables mainly include category labels or description tables.

[0107] S10123. Match the third processing template to the image data in the original data, and process the image data using the third processing template. The third processing template is used to extract the text content in the image data. For example, a high-precision OCR engine is used to extract all text content in various types of image data, such as PaddleOCR and Tesseract. In particular, for charts, it is also necessary to identify axis labels, legends, data point labels, etc., and then perform object detection on the image data to identify various regions in the image data. This identifies specific regions in the image data. Using object detection models, such as YOLO and Faster R-CNN, specific regions in the image are identified, such as "title area", "flowchart step area", "data annotation area", etc. Then, the text content is associated with the specific regions. For example, the extracted text content "Three: Content Review" is associated with the third graphic frame of the flowchart, that is, it is determined that the text content is located within the third graphic frame of the flowchart.

[0108] S10124. Match the fourth processing template to the structured business data in the original data, and process the structured business data using the fourth processing template. The fourth processing template is used to normalize the structured business data, thereby unifying numerical features of different dimensions into the same numerical range through methods such as Min-Max scaling or Z-Score standardization. For example, it unifies revenue of "hundred million yuan" and R&D investment of "ten thousand yuan" into the same numerical range, avoiding bias in model training due to differences in magnitude. Then, it encodes the non-numerical data in the structured business data, that is, it performs label encoding or one-hot encoding on non-numerical category features, such as "industry type: manufacturing" and "enterprise size: large". After that, it performs sliding window segmentation on the structured business data, and uses the sliding window to extract the time domain features and frequency domain features of the structured business data. This allows information such as mean, variance, and trend to be extracted from time series data.

[0109] In this embodiment of the invention, step S102 specifically includes:

[0110] S1021. Input the preprocessed text data into the text data encoder to obtain the text feature vector output by the text data encoder. and text feature vectors The context vector.

[0111] S1022. Input the preprocessed tabular data into the tabular data encoder to obtain the overall feature vector output by the tabular data encoder. .

[0112] S1023. Input the preprocessed image data into the image data encoder to obtain the local feature vector output by the image data encoder and the global feature vector concatenated from all the local feature vectors. .

[0113] S1024. Input the preprocessed structured business data into the structured data encoder to obtain the data feature vector output by the structured data encoder. .

[0114] S1025, Transfer text feature vectors Context vector, overall feature vector Local feature vectors, global feature vectors and data feature vectors Projected into semantic vector space That is, the feature vectors of all modalities are projected onto the same dimension to form a preliminary unified representation.

[0115] S1026. Input the projected feature vectors of various modalities into the cross-modal fusion encoder, and use the cross-modal fusion encoder to perform semantic alignment of the feature vectors of various modalities, and establish semantic association between the feature vectors of various modalities.

[0116] By concatenating feature vectors from various modalities into a multimodal vector, and then using a cross-modal fusion encoder to perform cross-modal attention learning, the semantic relationships between feature vectors from various modalities are learned, and semantic alignment between data such as text, tables, and images is achieved through an attention mechanism.

[0117] For example, the text data might state "Support amount not exceeding 1 million," while the table data has a column stating "Maximum amount = 1 million yuan," and a flowchart might also indicate "Funding subsidy ≤ 1 million yuan." A cross-modal semantic coding model converts text, tables, and images into unified vector representations, and then aligns this information through cross-modal alignment. Thus, the 1 million in the text data, the 1 million in the table data, and the ≤ 1 million in the image data all convey the same information: the funding support amount does not exceed 1 million yuan.

[0118] More specifically, the cross-modal fusion encoder includes a cross-modal attention layer and a multi-layered stacked Transformer module. The cross-modal attention layer determines the attention scores between feature vectors of various modalities based on an attention mechanism, queries and focuses on feature vectors of various modalities based on the attention mechanism, and calculates the attention scores between feature vectors of each modality and feature vectors of other modalities, thereby learning cross-modal associations such as the high correlation between the term "funding amount" in text data and the column "maximum amount" in table data. The multi-layered stacked Transformer module performs semantic alignment on the feature vectors of each modality after attention learning and fuses them to obtain a multi-modal vector. In the Transformer module, deep, multi-directional, multimodal information interaction is carried out to perform semantic alignment and deep fusion of feature vectors of each modality. In this way, the information describing the same concept or entity in different modalities will be in very close positions in the vector space.

[0119] In a cross-modal attention layer, the feature vector of each modality can be used as a query to retrieve and focus on features (keys and values) from all other modalities. For example, text feature vectors... It can be used as a query to calculate the overall feature vector. Global feature vectors Attention score.

[0120] In this embodiment of the invention, step S103 specifically includes:

[0121] S1031. Obtain historical tasks and extract historical task semantics from them.

[0122] In this embodiment of the invention, historical task semantics are extracted from historical tasks using a trained semantic extraction model, context encoder, etc.

[0123] In particular, historical data corresponding to historical tasks can also be obtained. It is understood that historical data also contains multimodal data. Processing the historical data in steps S101 and S102 will also establish semantic relationships between various modal data corresponding to the historical data. By performing pooling operations on the various modal data that have established semantic relationships, the historical context vector of the historical data can be obtained. Combining the historical semantics and the historical context vector, the aforementioned historical task semantics can be obtained.

[0124] S1032. Based on the semantic content of the historical task semantics, classify the historical task semantics to obtain historical classification results, and train an output head for each historical classification result to obtain at least one preset reasoning path corresponding to each historical classification result.

[0125] The reasoning path is used to guide reasoning in different ways. By training multiple specific output heads for each historical classification result, such as monetary reasoning head, classification judgment head, and relational reasoning head, each output head can form a reasoning path. Correspondingly, each historical classification result can determine at least one preset reasoning path.

[0126] S1033. Obtain the target task and extract the task context semantics from the target task. Based on the semantic similarity between the context semantics and the historical classification results, match the preset reasoning path corresponding to the historical classification result with the highest semantic similarity for the target task, and obtain at least one candidate preset reasoning path.

[0127] Similarly, task context semantics are extracted from the target task, and the semantic similarity between the task context semantics and historical classification results is calculated. The historical classification result with the highest semantic similarity to the task context semantics is determined, and then at least one candidate preset reasoning path is matched for the target task.

[0128] S1034. Perform feature analysis on the task context vector to obtain the data weights for each modality.

[0129] In this embodiment of the invention, a gating network is set up. The input data of the gating network is the task context vector, and the output of the gating network is the data weights of various modalities. Different task context semantics may result in different data weights. The gating network can selectively enable or disable the contribution of certain modal information to inference. For example, when the task focuses on the amount of funds, the data weight of tabular data is increased; when the task focuses on the nature of the enterprise or behavioral characteristics, the data weight of text data and structured business data is increased.

[0130] The gated network is a combination of a single-layer perceptron and a sigmoid activation function. The input of the single-layer perceptron is used to receive the task context vector, and the output of the single-layer perceptron is connected to the sigmoid activation function. Feature analysis is performed using the single-layer perceptron, and the final output is the data weights of various modalities, which is the text feature vector. The corresponding first weight The first weight corresponding to the context vector Overall feature vector The corresponding first weight The first weight corresponding to the local feature vector Global feature vectors The corresponding first weight and data feature vectors The corresponding first weight .

[0131] S1035. Perform weighted fusion of multimodal data according to data weights to obtain context fusion features. .

[0132] In this embodiment of the invention, step S104 specifically includes:

[0133] S1041, Fusing contextual features Input the results into each candidate preset reasoning path to obtain the reasoning results output by each preset reasoning path.

[0134] To enhance inference performance, all inference paths employed methods such as beam search and Top-k / Top-p sampling to generate higher-quality inference results.

[0135] Beam search retains the k candidate sequences with the highest probabilities at each generation step (beam width=k). This avoids getting trapped in local optima. Top-k / Top-p sampling further introduces randomness on top of beam search. Top-k sampling randomly selects from the k candidate words with the highest probabilities; Top-p sampling randomly selects from the smallest set of candidate words with a cumulative probability exceeding p.

[0136] S1042, Fusing contextual features Each inference result is input into the consistency evaluation model, and a consistency score for each inference result is obtained from the consistency evaluation model.

[0137] To avoid potential logical inconsistencies or data contradictions in the generated inference results, this embodiment of the invention constructs a consistency evaluation model based on a binary classification or regression model. The input data for this consistency evaluation model is context fusion features. And each reasoning result, the output is the consistency score of each reasoning result.

[0138] More specifically, the consistency evaluation model evaluates the inference results from the following dimensions:

[0139] Condition satisfaction: Whether the conclusion of the assessment path meets all known hard conditions. For example, if the reasoning result shows that the conditions are met, but the company's R&D investment ratio is significantly lower than the requirements, then the score for this dimension is low.

[0140] Data consistency: Evaluate whether the data referenced in the path reasoning results are consistent with the values ​​in the original tabular data, structured business data, etc. If they are inconsistent, the score for this dimension will be low.

[0141] Reasoning logic rationality: This assesses whether the reasoning process from premises to conclusion conforms to common sense and domain logic. If it does not, the score for this dimension is low.

[0142] It is understandable that each dimension has a corresponding dimension weight, and the final consistency score is obtained by weighting the score of each dimension with its corresponding dimension weight.

[0143] S1043. The reasoning result with the highest consistency score is taken as the final reasoning output.

[0144] Please see Figure 2 The method may also include the following steps:

[0145] S201. Obtain raw data containing multiple modalities from the target business scenario and preprocess the raw data. See details below. Figure 1 As shown in step S101.

[0146] S202. Obtain the preprocessed raw data, map the data of various modalities to a preset semantic vector space, and establish semantic relationships between the data of various modalities. (See details below.) Figure 1 As shown in step S102.

[0147] S203. Obtain the target task, and determine the data weights for each modality and at least one candidate pre-defined inference path based on the task context semantics of the target task. (See details below.) Figure 1 As shown in step S103.

[0148] S204. Based on the target task, use candidate preset inference paths to infer data of various modalities, obtain the inference result of each preset inference path, evaluate the inference result according to the constructed consistency evaluation model, and determine the final inference output result of the target task from the inference results. (See details for further information.) Figure 1 As shown in step S104.

[0149] S205. Compare the final inference output with the preset benchmark answer to obtain the difference evaluation result, and adjust the model parameters of the cross-modal semantic coding model and the gating network according to the difference evaluation result.

[0150] In this embodiment of the invention, reinforcement learning or contrastive learning mechanisms are used to adjust the model parameters, enabling the method to continuously update the semantic alignment effect and data weights under different task types or samples, thereby achieving adaptive optimization and reducing subsequent similar errors.

[0151] The benchmark answer can be written by industry experts or generated from a high-quality knowledge graph. The deviation between the final inference output and the preset benchmark answer is calculated by using cross-entropy loss, mean squared error or a custom task-related loss function to obtain the difference evaluation result.

[0152] By employing reinforcement learning or contrastive learning mechanisms, the model can continuously learn from new labeled data or interactive feedback during actual deployment, gradually adapting to new task types and new business rules, thereby achieving continuous performance improvement and domain adaptation.

[0153] Adjusting the model parameters of the cross-modal semantic coding model and the gating network based on the difference assessment results allows for self-iteration and optimization using feedback from practical applications. This significantly reduces the cost of migrating and maintaining the model across different domains and extends the model's lifespan.

[0154] The following describes the multimodal context reasoning device for complex task scenarios provided by the embodiments of the present invention. The multimodal context reasoning device for complex task scenarios described below and the multimodal context reasoning method for complex task scenarios described above can be referred to in correspondence.

[0155] Due to the aforementioned technical problems, this invention also provides a multimodal contextual reasoning device for complex task scenarios, which aims to dynamically adjust information utilization strategies and significantly improve the robustness and accuracy of reasoning in complex and variable scenarios. Figure 3 This is a schematic diagram of the structure of a multimodal contextual reasoning method for complex task scenarios according to an embodiment of the present invention, such as... Figure 3 As shown, the device may include:

[0156] The data acquisition module 10 is used to acquire raw data containing multiple modalities of the target business scenario, thereby obtaining multimodal data, and preprocessing the raw data. The target business scenario refers to a business scenario highly relevant to the user's current business scenario, a business scenario the user expects to apply, or a business scenario with potential applications. The target business scenario can characterize the complex business scenarios required by the user. The raw data containing multiple modalities includes at least: text data such as specification documents, product manuals, announcements, implementation details, etc.; tabular data such as indicator tables, classification tables, etc.; image data such as flowcharts, system architecture diagrams, regional indicator maps, etc.; and structured business data such as enterprise information, enterprise profiles, financial indicators, R&D investment, and regional information.

[0157] It should be noted that, depending on the user's needs, the specific business scenario corresponding to the target business scenario can be configured by the user. At the same time, the target business scenario can include at least one business scenario, that is, there can be multiple target business scenarios.

[0158] Raw data containing multiple modalities. Multimodal data can be pre-stored in electronic devices or acquired by electronic devices from external sources. For example, electronic devices may acquire data from various external channels.

[0159] There are no restrictions on the specific form in which the raw data is acquired, as long as the electronic device can acquire the raw data.

[0160] In this embodiment of the invention, the purpose of preprocessing is to transform the acquired raw data into standardized, structured features that the model can process.

[0161] The semantic association module 20 is used to acquire the preprocessed raw data, map the data of various modalities to a preset semantic vector space, and establish semantic associations between the data of various modalities.

[0162] In this embodiment of the invention, a trained Transformer-based cross-modal semantic coding model is used to map feature information from different modalities into a shared, high-dimensional semantic vector space. Precise semantic relationships are then established between these modalities within this semantic vector space, thereby achieving deep fusion of multimodal data. The cross-modal semantic coding model comprises multiple parallel modal data encoders and a cross-modal fusion encoder connected to all of these modal data encoders.

[0163] The input of the cross-modal fusion encoder is connected to the output of all modal data encoders and serves as a shared encoder for these modal data encoders. The modal data encoders will have corresponding modal data encoders for each modal data, ensuring that the corresponding feature information can be obtained for each modal data.

[0164] The path selection module 30 is used to acquire the target task and determine the data weight of each modality of data and at least one candidate preset inference path based on the task context semantics of the target task. Each preset inference path is used to infer the target task based on the multimodal data of the target business scenario and obtain the inference result.

[0165] In this embodiment of the invention, the candidate preset inference paths and data weights are flexibly adjusted by the task context semantics of the target task, thereby significantly improving the inference accuracy under complex semantics and enhancing contextual inference capabilities. Each modality of data has a corresponding data weight. The input of the preset inference path is the context fusion feature obtained by fusing data weights and multimodal data. By guiding a more suitable preset inference path to the context fusion feature, adaptive, task-related inference is achieved. During the fusion process, different modal features are weighted and fused by task context semantics, and the information flow is determined according to the task context semantics. This allows for context adaptation based on different types of task or data features.

[0166] Even after semantic alignment, the importance of each modality may differ in different task contexts. To address this issue, this invention employs context adaptation to achieve dynamic recalibration of multimodal data and intelligent selection of preset inference paths.

[0167] The data reasoning module 40 is used to reason about data of various modalities based on the target task and using candidate preset reasoning paths. It obtains the reasoning result for each preset reasoning path, evaluates the reasoning result according to a pre-constructed consistency evaluation model, and determines the final reasoning output of the target task from the reasoning results. Each reasoning result includes both the reasoning result itself and the original data referenced by the reasoning conclusion. That is, the reasoning result not only contains the reasoning conclusion itself but also carries the source of the original data, thus forming an interpretable reasoning chain. Users can trace the data source corresponding to each segment in the reasoning conclusion.

[0168] The consistency evaluation model is used to evaluate the reasoning results from multiple dimensions such as condition satisfaction, data consistency, and the rationality of reasoning logic, thereby obtaining a score for each reasoning result, and taking the reasoning result with the highest score as the final reasoning output of the target task.

[0169] In this embodiment of the invention, all candidate reasoning paths are used to perform multiple inferences, and the interpretable optimal solution is selected from them. This provides clear analytical steps for the reasoning conclusion, making the model's decision-making process transparent, easy for users to understand and verify, which is crucial for deploying AI systems in high-risk areas.

[0170] This invention presents a multimodal contextual reasoning device for complex task scenarios. It maps feature information from different modalities into a shared, high-dimensional semantic vector space using a cross-modal semantic encoding model based on the Transformer architecture. Within this semantic vector space, it establishes precise semantic relationships between them, achieving deep fusion and precise alignment of multimodal data such as text, tables, images, and structured business data within a unified semantic space, thus enabling better reasoning in complex scenarios. By flexibly adjusting the preset reasoning paths and data weights of candidates based on the task context semantics of the target task, the reasoning process is no longer static and fixed. Instead, it dynamically adjusts information utilization strategies based on the specific target task, significantly improving the robustness and accuracy of reasoning in complex and variable scenarios, thereby enhancing contextual reasoning capabilities. Through the generation and consistency evaluation of multiple candidate reasoning paths, it not only finds the optimal solution but also provides clear problem-solving steps, making the model's decision-making process transparent and easy for users to understand and verify.

[0171] This invention provides a complete and systematic context-adaptive reasoning solution for complex scenarios, from data preprocessing to final output and optimization. It forms a closed loop, is easy to integrate into actual business systems, and has high practical value and broad prospects for industrial application.

[0172] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical commands in the memory 430 to execute a multimodal contextual reasoning method for complex task scenarios, the method including:

[0173] Acquire raw data containing multiple modalities from the target business scenario and preprocess the raw data;

[0174] The process involves acquiring the preprocessed raw data and mapping the data from various modalities to a predefined semantic vector space, establishing semantic relationships between the data from different modalities, and using a trained cross-modal semantic coding model for unified mapping. This model also establishes semantic relationships between the data from different modalities within the semantic vector space. The cross-modal semantic coding model includes multiple parallel modal data encoders and a cross-modal fusion encoder that is connected to all the modal data encoders.

[0175] Obtain the target task, and determine the data weight of each modality and at least one candidate preset inference path based on the task context semantics of the target task;

[0176] Based on the target task, the candidate preset reasoning paths are used to reason about data of various modalities to obtain the reasoning result of each preset reasoning path. The reasoning result is evaluated according to the constructed consistency evaluation model, and the final reasoning output result of the target task is determined from the reasoning result. Each reasoning result includes the reasoning result and the original data referenced by the reasoning conclusion. The consistency evaluation model is used to evaluate the reasoning result from the perspectives of condition satisfaction, data consistency, and the rationality of reasoning logic.

[0177] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the multimodal context reasoning method for complex task scenarios provided by the above methods, the method comprising:

[0179] Acquire raw data containing multiple modalities from the target business scenario and preprocess the raw data;

[0180] The process involves acquiring the preprocessed raw data and mapping the data from various modalities to a predefined semantic vector space, establishing semantic relationships between the data from different modalities, and using a trained cross-modal semantic coding model for unified mapping. This model also establishes semantic relationships between the data from different modalities within the semantic vector space. The cross-modal semantic coding model includes multiple parallel modal data encoders and a cross-modal fusion encoder that is connected to all the modal data encoders.

[0181] Obtain the target task, and determine the data weight of each modality and at least one candidate preset inference path based on the task context semantics of the target task;

[0182] Based on the target task, the candidate preset reasoning paths are used to reason about data of various modalities to obtain the reasoning result of each preset reasoning path. The reasoning result is evaluated according to the constructed consistency evaluation model, and the final reasoning output result of the target task is determined from the reasoning result. Each reasoning result includes the reasoning result and the original data referenced by the reasoning conclusion. The consistency evaluation model is used to evaluate the reasoning result from the perspectives of condition satisfaction, data consistency, and the rationality of reasoning logic.

[0183] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the aforementioned multimodal context reasoning methods for complex task scenarios, the methods comprising:

[0184] Acquire raw data containing multiple modalities from the target business scenario and preprocess the raw data;

[0185] The process involves acquiring the preprocessed raw data and mapping the data from various modalities to a predefined semantic vector space, establishing semantic relationships between the data from different modalities, and using a trained cross-modal semantic coding model for unified mapping. This model also establishes semantic relationships between the data from different modalities within the semantic vector space. The cross-modal semantic coding model includes multiple parallel modal data encoders and a cross-modal fusion encoder that is connected to all the modal data encoders.

[0186] Obtain the target task, and determine the data weight of each modality and at least one candidate preset inference path based on the task context semantics of the target task;

[0187] Based on the target task, the candidate preset reasoning paths are used to reason about data of various modalities to obtain the reasoning result of each preset reasoning path. The reasoning result is evaluated according to the constructed consistency evaluation model, and the final reasoning output result of the target task is determined from the reasoning result. Each reasoning result includes the reasoning result and the original data referenced by the reasoning conclusion. The consistency evaluation model is used to evaluate the reasoning result from the perspectives of condition satisfaction, data consistency, and the rationality of reasoning logic.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal contextual reasoning method for complex task scenarios, characterized in that, The method includes: Acquire raw data containing multiple modalities from the target business scenario and preprocess the raw data; The process involves acquiring preprocessed raw data and mapping various modalities of data uniformly into a predefined semantic vector space, establishing semantic relationships between different modalities. A trained cross-modal semantic coding model is then used for this unified mapping, establishing semantic relationships between different modalities within this semantic vector space. The cross-modal semantic coding model comprises multiple parallel modal data encoders and a cross-modal fusion encoder connected to all modal data encoders. The cross-modal semantic coding model includes: a text data encoder, a tabular data encoder, an image data encoder, a structured data encoder, and a cross-modal fusion encoder. Obtain the target task, and determine the data weight of each modality and at least one candidate preset inference path based on the task context semantics of the target task; Based on the target task, the system uses candidate preset inference paths to infer data from various modalities, obtaining the inference result of each preset inference path. The inference result is then evaluated based on the constructed consistency evaluation model, and the final inference output of the target task is determined from the inference result. Each inference result includes the inference conclusion and the original data referenced by the inference conclusion. Based on the referenced original data, the user can trace the data source corresponding to each paragraph in the inference conclusion. The consistency evaluation model is used to evaluate the inference result from the perspectives of condition satisfaction, data consistency, and the rationality of the inference logic. The process of acquiring the target task and determining the data weights for each modality and at least one candidate preset inference path based on the task context semantics of the target task specifically includes: Obtain historical tasks and extract historical task semantics from them; Based on the semantic content of the historical task semantics, the historical task semantics are classified to obtain historical classification results, and an output head is trained for each historical classification result to obtain at least one preset reasoning path corresponding to each historical classification result. Obtain the target task and extract the task context semantics from the target task. Based on the semantic similarity between the context semantics and the historical classification results, match the preset reasoning path corresponding to the historical classification result with the highest semantic similarity for the target task, and obtain at least one candidate preset reasoning path. Feature analysis is performed on the task context vector to obtain the data weights for each modality; The multimodal data is weighted and fused according to the data weights to obtain context fusion features; The process of reasoning about various modalities of data using candidate preset inference paths according to the target task, obtaining the inference result of each preset inference path, evaluating the inference result according to the constructed consistency evaluation model, and determining the final inference output result of the target task from the inference results specifically includes: The context fusion features are input into each candidate preset inference path to obtain the inference result output by each preset inference path; The context fusion features and each inference result are input into the consistency evaluation model to obtain the consistency score of each inference result output by the consistency evaluation model. The inference result with the highest consistency score is used as the final inference output.

2. The multimodal contextual reasoning method for complex task scenarios according to claim 1, characterized in that, The text data encoder uses a large language model as its backbone network. The tabular data encoder includes a content embedding layer, a row embedding layer, a column embedding layer, a vector fusion layer, and a Transformer encoder; The content embedding layer is used to project the content of each cell in the preprocessed tabular data to obtain a content feature vector; The row embedding layer projects the content of each row in the preprocessed table data to obtain the row feature vector; the column embedding layer projects the content of each column in the preprocessed table data to obtain the column feature vector; the vector fusion layer fuses the content feature vector, row feature vector, and column feature vector to obtain the table feature vector. The image data encoder uses an image processing network model as its backbone network. The structured data encoder is a multilayer perceptron composed of fully connected layers; The cross-modal fusion encoder includes a cross-modal attention layer and a Transformer module formed by multiple stacked layers.

3. The multimodal contextual reasoning method for complex task scenarios according to claim 2, characterized in that, The process of acquiring the preprocessed raw data, mapping the data of various modalities to a preset semantic vector space, and establishing semantic relationships between the data of various modalities specifically includes: The preprocessed text data is input into the text data encoder to obtain the text feature vector and the context vector of the text feature vector output by the text data encoder. The preprocessed tabular data is input into the tabular data encoder to obtain the overall feature vector output by the tabular data encoder; The preprocessed image data is input into the image data encoder to obtain the local feature vector output by the image data encoder and the global feature vector concatenated from all the local feature vectors. The preprocessed structured business data is input into the structured data encoder to obtain the data feature vector output by the structured data encoder. Project text feature vectors, context vectors, global feature vectors, local feature vectors, global feature vectors, and data feature vectors into the semantic vector space; The projected feature vectors of various modalities are input into the cross-modal fusion encoder, and the cross-modal fusion encoder is used to perform semantic alignment of the feature vectors of various modalities, thereby establishing semantic associations between the feature vectors of various modalities.

4. The multimodal contextual reasoning method for complex task scenarios according to claim 1, characterized in that, The acquisition of raw data containing multiple modalities from the target business scenario, and the preprocessing of the raw data, specifically include: Identify at least one target business scenario and obtain the raw data for each target business scenario; For each modality in the original data, a data processing template is matched, and the original data of the corresponding modality is processed using the matching data processing template to obtain the preprocessed original data.

5. The multimodal contextual reasoning method for complex task scenarios according to claim 4, characterized in that, The process involves matching a data processing template to the data of each modality in the original data, and then using the matching data processing template to process the original data of the corresponding modality to obtain preprocessed original data. Specifically, this includes: Match the first processing template to the text data in the original data, and process the text data using the first processing template; Match the second processing template to the tabular data in the original data, and use the second processing template to process the tabular data; A third processing template is matched to the image data in the original data, and the image data is processed using the third processing template; The fourth processing template is matched to the structured business data in the original data, and the structured business data is processed using the fourth processing template.

6. A multimodal contextual reasoning device for complex task scenarios, characterized in that, The device includes: The data acquisition module is used to acquire raw data containing multiple modalities from the target business scenario and to preprocess the raw data. The semantic association module is used to acquire the preprocessed raw data, map the data of various modalities to a preset semantic vector space, and establish semantic associations between the data of various modalities. It utilizes a trained cross-modal semantic coding model for unified mapping and establishes semantic associations between the data of various modalities within this semantic vector space. The cross-modal semantic coding model includes multiple parallel modal data encoders and a cross-modal fusion encoder connected to all the modal data encoders. The cross-modal semantic coding model includes: a text data encoder, a tabular data encoder, an image data encoder, a structured data encoder, and a cross-modal fusion encoder. The path selection module is used to obtain the target task and determine the data weight of each modality and at least one candidate preset inference path based on the task context semantics of the target task. The data reasoning module is used to reason about data of various modalities based on the target task and using candidate preset reasoning paths to obtain the reasoning result of each preset reasoning path. The reasoning result is evaluated according to the constructed consistency evaluation model, and the final reasoning output result of the target task is determined from the reasoning result. Each reasoning result includes the reasoning conclusion and the original data referenced by the reasoning conclusion. Based on the referenced original data, the user can trace the data source corresponding to each paragraph in the reasoning conclusion. The consistency evaluation model is used to evaluate the reasoning result from the perspectives of condition satisfaction, data consistency, and the rationality of reasoning logic. The path selection module specifically includes: Obtain historical tasks and extract historical task semantics from them; Based on the semantic content of the historical task semantics, the historical task semantics are classified to obtain historical classification results, and an output head is trained for each historical classification result to obtain at least one preset reasoning path corresponding to each historical classification result. Obtain the target task and extract the task context semantics from the target task. Based on the semantic similarity between the context semantics and the historical classification results, match the preset reasoning path corresponding to the historical classification result with the highest semantic similarity for the target task, and obtain at least one candidate preset reasoning path. Feature analysis is performed on the task context vector to obtain the data weights for each modality; The multimodal data is weighted and fused according to the data weights to obtain context fusion features; The data reasoning module specifically includes: The context fusion features are input into each candidate preset inference path to obtain the inference result output by each preset inference path; The context fusion features and each inference result are input into the consistency evaluation model to obtain the consistency score of each inference result output by the consistency evaluation model. The inference result with the highest consistency score is used as the final inference output.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal context reasoning method for complex task scenarios as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal context reasoning method for complex task scenarios as described in any one of claims 1 to 5.

Images