Simulation deduction method, device and equipment, storage medium and product

Abstract

The invention relates to the field of simulation deduction, and particularly provides a simulation deduction method, device, equipment, storage medium and product, the method comprises the following steps: obtaining data sources of different modals in a target domain, the target domain comprising urban traffic, educational training, emergency response, automatic driving or medical disease analysis; the data source comprises at least two of image data, text data and voice data; data features in the data source are extracted and feature fusion is carried out to obtain fusion features, and the fusion features comprise comprehensive feature vectors corresponding to all data feature vectors in the data source; and performing simulation deduction on the fusion feature through a preset reasoning method and a decision model to obtain a simulation deduction decision result, the simulation deduction decision result including a reasoning result and a decision scheme. By means of the method, the effect of simulating the human cognitive process to conduct the accurate and real simulation deduction process can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of simulation reasoning, in particular, to a simulation reasoning method, device, equipment, storage medium and product. BACKGROUND

[0002] Currently, in the fields of modern complex society, economy, environment, education, etc., simulation reasoning in the cognitive domain is of great significance for decision support, risk assessment and strategic planning. However, traditional cognitive simulation systems are usually constructed based on single modal data (such as text or structured rules), and traditional cognitive domain simulation reasoning methods mainly rely on single data modalities, such as text analysis, image recognition or speech processing, etc. These methods have obvious limitations in dealing with complex scenarios. Single modal data cannot fully reflect the interaction of multi-element information in the cognitive domain, and cannot effectively fuse multi-modal information (such as visual scenes, voice interaction), resulting in that the reasoning result deviates from the real scene, and its accuracy and reliability are limited. The data processing and model construction process of traditional methods rely on pre-defined rule base, which is difficult to handle fuzzy and uncertain cognitive situations, and even more difficult to achieve accurate modeling of complex dynamic environment in the cognitive domain. For example, in the military field, relying only on written intelligence for battlefield situation reasoning may ignore the key clues contained in images, voices and other information; in the economic field, only analyzing numerical data while ignoring text news and market sentiment may lead to misjudgment of market trends. The black box model is difficult to provide the logical chain of the reasoning process, which limits the credibility in key fields.

[0003] Multi-modal large models have made breakthroughs in cross-modal alignment and context reasoning, but applying multi-modal large models to cognitive domain simulation reasoning still faces many challenges, including efficient fusion of multi-modal data, dynamic adaptability of models, and visualization and interpretability of simulation reasoning results, etc. Therefore, a new method that can simulate human cognitive process and perform reliable reasoning is urgently needed.

[0004] Therefore, how to simulate the human cognitive process to perform accurate and realistic simulation reasoning is a technical problem to be solved. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a simulation reasoning method, which can achieve the effect of simulating the human cognitive process to perform accurate and realistic simulation reasoning.

[0006] In a first aspect, embodiments of this application provide a simulation and deduction method, comprising: acquiring data sources of different modalities in a target domain, wherein the target domain includes: urban traffic, education and training, emergency response, autonomous driving, or medical disease analysis, and the data sources include: at least two of image data, text data, and voice data; extracting data features from the data sources and performing feature fusion to obtain fused features, wherein the fused features include a comprehensive feature vector corresponding to all data feature vectors in the data sources; and performing simulation and deduction on the fused features through a preset reasoning method and decision model to obtain simulation and deduction decision results, wherein the simulation and deduction decision results include reasoning results and decision schemes.

[0007] In the above embodiments, this application obtains data sources of different modalities in the target domain, extracts cross-modal features and performs feature fusion, and finally combines decision models and reasoning to achieve accurate simulation of cognitive processes in complex scenarios, thereby achieving the effect of simulating human cognitive processes for accurate and realistic simulation and deduction.

[0008] In some embodiments, the fused features are simulated and deduced using a preset reasoning method and decision model to obtain simulation and deduction decision results, including: simulating and deducing the fused features using preset rules, machine learning methods, or deep learning methods to obtain the simulation and deduction process; and performing decision analysis on the simulation and deduction process using a decision model to obtain the simulation and deduction decision results.

[0009] In the above embodiments, this application uses preset rules, machine learning methods, or deep learning methods to simulate and extrapolate the fused features, thereby accurately obtaining the simulation and extrapolation decision results.

[0010] In some embodiments, after simulating and deducing the fused features through a preset reasoning method and decision model to obtain the simulation deduction decision results, the method further includes: displaying the simulation deduction decision results in a user-preset output format, wherein the output format includes visual output of text and images as well as voice output.

[0011] In the above embodiments, this application displays the simulation and decision-making results in a user-preset output format, which facilitates the visualization output of text and images as well as voice output, making it convenient for users to view.

[0012] In some embodiments, extracting data features from the data source and performing feature fusion to obtain fused features includes: extracting feature vectors of all data in the data source through a neural network architecture to obtain a set of feature vectors; aligning all feature vectors in the set of feature vectors through a constructed shared space to obtain a set of feature vectors of the same dimension; and fusing all feature vectors in the set of feature vectors of the same dimension through a preset fusion algorithm to obtain fused features, wherein the fusion algorithm includes concatenation, weighted summation, or attention mechanism.

[0013] In the above embodiments, this application can accurately obtain fused features by fusing all feature vectors in a feature vector set of the same dimension through a preset fusion algorithm.

[0014] In some embodiments, extracting data features from the data source and performing feature fusion to obtain fused features includes: dynamically adjusting the weights of each data feature in the data source using an adaptive weighting mechanism to obtain a weight allocation; and weighting and fusing the data features in the data source according to the weight allocation to obtain fused features.

[0015] In the above embodiments, this application weights and fuses data features from the data source according to the weight allocation, and obtains fused features by reasonably allocating weights to obtain the final fused features.

[0016] In some embodiments, the fused features are simulated and deduced using a preset inference method and decision model to obtain simulation and deduction decision results, including: generating an inference engine based on a preset reinforcement learning decision model and defining a Markov decision process according to the inference method; and performing simulation and decision processing on the fused features based on the inference engine and the Markov decision process to obtain simulation and deduction decision results.

[0017] In the above embodiments, this application uses a constructed inference engine and Markov decision process to perform simulation inference and decision processing on the fused features, which can accurately obtain the simulation inference decision results.

[0018] Secondly, embodiments of this application provide a simulation and deduction system, including: Data input layer, multimodal fusion layer, cognitive reasoning layer, and output layer; The data input layer is used to acquire data sources of different modalities in the target domain; The multimodal fusion layer is used to extract data features from the data source and perform feature fusion to obtain fused features; The cognitive reasoning layer is used to simulate and deduce the fused features through preset reasoning methods and decision models, and obtain the simulation and deduction decision results; The output layer is used to display the simulation and decision-making results in a user-preset output format.

[0019] Optionally, the data input layer is specifically used for: Obtain initial data sources for different modalities in the target domain; The initial data source is subjected to noise reduction and irrelevant information removal operations to obtain the data source.

[0020] Thirdly, embodiments of this application provide a simulation and deduction apparatus, comprising: A multimodal data acquisition module is used to acquire data sources of different modalities in a target domain, wherein the target domain includes: urban traffic, education and training, emergency response, autonomous driving or medical disease analysis, and the data source includes at least two of: image data, text data and voice data; A multimodal data fusion module is used to extract data features from the data source and perform feature fusion to obtain fused features, wherein the fused features include a comprehensive feature vector corresponding to all data feature vectors in the data source; The simulation and deduction module is used to simulate and deduce the fused features through preset reasoning methods and decision models, and obtain simulation and deduction decision results, wherein the simulation and deduction decision results include reasoning results and decision schemes.

[0021] Optionally, the simulation and deduction module is specifically used for: The fused features are simulated and deduced using preset rules, machine learning methods, or deep learning methods to obtain the simulation and deduction process; The simulation process is analyzed using a decision model to obtain the simulation decision results.

[0022] Optionally, the device also includes: The display module is used by the simulation and deduction module to simulate and deduce the fused features through preset reasoning methods and decision models, and after obtaining the simulation and deduction decision results, to display the simulation and deduction decision results in a user-preset output format. The output format includes text and image visualization output as well as voice output.

[0023] Optionally, the multimodal data fusion module is specifically used for: The feature vector set is obtained by extracting the feature vectors of all data in the data source through a neural network architecture. By aligning all feature vectors in the constructed shared space feature vector set, a feature vector set of the same dimension is obtained; By fusing all feature vectors in a set of feature vectors of the same dimension using a preset fusion algorithm, a fused feature is obtained. The fusion algorithm may include concatenation, weighted summation, or attention mechanism.

[0024] Optionally, the multimodal data fusion module is specifically used for: An adaptive weighting mechanism is used to dynamically adjust the weights of each data feature in the data source to obtain the weight allocation. The data features in the data source are weighted and merged according to the weight allocation to obtain the merged features.

[0025] Optionally, the simulation and deduction module is specifically used for: An inference engine is generated based on a pre-defined reinforcement learning decision model, and a Markov decision process is defined according to the inference method. Simulation and decision processing of fused features are performed based on inference engine and Markov decision process to obtain simulation and decision results.

[0026] Fourthly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the method provided in the first aspect above are performed.

[0027] Fifthly, embodiments of this application provide a readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method provided in the first aspect above.

[0028] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating a simulation deduction method provided in this application embodiment; Figure 2 A schematic block diagram of the structure of a simulation and deduction system provided in this application embodiment; Figure 3 A schematic block diagram of a hierarchical cognitive reasoning architecture provided in this application embodiment; Figure 4 This is a schematic diagram of a dynamic memory update mechanism provided in an embodiment of this application; Figure 5 A schematic block diagram of a simulation and deduction device provided for an embodiment of this application; Figure 6 This is a schematic block diagram of a simulation and deduction device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0032] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0033] This application is applied to simulation scenarios, specifically, through cross-modal feature alignment, dynamic memory networks and adversarial optimization mechanisms, to achieve accurate simulation of cognitive processes in complex scenarios such as urban traffic, education and training, emergency response, autonomous driving, and medical diagnosis.

[0034] Currently, in complex modern social, economic, environmental, and educational fields, cognitive domain simulation is of great significance for decision support, risk assessment, and strategic planning. However, traditional cognitive simulation systems are typically built upon single-modal data (such as text or structured rules), and traditional cognitive domain simulation methods mainly rely on single data modalities, such as text analysis, image recognition, or speech processing. These methods have significant limitations when dealing with complex scenarios. Single-modal data cannot comprehensively reflect the interaction relationships of diverse information in the cognitive domain, nor can it effectively integrate multimodal information (such as visual scenes and speech interactions), leading to simulation results that deviate from the real-world scenario, thus limiting their accuracy and reliability. The data processing and model building processes of traditional methods rely on predefined rule bases, making it difficult to handle ambiguous and uncertain cognitive situations, and even more difficult to achieve accurate modeling of complex and dynamic environments in the cognitive domain. For example, in the military field, relying solely on textual intelligence for battlefield situation simulation may overlook key clues contained in other information such as images and speech; in the economic field, analyzing only numerical data while ignoring textual news and market sentiment may lead to misjudgments of market trends. Black-box models struggle to provide a logical chain of reasoning processes, limiting their reliability in critical domains. While multimodal large models have achieved breakthroughs in cross-modal alignment and contextual reasoning, applying them to cognitive domain simulations still faces numerous challenges, including efficient fusion of multimodal data, dynamic adaptability of the model, and visualization and interpretability of simulation results. Therefore, a novel method capable of simulating human cognitive processes and providing reliable inferences is urgently needed.

[0035] To this end, this application acquires data sources of different modalities within a target domain, including urban traffic, education and training, emergency response, autonomous driving, and medical disease analysis. The data sources include at least two of the following: image data, text data, and voice data. Features are extracted from the data sources and fused to obtain fused features, which include a comprehensive feature vector corresponding to all feature vectors in the data sources. The fused features are then simulated using a pre-defined inference method and decision model to obtain simulation and decision-making results, including inference results and decision schemes. By acquiring data sources of different modalities within the target domain, extracting cross-modal features, and fusing them, the application ultimately combines a decision model and inference to achieve accurate simulation of cognitive processes in complex scenarios, thus achieving the effect of accurately and realistically simulating human cognitive processes.

[0036] In this embodiment of the application, the executing entity can be the simulation simulation device in the simulation simulation system. In practical applications, the simulation simulation device can be electronic devices such as terminal devices and servers, and there are no restrictions here.

[0037] The following is combined with Figure 1The simulation and deduction method of the embodiments of this application will be described in detail.

[0038] Please refer to Figure 1 , Figure 1 A flowchart of a simulation deduction method provided in this application embodiment is shown below. Figure 1 The simulation and deduction methods shown include: Step 110: Obtain data sources for different modalities in the target domain.

[0039] The target areas include: urban transportation, education and training, emergency response, autonomous driving or medical disease analysis, and the data sources include: at least two of the following: image data, text data and voice data, and may also include fire drills and disaster drills.

[0040] Specifically, different acquisition methods can be implemented for different scenarios. For example, image data: image information acquired through devices such as cameras and scanners, used to reflect visual features in the cognitive scene. Text data: text information acquired through keyboard input, speech recognition, etc., used to describe the language content in the cognitive process. Speech data: speech information acquired through devices such as microphones, used to reflect speech features and emotional information in the cognitive process.

[0041] Optionally, data sources of different modalities in the target domain can be obtained. For example, image data can be obtained from traffic cameras and radar in the urban traffic field, lecture recordings and audio recordings can be obtained in the education and training field, combat information can be obtained from user-inputted text and images in emergency response, image data can be obtained from vehicle-mounted cameras or radar in the autonomous driving field, and image and text data can be obtained from medical device scanning in the medical disease analysis field.

[0042] Step 120: Extract data features from the data source and perform feature fusion to obtain fused features.

[0043] The fused feature includes a composite feature vector corresponding to all data feature vectors in the data source. The composite feature vector can be obtained by concatenating or fusing according to preset weights.

[0044] In some embodiments of this application, data features are extracted from the data source and fused to obtain fused features, including: extracting feature vectors of all data in the data source through a neural network architecture to obtain a set of feature vectors; aligning all feature vectors in the set of feature vectors through a constructed shared space to obtain a set of feature vectors of the same dimension; and fusing all feature vectors in the set of feature vectors of the same dimension through a preset fusion algorithm to obtain fused features, wherein the fusion algorithm includes concatenation, weighted summation, or attention mechanism.

[0045] In the above process, this application can accurately obtain fused features by fusing all feature vectors in a feature vector set of the same dimension through a preset fusion algorithm.

[0046] Feature vectors in the feature vector set can be obtained through the model's feature extraction layer. Feature extraction includes the processes of feature extraction, feature alignment, and feature fusion. Specifically, it involves using neural network architectures such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Transformers to extract feature vectors from image, text, and speech data. These feature vectors reflect the inherent properties and interrelationships of data from different modalities. The extracted features are aligned to ensure they can be compared and fused along the same dimension. Feature alignment can be achieved by learning a shared latent space, allowing feature vectors from different modalities to map and correspond to each other within this space. Based on feature alignment, the feature vectors from different modalities are fused to form a unified feature representation. Feature fusion can employ methods such as concatenation, weighted summation, and attention mechanisms. Through feature fusion, a comprehensive feature vector containing information from multiple modalities can be obtained, providing strong support for subsequent cognitive reasoning.

[0047] In some embodiments of this application, data features are extracted from the data source and fused to obtain fused features, including: dynamically adjusting the weights of each data feature in the data source using an adaptive weighting mechanism to obtain a weight allocation; and weighting and fusing the data features in the data source according to the weight allocation to obtain fused features.

[0048] In the above process, this application weights and fuses data features from the data source according to the weight allocation, and obtains fused features by reasonably allocating weights to obtain the final fused features.

[0049] The weight allocation can be calculated using a preset algorithm.

[0050] Optionally, data features are extracted from the data source and feature fusion is performed to obtain fused features, including visual feature extraction, text feature extraction formulas, and cross-modal alignment attention formulas. Specifically: Visual feature extraction can utilize the Vision Transformer (ViT) model to process image and video data. Given an input image... (high ,width Number of channels By dividing the input image into blocks, Divided into Non-overlapping image patches ( ), denoted as Each image patch, after linear projection and positional encoding, generates an embedding vector. The specific formula is as follows: ; in, Linear projection matrix, which maps image patches to... Hidden space. The size of the image patch (e.g., 16×16), with each patch having dimensions of... . : No. The flattened vector of each image patch. : Location encoding vector, used to preserve the spatial location information of image patches.

[0051] For example, if the input image size is Segment size The total number of blocks is: The dimension of each block is Linear projection matrix The dimension is (generally or ).

[0052] Specifically, the text feature extraction formula is as follows: For the input text Context-related features are extracted using a pre-trained language model (such as BERT). The process is as follows: Word embedding layer: The input text is... Divided into A sequence of words (or sub-words) Each word is embedded through a word embedding matrix. Convert to vector: , ; in, For vocabulary size, For words A one-hot vector.

[0053] Location coding: Add learnable location coding : ; Transformer encoder: after Layer self-attention mechanism processing, the first The layer output is: ; The final text feature is the concatenation of the hidden states at all positions in the last layer: ; in: Feature dimension (e.g., 768 for BERT-base). : Maximum text length (e.g., 512). : Context-dependent text feature matrix, with each column representing word-level features.

[0054] Specifically: The cross-modal alignment attention formula is as follows: Visual features ( (Number of image blocks) and text features Calculate the attention weights between the visual and text modalities through cross-modal attention alignment. : ; in: Learnable query and key projection matrices. : Scaling factor, to prevent the gradient from vanishing due to excessively large dot product values. Along the text dimension Normalization represents the attention weight of each image patch to the text word.

[0055] Aligned unified cognitive representation: ; in, Value is the projection matrix.

[0056] Step 130: Simulate and extrapolate the fused features using a preset reasoning method and decision model to obtain the simulation and extrapolation decision results.

[0057] The simulation and decision-making results include inference results and decision schemes. Different target domains correspond to different simulation and decision-making results. For example, urban traffic can construct traffic network models and combine real-time traffic data (such as road congestion and weather conditions) to optimize resource transportation routes. Education and training can establish personalized learning path planning and skills mastery prediction for trainees. Emergency response can quickly dispatch emergency resources (such as rescue teams, medical supplies, food, and water) to ensure the efficiency and accuracy of resource allocation. Autonomous driving can simulate the interaction behavior of pedestrians and vehicles in complex traffic scenarios, or medical disease analysis can combine patient vital signs with medical knowledge bases to optimize treatment processes.

[0058] In some embodiments of this application, the fused features are simulated and deduced using a preset reasoning method and decision model to obtain simulation and deduction decision results, including: simulating and deducing the fused features using preset rules, machine learning methods, or deep learning methods to obtain the simulation and deduction process; and performing decision analysis on the simulation and deduction process using a decision model to obtain the simulation and deduction decision results.

[0059] In the above process, this application uses preset rules, machine learning methods, or deep learning methods to simulate and extrapolate the fused features, thereby accurately obtaining the simulation and extrapolation decision results.

[0060] The reasoning methods can be rule-based, machine learning-based, or deep learning-based. Deep learning-based methods, in particular, can leverage neural network models to learn complex relationships between data, enabling more intelligent reasoning and decision-making. The decision-making model can be designed appropriately based on specific application scenarios and needs. For example, in the military field, a multi-objective optimization-based decision-making model can be designed to balance the priorities and resource allocation among different combat missions; in the education field, a student behavior analysis-based decision-making model can be designed to personalize learning resources and learning paths.

[0061] In some embodiments of this application, the fused features are simulated and deduced using a preset inference method and decision model to obtain simulation and deduction decision results, including: generating an inference engine based on a preset reinforcement learning decision model and defining a Markov decision process according to the inference method; and performing simulation and decision processing on the fused features based on the inference engine and the Markov decision process to obtain simulation and deduction decision results.

[0062] In the above process, this application uses a constructed inference engine and Markov decision process to simulate and process the fused features, thereby accurately obtaining the simulation and decision results.

[0063] In some embodiments of this application, after simulating and deducing the fused features through a preset reasoning method and decision model to obtain the simulation deduction decision results, the method further includes: displaying the simulation deduction decision results in a user-preset output format, wherein the output format includes visual output of text and images as well as voice output.

[0064] In the above process, this application can display the simulation and decision-making results in a user-preset output format, which can facilitate the visualization output of text and images as well as voice output, making it convenient for users to view.

[0065] In the above Figure 1In the process described, this application acquires data sources of different modalities within the target domain, including urban traffic, education and training, emergency response, autonomous driving, or medical disease analysis. The data sources include at least two of image data, text data, and voice data. Features are extracted from the data sources and fused to obtain fused features, which include a comprehensive feature vector corresponding to all data feature vectors in the data sources. The fused features are then simulated and extrapolated using a pre-defined inference method and decision model to obtain simulation and extrapolation decision results, including inference results and decision schemes. By acquiring data sources of different modalities within the target domain, extracting cross-modal features, and fusing them, the application ultimately combines a decision model and inference to achieve accurate simulation of cognitive processes in complex scenarios, thus achieving the effect of accurately and realistically simulating human cognitive processes.

[0066] The following is combined with Figure 2 The implementation method of simulation and deduction in the embodiments of this application will be described in detail.

[0067] Please refer to Figure 2 , Figure 2 A schematic block diagram of the structure of a simulation and deduction system provided in this application embodiment is shown below. Figure 2 The simulation system shown includes: Data input layer, multimodal fusion layer, cognitive reasoning layer, and output layer; The data input layer is used to acquire data sources of different modalities in the target domain; The multimodal fusion layer is used to extract data features from the data source and perform feature fusion to obtain fused features; The cognitive reasoning layer is used to simulate and deduce the fused features through preset reasoning methods and decision models, and obtain the simulation and deduction decision results; The output layer is used to display the simulation and decision-making results in a user-preset output format.

[0068] Optionally, the data input layer is specifically used for: Obtain initial data sources for different modalities in the target domain; The initial data source is subjected to noise reduction and irrelevant information removal operations to obtain the data source.

[0069] also, Figure 2 The specific methods and steps performed by the system shown can be found in [reference needed]. Figure 1 The method shown will not be elaborated further here.

[0070] Optional, for Figure 2 The cognitive reasoning layer shown is further divided into a hierarchical cognitive reasoning framework, which can be specifically described through... Figure 3The hierarchical cognitive reasoning framework shown is explained in detail.

[0071] Please refer to Figure 3 , Figure 3 A schematic block diagram of a hierarchical cognitive reasoning architecture provided in this application embodiment is shown below. Figure 3 The hierarchical cognitive reasoning framework shown includes: Long-term decision-making layer, working memory layer, and perception layer; The long-term decision-making layer includes a visualization interaction module and an inference engine module; the working memory layer includes a gating mechanism module, a logical unit update module, and a cognitive state modeling module; and the perception layer includes a multimodal data fusion module.

[0072] Specifically: The perception layer is responsible for the initial processing and feature fusion of multimodal information (data source). An adaptive weighting mechanism is used to dynamically adjust the contribution of each modality. ; The formula for calculating fusion features is: ; in: Modality Weight parameter vectors (such as visual, text, and speech). : Context vector, generated by encoding the current environment state. Modality The normalized weights reflect their importance to the current scenario. Modality eigenvectors. : The fused multimodal features. The connection relationship is as follows: Input: Independent features received from the multimodal perception layer. Output: Transmit To the working memory layer.

[0073] Specifically: The working memory layer can model short-term memory and contextual association based on graph neural networks (GNNs), updating node states through message passing. The update formula is: ; in: : No. Layer nodes The hidden state. :node The set of neighbors. :node and Relationship edge features (such as spatiotemporal relationships and semantic similarity). : Learnable weight matrix. Activation function (e.g., ReLU). Connections are as follows: Input: fused features from the receiving perceptual layer. , as initial node features Output: Updated node state Transmitted to the long-term decision-making level.

[0074] Specifically: the long-term decision-making layer can generate an inference engine based on a reinforcement learning framework, define a Markov Decision Process (MDP), and output the decision results through a visual interface. Define the state. Current cognitive state, action For possible decisions, the reward function Assess decision quality. Optimize the policy network using the policy gradient method. : state: ; action: (Discrete or continuous space); Reward function: ; Policy Networks: ; Optimization goal: ; Policy gradient update: ; The connection is as follows: Input: Receives the node state of the working memory layer. Output: Generates actions. It then feeds back into the environment, forming a closed loop.

[0075] Optionally, to improve the robustness of the perception layer, this application proposes an adversarial training method. Specifically, this involves injecting perturbations into the input of the perception layer to enhance the system's robustness against noise and deceptive information. In terms of technical implementation, the input... Add perturbation ,satisfy : ; in, For loss function, For the model, Assigning true labels. Optimizing the model by minimizing adversarial risk: ; in: The upper limit of the disturbance amplitude (e.g.) ). Cross-entropy loss or task-related loss function. Solution method: Iteratively generate adversarial examples using PGD (Projected Gradient Descent). E represents the process of model or data optimization.

[0076] also, Figure 3 The specific methods and steps performed by the architecture shown can be found in [reference needed]. Figure 1 The method shown and Figure 2 The system shown will not be described in detail here.

[0077] Optional, Figure 3 The working memory layer shown can be configured with a dynamic memory update mechanism to simulate the dynamic characteristics of human memory and achieve reasoning simulation. Specifically, this can be achieved through... Figure 4 accomplish.

[0078] Please refer to Figure 4 , Figure 4 A schematic diagram of a dynamic memory update mechanism provided in this application is shown below. Figure 4 The dynamic memory update mechanism shown includes: Forget gates, input gates, and output gates; also includes: memory cell update and hidden state update operations.

[0079] Optional, input gate: The input gate is responsible for receiving information input from the external environment. This information may be multimodal, such as images, text, and speech, but at the memory unit level, it has usually been transformed into a unified feature vector or representation. The input gate is determined based on the current input. and the hidden state of the previous moment Calculate the activation value of the input gate. The input gate determines how much new information can enter the cellular state.

[0080] Forgotten Gate: Also based on and Calculate the activation value of the forget gate. The forget gate determines the old information (i.e., the cell state at the previous moment). How much should be forgotten?

[0081] Output gate: ;according to and Calculate the activation value of the output gate. The output gate determines the current cell state. How much information should be output as the current hidden state? .

[0082] Memory unit update: Combining the results of the input gate and forget gate, along with a candidate cell state (usually obtained through an activation function), (Calculation), update cell state Cellular states are central to LSTM units and are used for storing long-term memory information.

[0083] Hidden status update: ; cell state pass The activation function (sometimes called the output activation function) is multiplied by the output gate. A new hidden state is obtained. .

[0084] in: Learnable weight matrix. Element-wise multiplication (Hadamard product). Sigmoid function, output range Control information is transmitted through proportions.

[0085] also, Figure 4 The specific methods executed by the structure shown can be found in [reference]. Figure 1 The steps shown will not be elaborated further here.

[0086] The previous text passed Figure 1 The simulation method is described below, combined with... Figures 5-6 Describe the apparatus used for simulation and deduction.

[0087] Please refer to Figure 5 This is a schematic block diagram of a simulation and deduction device 300 provided in an embodiment of this application. The device 500 can be a module, program segment, or code on an electronic device. This device 500 is related to the above... Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The various steps involved in the method embodiment, and the specific functions of the device 500, can be found in the description below. To avoid repetition, detailed descriptions are appropriately omitted here.

[0088] Optionally, the device 500 includes: The multimodal data acquisition module 510 is used to acquire data sources of different modalities in a target domain, wherein the target domain includes: urban traffic, education and training, emergency response, autonomous driving or medical disease analysis, and the data source includes at least two of: image data, text data and voice data; The multimodal data fusion module 520 is used to extract data features from the data source and perform feature fusion to obtain fused features, wherein the fused features include a comprehensive feature vector corresponding to all data feature vectors in the data source; The simulation and deduction module 530 is used to perform simulation and deduction on the fused features through a preset reasoning method and decision model, and obtain simulation and deduction decision results, wherein the simulation and deduction decision results include reasoning results and decision schemes.

[0089] Optionally, the simulation and deduction module is specifically used for: The fused features are simulated and deduced using preset rules, machine learning methods, or deep learning methods to obtain the simulation and deduction process; the simulation and deduction process is then analyzed using a decision model to obtain the simulation and deduction decision results.

[0090] Optionally, the device also includes: The display module is used by the simulation and deduction module to simulate and deduce the fused features through preset reasoning methods and decision models, and after obtaining the simulation and deduction decision results, to display the simulation and deduction decision results in a user-preset output format. The output format includes text and image visualization output as well as voice output.

[0091] Optionally, the multimodal data fusion module is specifically used for: The feature vectors of all data in the data source are extracted by a neural network architecture to obtain a feature vector set; all feature vectors in the feature vector set are aligned by a constructed shared space to obtain a feature vector set of the same dimension; all feature vectors in the feature vector set of the same dimension are fused by a preset fusion algorithm to obtain fused features. The fusion algorithm includes concatenation, weighted summation or attention mechanism.

[0092] Optionally, the multimodal data fusion module is specifically used for: An adaptive weighting mechanism is used to dynamically adjust the weights of each data feature in the data source to obtain the weight allocation; the data features in the data source are then weighted and merged according to the weight allocation to obtain the merged features.

[0093] Optionally, the simulation and deduction module is specifically used for: An inference engine is generated based on a pre-defined reinforcement learning decision model, and a Markov decision process is defined according to the inference method. The fused features are simulated and processed based on the inference engine and the Markov decision process to obtain the simulation and decision results.

[0094] Please refer to Figure 6 This is a schematic block diagram of a simulation and deduction device provided in an embodiment of this application. The device may include a memory 610 and a processor 620. Optionally, the device may further include a communication interface 630 and a communication bus 640. This device is similar to the one described above. Figure 1 The method implementation corresponds to this and can be executed. Figure 1 The specific functions of the device involved in the method embodiments can be found in the following description.

[0095] Specifically, memory 610 is used to store computer-readable instructions.

[0096] Processor 620 is used to process readable instructions stored in memory and is capable of executing... Figure 1 Each step in the method.

[0097] The communication interface 630 is used for signaling or data communication with other node devices. For example, it is used for communication with a server or terminal, or for communication with other device nodes, but the embodiments of this application are not limited thereto.

[0098] Communication bus 640 is used to enable direct communication between the above components.

[0099] In this embodiment, the communication interface 630 of the device is used for signaling or data communication with other node devices. The memory 610 can be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 610 can also be at least one storage device located remotely from the aforementioned processor. The memory 610 stores computer-readable instructions, which, when executed by the processor 620, enable the electronic device to perform the aforementioned... Figure 1 The method process is shown. The processor 620 can be used on the device 300 and is used to perform the functions in this application. Exemplarily, the processor 620 described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and the embodiments of this application are not limited thereto.

[0100] This application embodiment also provides a readable storage medium, wherein when the computer program is executed by a processor, it performs the following... Figure 1 The method process executed by the electronic device in the illustrated method embodiment.

[0101] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0102] In summary, this application provides a simulation and deduction method, apparatus, device, storage medium, and product. The method includes: acquiring data sources of different modalities in a target domain, wherein the target domain includes: urban traffic, education and training, emergency response, autonomous driving, or medical disease analysis; and the data sources include: at least two of image data, text data, and voice data; extracting data features from the data sources and performing feature fusion to obtain fused features, wherein the fused features include a comprehensive feature vector corresponding to all data feature vectors in the data sources; and performing simulation and deduction on the fused features using a preset reasoning method and decision model to obtain simulation and deduction decision results, wherein the simulation and deduction decision results include reasoning results and decision schemes. This method can achieve the effect of accurately and realistically simulating human cognitive processes.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0104] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0105] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0107] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

[0108] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A simulation and deduction method, characterized in that, include: Acquire data sources of different modalities in a target domain, wherein the target domain includes: urban traffic, education and training, emergency response, autonomous driving or medical disease analysis, and the data sources include: at least two of image data, text data and voice data; Extract data features from the data source and perform feature fusion to obtain fused features, wherein the fused features include a comprehensive feature vector corresponding to all data feature vectors in the data source; The fused features are simulated and deduced using a preset reasoning method and decision model to obtain simulation and deduction decision results, wherein the simulation and deduction decision results include reasoning results and decision schemes; The step of simulating and deducing the fused features using a preset reasoning method and decision model to obtain simulation and deduction decision results includes: An inference engine is generated based on a pre-defined reinforcement learning decision model, and a Markov decision process is defined according to the inference method. Based on the inference engine and the Markov decision process, the fused features are simulated and processed for decision-making to obtain the simulation and decision-making results.

2. The method according to claim 1, characterized in that, The step of simulating and deducing the fused features using a preset reasoning method and decision model to obtain simulation and deduction decision results includes: The fused features are simulated and deduced using preset rules, machine learning methods, or deep learning methods to obtain the simulation and deduction process; The simulation process is analyzed using the decision model to obtain the simulation decision results.

3. The method according to claim 1 or 2, characterized in that, After simulating and deducing the fused features using a preset reasoning method and decision model to obtain the simulation and deduction decision results, the method further includes: The simulation and decision-making results are displayed according to the user-preset output format, which includes text and image visualization output as well as voice output.

4. The method according to claim 1 or 2, characterized in that, The step of extracting data features from the data source and fusing those features to obtain fused features includes: The feature vectors of all data in the data source are extracted using a neural network architecture to obtain a set of feature vectors; By aligning all feature vectors in the feature vector set in the constructed shared space, a feature vector set of the same dimension is obtained; The fused feature is obtained by fusing all feature vectors in the same dimension feature vector set through a preset fusion algorithm, wherein the fusion algorithm includes concatenation, weighted summation or attention mechanism.

5. The method according to claim 1 or 2, characterized in that, The step of extracting data features from the data source and fusing those features to obtain fused features includes: An adaptive weighting mechanism is used to dynamically adjust the weights of each data feature in the data source to obtain the weight allocation. The data features in the data source are weighted and fused according to the weight allocation to obtain the fused features.

6. A simulation and deduction system, characterized in that, include: Data input layer, multimodal fusion layer, cognitive reasoning layer, and output layer; The data input layer is used to acquire data sources of different modalities in the target domain; The multimodal fusion layer is used to extract data features from the data source and perform feature fusion to obtain fused features; The cognitive reasoning layer is used to simulate and deduce the fused features through preset reasoning methods and decision models, and obtain simulation and deduction decision results; The output layer is used to display the simulation and decision-making results in a user-preset output format; The cognitive reasoning layer is specifically used for: An inference engine is generated based on a pre-defined reinforcement learning decision model, and a Markov decision process is defined according to the inference method. Based on the inference engine and the Markov decision process, the fused features are simulated and processed for decision-making to obtain the simulation and decision-making results.

7. The system according to claim 6, characterized in that, The data input layer is specifically used for: Obtain initial data sources for different modalities in the target domain; The initial data source is subjected to noise reduction and irrelevant information removal operations to obtain the data source.

8. A simulation and deduction device, characterized in that, include: A multimodal data acquisition module is used to acquire data sources of different modalities in a target domain, wherein the target domain includes: urban traffic, education and training, emergency response, autonomous driving or medical disease analysis, and the data source includes at least two of: image data, text data and voice data; A multimodal data fusion module is used to extract data features from the data source and perform feature fusion to obtain fused features, wherein the fused features include a comprehensive feature vector corresponding to all data feature vectors in the data source; The simulation and deduction module is used to simulate and deduce the fused features through preset reasoning methods and decision models, and obtain simulation and deduction decision results, wherein the simulation and deduction decision results include reasoning results and decision schemes; The simulation and deduction module is specifically used for: An inference engine is generated based on a pre-defined reinforcement learning decision model, and a Markov decision process is defined according to the inference method. Based on the inference engine and the Markov decision process, the fused features are simulated and processed for decision-making to obtain the simulation and decision-making results.

9. An electronic device, characterized in that, include: A memory and a processor, the memory storing computer-readable instructions that, when executed by the processor, perform the steps of the method as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, include: A computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-5.

11. A computer program product, characterized in that, The computer program product includes a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-5.

Images