Multimodal intelligent decision-making methods, devices, equipment, and media

Abstract

This application proposes a multimodal intelligent decision-making method, apparatus, device, and medium, relating to the field of artificial intelligence technology and applicable to the fintech and healthcare fields. The method includes: acquiring the speech of a target object; performing speech-to-text conversion on the target object's speech to obtain a target-converted text sequence; encoding the target-converted text sequence to obtain target-converted text features; extracting features from the target object's speech to obtain paralinguistic features; acquiring target environmental information about the target object's environment; extracting features from the target environmental information to obtain environmental features; performing feature enhancement based on the target-converted text features, paralinguistic features, and environmental features to obtain joint enhanced features; acquiring a semantic graph; generating a decision feature vector based on the semantic graph and the joint enhanced features; and performing intent decision-making based on the decision feature vector to obtain the intent decision result. This application can improve the accuracy of perceptual decision-making.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of fintech and healthcare, particularly to a multimodal intelligent decision-making method, device, equipment, and medium. Background Technology

[0002] In the field of artificial intelligence, Vision Language Action (VLA) models can be used for perception and decision-making. For example, in intelligent customer service scenarios in the fintech sector, perception and decision-making based on customer input (such as "I want to open a deposit account") can automatically execute self-service transactions and perform information queries. As another example, in medical image and text interpretation scenarios in the healthcare sector, perception and decision-making based on patient input (such as "Please help me analyze the images") can generate image analysis reports or provide treatment plans.

[0003] In applying VLA models, related technologies typically make perception decisions based on text or speech, but this makes it difficult to effectively capture key information in complex scenarios, thus limiting the accuracy of perception decisions. Summary of the Invention

[0004] The main objective of this application is to propose a multimodal intelligent decision-making method, device, equipment, and medium that can improve the accuracy of perception-based decision-making.

[0005] To achieve the above objectives, a first aspect of this application proposes a multimodal intelligent decision-making method, the method comprising:

[0006] Acquire the target object's speech; wherein the target object's speech is used to represent the target object's intent;

[0007] The speech of the target object is converted into speech to obtain a target converted text sequence. The target converted text sequence is then encoded into text to obtain target converted text features.

[0008] Feature extraction is performed on the speech of the target object to obtain paralinguistic features;

[0009] Obtain target environment information of the environment in which the target object is located, and extract features from the target environment information to obtain environmental features;

[0010] Feature enhancement is performed based on the target transformed text features, the paralinguistic features, and the environmental features to obtain joint enhanced features;

[0011] Obtain semantic graphs, including: speech semantic graphs, environmental semantic graphs, and text semantic graphs;

[0012] Based on the semantic graph and the joint enhanced features, a decision feature vector is generated;

[0013] Intent decision is performed based on the decision feature vector to obtain the intent decision result; wherein, the intent decision result is used to characterize the predicted action or predicted natural language response in response to the intent.

[0014] Optionally, a decision feature vector is generated based on the semantic graph and the joint enhanced features, including:

[0015] Based on the joint enhancement features, the semantic graph is completed to obtain the target semantic graph;

[0016] Feature extraction is performed on the target semantic graph to obtain target fused semantic features;

[0017] The decision feature vector is obtained by fusing the joint enhanced features and the target fusion semantic features.

[0018] Optionally, semantic completion is performed on the semantic graph based on the joint enhancement features to obtain the target semantic graph, including:

[0019] Using a pre-defined cross-modal semantic association analysis model, semantic association analysis is performed on the speech semantic graph, the environment semantic graph, and the text semantic graph to obtain a semantic association weight matrix;

[0020] Based on the joint enhancement features and the semantic association weight matrix, semantic completion is performed on the speech semantic graph and the text semantic graph to obtain the target semantic graph.

[0021] Optionally, the step of extracting features from the target semantic graph to obtain target fused semantic features includes:

[0022] Feature extraction is performed on the text semantic graph and the environment semantic graph to obtain initial fused semantic features;

[0023] The speech semantic graph is subjected to paralinguistic information extraction to obtain paralinguistic weights;

[0024] The initial fused semantic features are semantically adjusted based on the secondary language weights to obtain the target fused semantic features.

[0025] Optionally, the step of performing feature enhancement based on the target transformed text features, the sub-language features, and the environmental features to obtain joint enhanced features includes:

[0026] Based on the target converted text features and the paralinguistic features, feature association is performed to obtain speech enhancement features;

[0027] Based on the environmental features, a preset graph structure is learned to obtain environmental semantic features; wherein, the nodes of the graph structure are sensor data, and the edges of the graph structure are the relationships between the sensor data.

[0028] The joint enhancement features are obtained by highlighting key information based on the speech enhancement features and the environmental semantic features.

[0029] Optionally, the step of highlighting key information based on the speech enhancement features and the environmental semantic features to obtain the joint enhancement features includes:

[0030] The speech enhancement features are concatenated with the environmental semantic features to obtain the initial joint features;

[0031] The initial joint features are extracted by a preset gated loop unit to obtain intermediate joint features;

[0032] The intermediate joint features are processed by a preset attention unit to obtain the joint enhanced features.

[0033] Optionally, the target environmental information includes ambient sound information, ambient light information, and ambient temperature information;

[0034] The step of extracting features from the target environment information to obtain environmental features includes:

[0035] The environmental sound information is processed to distinguish sound sources, thereby obtaining single-source sound information;

[0036] The single-source sound information is encoded to obtain environmental sound features;

[0037] The ambient light information is encoded to obtain ambient light features;

[0038] The ambient temperature information is temperature-encoded to obtain ambient temperature characteristics;

[0039] The environmental features are obtained by fusing the environmental sound features, the environmental light features, and the environmental temperature features.

[0040] To achieve the above objectives, a second aspect of this application proposes a multimodal-based intelligent decision-making device, the device comprising:

[0041] A voice acquisition module is used to acquire the voice of a target object; wherein the voice of the target object is used to represent the intention of the target object.

[0042] The text feature extraction module is used to perform speech conversion on the speech of the target object to obtain a target converted text sequence, and to perform text encoding on the target converted text sequence to obtain target converted text features;

[0043] The sub-language feature extraction module is used to extract features from the speech of the target object to obtain sub-language features;

[0044] The environmental feature extraction module is used to obtain target environmental information of the environment in which the target object is located, and to extract features from the target environmental information to obtain environmental features;

[0045] The feature enhancement module is used to enhance features based on the target converted text features, the sub-language features, and the environmental features to obtain joint enhanced features;

[0046] The graph acquisition module is used to acquire semantic graphs, which include: speech semantic graphs, environmental semantic graphs, and text semantic graphs.

[0047] The decision feature generation module is used to generate a decision feature vector based on the semantic graph and the joint enhanced features;

[0048] The intent decision module is used to make intent decisions based on the decision feature vector to obtain intent decision results; wherein, the intent decision results are used to characterize the predicted action or predicted natural language response in response to the intent.

[0049] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the multimodal-based intelligent decision-making method described in the first aspect.

[0050] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal-based intelligent decision-making method described in the first aspect.

[0051] This application proposes a multimodal intelligent decision-making method, device, electronic device, and storage medium. Besides acquiring the target object's speech, it also acquires target environmental information. Based on the target object's speech, it determines target-converted text features and paralinguistic features, and based on the target environmental information, it determines environmental features. Then, it performs feature enhancement based on the target-converted text features, paralinguistic features, and environmental features to obtain joint enhanced features. These joint enhanced features can express the rich clues contained in the target object's tone, speech rate, and ambient sounds when uttering speech, including paralinguistic and environmental information. To further enhance the semantic association between paralinguistics, text, and environment, it optimizes the joint enhanced features using speech semantic graphs, environmental semantic graphs, and text semantic graphs to obtain a decision feature vector. This decision feature vector can more accurately express the target object's true intention. Finally, it performs intention decision-making based on the decision feature vector to obtain the intention decision result; the intention decision result is used to characterize the predicted action of the response intention or predict the predicted natural language response. In summary, this application, by first generating joint enhanced features based on speech, text, and environmental information and then using semantic graphs to generate decision feature vectors, can effectively capture key information in complex scenes and improve the accuracy of perceptual decision-making.

[0052] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0053] Figure 1 This is a flowchart of the multimodal-based intelligent decision-making method provided in the embodiments of this application;

[0054] Figure 2 yes Figure 1 The flowchart for step 105 in the document;

[0055] Figure 3 yes Figure 2 The flowchart for step 203 in the document;

[0056] Figure 4 yes Figure 1 The flowchart for step 107 in the document;

[0057] Figure 5 yes Figure 4 The flowchart for step 401 in the document;

[0058] Figure 6 yes Figure 4 The flowchart for step 402 in the document;

[0059] Figure 7 This is a block diagram of the module structure of the multimodal-based intelligent decision-making device provided in the embodiments of this application;

[0060] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0064] First, let's analyze some of the terms used in this application:

[0065] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0066] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information and image processing, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0067] Visual-Language-Action (VLA): This model integrates visual perception, language understanding, and action control to form an end-to-end decision-making system. VLA is an extension of multimodal large-scale models in the field of embodied intelligence. By establishing a mapping between visual signals, language commands, and physical actions, it achieves closed-loop decision-making from environmental understanding to behavioral output.

[0068] In the application of Visual Language Action (VLA) models, relying solely on text commands and conventional voice commands for perception and decision-making makes it difficult to fully capture key information in complex scenarios. Text commands typically lack descriptions of the real-time environmental state, and while voice commands can convey semantics, they overlook the rich cues contained in paralinguistic and environmental information such as tone of voice, speech rate, and ambient sounds. For example, in a smart home scenario, when a user says "turn on the light," combining their urgent tone (paralinguistic information) with the dim ambient light (environmental information) would allow the model to execute the command more accurately. Similarly, in industrial robot scenarios, the machine's operating sounds (environmental information) and the operator's emphasized tone (paralinguistic information) can help the model understand the urgency of the command. However, existing VLA models underutilize environmental and paralinguistic information, limiting the accuracy and flexibility of perception and decision-making. Innovative technologies are urgently needed to achieve deep integration and utilization of information.

[0069] Based on this, embodiments of this application propose a multimodal intelligent decision-making method, a multimodal intelligent decision-making device, an electronic device, and a computer-readable storage medium. By fusing environmental and paralinguistic information and achieving efficient processing of multidimensional information, the perception and decision-making capabilities of the VLA model are improved. Specifically, it adopts a method of first generating joint enhanced features based on speech, text, and environmental information, and then generating decision feature vectors with the help of semantic graphs. This method can effectively capture key information in complex scenarios and improve the accuracy of perception and decision-making.

[0070] The multimodal intelligent decision-making method provided in this application can be applied to terminals and servers, or it can be software running on the server. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The software can be an application that implements the multimodal intelligent decision-making method, but it is not limited to the above forms.

[0071] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0072] This application provides a multimodal intelligent decision-making method, a multimodal intelligent decision-making device, an electronic device, and a computer-readable storage medium, which are specifically described through the following embodiments. First, the multimodal intelligent decision-making method in the embodiments of this application is described.

[0073] It should be noted that in each specific embodiment of this application, when it is necessary to process data related to the user's identity or characteristics, such as image data, voice data, and text data, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards.

[0074] Reference Figure 1 , Figure 1 This is an optional flowchart of a multimodal-based intelligent decision-making method provided in the embodiments of this application, which may include, but is not limited to, steps 101 to 108.

[0075] Step 101: Obtain the target object's speech; wherein, the target object's speech is used to represent the target object's intent;

[0076] Step 102: Perform speech conversion on the target object speech to obtain the target converted text sequence, and perform text encoding on the target converted text sequence to obtain the target converted text features;

[0077] Step 103: Extract features from the speech of the target object to obtain paralinguistic features;

[0078] Step 104: Obtain target environment information of the environment in which the target object is located, extract features from the target environment information, and obtain environmental features;

[0079] Step 105: Perform feature enhancement based on the target transformed text features, paralinguistic features, and environmental features to obtain joint enhanced features;

[0080] Step 106: Obtain the semantic graph, which includes: speech semantic graph, environment semantic graph, and text semantic graph;

[0081] Step 107: Generate decision feature vectors based on the semantic graph and joint enhancement features;

[0082] Step 108: Make an intention decision based on the decision feature vector to obtain the intention decision result; wherein, the intention decision result is used to characterize the predicted action of the response intention or the predicted natural language response.

[0083] Steps 101 to 108, as illustrated in this embodiment, involve acquiring not only the target object's speech but also target environment information. Based on the target object's speech, target-converted text features and paralinguistic features are determined, and environmental features are determined based on the target environment information. Then, feature enhancement is performed based on the target-converted text features, paralinguistic features, and environmental features to obtain joint enhanced features. These joint enhanced features can express the rich clues contained in the target object's tone, speech rate, and environmental sounds when uttering speech, including paralinguistic and environmental information. To further enhance the semantic association between paralinguistics, text, and environment, the joint enhanced features are optimized using speech semantic graphs, environmental semantic graphs, and text semantic graphs to obtain a decision feature vector. This decision feature vector can more accurately express the target object's true intention. Finally, intention decision is performed based on the decision feature vector to obtain the intention decision result. The intention decision result is used to characterize the predicted action of the response intention or predict the natural language response. In summary, this application, by first generating joint enhanced features based on speech, text, and environmental information and then generating decision feature vectors using semantic graphs, can effectively capture key information in complex scenes and improve the accuracy of perceptual decision-making.

[0084] For example, in a smart customer service scenario within the fintech field, a customer at the counter might input the message "I want to make a deposit" into the smart customer service system. In addition to this voice message, the smart customer service system also acquires information about the customer's environment, such as ambient sound information including the voices of third parties. Based on the voice message and environmental information, the smart customer service system makes perceptual decisions to arrive at an intent decision result. This intent decision result is used to enable the smart customer service system to automatically execute self-service transactions or perform information inquiries.

[0085] For example, in the medical image and text interpretation scenario within the healthcare field, a patient might input the following into a consultation window into a smart medical assistant: "Please help me analyze the images." In addition to this voice input, the smart medical assistant also gathers information about the client's environment, such as pedestrian sounds and the sounds of medical equipment. Based on the voice input and environmental information, the smart medical assistant performs perceptual decision-making to arrive at an intent decision result. This intent decision result is then used to generate an image analysis report or provide a treatment plan.

[0086] In step 101 of some embodiments, the target object's speech is acquired. The target object's speech is used to represent the target object's intent. For example, if the target object's speech is "I want to make a deposit," then the target object's intent is "make a deposit." As another example, if the target object's speech is "Please help me analyze the image," then the target object's intent is "analyze the image."

[0087] In step 102 of some embodiments, the target speech is first converted to obtain a target converted text sequence, and then the target converted text sequence is encoded to obtain target converted text features. For example, for the speech conversion process, an end-to-end speech recognition model (such as DeepSpeech) can be used to convert the speech signal into a text sequence T_{voice}, and noise reduction processing can be performed using a bidirectional long short-term memory network combined with an attention mechanism to improve the accuracy of speech conversion. For the text encoding process, the input text sequence T_{voice} is segmented, part-of-speech tagging, and lexical reconstruction using a natural language processing toolkit (such as NLTK), and converted into a low-dimensional vector representation using word embedding technology (such as Word2Vec).

[0088] In step 103 of some embodiments, feature extraction is performed on the target speech to obtain paralinguistic features. For example, an array microphone is used to collect the target speech, simultaneously capturing paralinguistic information such as intonation, speech rate, and stress. An acoustic feature extraction model based on a deep convolutional neural network (DCNN) is used to analyze the target speech and extract acoustic features such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency, and short-time energy to characterize paralinguistic information. Let the input speech signal be V, and the extracted paralinguistic features be F_{paralinguistic}, calculated as: F_{paralinguistic} = DCNN(V).

[0089] In step 104 of some embodiments, target environment information of the environment in which the target object is located is obtained, and feature extraction is performed on the target environment information to obtain environmental features. The target environment information includes environmental sound information, environmental light information, and environmental temperature information.

[0090] In one embodiment, step 104, which involves extracting features from the target environment information to obtain environmental features, includes: distinguishing sound sources from environmental sound information to obtain single-source sound information; encoding the single-source sound information to obtain environmental sound features; encoding the ambient light information to obtain ambient light features; encoding the ambient temperature information to obtain ambient temperature features; and fusing the environmental sound features, ambient light features, and ambient temperature features to obtain environmental features.

[0091] For example, multiple types of environmental sensors (such as light sensors, sound sensors, and temperature sensors) are deployed to collect environmental data. For environmental sound information, different sound sources (such as machine operation sounds and the level of human voice noise) are distinguished using voiceprint recognition technology, and multilayer perceptron (MLP) is used to perform dimensionality reduction encoding of the spectral characteristics of environmental sound; light, temperature, and other data are normalized; finally, the extracted environmental feature is F_{environment}.

[0092] In step 105 of some embodiments, feature enhancement is performed based on the target converted text features, paralinguistic features, and environmental features to obtain joint enhanced features. For example, based on timestamps and semantic associations, the text features, paralinguistic features F_{paralinguistic}, and environmental features F_{environment} of the speech-converted text sequence T_{voice} are aligned to obtain joint enhanced features.

[0093] In one embodiment, step 105 may include: acquiring the target object text, wherein the target object speech is used to represent the target object's intent; extracting features from the target object text to obtain object semantic text features; and performing feature enhancement based on the object text features, target-converted text features, paralinguistic features, and environmental features to obtain joint enhanced features. For example, word embedding is performed on the target object text T_{text} to obtain initial text features; a Transformer encoder is used to extract features from the initial text features to obtain object text semantic features F_{text}; based on timestamps and semantic associations, the object text semantic features, the target-converted text features of the speech-converted text sequence T_{voice}, the paralinguistic features F_{paralinguistic}, and the environmental features F_{environment} are aligned to obtain joint enhanced features.

[0094] In one embodiment, reference is made to Figure 2 Step 105 may include:

[0095] Step 201: Perform feature association based on the target converted text features and paralinguistic features to obtain speech enhancement features;

[0096] Step 202: Perform graph structure learning on the preset graph structure based on environmental features to obtain environmental semantic features; wherein, the nodes of the graph structure are sensor data, and the edges of the graph structure are the relationships between sensor data.

[0097] Step 203: Highlight key information based on speech enhancement features and environmental semantic features to obtain joint enhancement features.

[0098] The advantage of the embodiments of steps 201 to 203 above is that by generating speech enhancement features and environmental semantic features separately and then generating joint enhancement features, key information features can be highlighted, which is conducive to improving decision-making accuracy.

[0099] In step 201, the target converted text feature can be a speech-semantic feature vector F_{voice-semantic}, specifically obtained by inputting the speech-converted text sequence T_{voice} into a pre-trained speech-semantic coding model (such as BERT-ASR). The paralinguistic feature F_{paralinguistic} is concatenated with the speech-semantic feature vector F_{voice-semantic}, and then the fused speech enhancement feature F_{voice-enhanced} is further extracted through a Transformer encoder. This speech enhancement feature can reflect the correlation between semantics, emotion, emphasis, and other information in the extracted speech.

[0100] In step 202, different types of environmental sensor data can be constructed into a graph structure, where nodes represent sensor data and edges represent the relationships between sensor data. A graph neural network (GNN) can be used to learn the environmental features F_{environment} and the graph structure. The aim is to learn the spatial and temporal correlation features of the environmental features through the GNN, thereby obtaining the environmental semantic feature vector F_{environment-semantic}.

[0101] In one embodiment, reference is made to Figure 3 Step 203 may include:

[0102] Step 301: Concatenate the speech enhancement features with the environmental semantic features to obtain the initial joint features;

[0103] Step 302: Extract key information from the initial joint features using a preset gated loop unit to obtain intermediate joint features;

[0104] Step 303: Attention processing is performed on the intermediate joint features through a preset attention unit to obtain joint enhanced features.

[0105] Specifically, the voice enhancement feature F_{voice-enhanced} and the environment semantic feature vector F_{environment-semantic} can be fused in series to obtain the initial joint feature F_{initial}. A gated recurrent unit (GRU) combined with an attention mechanism is introduced to construct a feature enhancement module. The weights are dynamically adjusted according to the importance of different information to the current task, highlighting key information features, and the joint enhanced feature F_{enhanced} is obtained. The calculation process is as follows: F_{enhanced} = {GRU - Attention - Enhancement}(F_{initial}).

[0106] The advantage of the embodiments of steps 301 to 303 described above is that it can further dynamically adjust the weights according to the importance of different information to the current intention, highlight the characteristics of key information, and help improve the accuracy of decision-making.

[0107] In one embodiment, step 301 may include concatenating and fusing the voice enhancement feature F_{voice-enhanced}, the environment semantic feature vector F_{environment-semantic}, and the object text semantic feature F_{text} to obtain an initial joint feature F_{initial}.

[0108] In step 106 of some embodiments, a semantic graph is obtained. The semantic graph includes: a speech semantic graph, an environment semantic graph, and a text semantic graph. Specifically, a speech semantic graph can be constructed for paralinguistic information, an environment semantic graph for environment information, and a text semantic graph for text information. The speech semantic graph uses speech recognition text and paralinguistic features as nodes, and determines node relationships through semantic role labeling and sentiment analysis; the environment semantic graph uses environmental entities (such as devices, spatial regions) and environmental states as nodes, and causal relationships and spatial relationships as edges; the text semantic graph is constructed using traditional methods. Knowledge graph embedding technology is used to associate entities in each graph with external knowledge bases to expand semantic information.

[0109] In one example, the process of semantic role labeling mentioned above includes: in the object speech "Please hand me the book on the table", "hand" is the core of the action (predicate), the subject "I" is the agent, the object "book" is the subject (theme), and the location "table" is the location, thus forming an "action-participant" semantic role framework.

[0110] In one example, the sentiment analysis process described above includes: when the voice says "This function is too complicated!" accompanied by a high pitch and rapid speech rate, the sentiment label is "frustration"; while when the voice says "It's so easy to use!" with a slow speed and rising pitch, it is labeled "satisfaction".

[0111] In one example, the causal relationships in the environmental semantic graph include: if "the air conditioner temperature is set to 26°C" causes "the indoor temperature to rise to 25°C", then a causal edge "setting temperature → temperature change" is established; if "the user presses the switch" triggers "the device starts", then the causal edge is represented as "button action → device state change".

[0112] In one example, the spatial relationships in the environmental semantic graph include: if the robot is located 2 meters in front of the shelf, it is defined as "Shelf A - Front Position → Robot"; if the robot leaves the shelf after picking up the goods, it is defined as "Shelf A - Dynamic Detachment → Robot".

[0113] In step 107 of some embodiments, a decision feature vector is generated based on the semantic graph and joint enhancement features.

[0114] In one embodiment, reference is made to Figure 4 Step 107 may include:

[0115] Step 401: Complete the semantic graph based on the joint enhancement features to obtain the target semantic graph;

[0116] Step 402: Extract features from the target semantic graph to obtain target fused semantic features;

[0117] Step 403: Perform feature fusion based on the joint enhanced features and the target fusion semantic features to obtain the decision feature vector.

[0118] In step 401, nodes and / or edges added to the semantic graph can be updated based on the joint augmented features. For example, if the joint augmented features represent "the object's speech is 'placed books'", and there is no direct edge between "entity and shelf" in the context semantic graph, then the implicit relationship "book → placed on → shelf" can be completed using an external knowledge base (such as ConceptNet).

[0119] In one embodiment, reference is made to Figure 5 Step 401 may include:

[0120] Step 501: Using a pre-defined cross-modal semantic association analysis model, perform semantic association analysis on the speech semantic graph, the environment semantic graph, and the text semantic graph to obtain a semantic association weight matrix;

[0121] Step 502: Based on the joint enhancement features and semantic association weight matrix, perform semantic completion on the speech semantic graph and the text semantic graph to obtain the target semantic graph.

[0122] In step 501, a cross-modal semantic association analysis model based on a Multi-Head Graph Attention Network (MHGAT) can be designed, taking three semantic graphs as input. The multi-head attention mechanism learns the semantic association weights between speech, environment, and text information from different perspectives to uncover deep semantic connections. For example, when a voice command mentions "turn off the device," the specific target is determined by combining the device's operating sound in the environment and the device name in the text command. The calculation process is as follows: A = {MHGAT}(G_{voice}, G_{environment}, G_{text}), where G_{voice}, G_{environment}, and G_{text} are the speech semantic graph, environment semantic graph, and text semantic graph, respectively, and A is the semantic association weight matrix. The semantic association weight matrix is ​​used to represent the semantic associations mined from the semantic graphs.

[0123] In step 502, the semantics of speech, environment and text information are fused and completed based on the discovered semantic relationships.

[0124] The benefit of the embodiments of steps 501 to 502 described above is that by supplementing the semantic graph with the discovered semantic relationships, the semantic representation capability of the semantic graph is improved, which is conducive to improving the accuracy of decision-making.

[0125] In step 402, feature extraction is performed on the target semantic graph to obtain target fused semantic features. The target semantic graph includes a speech semantic graph, an environmental semantic graph, and a text semantic graph.

[0126] In one embodiment, reference is made to Figure 6 Step 402 may include:

[0127] Step 601: Extract features from the text semantic graph and the environment semantic graph to obtain initial fused semantic features;

[0128] Step 602: Extract paralinguistic information from the speech semantic graph to obtain paralinguistic weights;

[0129] Step 603: Adjust the initial fused semantic features according to the secondary language weights to obtain the target fused semantic features.

[0130] Specifically, by supplementing implicit conditions in the environment (such as device operating status) into the speech semantic graph and text semantic graph, and combining paralinguistic information to adjust the focus of semantic understanding, a complete and accurate multimodal semantic representation can be formed.

[0131] The advantage of the embodiments of steps 601 to 603 described above is that they enable cross-modal semantic deep association and fusion, which is beneficial to improving decision-making accuracy.

[0132] In one example, a multimodal intelligent decision-making method may include: extracting initial semantic vectors of speech / text / environment; generating joint representations through a gated fusion network (including sub-language weights); embedding inference of implicit relations based on the semantic graph and completing the graph; inputting the completed semantics into a control policy network and outputting action commands (such as torque distribution of robotic arm joints).

[0133] In step 403, feature fusion is performed based on the joint enhanced features and the target fusion semantic features to obtain the decision feature vector. For example, a Transformer-Fusion() fusion decision network is constructed, using the joint enhanced features F_{enhanced} and the target fusion semantic features F_{Semantic-Fusion-Result} as input. The network performs weighted fusion of the input features through a multi-head self-attention mechanism, learning the contribution weights of speech, environment, and text information to the decision in different scenarios, and generating the decision feature vector F_{decision}. The calculation process is as follows:

[0134] F_{decision}={Transformer-Fusion}(F_{enhanced},F_{Semantic-Fusio n-Result}).

[0135] In step 108 of some embodiments, an intent decision is made based on the decision feature vector to obtain an intent decision result. The intent decision result is used to characterize the predicted action of the response intent or to predict the natural language response.

[0136] For example, the decision feature vector F_{decision} is input into the perception and decision-making modules of the VLA model. In smart home scenarios, combined with visual information (such as indoor conditions captured by cameras), the model generates precise actions to control home appliances based on fused instructions; in industrial scenarios, it drives robots to perform actions that conform to environmental conditions and intent requirements; in intelligent question-answering scenarios, it generates natural language responses that include environmental and emotional factors, achieving precise perception and decision-making based on multi-dimensional fused information.

[0137] In one embodiment, multimodal intelligent decision-making may further include a multi-dimensional feedback collection step and a model optimization iteration step.

[0138] In the multi-dimensional feedback collection step: Multi-dimensional feedback information is collected during the decision-making process, including the deviation between the action execution result and the expected instruction, the object's satisfaction with the response content, the impact assessment of environmental changes on the decision, and the effectiveness of paralinguistic information utilization. Simultaneously, performance indicators of the model at each stage when processing various types of information are recorded, such as the accuracy of semantic association analysis, the effectiveness of feature extraction, and the adaptability of the decision to the environment.

[0139] In the model optimization iteration steps: feedback information is combined with the original multi-source information data and used as training samples to optimize the model. The backpropagation algorithm is used to update the parameters of the multi-source information preprocessing module, feature extraction module, semantic association analysis module, and fusion decision network. Based on the feedback, the parameters of the paralinguistic feature extraction model, the environmental information encoding method, the threshold of the semantic association analysis algorithm, and the attention weights of the fusion decision network are adjusted to continuously improve the model's ability to process speech-text commands with fused environmental and paralinguistic information and its decision-making accuracy, achieving continuous optimization and iteration of the model.

[0140] In summary, the present application achieves at least the following beneficial effects: 1. Comprehensive information perception: By integrating environmental and paralinguistic information, the model can acquire richer and more detailed scene cues, making up for the shortcomings of traditional instruction information and achieving comprehensive perception of complex environments. 2. Precise and intelligent decision-making: Deep semantic association and fusion enable the model to comprehensively consider multi-dimensional information, understand the true intent behind instructions, and make more accurate decisions that meet actual needs, reducing misjudgments and omissions. 3. Flexible scene adaptation: The feedback optimization mechanism enables the model to dynamically adjust information processing strategies according to environmental changes and object expression characteristics in different scenarios, enhancing adaptability and flexibility in diverse scenarios and improving user experience.

[0141] Please see Figure 7 This application also provides a multimodal-based intelligent decision-making device, which can implement the above-mentioned multimodal-based intelligent decision-making method. Figure 7 This application provides a module structure block diagram of a multimodal-based intelligent decision-making device, which includes:

[0142] The voice acquisition module 701 is used to acquire the voice of the target object; wherein, the voice of the target object is used to represent the intention of the target object;

[0143] The text feature extraction module 702 is used to perform speech conversion on the speech of the target object to obtain the target converted text sequence, and to perform text encoding on the target converted text sequence to obtain the target converted text features;

[0144] The para-language feature extraction module 703 is used to extract features from the speech of the target object to obtain para-language features;

[0145] The environmental feature extraction module 704 is used to obtain target environmental information of the environment in which the target object is located, and to extract features from the target environmental information to obtain environmental features;

[0146] The feature enhancement module 705 is used to enhance features based on target transformed text features, paralinguistic features and environmental features to obtain joint enhanced features;

[0147] The graph acquisition module 706 is used to acquire semantic graphs, which include: speech semantic graphs, environmental semantic graphs, and text semantic graphs.

[0148] The decision feature generation module 707 is used to generate decision feature vectors based on the semantic graph and joint enhanced features;

[0149] The intent decision module 708 is used to make intent decisions based on the decision feature vector and obtain intent decision results; wherein, the intent decision results are used to characterize the predicted action of the response intent or the predicted natural language response.

[0150] In one embodiment, the environmental feature extraction module 704 is specifically used for: distinguishing sound sources from environmental sound information to obtain single-source sound information; encoding the single-source sound information to obtain environmental sound features; encoding the ambient light information to obtain ambient light features; encoding the ambient temperature information to obtain ambient temperature features; and fusing the environmental sound features, ambient light features, and ambient temperature features to obtain environmental features.

[0151] In one embodiment, the feature enhancement module 705 is specifically used for: performing feature association based on target converted text features and paralinguistic features to obtain speech enhancement features; performing graph structure learning on the environmental features based on a preset graph structure to obtain environmental semantic features; wherein, the nodes of the graph structure are sensor data, and the edges of the graph structure are the association relationships between sensor data; and highlighting key information based on the speech enhancement features and environmental semantic features to obtain joint enhancement features.

[0152] In one embodiment, the decision feature generation module 707 is specifically used for: completing the semantic graph based on the joint enhancement features to obtain a target semantic graph; extracting features from the target semantic graph to obtain target fused semantic features; and fusing features based on the joint enhancement features and the target fused semantic features to obtain a decision feature vector.

[0153] It should be noted that the specific implementation of this multimodal-based intelligent decision-making device is basically the same as the specific implementation of the multimodal-based intelligent decision-making method described above, and will not be repeated here.

[0154] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned multimodal intelligent decision-making method. This electronic device can be any intelligent terminal, including tablet computers, in-vehicle computers, etc.

[0155] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0156] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0157] The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and called and executed by the processor 801 to execute the multimodal intelligent decision-making method of the embodiments of this application.

[0158] The 803 input / output interface is used to implement information input and output.

[0159] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0160] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0161] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0162] This application also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described multimodal intelligent decision-making method.

[0163] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0164] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0165] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0167] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0168] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0169] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A multi-modal based intelligent decision making method, characterized in that, The method includes: Acquire the target object's speech; wherein the target object's speech is used to represent the target object's intent; The speech of the target object is converted into speech to obtain a target converted text sequence. The target converted text sequence is then encoded into text to obtain target converted text features. Feature extraction is performed on the speech of the target object to obtain paralinguistic features; Obtain target environment information of the environment in which the target object is located, and extract features from the target environment information to obtain environmental features; Feature enhancement is performed based on the target transformed text features, the paralinguistic features, and the environmental features to obtain joint enhanced features; Obtain semantic graphs, including: speech semantic graphs, environmental semantic graphs, and text semantic graphs; Using a pre-defined cross-modal semantic association analysis model, semantic association analysis is performed on the speech semantic graph, the environment semantic graph, and the text semantic graph to obtain a semantic association weight matrix; Based on the joint enhancement features and the semantic association weight matrix, semantic completion is performed on the speech semantic graph and the text semantic graph to obtain the target semantic graph. Feature extraction is performed on the text semantic graph and the environment semantic graph to obtain initial fused semantic features; The speech semantic graph is subjected to paralinguistic information extraction to obtain paralinguistic weights; The initial fused semantic features are semantically adjusted according to the secondary language weights to obtain the target fused semantic features; Based on the joint enhanced features and the target fusion semantic features, feature fusion is performed to obtain a decision feature vector; Intent decision is performed based on the decision feature vector to obtain the intent decision result; wherein, the intent decision result is used to characterize the predicted action or predicted natural language response in response to the intent.

2. The method according to claim 1, characterized in that, Feature extraction is performed on the speech of the target object to obtain paralinguistic features, including: An acoustic feature extraction model based on a deep convolutional neural network is used to analyze the speech of the target object, extracting Mel frequency cepstral coefficients, fundamental frequency, and short-time energy to obtain the paralinguistic features.

3. The method according to claim 1, characterized in that, The step of performing feature enhancement based on the target converted text features, the sub-language features, and the environmental features to obtain joint enhanced features includes: Based on the target converted text features and the paralinguistic features, feature association is performed to obtain speech enhancement features; Based on the environmental features, a preset graph structure is learned to obtain environmental semantic features; wherein, the nodes of the graph structure are sensor data, and the edges of the graph structure are the relationships between the sensor data. The joint enhancement features are obtained by highlighting key information based on the speech enhancement features and the environmental semantic features.

4. The method according to claim 3, characterized in that, The target-converted text features are speech-semantic feature vectors, obtained by inputting the speech-converted target-converted text sequence into a pre-trained speech-semantic coding model. The step of associating the target-converted text features and the paralinguistic features to obtain speech enhancement features includes: The sub-language features are concatenated with the speech semantic feature vector, and the fused speech enhancement features are extracted by a Transformer encoder.

5. The method according to claim 3, characterized in that, The step of highlighting key information based on the speech enhancement features and the environmental semantic features to obtain the joint enhancement features includes: The speech enhancement features are concatenated with the environmental semantic features to obtain the initial joint features; The initial joint features are extracted by a preset gated loop unit to obtain intermediate joint features; The intermediate joint features are processed by a preset attention unit to obtain the joint enhanced features.

6. The method according to any one of claims 1 to 4, characterized in that, The target environmental information includes environmental sound information, ambient light information, and ambient temperature information; The step of extracting features from the target environment information to obtain environmental features includes: The environmental sound information is processed to distinguish sound sources, thereby obtaining single-source sound information; The single-source sound information is encoded to obtain environmental sound features; The ambient light information is encoded to obtain ambient light features; The ambient temperature information is temperature-encoded to obtain ambient temperature characteristics; The environmental features are obtained by fusing the environmental sound features, the environmental light features, and the environmental temperature features.

7. A multimodal intelligent decision-making device, characterized in that, The device includes: A voice acquisition module is used to acquire the voice of a target object; wherein the voice of the target object is used to represent the intention of the target object. The text feature extraction module is used to perform speech conversion on the speech of the target object to obtain a target converted text sequence, and to perform text encoding on the target converted text sequence to obtain target converted text features; The sub-language feature extraction module is used to extract features from the speech of the target object to obtain sub-language features; The environmental feature extraction module is used to obtain target environmental information of the environment in which the target object is located, and to extract features from the target environmental information to obtain environmental features; The feature enhancement module is used to enhance features based on the target converted text features, the sub-language features, and the environmental features to obtain joint enhanced features; The graph acquisition module is used to acquire semantic graphs, which include: speech semantic graphs, environmental semantic graphs, and text semantic graphs. The decision feature generation module is used to perform semantic association analysis on the speech semantic graph, the environment semantic graph, and the text semantic graph using a preset cross-modal semantic association analysis model to obtain a semantic association weight matrix; perform semantic completion on the speech semantic graph and the text semantic graph based on the joint enhancement features and the semantic association weight matrix to obtain a target semantic graph; extract features from the text semantic graph and the environment semantic graph to obtain initial fused semantic features; extract paralinguistic information from the speech semantic graph to obtain paralinguistic weights; perform semantic adjustment on the initial fused semantic features based on the paralinguistic weights to obtain target fused semantic features; and perform feature fusion based on the joint enhancement features and the target fused semantic features to obtain a decision feature vector. The intent decision module is used to make intent decisions based on the decision feature vector to obtain intent decision results; wherein, the intent decision results are used to characterize the predicted action or predicted natural language response in response to the intent.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images