Decision-making method and device based on multi-modal collaborative optimization, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of robot agent decision making, financial science and technology, medical health and the like, and discloses a decision making method, device, equipment and medium based on multi-mode collaborative optimization. And corresponding features are generated through an image encoder, a text encoder, an auditory encoder and a tactile encoder. Visual and text coding features are optimized through positive and negative sample image pairs and text pairs, fusion weights are distributed based on cross-modal similarity, all modal features are weighted by applying the weights, collaborative decision features are generated, and finally an action decision group is generated through an action decoder. According to the method, the decision feature generation mode is optimized by fusing the multi-modal perception information, and the execution precision and reliability are improved. Through cross-modal similarity optimization and reasonable weight distribution, the accuracy and efficiency of task execution in a complex environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a decision-making method, apparatus, device and storage medium based on multimodal collaborative optimization. Background Art

[0002] Existing embodied intelligence technologies face numerous challenges when performing tasks, particularly at the perception and decision-making levels. Traditional image and text encoders have limitations when processing complex sensory data, resulting in inaccurate understanding of the environment and, in turn, impacting task execution accuracy. Existing technologies also have limitations in multimodal information fusion and decision execution, making it difficult to effectively integrate data from diverse sensory sources, hindering robot performance in complex environments. Furthermore, the output structure of existing motion decoders is overly simplified, failing to accurately describe key parameters in task execution, severely impacting task success.

[0003] In the area of ​​decision-making for robotic agents, existing technologies suffer from a disconnect between perception and decision execution. Traditional image encoders and text encoders fail to adequately address perception issues in complex environments, making it difficult for robots to accurately understand the details of the environment when performing complex tasks (such as path planning, object grasping, etc.). For example, robots cannot accurately identify obstacles in complex scenes or subtle changes in grasped objects, which directly affects the robot's decision-making ability and execution accuracy. At the same time, existing cross-modal information fusion methods are unable to effectively integrate multi-source perception data (vision, hearing, touch, and text), resulting in the generated decision features lacking sufficient accuracy and reliability, limiting the robot's ability to adapt to complex environments. Therefore, improving perception accuracy and the ability to fuse cross-modal information has become the key to improving the robot's intelligent decision-making capabilities.

[0004] In the field of healthcare, especially in intelligent medical devices and robot-assisted treatment, the challenges faced by existing technologies are mainly reflected in the accurate extraction of sensory data and the fusion of multimodal data. Medical equipment requires accurate image analysis, voice recognition and tactile feedback to support decision-making, but existing image and text encoders are often unable to handle complex and changing medical environments. For example, it is difficult to recognize details in medical images, and the ambiguity in voice commands may also lead to misunderstandings, which affects the accuracy of robots in assisting treatment. In addition, existing technologies have limited capabilities in integrating different sensory data (such as vision, voice, touch, etc.), resulting in medical robots being unable to make optimal decisions when performing complex tasks (such as precise operations, pathological analysis, etc.), thereby affecting the treatment effect. Therefore, improving perception accuracy and decision-making accuracy is the key to improving the performance of intelligent medical equipment.

[0005] In the field of FinTech, especially in automated financial services and intelligent customer support systems, existing technologies face significant challenges in data processing and decision execution. Financial data is diverse and complex, and existing image and text encoders struggle to accurately extract valuable features from massive amounts of data. This makes it particularly difficult to accurately interpret information or produce data biases, especially when processing complex customer requests, compliance checks, or risk assessments. Furthermore, when it comes to multimodal data fusion, traditional methods fail to effectively integrate multiple data sources, such as vision, speech, and text, resulting in insufficient precision in the decisions generated by the system. For example, when interpreting customer needs, customer service robots may provide incorrect service due to an inability to effectively understand the customer's voice or text instructions. Enhancing cross-modal data fusion and decision-making accuracy will significantly improve the accuracy of financial intelligence systems and enhance the customer experience. Summary of the Invention

[0006] The main purpose of the present invention is to provide a decision-making method, device, equipment and storage medium based on multimodal collaborative optimization, aiming to solve the technical problem that the existing technology cannot effectively integrate data features from different modalities in multimodal perception and decision execution, resulting in insufficient decision-making accuracy and execution effect in complex environments, and cannot meet the requirements of high-precision task execution.

[0007] To achieve the above objectives, the present invention provides a decision-making method based on multimodal collaborative optimization, comprising:

[0008] Acquire visual perception data, text instruction data, sound perception data, and tactile perception data, and generate visual coding features, text coding features, auditory coding features, and tactile coding features through an image encoder, a text encoder, an auditory encoder, and a tactile encoder, respectively;

[0009] Constructing positive and negative sample image pairs and positive and negative sample text pairs for the visual perception data and the text instruction data respectively, and optimizing the visual encoding features and the text encoding features based on the positive and negative sample image pairs and the positive and negative sample text pairs to generate optimized visual features and optimized text features;

[0010] Determining a first cross-modal similarity between the optimized visual feature and the auditory coding feature, a second cross-modal similarity between the optimized visual feature and the tactile coding feature, and a third cross-modal similarity between the optimized visual feature and the optimized textual feature;

[0011] assigning a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight based on the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity, respectively;

[0012] Applying the sound modality fusion weight, the tactile modality fusion weight, and the text modality fusion weight to weight the auditory coding features, the tactile coding features, and the optimized text features, respectively, to generate weighted auditory features, weighted tactile features, and weighted text features;

[0013] fusing the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate collaborative decision-making features;

[0014] The collaborative decision features are processed by an action decoder to generate an action decision group.

[0015] Furthermore, to achieve the above-mentioned purpose, the present invention provides a decision-making device based on multimodal collaborative optimization, comprising:

[0016] A perception data acquisition module is used to acquire visual perception data, text instruction data, sound perception data, and tactile perception data, and generate visual coding features, text coding features, auditory coding features, and tactile coding features through an image encoder, a text encoder, an auditory encoder, and a tactile encoder, respectively;

[0017] a feature optimization module, configured to construct positive and negative sample image pairs and positive and negative sample text pairs for the visual perception data and the text instruction data, respectively, and optimize the visual encoding features and the text encoding features based on the positive and negative sample image pairs and the positive and negative sample text pairs to generate optimized visual features and optimized text features;

[0018] a similarity calculation module, configured to determine a first cross-modal similarity between the optimized visual feature and the auditory coding feature, a second cross-modal similarity between the optimized visual feature and the tactile coding feature, and a third cross-modal similarity between the optimized visual feature and the optimized textual feature;

[0019] a weight assignment module, configured to assign a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight based on the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity, respectively;

[0020] A weighted feature generation module, configured to apply the sound modality fusion weight, the tactile modality fusion weight, and the text modality fusion weight to weight the auditory coding features, the tactile coding features, and the optimized text features, respectively, to generate weighted auditory features, weighted tactile features, and weighted text features;

[0021] A feature fusion module, configured to fuse the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate collaborative decision-making features;

[0022] The action decoding module is used to process the collaborative decision features through an action decoder to generate an action decision group.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a decision-making program based on multimodal collaborative optimization stored in the memory and executable on the processor. When the decision-making program based on multimodal collaborative optimization is executed by the processor, the steps of the decision-making method based on multimodal collaborative optimization as described above are implemented.

[0024] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a decision-making program based on multimodal collaborative optimization is stored. When the decision-making program based on multimodal collaborative optimization is executed by a processor, the steps of the decision-making method based on multimodal collaborative optimization as described above are implemented.

[0025] Beneficial effects: The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as robot intelligent body decision-making, financial technology, and medical health. It discloses a decision-making method, device, equipment, and medium based on multimodal collaborative optimization, including: obtaining visual perception data, text instruction data, sound perception data, and tactile perception data, and using an image encoder, a text encoder, an auditory encoder, and a tactile encoder to generate corresponding features; optimizing visual coding features and text coding features based on constructed positive and negative sample image pairs and text pairs; allocating different modal fusion weights based on cross-modal similarity, and applying weights to perform weighted processing on each modal feature; generating collaborative decision features through feature fusion, and then generating an action decision group through an action decoder. The present invention improves execution accuracy and reliability by effectively fusing multimodal perception information and optimizing the generation method of decision features. By optimizing cross-modal similarity and reasonably allocating fusion weights, it ensures that the role of each modal feature in decision-making is maximized, solving the problem that each modal feature cannot fully collaborate in the prior art, thereby achieving more accurate and efficient task execution in a complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0027] Figure 1 Schematic diagram of an application environment of a decision-making method based on multimodal collaborative optimization in one embodiment of the present invention;

[0028] Figure 2 This is a flow chart of an embodiment of a decision-making method based on multimodal collaborative optimization according to the present invention;

[0029] Figure 3 Schematic diagram of functional modules of a preferred embodiment of a decision-making device based on multimodal collaborative optimization of the present invention;

[0030] Figure 4A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0031] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] The decision-making method based on multimodal collaborative optimization provided by the embodiment of the present invention can be applied in the following situations: Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can obtain visual perception data, text instruction data, sound perception data, and tactile perception data from the user terminal, and use an image encoder, a text encoder, an auditory encoder, and a tactile encoder to generate corresponding features; optimize the visual encoding features and text encoding features based on the constructed positive and negative sample image pairs and text pairs; assign different modal fusion weights based on cross-modal similarity, and apply the weights to weighted processing of each modal feature; generate collaborative decision features through feature fusion, and then generate an action decision group through an action decoder. The present invention improves execution accuracy and reliability by effectively fusing multimodal perception information and optimizing the generation method of decision features. By optimizing cross-modal similarity and reasonably allocating fusion weights, the role of each modal feature in decision-making is maximized, solving the problem of insufficient coordination of each modal feature in the prior art, thereby achieving more accurate and efficient task execution in complex environments. The user terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server terminal can be implemented using an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0034] See also Figure 2 , Figure 2 This is a flowchart of an embodiment of a decision-making method based on multimodal collaborative optimization provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here.

[0035] like Figure 2 As shown, the decision-making method based on multimodal collaborative optimization proposed in the present invention includes the following steps:

[0036] S10, acquiring visual perception data, text instruction data, sound perception data, and tactile perception data, and generating visual coding features, text coding features, auditory coding features, and tactile coding features through an image encoder, a text encoder, an auditory encoder, and a tactile encoder, respectively;

[0037] In this embodiment, acquiring visual perception data, text command data, sound perception data, and tactile perception data is a preliminary step in realizing a multimodal perception system. Visual perception data is usually acquired by image acquisition devices such as cameras and lidars, providing environmental image information. Collection is performed using high-resolution cameras or depth cameras, which operate under various lighting and weather conditions to ensure the stability of visual perception. Visual data can provide robots with information about the shape, size, and position of objects, which is crucial for performing tasks such as path planning and object recognition. The acquisition of image data requires efficient and stable equipment to adapt to different working scenarios and dynamic changes in the environment.

[0038] Textual instruction data is crucial information for robots to interact with external systems or users. Typically, textual instructions are written in natural language and may include commands, task descriptions, or operational instructions. These instructions need to be decoded by text processing models and converted into machine-executable actions. In practical applications, textual data contains rich grammatical, semantic, and contextual information, which machines must parse using natural language processing (NLP) modules. By converting this semantic information, robots can accurately understand and execute complex tasks.

[0039] Sound perception data captures sound signals from the environment through devices such as microphones. Sound perception is crucial for robots to understand voice commands, detect sound changes, and identify unusual sound signals. In practical applications, sound data not only helps robots recognize commands but also serves as a crucial signal for environmental monitoring. For example, in autonomous navigation systems, sound perception helps robots respond to commands or warning sounds, enhancing their environmental adaptability. Sound data is typically processed using speech recognition technology to distinguish between voice commands and background noise.

[0040] Tactile perception data is acquired through devices such as tactile sensors and pressure sensors. These sensors record physical feedback generated by contact with objects, such as pressure, displacement, and hardness. Tactile sensing data is particularly important when robots perform delicate manipulation tasks, such as grasping and assembly. Tactile perception allows robots to accurately determine an object's shape, mass, and surface characteristics. Accurate sensor feedback is key to ensuring operational stability and precision.

[0041] Image encoders, text encoders, auditory encoders, and tactile encoders encode different types of sensory data, converting them into unified feature vectors. Image encoders, typically implemented using convolutional neural networks (CNNs), extract spatial features (such as edges, textures, and shapes) from input images. These features enable robots to model the visual environment and identify objects and obstacles. Text encoders process input text data using word embedding techniques and models like Transformers, converting natural language commands into semantic features that the robot can use to perform actions. Auditory encoders extract time-frequency features from audio data using techniques like spectral analysis and temporal convolution, enabling the recognition of sound changes, voice commands, and unusual noises. Tactile encoders, based on tactile sensor data, convert perceived physical feedback into machine-understandable numerical values. These encoders transform data from different modalities into digital features, enabling robots to make decisions and perform actions based on multimodal data.

[0042] In specific implementations, the image encoder uses a convolutional neural network (CNN) to extract features from image data. CNNs extract features layer by layer through a hierarchical structure, effectively identifying complex features within images, such as edges, textures, and shapes. This structure is capable of processing high-dimensional image data and adapting to varying lighting conditions, weather conditions, and dynamic scene changes. For text data processing, a transformer-based model is used to convert input text into high-dimensional feature vectors, which are then semantically parsed using a multi-layer neural network, enabling the robot to understand complex instructions and task descriptions. Sound data is processed using spectral analysis and a temporal convolutional network, converting audio signals into time-frequency features. This helps the robot recognize voice commands or detect audio changes in the environment. Tactile perception data is acquired through tactile sensors, and a force feedback parsing module extracts physical contact information and generates a numerical representation that can be used for decision-making.

[0043] Example: In an object grasping scenario, a robot requires a multimodal perception system to achieve precise manipulation of the target object. By acquiring visual, acoustic, tactile, and textual command data, the robot gains a comprehensive understanding of the current environment and the properties of the target object. The image encoder processes the visual data to extract the object's shape, size, and relative position to the environment, ensuring the robot can accurately locate the target object. Acoustic data helps the robot detect dynamic changes in the environment, such as the movement of other objects or potential obstacles. Tactile data enables the robot to detect the hardness, surface texture, and stability of the target object, preventing damage or dropping during the grasping process. Textual command data provides additional guidance, ensuring the robot understands the semantics of the command and performs the appropriate action. All of this data is generated through the corresponding encoders (image, auditory, tactile, and textual), ultimately enabling the robot to acquire multidimensional perception capabilities for complex environments. Leveraging these optimized features, the robot can dynamically adjust its operational strategy during grasping tasks, improving execution success rate and task robustness.

[0044] In the healthcare sector, medical robots often need to process multiple types of sensory data, such as visual images, auditory signals, tactile feedback, and text commands. For example, during surgery, a robot uses an image encoder to analyze image data of the lesion site, accurately locate the lesion, and provide surgical guidance. Furthermore, tactile feedback (via tactile sensory data) helps the robot sense the force of contact with the patient's tissue during surgery, avoiding excessive manipulation or inappropriate pressure, and minimizing harm to the patient. Auditory sensory data can be used to identify important acoustic signals during surgery, such as equipment operating status and patient body sounds, enabling the robot to adjust its operating methods in real time. Using text command data, the robot can also receive instructions from the doctor and generate a matching action plan. After converting this sensory data into features using the corresponding encoders, the robot can make real-time decisions and optimize operations based on the specific medical task, effectively improving the accuracy and safety of medical procedures.

[0045] In the FinTech sector, robots must process complex trading data and market information to make fast and accurate investment decisions. By acquiring visual, acoustic, tactile, and textual command data, robots can obtain real-time financial market information from multiple dimensions. For example, image encoders can process stock market graphs to help robots identify market trends. Auditory data can be used to monitor market sentiment in real time, such as by analyzing market sentiment changes through audio data from news media or social networks. Tactile data can be used to simulate sensory feedback from financial transactions, enhancing robots' decision-making capabilities in simulated trading environments. Textual command data is primarily used to receive market analysis reports, stock market news, or trading instructions and convert them into decision-making features, enabling robots to make more intelligent investment decisions based on multimodal data. After generating corresponding features from this sensory data through encoders, robots can comprehensively analyze information from all these sources to optimize investment decisions and reduce the risk of misjudgment or lag associated with a single data source.

[0046] This embodiment enables robots to make efficient decisions in complex and dynamic environments through the comprehensive integration of visual, textual, audio, and tactile data. Through optimized coding features, robots can simultaneously process multiple types of information, ensuring enhanced environmental perception when performing tasks. Visual data provides the robot with environmental image information, textual data provides instructions and task descriptions, audio data helps the robot understand changes in environmental sound, and tactile data provides physical contact feedback. The integration of various sensory data enhances the robot's understanding of the environment and enables the robot to perform tasks with greater precision and adaptability.

[0047] S20, constructing positive and negative sample image pairs and positive and negative sample text pairs for the visual perception data and the text instruction data respectively, and optimizing the visual coding features and the text coding features based on the positive and negative sample image pairs and the positive and negative sample text pairs to generate optimized visual features and optimized text features;

[0048] In this embodiment, visual perception data typically includes image data from a camera or other visual sensor. These images typically contain objects, background, and other relevant information in the scene. Textual instruction data includes text messages from the user or control system, used to instruct the robot or system to perform specific tasks. This perception data needs to be processed by an encoder to be converted into feature representations that can be used for further analysis and decision-making.

[0049] The generation of visual encoding features is achieved through an image encoder. This encoder, which can be based on a convolutional neural network (CNN) architecture, converts image data into a low-dimensional feature vector that effectively captures important information in the image. For complex scenes or images rich in detail, the image encoder can extract edges, shapes, colors, and other visual features. The output of the image encoder is a feature vector representing the image content, which will be fused with data from other modalities in subsequent steps.

[0050] The generation of text encoding features is achieved through a text encoder. Typically based on a recurrent neural network (RNN), a long short-term memory (LSTM), or a transformer network, a text encoder can extract semantic information from text data. After word segmentation, embedding, and encoding, the text encoder converts the text data into a feature vector reflecting the key semantic and structural features of the text. These text encoding features serve as the basis for subsequent processing and are used for comparison and optimization with other types of data features.

[0051] After generating these initial features, constructing positive and negative pairs is a key step. In visual perception data, constructing positive and negative image pairs is based on image similarity. Positive pairs typically refer to images with similar objects, scenes, or features, while negative pairs refer to images with significant differences or no correlation. In this way, the model can learn to distinguish between different visual information, thereby improving the expressive power of the image encoder.

[0052] Similarly, the positive and negative text pairs in the text instruction data are constructed based on the semantic relevance of the text. Positive text pairs typically contain similar or related semantic content, while negative text pairs contain irrelevant or opposing text content. This construction method enables the text encoder to better understand semantic similarity and distinguish irrelevant information during the optimization process.

[0053] Based on these positive and negative image pairs and positive and negative text pairs, the optimization of visual and text encoding features can be performed through contrastive learning. By calculating the similarity difference between positive and negative image pairs and positive and negative text pairs, the model parameters are optimized to make the features of positive pairs closer and the features of negative pairs more separated. In this way, the model can better capture the potential relationship between image and text data and effectively optimize in the feature space.

[0054] Ultimately, the optimized visual encoding features and text encoding features will generate optimized visual features and optimized text features. These optimized features can not only more accurately represent the key information in the input data, but also provide a more accurate foundation for subsequent multimodal fusion and decision-making tasks.

[0055] In actual implementation, the construction of positive and negative sample pairs can be achieved through clustering or contrastive learning strategies. For example, positive samples of image pairs can be selected through similarity-based clustering methods, while negative samples are generated by randomly selecting samples from the dataset that are different from the target image. Similarly, for text data, positive sample text pairs can be selected by calculating semantic similarity or using related text in the labeled dataset, while negative samples can be selected from irrelevant or opposite text. During the optimization process, the feature similarity of positive and negative samples is calculated using the contrastive loss function, and the parameters of the image encoder and text encoder are continuously adjusted using the gradient descent method to improve the similarity of positive sample pairs and reduce the similarity of negative sample pairs. In specific implementations, triple loss or contrastive loss functions can be used to optimize the performance of the model.

[0056] Example: In a robotic agent decision-making scenario, the robot needs to determine the position and shape of an object using visual perception data, obtain operational instructions using textual command data, and adapt to the environment using sound and tactile perception data. In grasping tasks, the robot can accurately identify the target object, understand and execute instructions using optimized visual and textual features, and avoid misgrasping caused by environmental noise or ambiguous instructions. The optimized image and text features provide the robot with more precise perception and operational strategies, thereby improving the success rate of the task.

[0057] In healthcare, medical robots use optimized visual and text features to accurately identify lesions during surgery, understand the doctor's instructions, and adapt to changes in the patient's body shape. These optimized features not only help the robot perform surgeries with precision but also enable it to quickly adjust its strategy in the event of unexpected situations, ensuring efficient and safe surgery.

[0058] In the fintech sector, when processing investment decisions, the system uses optimized image and text features to better analyze market intelligence, understand analytical reports, and respond quickly. Optimized features enable the system to extract key information from multiple sources, providing more reliable data support for investment decisions.

[0059] Through the above steps, this embodiment effectively optimizes visual and text encoding features, giving the robot stronger understanding and reasoning capabilities in multimodal perception and decision-making tasks. The optimized features can more accurately represent environmental information and task instructions, thereby improving the system's operational accuracy and reliability in complex scenarios.

[0060] S30, determining a first cross-modal similarity between the optimized visual feature and the auditory coding feature, a second cross-modal similarity between the optimized visual feature and the tactile coding feature, and a third cross-modal similarity between the optimized visual feature and the optimized text feature;

[0061] In this embodiment, the calculation of cross-modal similarity is key to achieving the fusion and optimization of data from different modalities. Cross-modal similarity measures the correlation and consistency between different types of data (such as visual, auditory, tactile, and text). In the current processing step, optimizing the similarity calculation between visual features and features from other modalities (such as auditory, tactile, and text) is the core task of multimodal data processing.

[0062] Optimized visual features are derived from visual perception data processed by an image encoder. These features typically represent key image information, such as object shape, color, and texture, in the form of vectors. Through further optimization, these visual features can more accurately represent key visual information in an image. These optimized visual features serve as the basis for cross-modal similarity calculations with features from other modalities.

[0063] Auditory coding features are generated by processing sound perception data using an auditory encoder. These features reflect key information in the audio signal, such as speech content and ambient sounds. Optimizing auditory coding features helps improve the accuracy of speech recognition and sound classification tasks. In cross-modal similarity calculations, auditory features are compared with optimized visual features to determine the degree of correlation between the two.

[0064] Tactile features are extracted from tactile perception data using a tactile encoder. They typically contain information about the intensity, shape, and surface texture of contact with an object. Tactile features provide feedback to robots during environmental interactions, which is crucial for executing actions. During cross-modal similarity calculations, tactile features are compared with optimized visual features to assess the relationship between vision and tactile perception.

[0065] Text features are extracted from text instruction data using a text encoder and typically contain semantic information and sentence structure. In multimodal tasks, textual information can provide robots with critical information such as task instructions and target descriptions. By calculating cross-modal similarity with optimized visual features, the system can determine the relationship between visual and textual information, thereby optimizing task execution accuracy and effectiveness.

[0066] Cosine similarity is a standard method commonly used to calculate cross-modal similarity. This method measures the similarity between two vectors by calculating the angle between them. The value range is -1 to 1, with larger values ​​indicating greater similarity and lower values ​​indicating less similarity.

[0067] For the first cross-modal similarity (the similarity between the optimized visual features and the auditory coding features), the calculation process is: first extract the feature vectors of the optimized visual features and the auditory coding features, and then use the cosine similarity formula to calculate the similarity value between them.

[0068] For the second cross-modal similarity (the similarity between the optimized visual features and the tactile coding features), the calculation process is similar to the first cross-modal similarity. First, the feature vectors of the optimized visual features and the tactile coding features are extracted, and then calculated using the cosine similarity formula.

[0069] The third cross-modal similarity (the similarity between the optimized visual features and the optimized text features) also follows the same calculation process. First, the feature vectors of the optimized visual features and the optimized text features are extracted, and their relationship is calculated through cosine similarity.

[0070] The calculation result of each similarity is a numerical value, which represents the similarity between different modalities, and can be used for subsequent tasks such as weight allocation and feature fusion.

[0071] In actual implementation, optimizing the similarity calculation between visual features and other modal features can be accomplished using a variety of deep learning models. Visual feature extraction is usually performed through convolutional neural networks (CNN) or more advanced visual models (such as ResNet, VGG). For the extraction of auditory features, models based on convolutional neural networks (CNN) or recurrent neural networks (RNN) can be used to process audio data. The extraction of tactile features usually relies on deep learning models or sensor data processing models for analysis. The extraction of text features can be accomplished through Transformer or BERT models to ensure that semantic information is fully extracted.

[0072] When calculating cross-modal similarity, the following steps can be performed: extract and optimize feature vectors of visual features, auditory encoding features, tactile encoding features, and optimized text features. Use the standard cosine similarity calculation method to calculate the similarity between visual features and other modal features. Specifically, the similarity value is obtained by calculating the angle between the two vectors. A larger value indicates a higher similarity, and a smaller value indicates a greater difference. To ensure the accuracy of the system, the extraction models of visual features and other modal features can be adjusted and optimized based on specific tasks and actual application scenarios, and more suitable algorithms and techniques can be used for feature processing and optimization.

[0073] Example: In a grasping task, a robot agent needs to comprehensively determine how to perform the task using sensory data from four modalities: vision, hearing, touch, and text instructions. First, the robot uses an image encoder to extract the object's visual features and a text encoder to obtain the target task instructions. Simultaneously, the auditory and tactile encoders provide sensory data about ambient sound and object contact, respectively. These multimodal features are optimized to ensure that the visual features accurately represent the object's shape, position, and size, the text features accurately convey the meaning of the task instructions, and the auditory and tactile features provide real-time environmental feedback. The robot then calculates cross-modal similarity between the optimized visual features and features from other modalities (such as hearing, touch, and text) to assess the relationship between the different modalities. Based on this, the robot understands the contribution of each modality to the decision-making process and makes decisions based on the weights of each modality. Finally, the system calculates weighted features, fuses them, and generates collaborative decision-making features to guide the robot in performing specific grasping operations. This process ensures that the robot can grasp objects efficiently and accurately in complex environments, thereby improving the success rate of grasping.

[0074] In healthcare, particularly in surgical robotics, the system fuses data from four modalities: vision, hearing, touch, and text, to enhance the robot's understanding of the surgical scene. A visual encoder extracts image features from surgical videos, while a text encoder interprets surgical instructions and the surgeon's voice commands. The auditory encoder and tactile encoder capture feedback from the surgeon's voice and changes in the patient's position, respectively. The system calculates cross-modal similarities between these modal features to analyze the correlations between visual, auditory, and tactile information. This allows the system to integrate diverse environmental information to provide precise guidance to the surgical robot. For example, visual features help the robot identify the surgical area, tactile information helps the robot perceive force feedback during surgery, and auditory information captures the surgeon's instructions and voice prompts. Ultimately, these fused collaborative decision-making features ensure the robot performs precise surgical procedures according to the surgeon's instructions, reducing the risk of errors and improper operation.

[0075] In the FinTech sector, especially in automated trading systems, systems must process large amounts of market data, news, economic reports, and other information. The system first uses a visual encoder to analyze stock market image data, a text encoder to parse financial news and market commentary, and then leverages auditory and tactile sensory data to collect real-time sound signals and sensor data related to market changes. The system then calculates cross-modal similarities between data from various modalities to assess the correlation between market data, news, and sensor data. This enables the system to comprehensively assess market trends based on diverse information sources and make efficient trading decisions. For example, by fusing visual and textual features, the system can quickly identify market trends, while tactile and auditory feedback enables rapid response to unexpected events. Ultimately, the fused collaborative decision-making features provide the system with precise decision-making information, helping financial trading platforms optimize investment strategies and improve the accuracy and efficiency of trading decisions.

[0076] This embodiment, through cross-modal similarity calculation, can provide a more accurate feature representation for multimodal systems, enabling different types of data (such as vision, hearing, touch, and text) to be effectively combined, thereby improving the system's performance in complex tasks. The calculated cross-modal similarity can provide reliable support for subsequent multimodal fusion, decision-making processes, and task execution.

[0077] S40, assigning a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight based on the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity, respectively;

[0078] In this embodiment, the calculated cross-modal similarity is first processed, mainly for optimizing the similarity values ​​between visual features and other modal features (such as auditory coding features, tactile coding features and optimized text features). These similarity values ​​reflect the strength of association between the modalities. In the process of calculating these similarities, the cosine similarity formula has been used. Cosine similarity converts the feature vectors of each pair of modalities into numerical values, indicating their degree of similarity in the feature space. The first cross-modal similarity, the second cross-modal similarity and the third cross-modal similarity obtained represent the similarity between vision and hearing, vision and touch, and vision and text, respectively.

[0079] Next, these similarity values ​​are used to assign fusion weights for the different modalities. The core purpose of this process is to assign a weight to each modality based on its correlation with the visual modality, ensuring that in the subsequent decision-making process, the features of the different modalities are weighted according to their importance in the decision. The weights are assigned using a standardized method, one common method being the softmax function. The softmax function converts similarity values ​​into weights and ensures that the sum of all weights is 1, giving each modality an appropriate influence. The higher the similarity, the greater the weight; the lower the similarity, the smaller the weight. In this way, the system can reasonably allocate resources based on the importance of each modality in the decision-making process, thereby improving the accuracy of decisions.

[0080] The actual weight assignment process takes the calculated cross-modal similarity values ​​as input and applies the softmax function to convert them into fusion weights. These weights are then used in the subsequent weighted feature generation process. Ultimately, the system can fuse the weighted features of different modalities to generate a collaborative decision feature for subsequent decision generation.

[0081] In practice, the system calculates the fusion weight of each modality based on the similarity between the different modalities. In a specific example, assuming that the similarity between the visual modality and the tactile modality is high, while the similarity between the text modality and the visual modality is low, the system will assign corresponding weights based on these similarity values.

[0082] For example, in a robotic grasping task, the calculated first cross-modal similarity (similarity between vision and hearing) is 0.6, the second cross-modal similarity (similarity between vision and touch) is 0.85, and the third cross-modal similarity (similarity between vision and text) is 0.3. Using these similarity values, the system will apply a softmax function to normalize them.

[0083] Assume that after softmax normalization, the weights obtained are:

[0084] Sound modality fusion weight (auditory modality related to vision): 0.25;

[0085] Tactile modality fusion weight (tactile modality related to vision): 0.45;

[0086] Text modality fusion weight (text modality related to vision): 0.30.

[0087] These weights represent the relative importance of each modality in the final decision-making process. During the system's weighting process, the visual and tactile modalities have higher weights, which means that in grasping tasks, the system will pay more attention to the characteristics of the visual and tactile modalities, thereby improving the accuracy and reliability of grasping tasks. The text modality has a lower weight, indicating that its contribution to the current grasping task is relatively small. Through this weighting process, the system can flexibly adjust the influence of each modality to ensure that in different task scenarios, the decision-making process can fully utilize the most relevant modal features and optimize the accuracy and efficiency of task execution.

[0088] This embodiment assigns a reasonable weight to each modality based on similarity, ensuring that the information influence of different modalities is reasonable during the decision-making process. In this way, the system can dynamically adjust based on the relevance of modal features, improving the accuracy of decision results. The weighted fusion of different modal features will result in more accurate results, avoiding excessive or insufficient information during the decision-making process, thereby improving the adaptability and intelligence of multimodal systems when handling complex tasks.

[0089] S50, applying the sound modality fusion weight, the tactile modality fusion weight, and the text modality fusion weight to weight the auditory coding features, the tactile coding features, and the optimized text features, respectively, to generate weighted auditory features, weighted tactile features, and weighted text features;

[0090] In this embodiment, the features of each modality are weighted by the corresponding modality fusion weight, ensuring that the contribution of each modality's features is adjusted according to its importance. The core of the weighting operation is to generate weighted features by multiplying the encoded features of each modality with its corresponding fusion weight. This process is crucial to ensuring that the contributions of different modalities in the final decision are balanced.

[0091] First, the sound modality fusion weight is multiplied by the auditory encoding feature to calculate the weighted auditory feature. This means that through this weighting operation, the system emphasizes the importance of auditory perception in the current task. If the sound modality fusion weight is high, the auditory feature will play a greater role in the final decision.

[0092] Next, the tactile modality fusion weights are multiplied by the tactile encoding features to generate weighted tactile features. This weighted processing of tactile modalities ensures that the influence of tactile perception on decision-making is reflected in tasks involving physical interaction or contact. By adjusting the weights of tactile modalities, the system can optimize the contribution of tactile perception based on task requirements.

[0093] Finally, the text modality fusion weights are multiplied by the optimized text features to generate weighted text features. In a task, if text information (such as instructions and descriptions) is crucial for task execution, the weight of the optimized text features will be increased accordingly, ensuring that the influence of text instructions is properly considered in the decision-making process.

[0094] The goal of this series of operations is to enable the system to flexibly adjust the influence of each modality on the final decision based on its importance in the task. The resulting weighted auditory features, weighted tactile features, and weighted text features provide sufficient modality information for subsequent feature fusion and decision making.

[0095] Example: In a robotic grasping task, the system can effectively improve the robot's environmental perception by weighting auditory, tactile, and textual modalities. For example, suppose the robot is performing a grasping task, and the grasping target is located in a complex environment surrounded by noise or obstacles. In this case, the acoustic modality may provide critical spatial positioning information, the tactile modality provides data on grasping strength and stability, and the textual modality provides information about instructions. By weighting these different perceptual features, the system can ensure that important perceptual information is prominently considered during the decision-making process, thereby ensuring the successful execution of the grasping task.

[0096] In healthcare, similar weighting can be applied to patient monitoring and diagnostic systems. For example, among the data collected by multimodal sensors, visual data is used to monitor patient expressions and facial changes, audio data is used to record the patient's voice or breathing, tactile data is used to perceive the patient's physiological feedback, and text data provides the doctor's instructions and the patient's description. By weighting these modal data, the system can adjust the influence of the modality according to different medical needs, improving diagnostic accuracy and the efficiency of medical services.

[0097] In the financial sector, by weighting customer behavioral data (such as transaction data, interaction records, and voice messages), the system can better understand customer needs and behavior patterns, thereby providing personalized services and risk assessments. For example, in the credit assessment process, text data (such as questionnaires completed by customers) and audio data (such as customer voice interactions) may provide important information. By weighting these different modalities of data, the system can more accurately assess a customer's credit risk, thereby making more reasonable credit decisions.

[0098] This embodiment weights the features of different modalities, allowing the system to flexibly adapt to different task scenarios. In specific scenarios, certain modalities may be given greater weight. For example, in a visual navigation task, the visual and tactile modalities have higher weights, while in a voice command task, the text modality has a higher weight. In this way, the system can automatically adjust its perception strategy based on the characteristics of the task, optimizing the final decision-making effect.

[0099] S60, fusing the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate collaborative decision-making features;

[0100] In this embodiment, the goal is to fuse weighted features from different modalities to generate the final collaborative decision-making features. The core of this process is the integration of multimodal data to ensure that information from different perceptual channels can be fully utilized in subsequent decision-making. First, the weighted auditory features, tactile features, text features and optimized visual features are fused to generate a unified feature set. The "weighted features" here mean that the features of each modality have been adjusted according to their importance or weight before being input into the fusion module.

[0101] The key steps in the fusion process are feature concatenation, dimension alignment, and feature fusion. Concatenation combines weighted features from different modalities into a unified feature vector, while dimension alignment ensures consistent data dimensions across all modalities, enabling effective fusion in subsequent steps. Finally, feature fusion combines information from each modality to generate a single feature representation encompassing all modalities. This representation reflects the combined influence of multiple perceptual channels on decision making.

[0102] Through this fusion, the system can comprehensively consider the information perceived by each modality, improving the accuracy and robustness of its decisions. This is especially true in complex environments, where different perception modalities may provide complementary information. A single modality may not be able to fully convey all key information. Therefore, the fused features can more comprehensively represent the state of the environment and task.

[0103] Example: In a robotic grasping task, the robot needs to use visual, tactile, and auditory information to determine the location and state of an object and execute the grasping action based on text instructions. Suppose the robot receives a command: "Grab the apple on the table." During execution, the visual modality provides the robot with image data of the table and apple, the tactile modality helps the robot determine the contact force between its fingers and the apple, and the auditory modality may provide acoustic cues about the apple's placement. The textual modality provides specific instructions for the task. By weighting and fusing this modal information, the system can generate a fused feature that reflects the combined understanding of visual, tactile, auditory, and textual information. In this case, the robot can determine the apple's location and state based on visual and tactile information, determine whether the apple is moving based on acoustic information, and determine the target of the grasping action based on the textual instruction. Ultimately, the generated collaborative decision-making feature guides the robot in executing the grasping action, ensuring accuracy and stability.

[0104] In healthcare, multiple sensors collect patient data, such as visual (facial expression detection), tactile (tactile feedback), auditory (voice data), and text (doctor's instructions or symptom descriptions). By integrating this data, the system can more accurately assess the patient's health status and provide a diagnosis or treatment recommendation based on the comprehensive information.

[0105] In the financial sector, the system generates multimodal feature representations of customer behavior by analyzing their transaction behaviors (visual, tactile, textual, and audio data). By integrating this multi-dimensional information, the system can better assess customer needs, predict customer behavior, and provide personalized financial products or services.

[0106] This embodiment fuses weighted features from multiple modalities, allowing the system to fully consider information from different sensory channels during the decision-making process, enhancing its understanding and ability to respond to complex tasks. The fused features not only improve the accuracy of decision-making, but also enhance the system's adaptability in changing environments, especially during task execution, enabling more precise responses based on more comprehensive sensory input. This approach is particularly suitable for tasks that require the integration of multiple information, including vision, hearing, touch, and language, in multimodal perception and decision-making scenarios.

[0107] S70: Process the collaborative decision features through an action decoder to generate an action decision group.

[0108] In this embodiment, the generated collaborative decision features are converted into specific action decision groups through the action decoder. Collaborative decision features are high-dimensional feature representations that integrate multimodal perceptual information, which provides a comprehensive understanding of the environment and tasks through different perceptual channels (visual, auditory, tactile, and textual). The role of the action decoder is to extract specific action information from these features and generate task-related decision outputs.

[0109] The core function of the action decoder is to take collaborative decision features as input, process them, and output a specific set of action decisions. This process is similar to how a decoder generates action sequences from environmental states in reinforcement learning. Collaborative decision features themselves contain information from multiple perceptual modalities. Therefore, the action decoder needs to process this multimodal information, understand the relationships between them, and generate the desired action output based on the task requirements.

[0110] The action decoder first receives input collaborative decision-making features. These features have been weighted and fused through the previous steps and contain information from multiple modalities. Based on these input features, the decoder extracts the parameters required for task execution. The output of the action decoder typically includes action parameters and action timing information, which directly guide task execution.

[0111] In practice, the collaborative decision features are first passed as input to the decoder. To process this multimodal information, the action decoder's input layer preprocesses it to ensure effective fusion of features from different modalities. Data from each modality, such as visual and textual data, is weighted and processed at different levels of the decoder before being integrated through an appropriate network structure (such as a fully connected layer, LSTM, or Transformer layer) to generate the final action decision output.

[0112] The decoder's output typically consists of multiple components, depending on the specific requirements of the task. For example, in a robotic grasping task, the action decision set might include grasping force, grasping angle, arm motion trajectory, and execution sequence. To accommodate different tasks, the action decoder's architecture can be adjusted as needed, and the output decision parameters can include different types of control information, such as position, velocity, and acceleration.

[0113] During the model training phase, the action decoder continuously adjusts its parameters based on feedback. Using reinforcement learning (RL) or supervised learning (SL) methods, the decoder's parameters are optimized based on the success rate of each task to improve execution accuracy and efficiency. For example, if the robot fails to grasp a task, the decoder receives negative feedback and adjusts its parameters through a backpropagation mechanism, enabling more accurate decisions for the next grasping task.

[0114] Ultimately, the action decision set generated by the action decoder provides the robot or agent with precise execution instructions, ensuring the successful completion of the task according to the predetermined objectives. This mechanism of optimizing decisions through multimodal fusion can significantly improve the robot or agent's ability to perform tasks in complex environments, especially in complex scenarios involving multiple perception and control requirements.

[0115] In actual applications, after the action decoder receives the collaborative decision-making features, it will first parse these multimodal features to extract key operation parameters. In the robot grasping task, the decoder will extract relevant data from multiple modalities such as visual features, tactile feedback, sound information and text instructions, and generate detailed action instructions based on this information. For example, in a grasping task, visual perception data may contain information about the position, shape and distance of the object, tactile perception data provides information such as the roughness or softness of the object's surface, sound perception data may reflect the interaction sound between the object and the environment, and text instructions provide specific requirements for the operation (such as grasping force, action timing, etc.).

[0116] The action decoder fuses these multimodal data and analyzes the relationship between different features. For grasping tasks, the decoder generates specific operation instructions through the rules learned by the model. First, the decoder generates an estimate of the grasping force based on the visual data, which usually depends on the size, shape and material of the object. For example, for a heavier object, the decoder may generate a higher grasping force, and for a lighter or fragile object, a lower grasping force instruction. In addition, the decoder will determine the grasping angle based on the shape and position of the object to ensure that the robot's arm or grasping tool can contact the object and grasp it at the most appropriate angle.

[0117] Regarding timing, the decoder generates a specific action sequence and time window based on the action timing identifier, ensuring that each action is executed in the predetermined order and completed at the correct time. For example, the decoder might instruct the robot to first approach an object when grasping it, then perform the actual grasping action after determining the object's location. Parsing the timing identifier helps the robot precisely control the execution time of each action, avoiding conflicts or errors between actions.

[0118] After analyzing these features, the decoder generates a detailed set of action instructions, forming an action decision group. This action decision group contains all the information about gripping force, angle, execution sequence, and timing control. The robot will perform specific tasks based on these instructions. During implementation, these instructions are transmitted to the robot control system, which adjusts its execution units (such as the robotic arm and gripper) accordingly to ensure each step is precise and accurate.

[0119] Therefore, the core role of the action decoder is to parse specific operational instructions from multiple sensory modalities and control the robot to perform complex tasks based on these instructions. In this way, the robot can not only understand the characteristics of objects in the environment, but also generate appropriate operational strategies based on multimodal data to ensure efficient and accurate execution of tasks.

[0120] This embodiment processes collaborative decision-making features through an action decoder, transforming high-dimensional features from multimodal fusion into specific, actionable action decisions. This process effectively improves the robot or intelligent agent's ability to execute tasks in complex environments, enabling the agent to make reasonable decisions based on diverse sensory information. By optimizing the decoder, the system can continuously adjust and improve its execution strategy, increasing the success rate and efficiency of task execution.

[0121] The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as robot intelligent body decision-making, financial technology, and medical health. A decision-making method, device, equipment, and medium based on multimodal collaborative optimization are disclosed, including: obtaining visual perception data, text instruction data, sound perception data, and tactile perception data, and using an image encoder, a text encoder, an auditory encoder, and a tactile encoder to generate corresponding features; optimizing visual encoding features and text encoding features based on constructed positive and negative sample image pairs and text pairs; allocating different modal fusion weights based on cross-modal similarity, and applying the weights to perform weighted processing on each modal feature; generating collaborative decision features through feature fusion, and then generating an action decision group through an action decoder. The present invention improves execution accuracy and reliability by effectively fusing multimodal perception information and optimizing the generation method of decision features. By optimizing cross-modal similarity and rationally allocating fusion weights, it ensures that the role of each modal feature in decision-making is maximized, solving the problem that each modal feature cannot fully collaborate in the prior art, thereby achieving more accurate and efficient task execution in a complex environment.

[0122] In one embodiment, the above step S20 includes:

[0123] S201, selecting images of the same scene from different perspectives from the visual perception data to construct positive sample image pairs, and selecting images of different scenes to construct negative sample image pairs;

[0124] S202, inputting the positive sample image pair into an image encoder to obtain a positive sample visual coding feature group, and inputting the negative sample image pair into an image encoder to obtain a negative sample visual coding feature group;

[0125] S203, determining a similarity value of visual positive sample features in the positive sample visual coding feature group, and a similarity value of visual negative sample features in the negative sample visual coding feature group;

[0126] S204, constructing an image contrast loss function based on the visual positive sample feature similarity value and the visual negative sample feature similarity value, and updating parameters of the image encoder by minimizing the image contrast loss function;

[0127] S205, using the updated image encoder to process the visual perception data to generate optimized visual features;

[0128] S206, selecting semantically similar texts from the text instruction data to construct positive sample text pairs, and selecting semantically irrelevant texts to construct negative sample text pairs;

[0129] S207, inputting the positive sample text pair into a text encoder to obtain a positive sample text encoding feature group, and inputting the negative sample text pair into a text encoder to obtain a negative sample text encoding feature group;

[0130] S208, determining the text positive sample feature similarity value in the positive sample text encoding feature group, and the text negative sample feature similarity value in the negative sample text encoding feature group;

[0131] S209, constructing a text contrast loss function based on the text positive sample feature similarity value and the text negative sample feature similarity value, and updating the parameters of the text encoder by minimizing the text contrast loss function;

[0132] S210: Use the updated text encoder to process the text instruction data to generate optimized text features.

[0133] In this embodiment, when processing visual perception data and text instruction data, it is first necessary to construct positive and negative sample pairs in order to optimize the generated visual encoding features and text encoding features. First, from the visual perception data, different perspective images of the same scene are selected to construct positive sample image pairs. This is because the diversity of visual data is crucial for extracting reliable features. By selecting image pairs with similar physical features and visual attributes, it can provide strong support for subsequent learning. In addition, selecting images from different scenes to construct negative sample image pairs helps the training system distinguish features in different scenes, thereby improving the model's ability to recognize scene differences.

[0134] The positive and negative image pairs are then fed into the image encoder for processing. For the positive image pairs, the image encoder extracts visual encoding features and generates a positive visual encoding feature set. Similarly, for the negative image pairs, the encoder generates a negative visual encoding feature set. The positive visual encoding feature set represents visual features from similar scenes, while the negative visual encoding feature set represents features from different scenes. This comparison helps the model learn the differences between scenes.

[0135] Next, we need to calculate the similarity between the visual features of the positive samples within the positive sample visual encoding feature set, and the visual features of the negative samples within the negative sample visual encoding feature set. This similarity is calculated by taking the cosine similarity between the feature vectors of each pair of samples. Positive samples have higher similarity values, while negative samples have lower similarity values, which helps the training system better distinguish different scene features.

[0136] After obtaining the similarity values, an image contrast loss function is constructed based on these similarity values. The goal of the image contrast loss function is to minimize the difference in similarity between positive and negative samples, thereby optimizing the parameters of the image encoder. By minimizing the loss function, the image encoder gradually learns visual features that can distinguish different scenes, making the features of positive samples closer and the features of negative samples farther apart.

[0137] Once the parameters of the image encoder are optimized, the image encoder can process more visual perception data and generate more accurate optimized visual features.

[0138] For text instruction data, we first need to construct positive and negative sample text pairs. Positive sample pairs consist of semantically similar text, while negative sample pairs consist of semantically unrelated text. The selection of positive samples helps ensure that the model can learn the inherent connections between semantically similar texts, while negative samples are used to train the model to distinguish semantic differences.

[0139] Positive and negative text pairs are input into a text encoder for processing, generating a positive text encoding feature set and a negative text encoding feature set. The positive text encoding feature set represents the encoded representation of semantically similar text, while the negative text encoding feature set represents the encoded representation of semantically unrelated text. Through this processing, the text encoder can learn the characteristics of semantically similar and dissimilar texts and effectively encode them.

[0140] Next, we need to calculate the similarity between the positive and negative text encoding feature sets. By calculating the cosine similarity of the text features, we can see that the positive samples have a higher similarity value, while the negative samples have a lower similarity value. This similarity value provides a strong basis for subsequent training.

[0141] Based on these similarity values, a text contrast loss function can be constructed. The text contrast loss function works similarly to the image contrast loss function, which is to optimize the parameters of the text encoder by minimizing the similarity difference between positive and negative samples.

[0142] Finally, the text encoder processes the text instruction data based on the updated parameters to generate optimized text features. These optimized text features better express the relationship between text information and environmental perception, providing more accurate support for subsequent decision-making.

[0143] This embodiment constructs positive and negative sample pairs for visual perception data and text instruction data, and optimizes visual coding features and text coding features through image encoders and text encoders respectively, so that the system can better extract and distinguish features of different scenes and semantics. By optimizing image and text coding features, the accuracy and reliability of the perception system are significantly improved, especially in complex scenes, it can effectively identify object features and reduce understanding errors, thereby providing robots or intelligent systems with more accurate environmental perception capabilities. This feature optimization not only enhances the system's multimodal fusion capabilities, but also improves the system's ability to understand task instructions, allowing robots to perform complex tasks more efficiently, especially in application scenarios involving multimodal input.

[0144] In one embodiment, the above step S30 includes:

[0145] S301, extracting a visual feature vector of the optimized visual feature and an auditory feature vector of the auditory coding feature;

[0146] S302, determining a first cosine similarity value between the visual feature vector and the auditory feature vector, and using the first cosine similarity value as a first cross-modal similarity;

[0147] S303, extracting a tactile feature vector of the tactile coding feature;

[0148] S304: determining a second cosine similarity value between the visual feature vector and the tactile feature vector, and using the second cosine similarity value as a second cross-modal similarity;

[0149] S305, extracting the text feature vector of the optimized text feature;

[0150] S306 : Determine a third cosine similarity value between the visual feature vector and the text feature vector, and use the third cosine similarity value as a third cross-modal similarity.

[0151] In this embodiment, a visual feature vector is first extracted from the optimized visual features. This process is typically performed using a visual encoder. This visual feature vector represents the key features extracted from the visual perception data. After training, it can effectively capture salient information in the image, such as objects, colors, and textures in the scene. After extracting the visual feature vector, a corresponding auditory feature vector is then extracted from the auditory encoding features. This process is performed using an auditory encoder, and the auditory feature vector represents characteristics such as pitch, rhythm, and speech content in the audio data.

[0152] Next, the similarity between the visual and auditory feature vectors is calculated. This is typically done using cosine similarity. Cosine similarity is a common method for measuring the directional similarity between two vectors, measuring their similarity by calculating the angle between them. The similarity value generated in this step is the first cross-modal similarity, which indicates the strength of the relationship between the visual and auditory modalities.

[0153] The same process applies to calculating the similarity between tactile coding features and visual features. First, a tactile feature vector is extracted from the tactile coding features. This feature vector represents the data collected by the tactile sensor, such as the object's hardness and temperature. Next, the similarity between the visual feature vector and the tactile feature vector is calculated. This similarity value is the second cross-modal similarity, reflecting the correlation between the visual and tactile modalities.

[0154] Finally, a text feature vector is extracted from the optimized text features. The text encoder converts the text data into a feature vector, which contains semantic, syntactic, and other information about the text. Then, by calculating the cosine similarity between the visual feature vector and the text feature vector, we obtain the third cross-modal similarity, which represents the similarity between the visual and textual modalities.

[0155] Each cross-modal similarity calculation process can effectively measure the correlation between different modalities and provide the necessary numerical basis for subsequent modal fusion and weighted allocation.

[0156] This embodiment effectively measures the information relevance between modalities by calculating the similarity between visual, auditory, tactile, and textual features. By calculating cross-modal similarity, the system can dynamically adjust the contribution of different modalities to the final decision, thereby improving the accuracy and effectiveness of modal fusion. This not only enhances the system's adaptability when processing multimodal data, but also provides a reliable basis for subsequent modal weighting and optimization, thereby improving the system's execution accuracy and efficiency in complex tasks.

[0157] In one embodiment, the above step S40 includes:

[0158] S401: Applying an exponential function and a temperature parameter to the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity to generate a first transformed similarity value, a second transformed similarity value, and a third transformed similarity value;

[0159] S402, determining a sum of the first transformation similarity value, the second transformation similarity value, and the third transformation similarity value;

[0160] S403 : Dividing the first transformed similarity value, the second transformed similarity value, and the third transformed similarity value by the sum value respectively to generate a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight.

[0161] In this embodiment, the first, second, and third cross-modal similarities are further processed based on the cross-modal similarity values ​​calculated previously. The first, second, and third cross-modal similarities represent the similarities between vision and hearing, vision and touch, and vision and text modalities, respectively. In order to adjust the weights of different modalities in the subsequent modal fusion process, these three similarity values ​​are first subjected to exponential function transformation and temperature parameter processing.

[0162] Specifically, the main purpose of the exponential function and temperature parameter processing is to adjust the distribution of similarity values ​​so that modalities with higher similarity can occupy higher weights in the fusion process, while modalities with lower similarity are given lower weights. The temperature parameter (usually represented by the Greek letter τ) adjusts the sensitivity of the model. High temperature values ​​will reduce the similarity differences between all modalities, while low temperature values ​​will amplify the similarity differences. Therefore, in this process, it is first necessary to apply the exponential function and temperature parameter to the first, second, and third cross-modal similarities respectively to transform them, generating the corresponding first transformed similarity values, second transformed similarity values, and third transformed similarity values.

[0163] Next, the sum of these three transformed similarity values ​​is calculated. This sum measures the overall similarity between the modalities. This sum ensures that the contributions of all modalities are properly quantified and provides a standardized basis for the subsequent weight calculation.

[0164] Finally, based on the sum value, each transformed similarity value is divided by the sum value to obtain the fusion weights of the sound modality, tactile modality, and text modality. Specifically, the first transformed similarity value is divided by the sum value to obtain the sound modality fusion weight, the second transformed similarity value is divided by the sum value to obtain the tactile modality fusion weight, and the third transformed similarity value is divided by the sum value to obtain the text modality fusion weight. In this way, the weights of each modality are adjusted in a proportional manner to ensure that different modalities are appropriately weighted according to the size of their similarities during the fusion process, thereby improving the decision-making ability and accuracy of the final model.

[0165] This embodiment not only effectively adjusts the similarity differences between modalities by performing exponential function and temperature parameter transformation on cross-modal similarity, but also ensures that the contributions of different modalities to the final decision can be reasonably weighted according to their relevance. After weighted processing, the system can intelligently adjust the weight of modal fusion according to the similarity between different modalities, thereby improving the accuracy and flexibility of decision-making. This dynamic weighting method enables the system to be optimized according to the needs of specific tasks and the importance of different modal data, and can make more accurate decisions in complex tasks, thereby improving the overall performance of the system.

[0166] In one embodiment, the above step S60 includes:

[0167] S601, performing a feature splicing operation on the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate a spliced ​​feature;

[0168] S602, performing dimension alignment processing on the splicing features to generate dimension alignment features;

[0169] S603, processing the dimension alignment features through a feature fusion layer to generate fusion features;

[0170] S604: Perform feature normalization processing on the fused features to generate collaborative decision-making features.

[0171] In this embodiment, the previously weighted modal features are first fused. The weighted auditory features, weighted tactile features, weighted textual features, and optimized visual features have been processed in the previous steps and weighted using different modality weighting mechanisms. On this basis, these features need to be effectively fused to generate the final decision feature.

[0172] The first step is feature concatenation, which combines weighted auditory features, weighted tactile features, weighted textual features, and optimized visual features. This concatenation operation can be simply understood as combining features from different modalities according to certain rules, typically concatenating them along the feature dimension. This concatenation aggregates information from different modalities into a more comprehensive feature set, preserving the richness of multimodal data. This concatenated feature set provides a multimodal information foundation for subsequent processing steps.

[0173] Next, the concatenated features are dimensionally aligned. Data from different modalities may have different feature dimensions and scales. To ensure that the concatenated features can be effectively fused, they must be dimensionally aligned. This dimensional alignment operation aims to adjust the features from different modalities to a unified dimensional space. This eliminates scale differences between the modalities, allowing them to be processed in the same feature space. In this way, features from different sources can be effectively calculated and analyzed within the same model.

[0174] Next, the dimensionally aligned features are input into the feature fusion layer. This is a key module that further fuses multimodal feature information. At this layer, weighted averaging, addition, concatenation, or other more complex fusion methods are typically used to combine features from each modality to generate a comprehensive decision feature. The output of the fusion layer is a feature vector that contains information from all modalities. This feature vector fully represents information extracted from multiple perspectives, including vision, hearing, touch, and text.

[0175] Finally, the fused features are normalized. The goal of feature normalization is to adjust the fused feature data to a uniform scale, typically within the standardized range of zero mean and unit variance. This normalization ensures that different features have a relatively balanced impact on subsequent processing, preventing certain features from dominating the calculation due to their large range. Normalized features provide more stable and reliable input for subsequent decision-making stages.

[0176] This embodiment successfully integrates information from different modalities into a unified collaborative decision-making feature through the splicing, dimensional alignment, fusion and normalization of multimodal features. This feature set contains comprehensive information from visual, auditory, tactile and textual data, and has stronger expressive and decision-making capabilities. Through this method, the system can fully utilize the advantages of each modality data, overcome the limitations that may exist in a single modality, and thus improve the accuracy and robustness of the decision-making process. Especially in complex multimodal tasks, this feature fusion method can improve the overall performance of the system, making it more efficient and accurate in dealing with real-world problems.

[0177] In one embodiment, the above step S70 includes:

[0178] S701, processing the collaborative decision-making feature through an action parameter generator to generate an action parameter value;

[0179] S702, processing the collaborative decision-making feature through a time sequence encoder to generate an action time sequence identifier;

[0180] S703, processing the collaborative decision-making feature through a state predictor to generate an expected environmental state change;

[0181] S704: Combine the action parameter values, action sequence identifiers, and expected environmental state changes to generate an action decision group.

[0182] In this embodiment, the action decoder is used to process the collaborative decision features generated above to generate a complete action decision group. Specifically, the collaborative decision features contain information from different modalities (such as vision, hearing, touch, etc.), which have been fully integrated and normalized after the processing of the previous steps. With the help of the action decoder, the collaborative decision features will be further converted into actual operation parameters to form executable action decisions.

[0183] First, the action parameter generator processes the collaborative decision-making features and generates corresponding action parameter values. These action parameters include, but are not limited to, grasping force, rotation angle, and speed. The action parameter generator's primary task is to map the collaborative decision-making features to actual physical actions. For example, it determines the force and precision of the robot's operation based on current visual, auditory, and tactile information. The action parameter generator typically performs this mapping through fully connected layers or other types of neural network layers.

[0184] Second, the temporal encoder generates action temporal identifiers based on the collaborative decision-making features. The purpose of the temporal encoder is to capture the temporal relationships between actions, namely, the order in which they occur and the time intervals between them. During task execution, many operations must be performed in a specific temporal order, such as grasping an object, rotating it, and finally placing it. Therefore, the temporal encoder assigns an appropriate time label to each action, ensuring that the action sequence is carried out in the expected order.

[0185] Next, the state predictor takes the collaborative decision features as input and generates an expected environmental state change. The state predictor's function is to predict environmental changes based on the current decision features, such as the change in the position of objects or the change in path after the robot's operation. This expected environmental state change provides the robot with information about the environmental impact of the current operation, enabling it to make real-time adjustments during execution.

[0186] Finally, the action parameter values, action sequence identifiers, and expected environmental state changes are combined to generate a final action decision set. This decision set includes the robot's required actions, the order in which they should be executed, and the expected environmental changes after execution. This information serves as input to the execution module, allowing the robot to accurately complete the task based on these decisions.

[0187] This embodiment can effectively convert the fused multimodal information into specific action decisions that the robot can perform through the action decoder. This processing method enables the robot to understand the task requirements more accurately and generate action decisions that adapt to different scenarios. It makes full use of the collaborative information from multiple sensory channels such as vision, hearing, and touch to ensure that the robot can comprehensively consider various environmental factors when handling complex tasks, thereby improving the success rate and accuracy of task execution. Through the generated action decision group, the robot can respond flexibly in complex and changing environments and demonstrate higher adaptability and efficiency in actual operations.

[0188] In one embodiment, after the above step S70, the method further includes:

[0189] S801, determining an operation execution order based on the action timing identifiers in the action decision group;

[0190] S802, executing operations corresponding to the action parameter values ​​in the action decision group according to the execution order;

[0191] S803, recording the actual environmental state change after the operation is performed through the sensor, and measuring the actual action parameter value when the operation is performed;

[0192] S804, determining an execution deviation between the actual action parameter value and the action parameter value;

[0193] S805, detecting an actual execution time interval of the action sequence, and determining a continuity deviation between the actual execution time interval and an expected time interval of the action sequence identifier;

[0194] S806, determining the difference between the expected environmental state change amount and the actual environmental state change amount in the action decision group;

[0195] S807, constructing a parameter update function based on the execution deviation, continuity deviation and difference value;

[0196] S808: Based on the parameter update function, update the parameters of the motion decoder through a back-propagation module.

[0197] In this embodiment, the action decoder generates an action decision group containing multiple parameters, including action sequence identifiers, action parameter values, and expected environmental state changes. Based on these parameters, the executor then executes the corresponding tasks in a specific order, adjusting and optimizing them based on feedback during execution.

[0198] First, the action sequence identifiers are used to determine the order in which actions should be executed. These identifiers indicate the order in which actions should be executed. During this process, the timing information output by the action decoder ensures that the robot can perform the actions according to the predetermined sequence. The sequence identifiers help the robot correctly execute multiple actions by interpreting the temporal position of each action within the task. For example, when the robot performs a pick-and-place operation, the sequence identifiers ensure that the pick-up action occurs before the place-and-place action.

[0199] Next, following the execution order, the robot begins executing the actions corresponding to the action parameter values ​​in the action decision group. These parameters include force, angle, speed, and more. The robot applies these parameters during physical execution and adjusts its actions in real time through feedback mechanisms. For example, when grasping an object, the robot calculates the specific action based on the force and angle generated by the action parameters and accurately executes the action according to these values.

[0200] During the operation, sensors record the actual changes in the environmental state after the operation is executed. Sensors can include visual, tactile, or other environmental monitoring devices that can capture and record environmental changes generated during the operation in real time, such as changes in the position of objects and other dynamic changes in the environment.

[0201] The system then measures the actual action parameter values ​​during the operation. For example, if the robot is set to a certain force value during a grasping operation, the sensor measures the actual force applied by the robot and feeds this back to the system. The system then calculates the execution deviation between the actual action parameter values ​​and the intended values. This deviation reflects the difference between the robot's actual operation and the intended target when performing the task.

[0202] Furthermore, the actual execution time intervals of the action sequence are detected and the amount of continuity deviation between the actual execution time intervals and the expected time intervals in the action sequence identifiers is determined. Each operation has specific timing requirements, which are provided by the action sequence identifiers. Detecting the actual time intervals helps the system understand whether each action is executed smoothly and, if so, helps adjust the execution speed of subsequent tasks.

[0203] The system also needs to calculate the difference between the expected environmental state change in the action decision group and the actual environmental state change. The expected environmental state change is the system's prediction of environmental changes, while the actual environmental state change is the data recorded by the sensors. The difference between the two will reveal any errors in task execution.

[0204] Finally, a parameter update function is constructed based on the execution deviation, continuity deviation, and difference value. This function is used to update the parameters of the action decoder. Through the backpropagation module, the system will adjust the weights and parameters of the action decoder through an error feedback mechanism, thereby optimizing the future action decision generation process.

[0205] Example: In the realm of decision-making for robotic agents, particularly in complex grasping scenarios, accurately perceiving the environment and executing corresponding actions is a key challenge. By integrating multimodal data such as vision, hearing, touch, and textual instructions, robots can obtain rich environmental information and make efficient and accurate decisions based on it.

[0206] First, the robot uses visual sensors to acquire visual perception data in the environment, including the location, size, and shape of objects. Simultaneously, the robot also receives textual instructions from the user, such as "grab the red cup on the table." Furthermore, the robot uses sound sensing devices to capture sound information from the environment, such as mechanical sounds from the console or the sounds of objects colliding. It also uses tactile sensing devices to collect tactile information, such as the force and pressure of contact with objects. This data is processed by an image encoder, a text encoder, an auditory encoder, and a tactile encoder, respectively, to generate corresponding visual, textual, auditory, and tactile encoding features.

[0207] Next, the robot constructs positive and negative samples based on the visual perception data and text instruction data. Visual features are further optimized by selecting images of the same scene from different perspectives to construct positive image pairs, and by selecting images of different scenes to construct negative image pairs. Furthermore, text features are optimized by constructing positive text pairs based on semantically similar text within the text instructions, and negative text pairs based on semantically unrelated text. These optimized visual and text features are further processed by the image encoder and text encoder to generate optimized visual and text features.

[0208] To further enhance the robot's understanding of the environment, it calculates the first cross-modal similarity between optimized visual features and auditory encoding features, the second cross-modal similarity between optimized visual features and tactile encoding features, and the third cross-modal similarity between optimized visual features and optimized text features. These similarity values ​​allow the robot to measure the correlation between different perceptual modalities, providing the necessary reference for subsequent decision-making.

[0209] Next, the robot assigns fusion weights to the different modalities based on these three cross-modal similarity values. For example, the robot determines the fusion weight of the sound modality based on the similarity between vision and hearing, the fusion weight of the tactile modality based on the similarity between vision and touch, and the fusion weight of the text modality based on the similarity between vision and text. These weighted features are then used to generate weighted auditory features, weighted tactile features, and weighted text features, respectively.

[0210] After generating these weighted features, the robot fuses the weighted auditory, tactile, and textual features with the optimized visual features to generate a collaborative decision-making feature. This collaborative decision-making feature integrates multi-dimensional information from different sensory channels and serves as the basis for the robot's action decisions.

[0211] Finally, the robot uses the action decoder to convert the collaborative decision-making features into specific action decision groups. The action decoder first generates action parameter values ​​(e.g., grasping force and angle) based on the collaborative decision-making features, as well as an action timing identifier (determining the order of operations) and an expected environmental state change (predicting how the environment will change after the operation). After integrating this information, an action decision group is generated and transmitted to the robot to guide its subsequent task operations.

[0212] By parsing the action timing identifiers, the robot can sequentially execute each operation within a task. For example, a grasping operation first controls the robot's gripping force and angle based on the action parameter values. The timing identifiers are then used to determine the timing of the grasping process, ensuring that the grasping operation is not interfered with by other operations. During execution, the robot continuously records the actual changes in environmental conditions through sensors and compares them with the expected changes. The robot then calculates execution deviations, time interval deviations, and other factors to further optimize the accuracy of action execution.

[0213] In healthcare, accurate decision support systems are crucial for improving diagnosis and treatment outcomes. Robotic agents in medical settings must process complex multimodal data, combining different types of information—visual, textual, audio, and tactile—to assist doctors in making accurate diagnoses and treatment decisions.

[0214] First, the robot uses a variety of sensors to acquire visual perception data from the environment, including patient imaging data (such as X-rays, CT images, etc.) and visual information about the surrounding environment. At the same time, the robot also receives text instruction data from the doctor, such as "check whether the patient has a tumor in the right lung." In addition to visual and text data, the robot also uses sound perception equipment to capture sound information such as the patient's heartbeat and breathing, and uses tactile sensors to measure physiological indicators such as the patient's body temperature and pulse. All of this data is processed by the image encoder, text encoder, auditory encoder, and tactile encoder to generate corresponding visual coding features, text coding features, auditory coding features, and tactile coding features.

[0215] During processing, the robot constructs positive and negative sample pairs based on visual perception data and text instruction data. Visual features are optimized by selecting images from different angles from the patient's imaging data to construct positive image pairs, and by selecting images from different patients to construct negative image pairs. Furthermore, text features are optimized by constructing positive text pairs based on semantically similar sentences in the medical text instructions, and constructing negative text pairs based on semantically unrelated sentences. These optimized visual and text features are further processed by an image encoder and a text encoder to generate optimized visual and text features.

[0216] To enhance the robot's understanding of the environment, it calculates cross-modal similarities between the optimized visual features and other sensory data. For example, it calculates the first cross-modal similarity between the optimized visual features and auditory encoding features, the second cross-modal similarity between the optimized visual features and tactile encoding features, and the third cross-modal similarity between the optimized visual features and optimized text features. Using these similarity values, the robot can assess the correlation between different sensory data, providing a basis for subsequent diagnosis and treatment decisions.

[0217] Next, the robot assigns fusion weights to the different modalities based on these three cross-modal similarity values. For example, the robot assigns a weight to the audio modality based on the similarity between vision and hearing, a weight to the tactile modality based on the similarity between vision and touch, and a weight to the text modality based on the similarity between vision and text. Using these weighted features, the robot generates weighted auditory features, weighted tactile features, and weighted text features.

[0218] After generating these weighted features, the robot then fuses the weighted auditory, tactile, and textual features with the optimized visual features to generate a collaborative decision-making feature. This feature integrates multi-dimensional information from different sensory channels and forms the basis for the robot's medical decision support.

[0219] Finally, the robot uses the action decoder to convert the collaborative decision-making features into a specific medical operation decision group. For example, the robot may need to perform image processing, tissue analysis, or simulated surgical operations. The action decoder generates action parameter values ​​(such as applied pressure, operation strength, etc.), action timing identifiers (the order of operations), and expected environmental state changes (physiological reactions that may occur after treatment) based on the collaborative decision-making features. After integrating this information, the robot can perform precise medical tasks and assist doctors in providing efficient treatment.

[0220] During the execution process, the robot interprets the action sequence identifiers and sequentially executes each operation, such as image analysis, diagnosis confirmation, and treatment plan execution. Using sensors, the robot can record real-time changes in the patient's condition, such as body temperature and blood pressure, to assess the effectiveness of treatment. This process not only accurately determines the effectiveness of treatment but also optimizes subsequent operations based on this feedback, improving treatment effectiveness and patient recovery.

[0221] In the field of financial technology, robotic agents primarily use multimodal perception technology to improve the accuracy and efficiency of financial decision-making. With the rapid development of big data and artificial intelligence technologies, financial institutions need to quickly process large amounts of customer information, market data, and related financial documents to support real-time decision-making and optimize risk management.

[0222] First, the robot uses various sensory devices to acquire visual data from its environment, including images of bank customers, financial documents, and market reports. Simultaneously, the robot receives textual instructions related to the customer, such as "analyze the customer's risk level" or "provide a loan application report." In addition to visual and textual data, the robot also uses audio sensors to capture audio information, such as bank meeting discussions and customer voice feedback. Tactile sensors also sense physical interactions, such as customer interactions and keystrokes in the bank lobby. This data is processed using an image encoder, text encoder, auditory encoder, and tactile encoder, respectively, to generate corresponding visual, textual, auditory, and tactile encoding features.

[0223] During data processing, the robot constructs positive and negative sample pairs based on the visual and textual command data. Visual features are optimized by selecting images from different scenes from the customer's image data to construct positive image pairs, and by selecting images from different customers to construct negative image pairs. Furthermore, textual features are optimized by selecting semantically similar sentences from the textual command data to construct positive text pairs, and semantically unrelated sentences to construct negative text pairs. These optimized visual and textual features are further processed by the image encoder and text encoder to generate optimized visual and textual features.

[0224] To enhance the robot's understanding of the financial context, it also calculates cross-modal similarities between the optimized visual features and other sensory data. For example, it calculates the first cross-modal similarity between the optimized visual features and auditory encoding features, the second cross-modal similarity between the optimized visual features and tactile encoding features, and the third cross-modal similarity between the optimized visual features and optimized text features. Using these similarity values, the robot can assess the relationships between different sensory data, providing a basis for subsequent financial decision-making.

[0225] Based on this, the robot assigns different modal fusion weights based on the three cross-modal similarity values. For example, based on the similarity between vision and hearing, the robot assigns a higher weight to the sound modality; based on the similarity between vision and touch, the robot assigns a higher weight to the touch modality; and based on the similarity between vision and text, the robot assigns a higher weight to the text modality. Using these weighted features, the robot can generate weighted auditory features, weighted tactile features, and weighted text features.

[0226] Next, the robot combines these weighted features with the optimized visual features to generate collaborative decision-making features. These features integrate multi-dimensional information from different perception channels, providing strong support for the robot's financial task execution. This decision-making feature can help the robot make more accurate and efficient decisions across various financial operations, such as customer risk assessment, portfolio optimization, and loan approval.

[0227] The robot uses an action decoder to translate collaborative decision-making features into specific financial operation instructions. For example, the robot might need to analyze a customer's credit history, generate investment recommendations, or review loan applications. Based on the collaborative decision-making features, the action decoder generates specific operation parameter values, such as credit assessment scores, return on investment rates, and loan limits. Action sequence identifiers determine the order in which operations should be executed, such as performing a credit review before loan approval. Expected environmental state changes describe the expected effects of the operation, such as the change in a customer's risk level after loan approval.

[0228] During execution, the robot analyzes the action sequence identifiers and sequentially executes each action, such as customer risk assessment, loan approval, and investment recommendation. The robot also uses sensors to record environmental changes in real time and compare actual execution results with expected results. If deviations are detected, the robot can promptly adjust its operational strategy and optimize the decision-making process, thereby improving the efficiency and accuracy of loan approval, credit risk assessment, and investment management.

[0229] This combination of multimodal perception and feedback mechanisms enables robots to provide more precise customer service and intelligent decision-making support in the financial technology sector. By processing different types of financial data and optimizing based on customer needs and market dynamics, robots can achieve efficient and low-risk decision-making in complex and dynamically changing financial environments.

[0230] Through the above steps, the system of this embodiment can continuously optimize the robot's operating strategy based on real-time feedback information to ensure that each step of the robot's execution is more accurate and reliable. Unlike traditional methods, this solution can comprehensively evaluate the execution of each action, including deviations in timing, force, angle, etc., so as to continuously adjust and optimize the parameters of the decoder through back propagation. In this way, the robot can adapt to different tasks in a changing environment and ensure high-precision execution of tasks. In addition, through multiple feedback mechanisms (such as execution deviation, time interval deviation, environmental change differences, etc.), the system can dynamically adjust during task execution to ensure optimal performance.

[0231] In one embodiment, a decision-making device based on multimodal collaborative optimization is provided, and the decision-making device based on multimodal collaborative optimization corresponds one-to-one with the decision-making method based on multimodal collaborative optimization in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a decision-making device based on multimodal collaborative optimization according to the present invention. It includes a perception data acquisition module 10, a feature optimization module 20, a similarity calculation module 30, a weight assignment module 40, a weighted feature generation module 50, a feature fusion module 60, and an action decoding module 70. Each functional module is described in detail below:

[0232] The perception data acquisition module 10 is used to obtain visual perception data, text instruction data, sound perception data and tactile perception data, and generate visual coding features, text coding features, auditory coding features and tactile coding features through an image encoder, a text encoder, an auditory encoder and a tactile encoder respectively;

[0233] a feature optimization module 20 for constructing positive and negative sample image pairs and positive and negative sample text pairs for the visual perception data and the text instruction data, respectively, and optimizing the visual encoding features and the text encoding features based on the positive and negative sample image pairs and the positive and negative sample text pairs to generate optimized visual features and optimized text features;

[0234] a similarity calculation module 30 for determining a first cross-modal similarity between the optimized visual feature and the auditory coding feature, a second cross-modal similarity between the optimized visual feature and the tactile coding feature, and a third cross-modal similarity between the optimized visual feature and the optimized textual feature;

[0235] a weight assignment module 40 for assigning a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight based on the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity, respectively;

[0236] A weighted feature generation module 50 is configured to apply the sound modality fusion weight, the tactile modality fusion weight, and the text modality fusion weight to weight the auditory coding features, the tactile coding features, and the optimized text features, respectively, to generate weighted auditory features, weighted tactile features, and weighted text features;

[0237] A feature fusion module 60 is configured to fuse the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate collaborative decision-making features;

[0238] The action decoding module 70 is used to process the collaborative decision features through an action decoder to generate an action decision group.

[0239] In one embodiment, the feature optimization module 20 is specifically configured to:

[0240] Selecting images of the same scene from different perspectives from the visual perception data to construct positive sample image pairs, and selecting images of different scenes to construct negative sample image pairs;

[0241] Input the positive sample image pair into an image encoder to obtain a positive sample visual coding feature group, and input the negative sample image pair into an image encoder to obtain a negative sample visual coding feature group;

[0242] Determining a similarity value of visual positive sample features within the positive sample visual coding feature group and a similarity value of visual negative sample features within the negative sample visual coding feature group;

[0243] Constructing an image contrast loss function based on the visual positive sample feature similarity value and the visual negative sample feature similarity value, and updating parameters of an image encoder by minimizing the image contrast loss function;

[0244] Processing the visual perception data using the updated image encoder to generate optimized visual features;

[0245] Selecting semantically similar texts from the text instruction data to construct positive sample text pairs, and selecting semantically irrelevant texts to construct negative sample text pairs;

[0246] Inputting the positive sample text pair into a text encoder to obtain a positive sample text encoding feature group, and inputting the negative sample text pair into a text encoder to obtain a negative sample text encoding feature group;

[0247] Determining the similarity values ​​of the text positive sample features in the positive sample text encoding feature group and the similarity values ​​of the text negative sample features in the negative sample text encoding feature group;

[0248] Constructing a text contrast loss function based on the text positive sample feature similarity value and the text negative sample feature similarity value, and updating the parameters of the text encoder by minimizing the text contrast loss function;

[0249] The updated text encoder is used to process the text instruction data to generate optimized text features.

[0250] In one embodiment, the similarity calculation module 30 is specifically configured to:

[0251] Extracting a visual feature vector of the optimized visual feature and an auditory feature vector of the auditory coding feature;

[0252] determining a first cosine similarity value between the visual feature vector and the auditory feature vector, and using the first cosine similarity value as a first cross-modal similarity;

[0253] extracting a tactile feature vector of the tactile coding feature;

[0254] determining a second cosine similarity value between the visual feature vector and the tactile feature vector, and using the second cosine similarity value as a second cross-modal similarity;

[0255] Extracting a text feature vector of the optimized text feature;

[0256] A third cosine similarity value between the visual feature vector and the text feature vector is determined, and the third cosine similarity value is used as a third cross-modal similarity.

[0257] In one embodiment, the weight allocation module 40 is specifically configured to:

[0258] Applying an exponential function and a temperature parameter to the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity respectively to generate a first transformed similarity value, a second transformed similarity value, and a third transformed similarity value;

[0259] determining a sum of the first transformed similarity value, the second transformed similarity value, and the third transformed similarity value;

[0260] The first transformed similarity value, the second transformed similarity value, and the third transformed similarity value are respectively divided by the sum value to generate a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight.

[0261] In one embodiment, the feature fusion module 60 is specifically configured to:

[0262] Performing a feature splicing operation on the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate a spliced ​​feature;

[0263] Performing dimension alignment processing on the splicing features to generate dimension alignment features;

[0264] Processing the dimension alignment features through a feature fusion layer to generate fused features;

[0265] The fusion features are normalized to generate collaborative decision-making features.

[0266] In one embodiment, the action decoding module 70 is specifically configured to:

[0267] Processing the collaborative decision-making features through an action parameter generator to generate action parameter values;

[0268] Processing the collaborative decision-making features through a temporal encoder to generate an action temporal identifier;

[0269] Processing the collaborative decision-making features through a state predictor to generate an expected environmental state change;

[0270] The action parameter values, action sequence identifiers, and expected environmental state changes are combined to generate an action decision group.

[0271] In one embodiment, the action decoding module 70 is specifically configured to:

[0272] Determining an operation execution order based on the action timing identifiers in the action decision group;

[0273] Execute the operations corresponding to the action parameter values ​​in the action decision group according to the execution order;

[0274] The sensor records the actual change in the environmental state after the operation is performed, and measures the actual action parameter value when the operation is performed;

[0275] determining an execution deviation between the actual action parameter value and the action parameter value;

[0276] detecting an actual execution time interval of an action sequence, and determining a continuity deviation amount between the actual execution time interval and an expected time interval of the action sequence identifier;

[0277] Determining the difference between the expected environmental state change and the actual environmental state change in the action decision group;

[0278] Constructing a parameter update function based on the execution deviation, the continuity deviation and the difference value;

[0279] Based on the parameter update function, the parameters of the motion decoder are updated through a back-propagation module.

[0280] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a decision-making method based on multimodal collaborative optimization.

[0281] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. Among them, the processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a decision-making method based on multimodal collaborative optimization

[0282] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0283] Acquire visual perception data, text instruction data, sound perception data, and tactile perception data, and generate visual coding features, text coding features, auditory coding features, and tactile coding features through an image encoder, a text encoder, an auditory encoder, and a tactile encoder, respectively;

[0284] Constructing positive and negative sample image pairs and positive and negative sample text pairs for the visual perception data and the text instruction data respectively, and optimizing the visual encoding features and the text encoding features based on the positive and negative sample image pairs and the positive and negative sample text pairs to generate optimized visual features and optimized text features;

[0285] Determining a first cross-modal similarity between the optimized visual feature and the auditory coding feature, a second cross-modal similarity between the optimized visual feature and the tactile coding feature, and a third cross-modal similarity between the optimized visual feature and the optimized textual feature;

[0286] assigning a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight based on the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity, respectively;

[0287] Applying the sound modality fusion weight, the tactile modality fusion weight, and the text modality fusion weight to weight the auditory coding features, the tactile coding features, and the optimized text features, respectively, to generate weighted auditory features, weighted tactile features, and weighted text features;

[0288] fusing the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate collaborative decision-making features;

[0289] The collaborative decision features are processed by an action decoder to generate an action decision group.

[0290] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0291] Acquire visual perception data, text instruction data, sound perception data, and tactile perception data, and generate visual coding features, text coding features, auditory coding features, and tactile coding features through an image encoder, a text encoder, an auditory encoder, and a tactile encoder, respectively;

[0292] Constructing positive and negative sample image pairs and positive and negative sample text pairs for the visual perception data and the text instruction data respectively, and optimizing the visual encoding features and the text encoding features based on the positive and negative sample image pairs and the positive and negative sample text pairs to generate optimized visual features and optimized text features;

[0293] Determining a first cross-modal similarity between the optimized visual feature and the auditory coding feature, a second cross-modal similarity between the optimized visual feature and the tactile coding feature, and a third cross-modal similarity between the optimized visual feature and the optimized textual feature;

[0294] assigning a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight based on the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity, respectively;

[0295] Applying the sound modality fusion weight, the tactile modality fusion weight, and the text modality fusion weight to weight the auditory coding features, the tactile coding features, and the optimized text features, respectively, to generate weighted auditory features, weighted tactile features, and weighted text features;

[0296] fusing the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate collaborative decision-making features;

[0297] The collaborative decision features are processed by an action decoder to generate an action decision group.

[0298] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0299] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0300] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0301] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A decision-making method based on multimodal collaborative optimization, characterized in that: The following steps are involved: Acquire visual perception data, text instruction data, sound perception data, and tactile perception data, and generate visual coding features, text coding features, auditory coding features, and tactile coding features through an image encoder, a text encoder, an auditory encoder, and a tactile encoder, respectively; Constructing positive and negative sample image pairs and positive and negative sample text pairs for the visual perception data and the text instruction data respectively, and optimizing the visual encoding features and the text encoding features based on the positive and negative sample image pairs and the positive and negative sample text pairs to generate optimized visual features and optimized text features; Determining a first cross-modal similarity between the optimized visual feature and the auditory coding feature, a second cross-modal similarity between the optimized visual feature and the tactile coding feature, and a third cross-modal similarity between the optimized visual feature and the optimized textual feature; assigning a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight based on the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity, respectively; Applying the sound modality fusion weight, the tactile modality fusion weight, and the text modality fusion weight to weight the auditory coding features, the tactile coding features, and the optimized text features, respectively, to generate weighted auditory features, weighted tactile features, and weighted text features; fusing the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate collaborative decision-making features; The collaborative decision features are processed by an action decoder to generate an action decision group.

2. The decision-making method based on multimodal collaborative optimization according to claim 1, characterized in that: Constructing positive and negative sample image pairs and positive and negative sample text pairs for the visual perception data and the text instruction data respectively, and optimizing the visual encoding features and the text encoding features based on the positive and negative sample image pairs and the positive and negative sample text pairs to generate optimized visual features and optimized text features, including: Selecting images of the same scene from different perspectives from the visual perception data to construct positive sample image pairs, and selecting images of different scenes to construct negative sample image pairs; Input the positive sample image pair into an image encoder to obtain a positive sample visual coding feature group, and input the negative sample image pair into an image encoder to obtain a negative sample visual coding feature group; Determining a similarity value of visual positive sample features within the positive sample visual coding feature group and a similarity value of visual negative sample features within the negative sample visual coding feature group; Constructing an image contrast loss function based on the visual positive sample feature similarity value and the visual negative sample feature similarity value, and updating parameters of an image encoder by minimizing the image contrast loss function; Processing the visual perception data using the updated image encoder to generate optimized visual features; Selecting semantically similar texts from the text instruction data to construct positive sample text pairs, and selecting semantically irrelevant texts to construct negative sample text pairs; Inputting the positive sample text pair into a text encoder to obtain a positive sample text encoding feature group, and inputting the negative sample text pair into a text encoder to obtain a negative sample text encoding feature group; Determining the similarity values ​​of the text positive sample features in the positive sample text encoding feature group and the similarity values ​​of the text negative sample features in the negative sample text encoding feature group; Constructing a text contrast loss function based on the text positive sample feature similarity value and the text negative sample feature similarity value, and updating the parameters of the text encoder by minimizing the text contrast loss function; The updated text encoder is used to process the text instruction data to generate optimized text features.

3. The decision-making method based on multimodal collaborative optimization according to claim 1, characterized in that: Determining a first cross-modal similarity between the optimized visual feature and the auditory coding feature, a second cross-modal similarity between the optimized visual feature and the tactile coding feature, and a third cross-modal similarity between the optimized visual feature and the optimized textual feature, comprising: Extracting a visual feature vector of the optimized visual feature and an auditory feature vector of the auditory coding feature; determining a first cosine similarity value between the visual feature vector and the auditory feature vector, and using the first cosine similarity value as a first cross-modal similarity; extracting a tactile feature vector of the tactile coding feature; determining a second cosine similarity value between the visual feature vector and the tactile feature vector, and using the second cosine similarity value as a second cross-modal similarity; Extracting a text feature vector of the optimized text feature; A third cosine similarity value between the visual feature vector and the text feature vector is determined, and the third cosine similarity value is used as a third cross-modal similarity.

4. The decision-making method based on multimodal collaborative optimization according to claim 1, characterized in that: Allocating a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight based on the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity, respectively, includes: Applying an exponential function and a temperature parameter to the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity respectively to generate a first transformed similarity value, a second transformed similarity value, and a third transformed similarity value; determining a sum of the first transformed similarity value, the second transformed similarity value, and the third transformed similarity value; The first transformed similarity value, the second transformed similarity value, and the third transformed similarity value are respectively divided by the sum value to generate a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight.

5. The decision-making method based on multimodal collaborative optimization according to claim 1, characterized in that: The weighted auditory features, weighted tactile features, weighted text features, and optimized visual features are integrated to generate collaborative decision-making features, including: Performing a feature splicing operation on the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate a spliced ​​feature; Performing dimension alignment processing on the splicing features to generate dimension alignment features; Processing the dimension alignment features through a feature fusion layer to generate fused features; The fusion features are normalized to generate collaborative decision-making features.

6. The decision-making method based on multimodal collaborative optimization according to claim 1, characterized in that: The collaborative decision feature is processed by an action decoder to generate an action decision group, including: Processing the collaborative decision-making features through an action parameter generator to generate action parameter values; Processing the collaborative decision-making features through a temporal encoder to generate an action temporal identifier; Processing the collaborative decision-making features through a state predictor to generate an expected environmental state change; The action parameter values, action sequence identifiers, and expected environmental state changes are combined to generate an action decision group.

7. The decision-making method based on multimodal collaborative optimization according to claim 1, characterized in that: After the collaborative decision feature is processed by the action decoder to generate an action decision group, the method further includes: Determining an operation execution order based on the action timing identifiers in the action decision group; Execute the operations corresponding to the action parameter values ​​in the action decision group according to the execution order; The sensor records the actual change in the environmental state after the operation is performed, and measures the actual action parameter value when the operation is performed; determining an execution deviation between the actual action parameter value and the action parameter value; detecting an actual execution time interval of an action sequence, and determining a continuity deviation amount between the actual execution time interval and an expected time interval of the action sequence identifier; Determining the difference between the expected environmental state change and the actual environmental state change in the action decision group; Constructing a parameter update function based on the execution deviation, the continuity deviation and the difference value; Based on the parameter update function, the parameters of the motion decoder are updated through a back-propagation module.

8. A decision-making device based on multimodal collaborative optimization, characterized in that: The decision-making device based on multimodal collaborative optimization includes: A perception data acquisition module is used to acquire visual perception data, text instruction data, sound perception data, and tactile perception data, and generate visual coding features, text coding features, auditory coding features, and tactile coding features through an image encoder, a text encoder, an auditory encoder, and a tactile encoder, respectively; a feature optimization module, configured to construct positive and negative sample image pairs and positive and negative sample text pairs for the visual perception data and the text instruction data, respectively, and optimize the visual encoding features and the text encoding features based on the positive and negative sample image pairs and the positive and negative sample text pairs to generate optimized visual features and optimized text features; a similarity calculation module, configured to determine a first cross-modal similarity between the optimized visual feature and the auditory coding feature, a second cross-modal similarity between the optimized visual feature and the tactile coding feature, and a third cross-modal similarity between the optimized visual feature and the optimized textual feature; a weight assignment module, configured to assign a sound modality fusion weight, a tactile modality fusion weight, and a text modality fusion weight based on the first cross-modal similarity, the second cross-modal similarity, and the third cross-modal similarity, respectively; A weighted feature generation module, configured to apply the sound modality fusion weight, the tactile modality fusion weight, and the text modality fusion weight to weight the auditory coding features, the tactile coding features, and the optimized text features, respectively, to generate weighted auditory features, weighted tactile features, and weighted text features; A feature fusion module, configured to fuse the weighted auditory features, weighted tactile features, weighted text features, and optimized visual features to generate collaborative decision-making features; The action decoding module is used to process the collaborative decision features through an action decoder to generate an action decision group.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a decision program based on multimodal collaborative optimization stored in the memory and capable of running on the processor. When the decision program based on multimodal collaborative optimization is executed by the processor, the steps of the decision method based on multimodal collaborative optimization as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a decision-making program based on multimodal collaborative optimization, and when the decision-making program based on multimodal collaborative optimization is executed by the processor, the steps of the decision-making method based on multimodal collaborative optimization as described in any one of claims 1 to 7 are implemented.

Images