Robot control method and device based on multi-modal data fusion

Abstract

The invention discloses a robot control method and device based on multi-modal data fusion, and the method comprises the steps: responding to a control instruction inputted by a user, receiving and preprocessing sensing information fed back by a plurality of types of sensors preset on a robot, and obtaining multi-modal data; dynamically adjusting the weight of each modal data, and extracting the feature vector of each modal data; according to the dynamically adjusted weight of each modal data, fusing the extracted feature vectors of each modal data to generate a comprehensive feature vector for representing the comprehensive state of the current environment; according to the comprehensive feature vector, the control instruction is analyzed through a preset large language model, and a final decision result of the robot is obtained; and according to the final decision result, calling and executing an instruction set required to be executed by the robot so as to drive the robot to complete related actions. Therefore, by adopting the embodiment of the invention, the robot can effectively process and utilize information from different modes to the maximum extent, and better decision and performance of the robot are realized.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot control method and device based on multimodal data fusion. Background Art

[0002] Robots have become a bridge between the physical world and the digital world, and they play an increasingly important role in many fields such as home, industry, medical care, and education. These application scenarios require robots to understand and respond to complex natural language commands, while processing data from multiple sensors such as vision, hearing, and touch to achieve highly simulated and natural interactions.

[0003] In related technologies, the robot's operating system only supports single-modal information processing, such as relying only on visual or auditory information, and cannot fully integrate multiple sensor data such as vision, hearing, and touch, which limits the robot's perception and interaction capabilities in complex environments. At the same time, the existing system is difficult to achieve a deep understanding of user intentions, resulting in the robot's lack of naturalness and personalization when providing interactive experience, and cannot meet users' expectations for intelligent interaction. Summary of the invention

[0004] The embodiments of the present application provide a robot control method and device based on multimodal data fusion. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a robot control method based on multimodal data fusion, the method comprising: In response to the control command input by the user, the robot receives and pre-processes the perception information fed back by multiple types of sensors pre-set on the robot to obtain multi-modal data; Dynamically adjust the weight of each modal data and extract the feature vector of each modal data. The weight of each modal data is calculated based on the reliability score of each modal data. The reliability score of each modal data is determined based on the current application scenario of the robot. The current application scenario is analyzed based on a pre-trained environmental analysis model. The pre-trained environmental analysis model is obtained by machine learning based on historical sensor data when the robot performs tasks in different environments; According to the dynamically adjusted weights of each modal data, the extracted feature vectors of each modal data are fused to generate a comprehensive feature vector for representing the overall state of the current environment; According to the comprehensive feature vector, the control instructions are analyzed through the preset large language model to obtain the final decision result of the robot; Based on the final decision result, the instruction set required by the robot is called and executed to drive the robot to complete the relevant actions corresponding to the instruction set.

[0006] Optionally, dynamically adjust the weight of each modality data, including Get the initial default weights of various sensors as the initial weights of each modality data; Feed multimodal data into pre-trained environmental analysis models; Output the scene corresponding to the multimodal data as the current application scene of the robot; Determine the reliability score of each modal data based on the current application scenario of the robot; According to the initial weight of each modal data and the reliability score of each modal data, the weight of each modal data adjusted dynamically is calculated; the weight calculation formula of each modal data adjusted dynamically is:

[0007] in, For the The final adjusted weights of the modal data are is the adjustment factor, which is used to control the sensitivity of weight adjustment. It is Modal data in the current application scenario The reliability scoring function under is the index variable, which is used to traverse all modal data during the summation process. Indicates the total number of modal data.

[0008] Optionally, follow these steps to generate a pre-trained environment analysis model, including: Collect and preprocess historical sensor data when the robot performs tasks in different environments; For the pre-processed historical sensor data, the corresponding scene labels are marked to obtain multiple sample data; Use neural networks to create environmental analysis models; Input multiple sample data into the environmental analysis model for machine learning, and output the model loss value when learning is completed; When the model loss value reaches the minimum, a pre-trained environment analysis model is generated; or, when the loss value does not reach the minimum, the step of inputting multiple sample data into the environment analysis model is continued until the model loss value reaches the minimum; The model loss function of the environmental analysis model is:

[0009] in, is the model loss value, is a hyperparameter used to control the transition point of the loss function from square loss to linear loss. is the annotated scene label, is the scene label predicted by the neural network, Indicates the error between the annotated scene label and the predicted scene label, When the error is less than or equal to the hyperparameter, the loss function uses the square error, that is , When the error is greater than the hyperparameter, the loss function uses linear error, that is .

[0010] Optionally, each modality data includes visual modality data, auditory modality data, and tactile modality data; According to the current application scenario of the robot, determine the reliability score of each modal data, including: Obtain the current visual state, current auditory state, and current tactile state corresponding to the current application scenario of the robot; Loading a pre-established first mapping relationship between visual state and reliability score, a second mapping relationship between auditory state and reliability score, and a third mapping relationship between tactile state and reliability score; Obtaining a reliability score corresponding to the current visual state from the first mapping relationship; Obtaining a reliability score corresponding to the current hearing state from the second mapping relationship; Obtaining a reliability score corresponding to the current tactile state from the third mapping relationship; The reliability scores obtained above are used as the reliability scores of each modal data, where: Visual states include indoors or outdoors, auditory states include quiet or noisy, and tactile states include hard objects or soft objects.

[0011] Optionally, the feature vector of each modal data is obtained using a feature extraction technique; According to the dynamically adjusted weights of each modal data, the feature vectors extracted from each modal data are fused to generate a comprehensive feature vector for characterizing the overall state of the current environment, including: Obtaining a feature vector of each modal data and its weight from the extracted feature vector of each modal data and the dynamically adjusted weight of each modal data; The eigenvectors of each modal data and their weights are weighted summed to obtain the intermediate eigenvector of primary fusion. The calculation formula of the intermediate eigenvector of primary fusion is:

[0012] in, is the intermediate feature vector of primary fusion, is the total number of modal data, It is The feature vector of the modal data, It is The weight of each modal data; Use a deep learning network to perform nonlinear mapping on the primary fused feature vectors to extract higher-level feature representations; The extracted higher-level feature representations are concatenated to obtain a comprehensive feature vector that is used to characterize the overall state of the current environment.

[0013] Optionally, the control instruction is analyzed by a preset large language model according to the comprehensive feature vector to obtain the final decision result of the robot, including: When the control instruction is a text object, obtain the description text of the text object; In the case where the control instruction is a voice object, a speech recognition ASR system is used to convert the voice object into a description text; wherein the description text is used to describe the object content of the text object or the voice object; According to the description text and the comprehensive feature vector, the decision result corresponding to the control instruction is generated by the preset large language model as the final decision result of the robot.

[0014] Optionally, based on the description text and the comprehensive feature vector, a decision result corresponding to the control instruction is generated by a preset large language model, including: Constructing a prompt information set for characterizing context information of a current application scenario of the robot according to the comprehensive feature vector; Perform word segmentation on the description text to extract multiple task keywords; Multiple task keywords and prompt information sets are spliced ​​together to obtain a task sequence; The task sequence is input into the preset large language model, the task sequence is analyzed by the preset large language model, and the decision result corresponding to the control instruction is output.

[0015] Optionally, a prompt information set for characterizing context information of the current application scenario of the robot is constructed based on the comprehensive feature vector, including: Acquire multiple predefined context scene tags; the multiple context scene tags include lighting conditions and environmental noise; By synthesizing the eigenvectors, we can determine the statistical parameters of the lighting conditions, the quantiles of the ambient noise, Compare the statistical parameters, quantiles and preset parameter values, and use the feature vectors that meet the preset parameter values ​​as sub-features of each context scene label; Convert the sub-features of each context scene label into a natural language description to obtain the prompt information corresponding to each context scene label; The prompt information corresponding to each context scene label is stored as a JSON object to obtain a prompt information set for characterizing the context information of the current application scene of the robot.

[0016] Optionally, the statistical parameter of the lighting condition includes the average lighting intensity; the quantile of the ambient noise is a quartile calculated based on the characteristics of the ambient noise in the comprehensive feature vector; The calculation formula for average light intensity is:

[0017] in, is the average light intensity, is the number of features about lighting conditions in the comprehensive feature vector, It is the first of the general characteristics of lighting conditions. Features.

[0018] In a second aspect, an embodiment of the present application provides a robot control device based on multimodal data fusion, the device comprising: A multimodal data acquisition module is used to receive and pre-process the perception information fed back by multiple types of sensors pre-set on the robot in response to the control instructions input by the user to obtain multimodal data; A weight dynamic adjustment module is used to dynamically adjust the weight of each modal data and extract the feature vector of each modal data. The weight of each modal data is calculated based on the reliability score of each modal data. The reliability score of each modal data is determined based on the current application scenario of the robot. The current application scenario is analyzed based on a pre-trained environmental analysis model. The pre-trained environmental analysis model is obtained by machine learning based on historical sensor data when the robot performs tasks in different environments; A comprehensive feature vector generation module is used to fuse the feature vectors of each modal data extracted according to the dynamically adjusted weights of each modal data to generate a comprehensive feature vector for representing the overall state of the current environment; The decision module is used to analyze the control instructions according to the comprehensive feature vector through the preset large language model to obtain the final decision result of the robot; The instruction set calling module is used to call and execute the instruction set required by the robot according to the final decision result, so as to drive the robot to complete the relevant actions corresponding to the instruction set.

[0019] The technical solution provided by the embodiments of the present application may have the following beneficial effects: In the embodiments of the present application, on the one hand, the dynamic weight adjustment enables the fusion process to adjust the contribution of each modality according to the dynamic importance of each modality, so that the fused feature vector can more accurately reflect the comprehensive information of different modalities, and the fused comprehensive feature vector can more comprehensively characterize the current environmental state and provide the robot with richer contextual information; on the other hand, by utilizing the comprehensive feature vector, the control instructions are analyzed through a preset large language model, so that the robot can deeply understand the user's control instructions, and at the same time, the robot can provide a more natural and more user-expected interactive experience.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] Figure 1 It is a method flow diagram of a robot control method based on multimodal data fusion provided in an embodiment of the present application; Figure 2 This is a data representation of a weight adjustment provided in an embodiment of the present application; Figure 3 It is a data representation intent of a mapping relationship provided by this application; Figure 4 It is a schematic diagram of a feature fusion process provided by this application; Figure 5 It is a schematic diagram of a prompt information set provided by this application; Figure 6 It is a structural schematic diagram of a robot control device based on multimodal data fusion provided by the present application; Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.

[0024] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0025] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0026] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0027] Currently, the robot's operating system only supports single-modal information processing, such as relying solely on visual or auditory information, and is unable to fully integrate multiple sensor data such as vision, hearing, and touch.

[0028] The inventors realized that the current approach limits the robot's perception and interaction capabilities in complex environments. At the same time, the existing system is difficult to achieve a deep understanding of user intentions, resulting in the robot's lack of naturalness and personalization when providing interactive experience, and unable to meet users' expectations for intelligent interaction.

[0029] In order to solve the above problems, the present application provides a robot control method and device based on multimodal data fusion to solve the problems existing in the above-mentioned related technical issues. In the embodiment of the present application, on the one hand, the dynamic weight adjustment enables the fusion process to adjust the contribution of each modality according to its dynamic importance, so that the fused feature vector can more accurately reflect the comprehensive information of different modalities, and the fused comprehensive feature vector can more comprehensively characterize the current environmental state and provide the robot with richer context information; on the other hand, by using the comprehensive feature vector, the control instructions are analyzed by presetting a large language model, so that the robot can deeply understand the user's control instructions, and at the same time, the robot can provide a more natural and more user-expected interactive experience, which is described in detail using an exemplary embodiment below.

[0030] The following will be combined with the attached Figure 1 -Attached Figure 5, a robot control method based on multimodal data fusion provided by an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a robot control device based on multimodal data fusion based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0031] See also Figure 1 , which is a flowchart of a robot control method based on multimodal data fusion, applied to a server. Figure 1 As shown, the method of the embodiment of the present application may include the following steps: S101, in response to a control instruction input by a user, receiving and preprocessing perception information fed back by multiple types of sensors pre-set on the robot to obtain multi-modal data; Among them, the control instructions input by the user refer to the commands given by the user to the robot, which can be voice, text or other forms of instructions. Multiple types of sensors refer to different types of sensors installed on the robot, such as visual sensors (cameras), auditory sensors (microphones), tactile sensors (pressure sensors), environmental sensors (temperature, humidity sensors), etc. Perception information is the data collected by sensors about the robot's surroundings, which reflects the physical characteristics and status of the environment.

[0032] Among them, preprocessing is the preliminary processing of raw sensor data, including filtering, denoising, normalization, etc., to facilitate subsequent processing and analysis.

[0033] In some embodiments of the present application, it is assumed that there is a service robot that needs to perform tasks in a home environment, such as getting a drink according to the user's instructions. At this time, the user says through a voice command: "Robot, please go to the kitchen and get me a cup of Coke." The microphone on the robot receives the voice command. At the same time, the robot's visual sensor (camera) monitors the environment in real time, the tactile sensor (pressure sensor) senses the texture of the ground, and the environmental sensor (such as an infrared sensor) detects the indoor temperature. After filtering, denoising, normalizing, and other operations on the data fed back by the above sensors, multimodal data is obtained.

[0034] S102, dynamically adjusting the weight of each modal data and extracting the feature vector of each modal data, wherein the weight of each modal data is calculated based on the reliability score of each modal data, and the reliability score of each modal data is determined based on the current application scenario of the robot, and the current application scenario is analyzed according to a pre-trained environmental analysis model, and the pre-trained environmental analysis model is obtained by machine learning based on historical sensor data when the robot performs tasks in different environments; The weight refers to the contribution of each modal data to the final decision or understanding. The higher the weight, the greater the influence of the modal data in the decision-making process. The feature vector is an abstraction and simplification of the original modal data, which captures the core attributes of the data.

[0035] In an embodiment of the present application, the specific process of dynamically adjusting the weight of each modal data includes: obtaining the initial default weight of each type of sensor as the initial weight of each modal data; inputting the multimodal data into a pre-trained environmental analysis model; outputting the scene corresponding to the multimodal data as the current application scene of the robot; determining the reliability score of each modal data according to the current application scene of the robot; calculating the weight of each modal data dynamically adjusted according to the initial weight of each modal data and the reliability score of each modal data; the weight calculation formula of each modal data dynamically adjusted is:

[0036] in, For the The final adjusted weights of the modal data are is the adjustment factor, which is used to control the sensitivity of weight adjustment. It is Modal data in the current application scenario The reliability scoring function under is the index variable, which is used to traverse all modal data during the summation process. Indicates the total number of modal data.

[0037] For example Figure 2 As shown in the figure, assume that a service robot performs a task in an indoor environment, and its default weights are as follows: vision: 0.5 (visual information is very important in navigation and object recognition), hearing: 0.3 (auditory information is important for detecting sound cues and environmental noise), touch: 0.2 (tactile information is important for physical interaction and object grasping). The robot's environmental analysis model identifies the current environment as a "quiet indoor environment". According to this scenario, the robot's system dynamically adjusts the weights of each modal data: vision: increased to 0.6 (in indoor environments, visual information is particularly important for accurate navigation and object recognition), hearing: reduced to 0.2 (in quiet environments, the importance of auditory information is reduced), touch: reduced to 0.15 (in indoor environments, the importance of tactile information is relatively low), environmental sensors: increased to 0.05 (environmental sensors can provide additional information about the indoor environment, such as temperature and humidity, which are also important for task execution).

[0038] The initial default weights refer to the pre-assigned weights for different sensor data without additional information. The environmental analysis model is a pre-trained machine learning model that can analyze multimodal data and identify the current environmental scenario. The scenario refers to the specific environment in which the robot is located, such as indoors, outdoors, and in noisy environments. The current application scenario refers to the specific environmental scenario in which the robot is performing a task. The reliability score is an assessment of the credibility and effectiveness of each modal data in the current application scenario. The dynamically adjusted weights are the result of real-time adjustments to the weights of each modal data based on the reliability score of the current environmental scenario and data.

[0039] In the embodiment of the present application, dynamic weight adjustment enables the fusion process to adjust the contribution of each modality according to its dynamic importance, so that the fused feature vector can more accurately reflect the comprehensive information of different modalities. The fused comprehensive feature vector can more comprehensively characterize the current environmental state, provide the robot with richer contextual information, and facilitate the robot's in-depth understanding of the scene.

[0040] For example, if the environment is noisy, the weight of auditory information may be reduced, while the weight of visual and tactile information may be increased accordingly.

[0041] In an embodiment of the present application, the specific process of generating a pre-trained environmental analysis model includes: collecting and pre-processing historical sensor data when the robot performs tasks in different environments; marking corresponding scene labels for the pre-processed historical sensor data to obtain multiple sample data; using a neural network to create an environmental analysis model; inputting multiple sample data into the environmental analysis model for machine learning, and outputting a model loss value when the learning is completed; when the model loss value reaches the minimum, generating a pre-trained environmental analysis model; or, when the loss value does not reach the minimum, continuing to execute the step of inputting multiple sample data into the environmental analysis model until the model loss value reaches the minimum; the model loss function of the environmental analysis model is:

[0042] in, is the model loss value, is a hyperparameter used to control the transition point of the loss function from square loss to linear loss. is the annotated scene label, is the scene label predicted by the neural network, Indicates the error between the annotated scene label and the predicted scene label, When the error is less than or equal to the hyperparameter, the loss function uses the square error, that is , When the error is greater than the hyperparameter, the loss function uses linear error, that is .

[0043] Among them, historical sensor data refers to sensor data collected by the robot when performing tasks in the past. These data record the robot's perception information in different environments. Preprocessing is a series of processing steps performed on raw sensor data, such as denoising, normalization, feature extraction, etc. Scene labels correspond to labels for specific environments or situations, and are used to classify and identify different application scenarios, such as "indoor", "outdoor", "noisy environment", etc. Sample data is data that has been preprocessed and annotated with scene labels, and is used to train machine learning models. A neural network is a computational model that mimics the structure of a human brain neuron network. Machine learning is a technology that enables computer systems to use data to continuously improve performance, which here refers to training environmental analysis models to identify different environmental scenarios.

[0044] In the embodiment of the present application, the process of the environmental analysis model not only improves the robot's ability to perceive and recognize the environment, but also enhances the robot's adaptability and decision-making efficiency through continuous machine learning optimization, providing a solid foundation for the robot to operate effectively in complex environments.

[0045] Among them, each modality data includes visual modality data, auditory modality data and tactile modality data.

[0046] In an embodiment of the present application, according to the current application scenario of the robot, the specific process of determining the reliability score of each modal data includes: obtaining the current visual state, the current auditory state, and the current tactile state corresponding to the current application scenario of the robot; loading a pre-established first mapping relationship between the visual state and the reliability score, a second mapping relationship between the auditory state and the reliability score, and a third mapping relationship between the tactile state and the reliability score; from the first mapping relationship, obtaining the reliability score corresponding to the current visual state; from the second mapping relationship, obtaining the reliability score corresponding to the current auditory state; from the third mapping relationship, obtaining the reliability score corresponding to the current tactile state; using the reliability scores obtained above as the reliability scores of each modal data, wherein the visual state includes indoors or outdoors, the auditory state includes quiet or noisy, and the tactile state includes hard objects or soft objects.

[0047] Among them, the current visual state refers to the ambient lighting condition currently detected by the robot's visual sensor, such as "indoor" or "outdoor". The current auditory state refers to the ambient sound condition currently perceived by the robot's auditory sensor, such as "quiet" or "noisy". The current tactile state refers to the surface characteristics of the object currently contacted by the robot's tactile sensor, such as "hard object" or "soft object". A mapping relationship refers to a correspondence relationship that maps one type of data (such as sensor state) to another type of data (such as reliability score). The first mapping relationship refers to the correspondence between the visual state and the reliability score. The second mapping relationship refers to the correspondence between the auditory state and the reliability score. The third mapping relationship refers to the correspondence between the tactile state and the reliability score.

[0048] For example, the robot's sensor detects that it is currently in an indoor environment (visual state), the sound in the environment is low (auditory state), and the surface of the object it touches is hard (tactile state). The system loads three pre-established mapping relationships: The first mapping relationship: the correspondence between the visual state (indoor or outdoor) and the reliability score. The second mapping relationship: the correspondence between the auditory state (quiet or noisy) and the reliability score. The third mapping relationship: the correspondence between the tactile state (hard object or soft object) and the reliability score. The system obtains the corresponding reliability score from the first mapping relationship based on the current visual state (indoor), for example, the reliability score of the visual data in the indoor environment is higher. The system obtains the corresponding reliability score from the second mapping relationship based on the current auditory state (quiet), for example, the reliability score of the auditory data in the quiet environment is higher. The system obtains the corresponding reliability score from the third mapping relationship based on the current tactile state (hard object), for example, the reliability score of the tactile data of the hard object is higher.

[0049] For example Figure 3 As shown in the table, if the robot detects that the current environment is "indoor", the reliability score of the visual data is 0.8 according to the table. If the robot detects that the current environment is "outdoor", the reliability score of the visual data is 0.6. If the robot detects that the current environment is "quiet", the reliability score of the auditory data is 0.7. If the robot detects that the current environment is "noisy", the reliability score of the auditory data is 0.4. If the surface of the object touched by the robot is a "hard object", the reliability score of the tactile data is 0.9. If the surface of the object touched by the robot is a "soft object", the reliability score of the tactile data is 0.5.

[0050] S103, fusing the extracted feature vectors of each modal data according to the dynamically adjusted weights of each modal data to generate a comprehensive feature vector for representing the overall state of the current environment; Among them, the comprehensive feature vector refers to the feature vector that integrates multiple modal data. It can characterize the overall state of the current environment and provide richer information than a single modality.

[0051] Among them, the feature vector of each modal data is obtained by using feature extraction technology.

[0052] In some embodiments of the present application, the feature vectors of each modal data are fused according to the dynamically adjusted weights of each modal data to generate a comprehensive feature vector for characterizing the overall state of the current environment. The specific process includes: obtaining the feature vectors of each modal data and their weights from the extracted feature vectors of each modal data and the dynamically adjusted weights of each modal data; performing weighted summation on the feature vectors of each modal data and their weights to obtain an intermediate feature vector of primary fusion; using a deep learning network to perform nonlinear mapping on the feature vectors of the primary fusion to extract higher-level feature representations; and splicing the extracted higher-level feature representations to obtain a comprehensive feature vector for characterizing the overall state of the current environment.

[0053] Among them, the calculation formula of the intermediate feature vector of primary fusion is:

[0054] in, is the intermediate feature vector of primary fusion, is the total number of modal data, It is The feature vector of the modal data, It is The weight of each modal data; For example Figure 4 As shown in the figure, the feature vectors of each modal data extracted are weighted and summed with their corresponding weights to obtain a primary fused intermediate feature vector. This intermediate feature vector integrates information from different modalities. A deep learning network is used to perform nonlinear mapping on the primary fused feature vector to extract higher-level feature representations. This can be achieved by increasing the number of layers and complexity of the deep learning network to capture complex patterns and relationships in the data. The extracted higher-level feature representations are spliced ​​to form a comprehensive feature vector that can fully characterize the overall state of the current environment. This comprehensive feature vector can be used in the robot's decision-making system to guide its behavior and response.

[0055] S104, analyzing the control instruction according to the comprehensive feature vector through a preset large language model to obtain the final decision result of the robot; Among them, Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, dialogue, etc., and are an important path to artificial intelligence. The preferred large language model in this application is ChatGPT.

[0056] In an embodiment of the present application, the control instruction is analyzed according to the comprehensive feature vector through a preset large language model to obtain the specific process of the robot's final decision result, including: when the control instruction is a text object, obtaining a description text of the text object; when the control instruction is a voice object, using a speech recognition ASR system to convert the voice object into a description text; wherein the description text is used to describe the object content of the text object or the voice object; based on the description text and the comprehensive feature vector, a decision result corresponding to the control instruction is generated through a preset large language model as the robot's final decision result.

[0057] The text object refers to the control instruction in the form of text, such as a command or question input by a user. The voice object refers to the control instruction in the form of voice, such as a verbal instruction of a user.

[0058] In some embodiments of the present application, the specific process of generating a decision result corresponding to a control instruction through a preset large language model based on a description text and a comprehensive feature vector includes: constructing a prompt information set of context information for characterizing the current application scenario of the robot based on the comprehensive feature vector; performing word segmentation processing on the description text to extract multiple task keywords; concatenating multiple task keywords with the prompt information set to obtain a task sequence; inputting the task sequence into a preset large language model, analyzing the task sequence through the preset large language model, and outputting a decision result corresponding to the control instruction.

[0059] The context information prompt information set is used to describe the context of the robot's current application scenario, helping the large language model to better understand the background of the control command. Task keywords are important words or phrases directly related to the task extracted from the description text. These keywords help identify and understand the specific intention of the control command.

[0060] For example, the system constructs a context information prompt information set based on the comprehensive feature vector, such as: "Indoor environment, sufficient light, suitable temperature". The user enters a text instruction: "Please bring me the cup on the table". The system performs word segmentation on the text and extracts the task keywords: "cup", "take". The task keywords are combined with the context information prompt information set to form a task sequence: "Indoor environment, sufficient light, suitable temperature, please bring me the cup on the table". The task sequence is input into the preset large language model. The large language model analyzes the task sequence, understands the user's intention, and combines the context information to output the decision result: "The robot needs to move to the table, identify the cup, and bring it to the user."

[0061] In some embodiments of the present application, based on the comprehensive feature vector, the specific process of constructing a prompt information set of context information for characterizing the current application scenario of the robot includes: obtaining multiple pre-defined context scene labels; the multiple context scene labels include lighting conditions and environmental noise; through the comprehensive feature vector, determining the statistical parameters of the lighting conditions and the quantiles of the environmental noise, comparing the statistical parameters, the quantiles and the preset parameter values, and taking the feature vectors that meet the preset parameter values ​​as sub-features of each context scene label; converting the sub-features of each context scene label into a natural language description to obtain the prompt information corresponding to each context scene label; storing the prompt information corresponding to each context scene label as a JSON object to obtain a prompt information set of context information for characterizing the current application scenario of the robot.

[0062] For example, after analysis based on the comprehensive feature vector, if the average light intensity is higher than a certain threshold, it is described as "well-lit". If the quantile of the noise level shows that it is in a high noise state most of the time, it is described as "noisy". If the correlation coefficient between temperature and humidity is close to 1 or -1, it means that there is a strong positive or negative correlation between them, which can be described as "temperature and humidity are highly correlated". If the principal components extracted by PCA explain most of the variance, it can be described as "the terrain characteristics are mainly determined by several principal components". At this time, the above natural language description is stored as a JSON object to form a prompt information set, such as Figure 5 shown.

[0063] The statistical parameters of the lighting conditions include the average lighting intensity; the quantile of the ambient noise is the quartile calculated based on the characteristics of the ambient noise in the comprehensive feature vector.

[0064] S105, based on the final decision result, calling and executing the instruction set that the robot needs to execute, so as to drive the robot to complete the relevant actions corresponding to the instruction set.

[0065] The final decision result is the result obtained by the robot's decision system after analyzing the comprehensive feature vector and control instructions. It indicates the specific task or action that the robot should perform next. The instruction set refers to a set of specific, executable commands, which are the detailed steps required for the robot to execute the final decision result. The related actions refer to the specific physical actions performed by the robot according to the instruction set, such as moving, grasping, releasing, etc.

[0066] In the embodiment of the present application, the robot analyzes the user's control instructions and comprehensive feature vectors through a large language model, and obtains the final decision result: the robot needs to move to the bookshelf, identify the red book, and give it to the user. According to the final decision result, the robot's control system calls the corresponding instruction set, which includes: movement instruction: instruct the robot to move to the position of the bookshelf. Visual recognition instruction: activate the visual system to identify the red book on the bookshelf. Grasping instruction: control the mechanical arm to grab the book. Navigation instruction: instruct the robot to navigate back to the user's position. Release instruction: control the mechanical arm to hand the book to the user. The robot starts to execute each step in the instruction set: the robot moves to the bookshelf using wheels or tracks. The robot's camera starts working and finds the red book through image recognition technology. The robot's mechanical arm moves to the position of the book, adjusts the grasping force according to tactile feedback, and successfully grabs the book. The robot navigation system plans a path to guide the robot back to the user's position. The robot stops in front of the user, and after confirming the user's position according to the visual system, releases the mechanical arm and hands the book to the user. The robot completes the actions corresponding to all instruction sets and successfully executes the user's control instructions.

[0067] In the embodiments of the present application, on the one hand, the dynamic weight adjustment enables the fusion process to adjust the contribution of each modality according to the dynamic importance of each modality, so that the fused feature vector can more accurately reflect the comprehensive information of different modalities, and the fused comprehensive feature vector can more comprehensively characterize the current environmental state and provide the robot with richer contextual information; on the other hand, by utilizing the comprehensive feature vector, the control instructions are analyzed through a preset large language model, so that the robot can deeply understand the user's control instructions, and at the same time, the robot can provide a more natural and more user-expected interactive experience.

[0068] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0069] See also Figure 6, which shows a schematic diagram of the structure of a robot control device based on multimodal data fusion provided by an exemplary embodiment of the present application. The robot control device based on multimodal data fusion can be implemented as all or part of an electronic device through software, hardware or a combination of both. The device 1 includes a multimodal data acquisition module 10, a weight dynamic adjustment module 20, a comprehensive feature vector generation module 30, a decision module 40, and an instruction set calling module 50.

[0070] The multimodal data acquisition module 10 is used to receive and pre-process the perception information fed back by multiple types of sensors pre-set on the robot in response to the control command input by the user to obtain multimodal data; A weight dynamic adjustment module 20, for dynamically adjusting the weight of each modal data and extracting a feature vector of each modal data, wherein the weight of each modal data is calculated based on a reliability score of each modal data, wherein the reliability score of each modal data is determined based on the current application scenario of the robot, wherein the current application scenario is analyzed based on a pre-trained environmental analysis model, wherein the pre-trained environmental analysis model is obtained by machine learning based on historical sensor data when the robot performs tasks in different environments; A comprehensive feature vector generation module 30 is used to fuse the extracted feature vectors of each modal data according to the dynamically adjusted weights of each modal data to generate a comprehensive feature vector for representing the overall state of the current environment; The decision module 40 is used to analyze the control instruction according to the comprehensive feature vector through a preset large language model to obtain the final decision result of the robot; The instruction set calling module 50 is used to call and execute the instruction set required to be executed by the robot according to the final decision result, so as to drive the robot to complete the relevant actions corresponding to the instruction set.

[0071] It should be noted that the robot control device based on multimodal data fusion provided in the above embodiment only uses the division of the above functional modules as an example when executing the robot control method based on multimodal data fusion. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the robot control device based on multimodal data fusion provided in the above embodiment and the robot control method based on multimodal data fusion embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0072] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0073] In the embodiments of the present application, on the one hand, the dynamic weight adjustment enables the fusion process to adjust the contribution of each modality according to the dynamic importance of each modality, so that the fused feature vector can more accurately reflect the comprehensive information of different modalities, and the fused comprehensive feature vector can more comprehensively characterize the current environmental state and provide the robot with richer contextual information; on the other hand, by utilizing the comprehensive feature vector, the control instructions are analyzed through a preset large language model, so that the robot can deeply understand the user's control instructions, and at the same time, the robot can provide a more natural and more user-expected interactive experience.

[0074] The present application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the robot control method based on multimodal data fusion provided by the above-mentioned various method embodiments.

[0075] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the robot control method based on multimodal data fusion of the above-mentioned various method embodiments.

[0076] See also Figure 7 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0077] The communication bus 1002 is used to realize the connection and communication between these components.

[0078] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0079] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0080] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0081] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Figure 7 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a robot control application based on multimodal data fusion.

[0082] exist Figure 7In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 1001 can be used to call the robot control application based on multimodal data fusion stored in the memory 1005, and specifically perform the following operations: In response to the control command input by the user, the robot receives and pre-processes the perception information fed back by multiple types of sensors pre-set on the robot to obtain multi-modal data; Dynamically adjust the weight of each modal data and extract the feature vector of each modal data. The weight of each modal data is calculated based on the reliability score of each modal data. The reliability score of each modal data is determined based on the current application scenario of the robot. The current application scenario is analyzed based on a pre-trained environmental analysis model. The pre-trained environmental analysis model is obtained by machine learning based on historical sensor data when the robot performs tasks in different environments; According to the dynamically adjusted weights of each modal data, the extracted feature vectors of each modal data are fused to generate a comprehensive feature vector for representing the overall state of the current environment; According to the comprehensive feature vector, the control instructions are analyzed through the preset large language model to obtain the final decision result of the robot; Based on the final decision result, the instruction set required by the robot is called and executed to drive the robot to complete the relevant actions corresponding to the instruction set.

[0083] In one embodiment, when dynamically adjusting the weight of each modality data, the processor 1001 specifically performs the following operations: Get the initial default weights of various sensors as the initial weights of each modality data; Feed multimodal data into pre-trained environmental analysis models; Output the scene corresponding to the multimodal data as the current application scene of the robot; Determine the reliability score of each modal data based on the current application scenario of the robot; According to the initial weight of each modal data and the reliability score of each modal data, the weight of each modal data adjusted dynamically is calculated; the weight calculation formula of each modal data adjusted dynamically is:

[0084] in, For the The final adjusted weights of the modal data are is the adjustment factor, which is used to control the sensitivity of weight adjustment. It is Modal data in the current application scenario The reliability scoring function under is the index variable, which is used to traverse all modal data during the summation process. Indicates the total number of modal data.

[0085] In one embodiment, when the processor 1001 generates a pre-trained environment analysis model, the processor 1001 specifically performs the following operations: Collect and preprocess historical sensor data when the robot performs tasks in different environments; For the pre-processed historical sensor data, the corresponding scene labels are marked to obtain multiple sample data; Use neural networks to create environmental analysis models; Input multiple sample data into the environmental analysis model for machine learning, and output the model loss value when learning is completed; When the model loss value reaches the minimum, a pre-trained environment analysis model is generated; or, when the loss value does not reach the minimum, the step of inputting multiple sample data into the environment analysis model is continued until the model loss value reaches the minimum; The model loss function of the environmental analysis model is:

[0086] in, is the model loss value, is a hyperparameter used to control the transition point of the loss function from square loss to linear loss. is the annotated scene label, is the scene label predicted by the neural network, Indicates the error between the annotated scene label and the predicted scene label, When the error is less than or equal to the hyperparameter, the loss function uses the square error, that is , When the error is greater than the hyperparameter, the loss function uses linear error, that is .

[0087] In one embodiment, when the processor 1001 determines the reliability score of each modal data according to the current application scenario of the robot, the processor 1001 specifically performs the following operations: Obtain the current visual state, current auditory state, and current tactile state corresponding to the current application scenario of the robot; Loading a pre-established first mapping relationship between visual state and reliability score, a second mapping relationship between auditory state and reliability score, and a third mapping relationship between tactile state and reliability score; Obtaining a reliability score corresponding to the current visual state from the first mapping relationship; Obtaining a reliability score corresponding to the current hearing state from the second mapping relationship; Obtaining a reliability score corresponding to the current tactile state from the third mapping relationship; The reliability scores obtained above are used as the reliability scores of each modal data, where: Visual states include indoors or outdoors, auditory states include quiet or noisy, and tactile states include hard objects or soft objects.

[0088] In one embodiment, when the processor 1001 performs the fusion of the extracted feature vectors of each modal data according to the dynamically adjusted weights of each modal data to generate a comprehensive feature vector for representing the overall state of the current environment, the processor 1001 specifically performs the following operations: Obtaining a feature vector of each modal data and its weight from the extracted feature vector of each modal data and the dynamically adjusted weight of each modal data; The eigenvectors of each modal data and their weights are weighted summed to obtain the intermediate eigenvector of primary fusion. The calculation formula of the intermediate eigenvector of primary fusion is:

[0089] in, is the intermediate feature vector of primary fusion, is the total number of modal data, It is The feature vector of the modal data, It is The weight of each modal data; Use a deep learning network to perform nonlinear mapping on the primary fused feature vectors to extract higher-level feature representations; The extracted higher-level feature representations are concatenated to obtain a comprehensive feature vector that is used to characterize the overall state of the current environment.

[0090] In one embodiment, when the processor 1001 analyzes the control instruction according to the comprehensive feature vector and the preset large language model to obtain the final decision result of the robot, the following operations are specifically performed: When the control instruction is a text object, obtain the description text of the text object; In the case where the control instruction is a voice object, a speech recognition ASR system is used to convert the voice object into a description text; wherein the description text is used to describe the object content of the text object or the voice object; According to the description text and the comprehensive feature vector, the decision result corresponding to the control instruction is generated by the preset large language model as the final decision result of the robot.

[0091] In one embodiment, when the processor 1001 generates a decision result corresponding to the control instruction by using a preset large language model according to the description text and the comprehensive feature vector, the processor 1001 specifically performs the following operations: Constructing a prompt information set for characterizing context information of a current application scenario of the robot according to the comprehensive feature vector; Perform word segmentation on the description text to extract multiple task keywords; Multiple task keywords and prompt information sets are spliced ​​together to obtain a task sequence; The task sequence is input into the preset large language model, the task sequence is analyzed by the preset large language model, and the decision result corresponding to the control instruction is output.

[0092] In one embodiment, when the processor 1001 constructs a prompt information set for context information representing the current application scenario of the robot according to the comprehensive feature vector, the processor 1001 specifically performs the following operations: Acquire multiple predefined context scene tags; the multiple context scene tags include lighting conditions and environmental noise; By synthesizing the eigenvectors, we can determine the statistical parameters of the lighting conditions, the quantiles of the ambient noise, Compare the statistical parameters, quantiles and preset parameter values, and use the feature vectors that meet the preset parameter values ​​as sub-features of each context scene label; Convert the sub-features of each context scene label into a natural language description to obtain the prompt information corresponding to each context scene label; The prompt information corresponding to each context scene label is stored as a JSON object to obtain a prompt information set for characterizing the context information of the current application scene of the robot.

[0093] In the embodiments of the present application, on the one hand, the dynamic weight adjustment enables the fusion process to adjust the contribution of each modality according to the dynamic importance of each modality, so that the fused feature vector can more accurately reflect the comprehensive information of different modalities, and the fused comprehensive feature vector can more comprehensively characterize the current environmental state and provide the robot with richer contextual information; on the other hand, by utilizing the comprehensive feature vector, the control instructions are analyzed through a preset large language model, so that the robot can deeply understand the user's control instructions, and at the same time, the robot can provide a more natural and more user-expected interactive experience.

[0094] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program for robot control based on multimodal data fusion can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program for robot control based on multimodal data fusion can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0095] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A robot control method based on multimodal data fusion, characterized in that: The method comprises: In response to the control command input by the user, the robot receives and pre-processes the perception information fed back by multiple types of sensors pre-set on the robot to obtain multi-modal data; Dynamically adjust the weight of each modal data and extract the feature vector of each modal data, wherein the weight of each modal data is calculated based on the reliability score of each modal data, and the reliability score of each modal data is determined based on the current application scenario of the robot, and the current application scenario is analyzed according to a pre-trained environmental analysis model, and the pre-trained environmental analysis model is obtained by machine learning based on historical sensor data when the robot performs tasks in different environments; According to the dynamically adjusted weights of each modal data, the extracted feature vectors of each modal data are fused to generate a comprehensive feature vector for representing the overall state of the current environment; According to the comprehensive feature vector, the control instruction is analyzed by a preset large language model to obtain a final decision result of the robot; According to the final decision result, the instruction set required to be executed by the robot is called and executed to drive the robot to complete the relevant actions corresponding to the instruction set.

2. The method according to claim 1, characterized in that The dynamic adjustment of the weight of each modality data includes: Get the initial default weights of various sensors as the initial weights of each modality data; Inputting the multimodal data into a pre-trained environmental analysis model; Outputting the scene corresponding to the multimodal data as the current application scene of the robot; Determining the reliability score of each modal data according to the current application scenario of the robot; According to the initial weight of each modal data and the reliability score of each modal data, the weight of each modal data adjusted dynamically is calculated; the weight calculation formula of each modal data adjusted dynamically is: in, For the The final adjusted weights of the modal data are is the adjustment factor, which is used to control the sensitivity of weight adjustment. It is The modal data is in the current application scenario The reliability scoring function under is the index variable, which is used to traverse all modal data during the summation process. Indicates the total number of modal data.

3. The method according to claim 2, characterized in that Follow these steps to generate a pre-trained environment analysis model, including: Collecting and preprocessing historical sensor data of the robot when performing tasks in different environments; For the pre-processed historical sensor data, the corresponding scene labels are marked to obtain multiple sample data; Use neural networks to create environmental analysis models; Inputting the plurality of sample data into the environmental analysis model for machine learning, and outputting a model loss value when learning is completed; When the model loss value reaches a minimum, generating a pre-trained environment analysis model; or, when the loss value does not reach a minimum, continuing to perform the step of inputting the plurality of sample data into the environment analysis model until the model loss value reaches a minimum; The model loss function of the environmental analysis model is: in, is the model loss value, is a hyperparameter used to control the transition point of the loss function from square loss to linear loss. is the annotated scene label, is the scene label predicted by the neural network, Indicates the error between the annotated scene label and the predicted scene label, When the error is less than or equal to the hyperparameter, the loss function uses the square error, that is , When the error is greater than the hyperparameter, the loss function uses linear error, that is .

4. The method according to claim 2, characterized in that: The modal data include visual modal data, auditory modal data and tactile modal data; Determining the reliability score of each modal data according to the current application scenario of the robot includes: Obtaining a current visual state, a current auditory state, and a current tactile state corresponding to a current application scenario of the robot; Loading a pre-established first mapping relationship between visual state and reliability score, a second mapping relationship between auditory state and reliability score, and a third mapping relationship between tactile state and reliability score; Acquire a reliability score corresponding to the current visual state from the first mapping relationship; Acquire a reliability score corresponding to the current hearing state from the second mapping relationship; Acquire a reliability score corresponding to the current tactile state from the third mapping relationship; The reliability scores obtained above are used as the reliability scores of the modal data, where: The visual state includes indoor or outdoor, the auditory state includes quiet or noisy, and the tactile state includes hard object or soft object.

5. The method according to claim 1, characterized in that The feature vectors of each modal data are obtained by using feature extraction technology; The method of fusing the extracted feature vectors of each modal data according to the dynamically adjusted weights of each modal data to generate a comprehensive feature vector for characterizing the overall state of the current environment includes: From the extracted feature vectors of each modal data and the dynamically adjusted weights of each modal data, obtaining the feature vectors of each modal data and their weights; The feature vectors of each modality data and their weights are weighted summed to obtain the intermediate feature vector of primary fusion; wherein the calculation formula of the intermediate feature vector of primary fusion is: in, is the intermediate feature vector of primary fusion, is the total number of modal data, It is The feature vector of the modal data, It is The weight of each modal data; Use a deep learning network to perform nonlinear mapping on the primary fused feature vectors to extract higher-level feature representations; The extracted higher-level feature representations are concatenated to obtain a comprehensive feature vector that is used to characterize the overall state of the current environment.

6. The method according to claim 1, characterized in that The control instruction is analyzed according to the comprehensive feature vector by using a preset large language model to obtain a final decision result of the robot, including: In the case where the control instruction is a text object, obtaining a description text of the text object; In the case where the control instruction is a voice object, a speech recognition ASR system is used to convert the voice object into a description text; wherein the description text is used to describe the object content of the text object or the voice object; According to the description text and the comprehensive feature vector, a decision result corresponding to the control instruction is generated through the preset large language model as the final decision result of the robot.

7. The method according to claim 6, characterized in that The step of generating a decision result corresponding to the control instruction by using the preset large language model according to the description text and the comprehensive feature vector includes: Constructing a prompt information set for characterizing context information of a current application scenario of the robot according to the comprehensive feature vector; Performing word segmentation processing on the description text to extract multiple task keywords; splicing the plurality of task keywords with the prompt information set to obtain a task sequence; The task sequence is input into the preset large language model, the task sequence is analyzed by the preset large language model, and a decision result corresponding to the control instruction is output.

8. The method according to claim 7, characterized in that The step of constructing a prompt information set for characterizing context information of a current application scenario of the robot according to the comprehensive feature vector comprises: Acquire a plurality of predefined context scene tags; the plurality of context scene tags include lighting conditions and environmental noise; Determine the statistical parameters of the lighting conditions and the quantile of the environmental noise through the comprehensive feature vector. Compare the statistical parameter, the quantile and a preset parameter value, and use a feature vector satisfying the preset parameter value as a sub-feature of each context scene label; Convert the sub-features of each context scene label into a natural language description to obtain prompt information corresponding to each context scene label; The prompt information corresponding to each context scene label is stored as a JSON object to obtain a prompt information set for characterizing the context information of the current application scene of the robot.

9. The method according to claim 8, characterized in that The statistical parameter of the lighting condition includes the average lighting intensity; the quantile of the environmental noise is the quartile calculated based on the characteristics of the environmental noise in the comprehensive feature vector; The calculation formula of the average light intensity is: in, is the average light intensity, is the number of features related to lighting conditions in the comprehensive feature vector, It is the first of the general characteristics of lighting conditions. Features.

10. A robot control device based on multimodal data fusion, characterized in that: The device comprises: A multimodal data acquisition module is used to receive and pre-process the perception information fed back by multiple types of sensors pre-set on the robot in response to the control instructions input by the user to obtain multimodal data; A weight dynamic adjustment module, used to dynamically adjust the weight of each modal data and extract the feature vector of each modal data, wherein the weight of each modal data is calculated based on the reliability score of each modal data, and the reliability score of each modal data is determined based on the current application scenario of the robot, and the current application scenario is analyzed according to a pre-trained environmental analysis model, and the pre-trained environmental analysis model is obtained by machine learning based on historical sensor data when the robot performs tasks in different environments; A comprehensive feature vector generation module is used to fuse the feature vectors of each modal data extracted according to the dynamically adjusted weights of each modal data to generate a comprehensive feature vector for representing the overall state of the current environment; A decision-making module, used to analyze the control instruction according to the comprehensive feature vector through a preset large language model to obtain a final decision result of the robot; The instruction set calling module is used to call and execute the instruction set required to be executed by the robot according to the final decision result, so as to drive the robot to complete the relevant actions corresponding to the instruction set.

Images