Intelligent robot interaction method and system based on multi-modal technology and medium

Abstract

The invention discloses an intelligent robot interaction method and system based on a multi-modal technology, and a medium, mainly relates to the technical field of intelligent robot control, and is used for solving the problems that an existing scheme is lack of semantic perception in navigation technology, cannot respond to semantic-level instructions, and is limited in semantic understanding ability; in the aspect of an interaction technology, semantic information in multiple rounds of dialogues cannot be associated, and multi-modal observation information lacks a collaborative mechanism; and on the aspect of anomaly detection, threshold judgment or static rules are depended on, and the false alarm rate is high. Comprising the steps of collecting observation data to obtain a current pose of the intelligent robot; forming multi-modal data by using the RGB image information, the current pose, the continuous RGB images under the sliding window and the corresponding voice stream information, text information and three-dimensional point cloud; calculating the confidence degree of each state change at the current moment, selecting the state change with the highest confidence degree as a decision state, and judging whether abnormal condition response is carried out or not; and generating an interaction strategy based on the multi-modal data and the decision state.

Description

Technical Field

[0001] The present application relates to the field of intelligent robot control technology, and in particular to an intelligent robot interaction method, system and medium based on multimodal technology. Background Art

[0002] For the past decade, traditional intelligent robot control methods (e.g., patrol and interaction) have primarily relied on sensor navigation and pre-programmed behavioral logic. The following details the shortcomings of existing technologies from three perspectives: navigation technology, interaction capabilities, and anomaly detection. In terms of navigation technology, lidar-based navigation systems achieve autonomous mobility by generating high-precision point cloud maps and employing SLAM (Simultaneous Localization and Mapping) algorithms. However, due to a lack of semantic perception, lidar cannot identify the semantic attributes of objects in the environment. This results in the robot being limited to obstacle avoidance and path tracking tasks, and unable to respond to semantic-level commands. Furthermore, navigation systems based on visual SLAM suffer from limited semantic understanding capabilities.

[0003] In terms of interactive capabilities, existing robot interaction technologies mostly use rule engines or simple voice assistants, which have insufficient understanding of context and the system cannot associate semantic information in multiple rounds of conversations; multimodal observation information, such as voice, vision, gestures, etc., are processed independently and lack a collaborative mechanism, resulting in a high error rate and poor user experience.

[0004] In terms of anomaly detection, traditional anomaly detection methods rely on threshold judgment or static rules. For human fall detection, the false alarm rate is high in scenarios such as sitting or bending over. For object status monitoring, infrared sensors are used to detect the humidity of green plants, and false alarms occur frequently.

[0005] In summary, it can be seen that existing robot patrol and interaction technologies have systematic deficiencies in perception dimensions, dynamic adaptability, semantic understanding, and multimodal collaboration, making it difficult to meet the needs of efficient patrols, accurate anomaly detection, and natural interaction in complex scenarios (such as large shopping malls and smart hospitals). Summary of the Invention

[0006] The present application provides an intelligent robot interaction method, system, and medium based on multimodal technology to address the problems of existing solutions in navigation technology, such as the lack of semantic perception, inability to respond to semantic-level instructions, high sensitivity to light, and limited semantic understanding ability; in interaction technology, the inability to associate semantic information in multiple rounds of dialogue and the lack of a collaborative mechanism for multimodal observation information; and in anomaly detection, the reliance on threshold judgment or static rules, resulting in a high false alarm rate.

[0007] In a first aspect, the present application provides an intelligent robot interaction method based on multimodal technology, the method comprising: Observation data is collected using sensors on the intelligent robot body, wherein the sensors include at least: a visual sensor, a lidar sensor, and an inertial measurement unit; the current position of the intelligent robot is obtained through a multi-sensor fusion mapping and positioning algorithm, and a preset local area map is constructed; the text information in the RGB image information collected by the visual sensor is obtained using optical character recognition (OCR) technology; the RGB image information within a preset time period is collected using a sliding window mechanism; the voice stream information corresponding to the RGB image information within the preset time period is recorded using a preset voice board; the RGB image information is mapped from two dimensions to three dimensions using a depth map to obtain a three-dimensional point cloud; multimodal data is formed using the RGB image information, the current position, the continuous RGB images under the sliding window, and the corresponding voice stream information, text information, and three-dimensional point cloud; based on the CLIP joint encoder, the multimodal data is fused to obtain a feature vector; the feature vector and the dynamic behavior weight parameter of the finite state machine are used to calculate the confidence of each state change at the current moment, select the state change with the highest confidence as the decision state, and use the preset double confirmation mechanism to determine whether to respond to abnormal situations; and the large language model is used to generate an interaction strategy based on the multimodal data and the decision state.

[0008] In one implementation of the present application, multimodal data is formed using RGB image information, current posture, continuous RGB images under a sliding window, and corresponding voice stream information, text information, and three-dimensional point cloud, specifically including: For the time( ), forming the following multimodal data: ; in, As the first Intelligent body posture at all times, For the RGB image information collected at all times, and For the first Continuous images and corresponding voice stream information under the time sliding window, For the time The text information extracted from For the corresponding 3D point cloud at the moment.

[0009] In one implementation of the present application, based on the CLIP joint encoder, multimodal data is fused to obtain a feature vector, specifically including: By formula: , get the feature vector ; in, Represents multimodal data, Represents the CLIP joint encoder function.

[0010] In one implementation of the present application, the states involved in the finite state machine include at least: a standby state, a trigger state, an active interaction state, and a passive response state; Among them, the standby state is the state in which the intelligent robot is patrolling or stationary and continuously collecting multimodal observation information; The trigger state is the state where the interaction signal is detected and the pre-response phase is entered; The active interaction state is the state in which the behavior is initiated according to the context; The passive response state is the state of processing user input and initiating behavior.

[0011] In one implementation of the present application, the confidence level of each state change at the current moment is calculated using the feature vector and the finite state machine state change weight parameter, and the state change with the highest confidence level is selected as the decision state, specifically including: Calculate the current time , the confidence level of each state change ( ) is calculated as: ; in, and For the large model, Parameters of state change, K Indicates the total number of state changes, The state change weight parameter introduced to distinguish state priorities; Select the state change with the highest confidence as the decision state: ; in, Indicates the decision status.

[0012] In one implementation of the present application, a preset double confirmation mechanism is used to determine whether to respond to an abnormal situation, specifically including: Get the confidence of each state change; When the state change in response to an abnormal event is not the state change with the highest confidence, the confidence corresponding to the state change in response to an abnormal event exceeds the preset threshold, and a person is detected in the image at the current moment, the HRNet posture estimation model is used to calculate the joint angle abnormality value corresponding to the image at the current moment to determine whether to respond to the abnormal situation.

[0013] In one implementation of the present application, a large language model is used to generate an interaction strategy based on multimodal data and the current decision state, specifically including: pass: , generate interaction strategies ; in, Indicates guiding the large model according to the current decision state Preset prompt words for interaction strategy inference and interaction behavior planning.

[0014] In one implementation of the present application, after generating an interaction strategy based on multimodal data and a current decision state using a large language model, the method further includes: Visual navigation is performed based on the preset local area map constructed in the current frame.

[0015] In a second aspect, the present application provides an intelligent robot interaction system based on multimodal technology, the system comprising: A multimodal perception module is used to collect observation data using sensors of the intelligent robot body; wherein the sensors include at least: a visual sensor, a lidar sensor, and an inertial measurement unit; the current position of the intelligent robot is obtained through a multi-sensor fusion mapping and positioning algorithm, and a preset local area map is constructed; the text information in the RGB image information collected by the visual sensor is obtained using OCR technology; the RGB image information within a preset time period is collected using a sliding window mechanism; the voice stream information corresponding to the RGB image information within the preset time period is collected using a preset voice board; the RGB image information is mapped from two dimensions to three dimensions using a depth map to obtain a three-dimensional point cloud; the RGB image information, the current position, the continuous RGB images under the sliding window and the corresponding voice stream information, text information and three-dimensional point cloud are used to form multimodal data; The analysis and decision-making module is used to fuse multimodal data based on the CLIP joint encoder to obtain feature vectors. It uses the feature vectors and the dynamic behavior weight parameters of the finite state machine to calculate the confidence level of each state change at the current moment, selects the state change with the highest confidence level as the decision state, and uses a preset double confirmation mechanism to determine whether to respond to abnormal situations. The human-like interaction module is used to generate interaction strategies based on multimodal data and decision states using a large language model.

[0016] In a third aspect, the present application provides a non-volatile computer storage medium having computer instructions stored thereon, which, when executed, implement an intelligent robot interaction method based on multimodal technology as described in any one of the above.

[0017] It can be seen from the above technical solutions that this application has the following advantages: This application utilizes large model technology with strong semantic understanding capabilities. It not only has recognition capabilities, but can also make judgments based on the status of people and objects and respond to semantic-level instructions.

[0018] Comprehensively analyze multimodal observation information including voice, vision, gesture, etc., realize the coupling of time and space information, and have the ability of accurate perception.

[0019] Combining large-scale model reasoning and finite state machine technology, it can achieve accurate response in complex environments, and has autonomous patrol, dynamic anomaly detection and anthropomorphic interaction functions, which improves the adaptability and practicality of robots and other intelligent bodies in security, navigation, medical and other scenarios.

[0020] To further illustrate, by fusing data from multiple sensors, including vision, lidar, and IMU, the system breaks through the traditional SLAM algorithm's reliance on a single modality, improving the stability of environmental modeling in scenes with varying lighting conditions (for example, lidar compensates for the performance degradation of vision sensors in low light). By introducing OCR text recognition technology, text information in the scene can be used as a new navigation feature source. For example, recognizing store names in a shopping mall enables semantic-level path planning, such as "navigate to Starbucks," addressing the pain point of traditional geometric navigation, which is unable to respond to semantic commands.

[0021] The sliding window mechanism collects continuous RGB images in a time series (e.g., a 30-frame sequence within 5 seconds) and combines them with the speech stream information from the corresponding time period to construct a spatiotemporal correlation model between vision and speech. For example, when a user says, "Avoid the obstacle I just saw," the system can use the visual information from the corresponding time window to accurately locate the obstacle.

[0022] 3D point cloud mapping technology aligns 2D visual features with 3D spatial data, giving semantic understanding a spatial dimension. For example, when identifying "the third door on the left," 3D coordinates can be used for precise positioning.

[0023] The synergy between the finite state machine and CLIP feature vectors enables dynamic coupling of environmental perception and behavioral strategies. Through confidence calculation, the system can quantitatively evaluate the degree of adaptation of different behaviors.

[0024] The integration of a large language model and multimodal data enables the system to retain contextual information about the conversation. For example, if a user says "check the temperature in the conference room," the subsequent command "lower it by two degrees" does not require repeating the context. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 This is a flow chart of an intelligent robot interaction method based on multimodal technology provided in an embodiment of the present application.

[0027] Figure 2 This is a schematic diagram of a finite state machine design provided in an embodiment of the present application.

[0028] Figure 3 This is a schematic diagram of the internal structure of an intelligent robot interaction system based on multimodal technology provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] It should be understood by those skilled in the art that the embodiments described below are merely preferred embodiments of the present disclosure and do not imply that the present disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely intended to explain the technical principles of the present disclosure and are not intended to limit the scope of protection of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present disclosure.

[0031] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0032] This application belongs to the field of intelligent robotics technology, specifically to a robot patrol control system and method that combines multimodal perception, large-scale model interactive decision-making, and traditional navigation technologies. By integrating visual, depth, voice, and text data, combined with the semantic reasoning capabilities of large-scale models and finite state machine (FSM) design, this system enables autonomous patrolling, dynamic anomaly detection, and anthropomorphic interaction in complex environments, enhancing the robot's adaptability and practicality in scenarios such as security, navigation, and healthcare.

[0033] Implementation of this application requires an embodied intelligent device with autonomous mobility. This device is equipped with a computing unit and sensors such as visual sensors, lidar sensors, and an IMU. The computing unit is equipped with either independent lidar mapping and positioning or multi-sensor fusion mapping and positioning technology environments, as well as available large language models and visual language model interfaces.

[0034] The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0035] The embodiment provides an intelligent robot interaction method based on multimodal technology, such as Figure 1 As shown, the method provided in the embodiment of the present application mainly includes the following steps: Step 110: Utilize the sensors of the intelligent robot to collect observation data; obtain the current posture of the intelligent robot through a multi-sensor fusion mapping and positioning algorithm, and construct a preset local area map; utilize OCR technology to obtain text information in the RGB image information collected by the visual sensor; utilize a sliding window mechanism to collect RGB image information within a preset time period; utilize a preset voice board to record voice stream information corresponding to the RGB image information within a preset time period; perform two-dimensional to three-dimensional mapping of the RGB image information using a depth map to obtain a three-dimensional point cloud; utilize the RGB image information, current posture, continuous RGB images under the sliding window, and corresponding voice stream information, text information, and three-dimensional point cloud to form multimodal data.

[0036] The sensors include at least: a visual sensor, a lidar sensor, and an inertial measurement unit; In some embodiments, multimodal data is generated using RGB image information, current posture, continuous RGB images under a sliding window, and corresponding voice stream information, text information, and a three-dimensional point cloud, specifically including: For the time( ), forming the following multimodal data: ; in, As the first Intelligent body posture at all times, For the RGB image information collected at all times, and For the first Continuous images and corresponding voice stream information under the time sliding window, For the time The text information extracted from For the corresponding 3D point cloud at the moment.

[0037] Based on the above description, those skilled in the art will understand that this step utilizes the intelligent robot's various sensors (visual sensors, lidar sensors, inertial measurement units, etc.) to collect observation data. This multi-sensor fusion mapping and positioning algorithm accurately determines the intelligent robot's current position. This provides a reliable basis for the robot's autonomous navigation and positioning in complex environments, helping it better plan paths, avoid obstacles, and perform various tasks.

[0038] Based on the collected data, a pre-set local area map is constructed, giving the robot a clearer understanding of its surroundings. This local map helps the robot understand the relative relationship between its position and the surrounding environment in real time, providing basic environmental information for subsequent decision-making and actions, and improving the robot's adaptability and operational capabilities in unknown environments.

[0039] Using OCR technology to extract text from RGB images captured by the visual sensor, the robot can identify valuable text content, such as signs and directions, from the image. This allows the robot to obtain more information about the environment and mission objectives, enhancing its understanding of the environment.

[0040] Using a preset voice board, the voice stream corresponding to the RGB image information within a preset time period is collected. Combined with the image information, the voice stream provides the robot with richer contextual information, helping it to more fully understand the scene. For example, in scenarios requiring voice interaction or voice prompts, it can better complete tasks.

[0041] The depth map is used to map the RGB image information from two dimensions to three dimensions, generating a three-dimensional point cloud. This 3D point cloud provides more precise spatial information, giving the robot a more intuitive understanding of the shape, size, and position of objects, and aiding in more complex operations such as grasping and obstacle avoidance.

[0042] Multimodal data is generated using RGB image information, the current position, continuous RGB images within a sliding window, and the corresponding audio stream, text information, and 3D point cloud. This multimodal data integrates multiple aspects of vision, position, audio, text, and 3D space, providing the robot with a more comprehensive and richer environmental perception. Data from different modalities complement each other, making up for the shortcomings of a single modality and improving the robot's perception and understanding of the environment.

[0043] A sliding window mechanism collects RGB image information within a preset time period and combines it with the corresponding voice stream information to form a continuous data record. This helps the robot analyze dynamically changing environments and capture time series information within them, such as object movement and event occurrences. This enables the robot to better adapt to dynamic environments and make more accurate decisions.

[0044] The formation of multimodal data provides a richer basis for robot decision-making. Robots can comprehensively analyze various information and more accurately judge the current environmental conditions and task requirements, thereby making more reasonable and intelligent decisions, improving the efficiency and success rate of task execution.

[0045] This multimodal data processing capability enables robots to be used in a wider range of applications. For example, in areas such as intelligent services, industrial inspections, and logistics and distribution, robots can better complete various complex tasks and meet the needs of different scenarios by analyzing and understanding multimodal data.

[0046] Step 120: Based on the CLIP joint encoder, multimodal data is integrated to obtain a feature vector; the feature vector and the dynamic behavior weight parameters of the finite state machine technology are used to calculate the confidence of each state change at the current moment, and the state change with the highest confidence is selected as the decision state. The preset double confirmation mechanism is used to determine whether to respond to the abnormal situation.

[0047] In the step, based on the CLIP joint encoder, multimodal data is fused to obtain a feature vector, which can be specifically: By formula: , get the feature vector ; in, Represents multimodal data, Represents the CLIP joint encoder function.

[0048] In addition, if Figure 2 As shown, the finite state machine, different states are defined as follows: Standby state: The robot is in patrol or stationary state, continuously collecting multimodal observation information.

[0049] Trigger state: When an interaction signal is detected (such as a user waving or a voice asking "Hello"), the pre-response phase begins.

[0050] Active interaction state: Initiates actions based on context, for example: Greeting: Combining speech (“Hello, how can I help you?”) with gestures (a nod of the head or a raise of the arm).

[0051] Guide service: Provides route guidance based on signboard information recognized by OCR.

[0052] Passive response state: handles user input, for example: Voice response: generates responses after parsing user questions.

[0053] Abnormal handling: When a person falls or an object is detected to be abnormal, an alarm is triggered and navigation is carried out to the target point.

[0054] In the step, the confidence of each state change at the current moment is calculated using the characteristic vector and the finite state machine state change weight parameter, and the state change with the highest confidence is selected as the decision state. Specifically, it can be: Calculate the current time , the confidence level of each state change ( ) is calculated as: ; in, and For the large model, Parameters of state changes (such as Figure 2 shown), K Indicates the total number of state changes, The state change weight parameter introduced to distinguish state priorities; Select the state change with the highest confidence as the decision state: ; in, Indicates the decision status.

[0055] In order to further ensure the priority response to abnormal posture situations, a double confirmation mechanism is proposed. The preset double confirmation mechanism is used to determine whether to respond to abnormal situations. Specifically, it includes: Get the confidence of each state change; When the state change in response to an abnormal event is not the state change with the highest confidence, the confidence corresponding to the state change in response to an abnormal event exceeds the preset threshold, and a person is detected in the image at the current moment, the HRNet posture estimation model is used to calculate the joint angle abnormality value corresponding to the image at the current moment to determine whether to respond to the abnormal situation.

[0056] Step 130: Generate an interaction strategy based on the multimodal data and decision state using the large language model.

[0057] Leveraging a large language model, we generate interaction strategies based on multimodal data and the current decision state. These strategies include: pass: , generate interaction strategies ; in, Indicates guiding the large model according to the current decision state Preset prompt words for interaction strategy inference and interaction behavior planning.

[0058] The following are examples of interaction strategies: Interactive generation: The user asks "Where is the restroom?", and the nearest location is retrieved and the response is generated: "Turn left and 20 meters, on the right side of area A2."

[0059] Anomaly detection: Triggers a voice alert "Fall detected, assistance on the way" and navigates to the target point.

[0060] Navigation task execution: Based on the user instruction of "need to heat up lunch", the system provides a plan of "go to the coffee break area - microwave oven" and performs navigation.

[0061] After generating an interaction strategy based on the multimodal data and the current decision state using the large language model, the method further includes: Visual navigation is performed based on the preset local area map constructed in the current frame.

[0062] In addition, for the current scene, if a three-dimensional scene atlas covering all the perceptual information of the scene has been constructed, navigation can be achieved based on the three-dimensional scene atlas. The three-dimensional scene atlas has a hierarchical feature. By constructing a grid layer containing geometric information, an object layer with rich semantic information, and high-level atlas layers such as room areas and floors, it effectively links the geometric and semantic information in the scene. This application proposes to use RAG technology to build a local knowledge base based on the three-dimensional scene atlas of the scene where the intelligent agent is located to achieve efficient retrieval of scene information. In addition, visual navigation can also be performed based on the local map constructed in the current frame to reach the target point of human-like interaction.

[0063] In addition, this application Figure 3 The embodiment of the present application provides an intelligent robot interaction system based on multimodal technology. Figure 3 As shown, the system provided in the embodiment of the present application mainly includes: The multimodal perception module 210 is used to collect observation data using sensors of the intelligent robot body; wherein the sensors include at least: a visual sensor, a lidar sensor, and an inertial measurement unit; obtain the current position of the intelligent robot through a multi-sensor fusion mapping and positioning algorithm, and construct a preset local area map; use optical character recognition (OCR) technology to obtain text information in the RGB image information collected by the visual sensor; use a sliding window mechanism to collect RGB image information within a preset time period; use a preset voice board to record voice stream information corresponding to the RGB image information within the preset time period; use a depth map to map the RGB image information from two dimensions to three dimensions to obtain a three-dimensional point cloud; and use the RGB image information, current position, continuous RGB images under the sliding window, and corresponding voice stream information, text information, and three-dimensional point cloud to form multimodal data; Analysis and decision module 220 is used to fuse multimodal data to obtain feature vectors based on the CLIP joint encoder; use the feature vectors and the dynamic behavior weight parameters of the finite state machine to calculate the confidence level of each state change at the current moment, select the state change with the highest confidence level as the decision state, and use a preset double confirmation mechanism to determine whether to respond to the abnormal situation; The human-like interaction module 230 is used to generate interaction strategies based on multimodal data and decision states using a large language model.

[0064] In addition, an embodiment of the present application further provides a non-volatile computer storage medium on which executable instructions are stored. When the executable instructions are executed, an intelligent robot interaction method based on multimodal technology as described above is implemented.

[0065] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An intelligent robot interaction method based on multimodal technology, characterized in that: The method comprises: Observation data is collected using sensors of the intelligent robot body; wherein the sensors include at least: a visual sensor, a lidar sensor, and an inertial measurement unit; the current position of the intelligent robot is obtained through a multi-sensor fusion mapping and positioning algorithm, and a preset local area map is constructed; for the RGB image information collected by the visual sensor, the text information in the RGB image information is obtained using OCR technology; the RGB image information within a preset time period is collected using a sliding window mechanism; the voice stream information corresponding to the RGB image information within the preset time period is recorded using a preset voice board; the RGB image information is mapped from two dimensions to three dimensions using a depth map to obtain a three-dimensional point cloud; multimodal data is formed using the RGB image information, the current position, the continuous RGB images under the sliding window, and the corresponding voice stream information, text information, and three-dimensional point cloud; Based on the CLIP joint encoder, multimodal data is integrated to obtain feature vectors. The feature vectors and dynamic behavior weight parameters of finite state machine technology are used to calculate the confidence level of each state change at the current moment. The state change with the highest confidence level is selected as the decision state. A preset double confirmation mechanism is used to determine whether to respond to abnormal situations. Leverage large language models to generate interaction strategies based on multimodal data and decision states.

2. The intelligent robot interaction method based on multimodal technology according to claim 1, characterized in that: Multimodal data is generated using RGB image information, current posture, continuous RGB images under the sliding window, and corresponding voice stream information, text information, and 3D point cloud, specifically including: For the time( ), forming the following multimodal data: ； in, As the first Intelligent body posture at all times, For the RGB image information collected at all times, and For the first Continuous images and corresponding voice stream information under the time sliding window, For the time The text information extracted from For the corresponding 3D point cloud at the moment.

3. The intelligent robot interaction method based on multimodal technology according to claim 1, characterized in that: Based on the CLIP joint encoder, multimodal data is fused to obtain feature vectors, including: By formula: , get the feature vector ; in, Represents multimodal data, Represents the CLIP joint encoder function.

4. The intelligent robot interaction method based on multimodal technology according to claim 1, characterized in that: The states involved in the finite state machine include at least: standby state, trigger state, active interaction state and passive response state; Among them, the standby state is the state in which the intelligent robot is patrolling or stationary and continuously collecting multimodal observation information; The trigger state is the state where the interaction signal is detected and the pre-response phase is entered; The active interaction state is the state in which the behavior is initiated according to the context; The passive response state is the state of processing user input and initiating behavior.

5. The intelligent robot interaction method based on multimodal technology according to claim 1, characterized in that: Using the eigenvector and the finite state machine state change weight parameter, the confidence of each state change at the current moment is calculated, and the state change with the highest confidence is selected as the decision state, specifically including: Calculate the current time , the confidence level of each state change ( ) is calculated as: ； in, and For the large model, The state change parameters, K Indicates the total number of state changes, The state change weight parameter introduced to distinguish state priorities; Select the state change with the highest confidence as the decision state: ； in, Indicates the decision status.

6. The intelligent robot interaction method based on multimodal technology according to claim 5, characterized in that: Use the preset double confirmation mechanism to determine whether to respond to abnormal situations, including: Get the confidence of each state change; When the state change in response to an abnormal event is not the state change with the highest confidence, the confidence corresponding to the state change in response to an abnormal event exceeds the preset threshold, and a person is detected in the image at the current moment, the HRNet posture estimation model is used to calculate the joint angle abnormality value corresponding to the image at the current moment to determine whether to respond to the abnormal situation.

7. The intelligent robot interaction method based on multimodal technology according to claim 1, characterized in that: Leveraging a large language model, we generate interaction strategies based on multimodal data and the current decision state. These strategies include: pass: , generate interaction strategies ; in, Indicates guiding the large model according to the current decision state Preset prompt words for interaction strategy inference and interaction behavior planning.

8. The intelligent robot interaction method based on multimodal technology according to claim 1, characterized in that: After generating an interaction strategy based on the multimodal data and the current decision state using the large language model, the method further includes: Visual navigation is performed based on the preset local area map constructed in the current frame.

9. An intelligent robot interaction system based on multimodal technology, characterized in that: The system comprises: A multimodal perception module is used to collect observation data using sensors of the intelligent robot body; wherein the sensors include at least: a visual sensor, a lidar sensor, and an inertial measurement unit; the current position of the intelligent robot is obtained through a multi-sensor fusion mapping and positioning algorithm, and a preset local area map is constructed; the text information in the RGB image information collected by the visual sensor is obtained using OCR technology; the RGB image information within a preset time period is collected using a sliding window mechanism; the voice stream information corresponding to the RGB image information within the preset time period is collected using a preset voice board; the RGB image information is mapped from two dimensions to three dimensions using a depth map to obtain a three-dimensional point cloud; the RGB image information, the current position, the continuous RGB images under the sliding window and the corresponding voice stream information, text information and three-dimensional point cloud are used to form multimodal data; The analysis and decision-making module is used to fuse multimodal data based on the CLIP joint encoder to obtain feature vectors. It uses the feature vectors and the dynamic behavior weight parameters of the finite state machine to calculate the confidence level of each state change at the current moment, selects the state change with the highest confidence level as the decision state, and uses a preset double confirmation mechanism to determine whether to respond to abnormal situations. The human-like interaction module is used to generate interaction strategies based on multimodal data and decision states using a large language model.

10. A non-volatile computer storage medium, characterized in that Computer instructions are stored thereon, and when the computer instructions are executed, the intelligent robot interaction method based on multimodal technology as described in any one of claims 1 to 8 is implemented.

Images