Visual interaction method and system for robot with body, terminal and medium
By acquiring and processing a variety of visual image data, combining multimodal feature fusion algorithm and three-dimensional reconstruction technology, the problem of low spatial perception accuracy of embodied robot vision systems in complex scenarios is solved, and the interaction efficiency and safety of the robot and the environment are improved. sex.
Patent Information
- Application Number
- CN202411403172.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2024-11-29
AI Technical Summary
The existing embodied robot vision system has low spatial perception accuracy in complex scenarios, which affects user experience and interaction efficiency.
RGB images, depth images and infrared image data are obtained, and processed through pre-trained image processing models, and deep fusion is performed using multimodal feature fusion algorithm to perform three-dimensional reconstruction and scene analysis, and identify key interactive objects and obstacles. Generate interaction schemes and behavioral strategies.
It improves the robot's perception accuracy and interaction efficiency of complex environments, and enhances the safety and intelligence of the robot's interaction with the environment.
Smart Images

Figure CN119036461A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of motor control, and in particular to a method, system, terminal and medium for embodied robot visual interaction. Background Art
[0002] At present, with the rapid development of robotics technology, embodied robots have been applied in many fields. In the development of embodied intelligent robots, the visual system, as its core component, plays a vital role in the robot's environmental perception, scene understanding and intelligent interaction. Traditional robot vision systems mostly use planar imaging technology, which faces problems such as low spatial perception accuracy and limited understanding of complex scenes, affecting user experience.
[0003] The above-mentioned existing technical solutions have the following defects: At present, the mainstream embodied robots on the market usually adopt traditional computer vision algorithms for interaction. These algorithms have low recognition accuracy when processing complex scenes, so there is room for improvement. Summary of the invention
[0004] In order to improve the accuracy of embodied robot visual interaction, the present application provides an embodied robot visual interaction method, system, terminal and medium.
[0005] The above-mentioned invention objective of the present application is achieved through the following technical solutions: A method for visual interaction of an embodied robot, the method comprising: Acquire visual image data of the robot, wherein the visual image data includes an RGB image, a depth image, and an infrared image; Inputting the visual image data into a pre-trained image processing model for processing to obtain feature point data of the visual image data; Using a multimodal feature fusion algorithm to deeply fuse the feature point data of the visual image data to generate spatial perception data of the robot's surrounding environment; Based on the spatial perception data, three-dimensional reconstruction and scene analysis are performed to identify key interactive objects and obstacles in the environment where the robot is located, and obtain identification and analysis results; Generate an interaction plan for the robot according to the recognition and analysis results, wherein the interaction plan includes path planning, interaction actions with objects, and voice intonation; Adjust the robot's behavior strategy according to the robot's interaction plan By adopting the above technical solution, by acquiring the robot's visual image data, it is possible to capture a variety of image information in the environment, including RGB images, depth images and infrared images, thereby improving the perception accuracy and depth of the robot's surrounding environment; at the same time, the visual image data is input into a pre-trained image processing model for processing, which can understand and analyze the environment based on rich feature point data, thereby improving the robot's environmental perception ability; by using a multimodal feature fusion algorithm to deeply fuse the feature point data, it is possible to fully integrate the data of different sensors, thereby improving the robot's spatial perception ability in complex environments; finally, three-dimensional reconstruction and scene analysis based on spatial perception data can more accurately identify key interactive objects and obstacles around the robot, thereby improving the interaction efficiency and safety of the robot and the environment.
[0006] In a preferred example, the present application may be further configured as follows: the obtaining of the robot's visual image data includes: Acquire external environmental conditions, and based on the external environmental conditions, use a visual sensor to collect data to obtain sensor data; The sensor data are synchronously calibrated and aligned to obtain visual image data of the robot.
[0007] By adopting the above technical solution, by acquiring external environmental conditions, the collection method of the visual sensor can be dynamically adjusted according to different environmental conditions, thereby improving the adaptability of data collection; by using visual sensors to collect data and synchronously calibrating and aligning the sensor data, it can be ensured that the acquired data is consistent and accurate, thereby improving the quality of the robot's visual image data; the final visual image data can provide an accurate and reliable data basis for subsequent image processing steps, thereby improving the robot's visual processing effect in complex environments.
[0008] In a preferred example, the present application may be further configured as follows: before inputting the visual image data into a pre-trained image processing model for processing to obtain feature point data of the visual image data, the embodied robot visual interaction method further includes: Collecting multimodal data, preprocessing and labeling the multimodal data to obtain a training set; The training set is used to perform forward propagation and back propagation training on the image processing model built based on the convolutional neural network and the self-attention mechanism, and the transfer learning technology is applied to optimize the image processing model after the forward propagation and back propagation training to obtain the pre-trained image processing model.
[0009] By adopting the above technical solutions, by collecting multimodal data and preprocessing and annotating it, it is possible to provide high-quality training sets for the training of image processing models, thereby improving the effectiveness and accuracy of model training; using an image processing model based on a convolutional neural network and a self-attention mechanism for forward and back propagation training, the model can learn more features through training with a large amount of data, thereby improving the model's ability to process visual image data; optimizing the image processing model through transfer learning technology can speed up the training of the model and improve the generalization ability of the model, so that the robot can maintain a high image processing accuracy in different environments.
[0010] In a preferred example, the present application may be further configured as follows: the visual image data is input into a pre-trained image processing model for processing, and the feature point data of the visual image data is obtained, including: Extracting feature points from the visual image data using a multimodal feature extraction algorithm to obtain an extracted feature point data set, wherein the extracted feature point data set includes edge features, texture features, depth boundaries, and thermal sensing areas; The data in the extracted feature point data set is matched and fused through a multi-layer fusion network to obtain the feature point data of the visual image data.
[0011] By adopting the above technical solution, by using a multimodal feature extraction algorithm to extract feature points of visual image data, it is possible to extract a variety of information such as edge features, texture features, depth boundaries and thermal sensing areas, thereby improving the perception dimension and accuracy of the environment; by matching and fusing feature point data sets through a multi-layer fusion network, it is possible to fully integrate data from different modalities and improve the robot's ability to understand complex environments; the feature point data of the visual image data finally obtained can provide more accurate environmental perception information for the robot's subsequent interaction strategies, thereby improving the intelligence and adaptability of the robot's interaction with the environment.
[0012] In a preferred example, the present application can be further configured as follows: performing three-dimensional reconstruction and scene analysis based on the spatial perception data, identifying key interactive objects and obstacles in the environment where the robot is located, and obtaining identification and analysis results including: Based on the spatial perception data, the depth information of objects and obstacles in the environment where the robot is located is converted into a three-dimensional model through a three-dimensional point cloud generation algorithm; Performing semantic segmentation on the three-dimensional model to obtain semantic segmentation data; The semantic segmentation data is analyzed to obtain the recognition analysis result.
[0013] By adopting the above technical solution, three-dimensional reconstruction and scene analysis are performed based on spatial perception data, and the depth information of objects and obstacles in the robot's environment can be converted into an intuitive three-dimensional model, thereby improving the precise perception of the environment; semantic segmentation of the three-dimensional model can automatically identify the categories and positions of different objects and obstacles, thereby improving the depth of understanding of the environment; through the analysis of semantic segmentation data, the recognition and analysis results in the robot environment can be obtained quickly and accurately, thereby improving the robot's scene recognition and decision-making capabilities.
[0014] In a preferred example, the present application may be further configured as follows: generating the interaction scheme of the robot according to the recognition analysis result includes: Determining the priority of the interaction strategy according to the recognition and analysis results, wherein the recognition and analysis results include the spatial distribution of the interaction object, obstacles, and the surrounding environment; Based on the priority of the interaction strategy, the interaction plan is calculated and generated using a real-time path planning algorithm.
[0015] By adopting the above technical solution, the robot's interaction plan is generated according to the recognition and analysis results. It can quickly formulate an interaction strategy that adapts to the current scene according to the interactive objects, obstacles and spatial distribution in the environment, thereby improving the robot's interaction efficiency in complex environments; the interaction plan is generated based on the real-time path planning algorithm, which can ensure that the robot remains efficient and safe in path planning and interaction with objects, thereby improving the success rate of the robot's task execution and user experience.
[0016] In a preferred example, the present application may be further configured as follows: the embodied robot visual interaction method further includes: Acquire interactive feedback information, and adjust and optimize the behavior strategy of the robot in real time according to the interactive feedback information; When executing user instructions, obtaining the user's task requirements and generating a task execution plan according to the task requirements; During the task execution process, the task progress is monitored in real time. When a task deviation is detected, the deviation data is obtained, and the task execution plan is adjusted according to the deviation data. At the same time, the task status is fed back to the user through the display interface.
[0017] By adopting the above technical solution, interactive feedback information is obtained, and the behavior strategy of the robot is adjusted and optimized in real time according to the interactive feedback information. Through real-time adjustment, the robot can better adapt to environmental changes and user needs, and improve the efficiency and quality of interaction; when executing user instructions, the user's task requirements are obtained, and a task execution plan is generated according to the task requirements. The task execution plan provides the robot with a clear action guide to ensure that the task can be carried out in an orderly manner; during the task execution process, the task progress is monitored in real time. When a task deviation is detected, the deviation data is obtained, and the task execution plan is adjusted according to the deviation data. At the same time, the task status is fed back to the user through the display interface. Through adjustment, the robot can correct the deviation and return to the correct task execution trajectory to ensure the realization of the task goals, and the user can understand the task execution status in a timely manner, thereby enhancing user experience and satisfaction.
[0018] The second object of the invention is achieved by the following technical solutions: An embodied robot visual interaction system, the embodied robot visual interaction system comprising: An image acquisition module is used to acquire visual image data of the robot, wherein the visual image data includes RGB images, depth images and infrared images; A model analysis module, used for inputting the visual image data into a pre-trained image processing model for processing to obtain feature point data of the visual image data; A fusion module, used to perform deep fusion of feature point data of the visual image data using a multimodal feature fusion algorithm to generate spatial perception data of the robot's surrounding environment; A recognition and analysis module, used to perform three-dimensional reconstruction and scene analysis based on the spatial perception data, identify key interactive objects and obstacles in the environment where the robot is located, and obtain recognition and analysis results; Obtaining an interaction scheme module, for generating an interaction scheme of the robot according to the recognition and analysis results, wherein the interaction scheme includes path planning, interaction actions with objects, and voice intonation; An adjustment module is used to adjust the behavior strategy of the robot according to the interaction scheme of the robot.
[0019] By adopting the above technical solution, by acquiring the robot's visual image data, it is possible to capture a variety of image information in the environment, including RGB images, depth images and infrared images, thereby improving the perception accuracy and depth of the robot's surrounding environment; at the same time, the visual image data is input into a pre-trained image processing model for processing, which can understand and analyze the environment based on rich feature point data, thereby improving the robot's environmental perception ability; by using a multimodal feature fusion algorithm to deeply fuse the feature point data, it is possible to fully integrate the data of different sensors, thereby improving the robot's spatial perception ability in complex environments; finally, three-dimensional reconstruction and scene analysis based on spatial perception data can more accurately identify key interactive objects and obstacles around the robot, thereby improving the interaction efficiency and safety of the robot and the environment.
[0020] The third objective of the present application is achieved through the following technical solutions: A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned embodied robot visual interaction method when executing the computer program.
[0021] The fourth objective of the present application is achieved through the following technical solutions: A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned embodied robot visual interaction method.
[0022] In summary, the present application includes at least one of the following beneficial technical effects: 1. Obtaining the robot's visual image data can capture a variety of image information in the environment, including RGB images, depth images, and infrared images, thereby improving the perception accuracy and depth of the robot's surrounding environment; at the same time, inputting the visual image data into a pre-trained image processing model for processing can understand and analyze the environment based on rich feature point data, thereby improving the robot's environmental perception ability; 2. By using a multimodal feature fusion algorithm to deeply fuse feature point data, the data from different sensors can be fully integrated, thereby improving the robot's spatial perception ability in complex environments; finally, three-dimensional reconstruction and scene analysis based on spatial perception data can more accurately identify key interactive objects and obstacles around the robot, thereby improving the efficiency and safety of the robot's interaction with the environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a flow chart of a method for visual interaction of an embodied robot in one embodiment of the present application; Figure 2 is a flowchart for implementing step S10 in a method for visual interaction of an embodied robot in one embodiment of the present application; Figure 3 is a flowchart for implementing step S20 in a method for visual interaction of an embodied robot in one embodiment of the present application; Figure 4 is a flowchart for implementing step S20 in a method for visual interaction of an embodied robot in an embodiment of the present application; Figure 5 is a flowchart for implementing step S40 in a method for visual interaction of an embodied robot in an embodiment of the present application; Figure 6 is a flowchart for implementing step S50 in a method for visual interaction of an embodied robot in an embodiment of the present application; Figure 7 is a flowchart for implementing a method for visual interaction of an embodied robot in one embodiment of the present application; Figure 8 It is a principle block diagram of an embodied robot visual interaction system in one embodiment of the present application; Fig. 9 It is a schematic diagram of a device in an embodiment of the present application. DETAILED DESCRIPTION
[0024] The present application is further described in detail below in conjunction with the accompanying drawings.
[0025] In one embodiment, if Figure 1 As shown, the present application discloses an embodied robot visual interaction method, which specifically includes the following steps: S10: Acquire visual image data of the robot, where the visual image data includes RGB images, depth images, and infrared images.
[0026] Specifically, the cameras and sensors equipped on the robot collect RGB images, depth images that show information about the distance of objects, and infrared images that can capture images under different lighting conditions. These different types of image data provide the robot with comprehensive information about its surroundings.
[0027] S20: Input the visual image data into a pre-trained image processing model for processing to obtain feature point data of the visual image data.
[0028] Specifically, the collected visual image data is input into a pre-trained image processing model. The model can extract key feature points from the image data to provide a basis for subsequent analysis and decision-making. The image processing model outputs feature point data, which is the key information in the image, such as the edges, corners and textures of objects. Feature point data helps the robot better understand the geometry and physical properties of its surrounding environment.
[0029] S30: Use a multimodal feature fusion algorithm to deeply fuse the feature point data of the visual image data to generate spatial perception data of the robot's surrounding environment.
[0030] Specifically, a multimodal feature fusion algorithm is applied to integrate the feature point data of RGB images, depth images and infrared images. Based on the fused feature point data, spatial perception data is generated. These data describe the three-dimensional structure and layout of the robot's surrounding environment. The fused data provides richer and more accurate environmental information, which helps to improve the robot's spatial perception ability.
[0031] S40: Based on the spatial perception data, perform three-dimensional reconstruction and scene analysis to identify key interactive objects and obstacles in the robot's environment and obtain recognition and analysis results.
[0032] Specifically, by using spatial perception data for three-dimensional reconstruction, creating a three-dimensional model of the environment, and performing scene analysis, the robot can identify key interactive objects in the environment, such as people, doors, tools, and obstacles, and provide information for safe navigation and effective interaction. Through scene analysis, identification and analysis results are obtained. These results include important elements in the environment and their characteristics, providing the robot with a basis for making decisions and actions.
[0033] S50: Generate an interaction plan for the robot according to the recognition and analysis results. The interaction plan includes path planning, interaction actions with objects, and voice intonation.
[0034] Specifically, based on the recognition and analysis results, the robot's interaction plan is generated, including path planning, interactive actions with objects, and voice intonation, such as how the robot reaches the destination, how to grab and push objects, etc. The interaction plan enables the robot to interact with the environment and people in an appropriate manner.
[0035] S60: Adjust the robot's behavior strategy according to the robot's interaction plan.
[0036] Specifically, according to the interaction plan, the robot's behavioral strategies, such as motion control, task execution order, and choice of interaction method, are dynamically adjusted to adapt to the current environment and task requirements. The robot can flexibly respond to different environments and situations, improving its autonomy and adaptability.
[0037] By adopting the above technical solution, the robot's visual image data is obtained, and a variety of image information in the environment can be captured, including RGB images, depth images and infrared images, thereby improving the perception accuracy and depth of the robot's surrounding environment; at the same time, the visual image data is input into a pre-trained image processing model for processing, which can understand and analyze the environment based on rich feature point data, thereby improving the robot's environmental perception ability; by using a multimodal feature fusion algorithm to deeply fuse the feature point data, the data of different sensors can be fully integrated, thereby improving the robot's spatial perception ability in complex environments; finally, three-dimensional reconstruction and scene analysis based on spatial perception data can more accurately identify key interactive objects and obstacles around the robot, thereby improving the interaction efficiency and safety of the robot and the environment.
[0038] In one embodiment, if Figure 2 As shown, in step S10, the visual image data of the robot is obtained, which specifically includes: S11: Acquire external environmental conditions, and based on the external environmental conditions, use a visual sensor to collect data to obtain sensor data.
[0039] Specifically, because the external environment will affect the performance of the robot's sensors and the accuracy of data collection, it is first necessary to perceive the external environmental conditions through environmental sensors. These conditions may include light intensity, weather conditions, temperature, humidity, etc., and summarize the acquired data to obtain the external environmental conditions. Based on the external environmental conditions, the robot uses the onboard visual sensors to collect data. The visual sensors may include various types of cameras, such as RGB cameras, depth cameras, infrared cameras, etc., to finally obtain sensor data.
[0040] S12: synchronously calibrate and align the sensor data to obtain the robot's visual image data.
[0041] Specifically, since the robot may use multiple visual sensors to collect data from different angles and distances, these data need to be calibrated synchronously. By synchronously calibrating and aligning the sensor data, the robot obtains a unified set of visual image data, namely the robot's visual image data, which provides the robot with a comprehensive and accurate view of the external environment, which can be used for subsequent image processing, feature extraction, scene understanding and other tasks.
[0042] In one embodiment, if Figure 3 As shown, before step S20, that is, before the visual image data is input into a pre-trained image processing model for processing to obtain feature point data of the visual image data, the embodied robot visual interaction method further includes: S201: Collect multimodal data, preprocess and label the multimodal data, and obtain a training set.
[0043] Specifically, data from different sources and modalities are collected, such as RGB, depth, infrared images, audio, text, etc. Multimodal data provides more comprehensive information, which helps to improve the model's understanding and processing capabilities. The collected multimodal data is preprocessed by cleaning, formatting, standardization and other preprocessing operations, and the data is labeled, such as identifying objects in the image and assigning labels. The preprocessed and labeled data is organized into a training set, which is a data set used to train machine learning models. The preprocessed and labeled data is more suitable for training machine learning models, which helps to improve the accuracy and generalization ability of the model.
[0044] S202: Use the training set to perform forward propagation and back propagation training on the image processing model built based on the convolutional neural network and the self-attention mechanism, and apply the transfer learning technology to optimize the image processing model after the forward propagation and back propagation training to obtain a pre-trained image processing model.
[0045] Specifically, the training set is used to train the image processing model based on CNN and self-attention mechanism, including forward propagation and back propagation, that is, calculating the prediction results and adjusting the model parameters according to the difference between the prediction results and the actual results. Through training, the model learns how to extract features from image data and perform effective image processing. Using transfer learning technology, the model parameters or feature extractors that have been trained on other tasks are applied to the current model to accelerate the training process and improve model performance. After training and optimization, a pre-trained image processing model is obtained, which can process and analyze image data.
[0046] In one embodiment, if Figure 4 As shown, in step S20, the visual image data is input into a pre-trained image processing model for processing to obtain feature point data of the visual image data, specifically including: S21: Extract feature points from visual image data using a multimodal feature extraction algorithm to obtain an extracted feature point data set, where the extracted feature point data set includes edge features, texture features, depth boundaries, and thermal sensing areas.
[0047] Specifically, a multimodal feature extraction algorithm is applied to process different types of visual image data, such as RGB images, depth images, and infrared images, and key feature points are extracted from various image data. These feature points contain important information of the image, such as edges, textures, depth boundaries, and thermal areas. The extracted feature points are organized into a data set that contains various feature points extracted from multimodal images.
[0048] S22: Match and fuse the data in the extracted feature point data set through a multi-layer fusion network to obtain feature point data of the visual image data.
[0049] Specifically, a multi-layer fusion network is used to process the extracted feature point data set. The network includes multiple levels of neurons and hierarchical structures. Through the multi-layer fusion network, feature points of different types and sources can be effectively matched and fused to improve the consistency and accuracy of the feature points. The fusion network outputs the final feature point data, that is, the feature point data of the visual image data.
[0050] In one embodiment, if Figure 5 As shown, in step S40, three-dimensional reconstruction and scene analysis are performed based on the spatial perception data to identify key interactive objects and obstacles in the robot's environment, and obtain identification and analysis results, which specifically include: S41: Based on the spatial perception data, the depth information of objects and obstacles in the robot's environment is converted into a three-dimensional model through a three-dimensional point cloud generation algorithm.
[0051] Specifically, a 3D point cloud generation algorithm is used to process the depth information of objects and obstacles in the robot's environment and convert this information into a 3D model. The algorithm can extract points in space from the depth map and construct a 3D representation. S42: Perform semantic segmentation on the three-dimensional model to obtain semantic segmentation data.
[0052] Specifically, a scene segmentation algorithm based on graph neural network is used to perform semantic segmentation on the generated three-dimensional model. Graph neural network can process graph structure data, such as point clouds, and can capture complex topological structures and spatial relationships. In this process, the algorithm will learn how to distinguish different objects and obstacles, assign a semantic label to each pixel, and perform pixel-level classification by learning the topological relationship between nodes, thereby obtaining semantic segmentation data.
[0053] S43: Analyze the semantic segmentation data to obtain recognition analysis results.
[0054] Specifically, the semantic segmentation data is analyzed to identify key interactive objects and obstacles in the environment. This step may involve further processing of the segmentation results, such as object recognition, boundary detection, etc., to obtain recognition and analysis results. The final recognition and analysis results can help the robot better understand its environment and provide a basis for subsequent decisions and actions.
[0055] In one embodiment, if Figure 6 As shown, in step S50, the robot's interaction plan is generated according to the recognition and analysis results, which specifically includes: S51: Determine the priority of the interaction strategy according to the recognition analysis result, where the recognition analysis result includes the spatial distribution of the interaction object, obstacles, and the surrounding environment.
[0056] Specifically, based on the robot's recognition and analysis of the environment, the priority of interaction with each object in the environment is determined. The analysis results include interaction objects, such as people, animals or other robots, obstacles, such as walls and furniture, and the spatial distribution of the surrounding environment. By evaluating the importance, urgency and accessibility of interaction objects, an ordered list of strategies is provided for the robot's action plan to ensure that the robot can handle the most critical tasks first.
[0057] S52: Based on the priority of the interaction strategy, an interaction plan is calculated and generated using a real-time path planning algorithm.
[0058] Specifically, based on the determined interaction strategy priority, a real-time path planning algorithm is used to calculate and generate specific interaction plans. This usually involves finding the best path from the robot's current position to the interaction object while avoiding obstacles. The real-time path planning algorithm can take into account the robot's current environmental perception information and dynamically adapt to environmental changes, such as the appearance of moving obstacles or new interaction objects, ensuring that the robot can effectively and safely reach the destination and perform the predetermined interaction task.
[0059] In one embodiment, if Figure 7 As shown, the embodied robot visual interaction method also includes: S70: Obtain interactive feedback information, and adjust and optimize the robot's behavior strategy in real time according to the interactive feedback information.
[0060] Specifically, when the robot interacts with the environment or users, the system collects feedback information, such as sensor data, user input, environmental changes, etc. The system analyzes the interactive feedback information and adjusts the robot's behavior strategy in real time based on the analysis results. Through real-time adjustment, the robot can better adapt to environmental changes and user needs, and improve the efficiency and quality of interaction.
[0061] S80: When executing the user instruction, obtain the user's task requirements and generate a task execution plan according to the task requirements.
[0062] Specifically, when the user gives an instruction to the robot, the system obtains the user's specific task requirements. Based on the user's task requirements, the system generates a detailed task execution plan, including action steps, resource allocation, time planning, etc. The task execution plan provides the robot with a clear action guide to ensure that the task can be carried out in an orderly manner.
[0063] S90: During the task execution process, the task progress is monitored in real time. When a task deviation is detected, the deviation data is obtained, and the task execution plan is adjusted according to the deviation data. At the same time, the task status is fed back to the user through the display interface.
[0064] Specifically, during the task execution process, the system monitors the progress of the task in real time, including progress tracking, performance evaluation, etc. When the system detects deviations in task execution, it collects deviation data and adjusts the task execution plan based on these data, and displays the real-time status and progress of the task to the user through the user interface, such as screens, indicator lights or other display devices.
[0065] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0066] In one embodiment, an embodied robot visual interaction system is provided, and the embodied robot visual interaction system corresponds one-to-one to an embodied robot visual interaction method in the above embodiment. Figure 8 As shown, the embodied robot visual interaction system includes an image acquisition module, a model analysis module, a fusion module, a recognition analysis module, an interaction solution module, and an adjustment module. The detailed description of each functional module is as follows: The image acquisition module is used to acquire the robot's visual image data, which includes RGB images, depth images and infrared images.
[0067] The model analysis module is used to input the visual image data into a pre-trained image processing model for processing to obtain feature point data of the visual image data.
[0068] The fusion module is used to use a multimodal feature fusion algorithm to deeply fuse the feature point data of the visual image data to generate spatial perception data of the robot's surrounding environment.
[0069] The recognition and analysis module is used to perform three-dimensional reconstruction and scene analysis based on spatial perception data, identify key interactive objects and obstacles in the robot's environment, and obtain recognition and analysis results.
[0070] The interaction scheme module is used to generate the robot's interaction scheme according to the recognition and analysis results. The interaction scheme includes path planning, interaction actions with objects and voice intonation.
[0071] The adjustment module is used to adjust the robot's behavior strategy according to the robot's interaction plan.
[0072] Optionally, the embodied robot visual interaction system further includes: The environmental data acquisition submodule is used to acquire external environmental conditions. Based on the external environmental conditions, the visual sensor is used to collect data to obtain sensor data. The calibration submodule is used to synchronously calibrate and align the sensor data to obtain the robot's visual image data.
[0073] Optionally, the image acquisition module includes: A training data collection module is used to collect multimodal data, preprocess and annotate the multimodal data, and obtain a training set; The model training module is used to use the training set to perform forward propagation and back propagation training on the image processing model built based on the convolutional neural network and the self-attention mechanism, and to apply the transfer learning technology to optimize the image processing model after the forward propagation and back propagation training to obtain a pre-trained image processing model.
[0074] Feedback module, used to obtain interactive feedback information and make real-time adjustments and optimizations to the robot’s behavior strategy based on the interactive feedback information; Generate an execution module, which is used to obtain the user's task requirements when executing user instructions and generate a task execution plan based on the task requirements; The deviation adjustment module is used to monitor the progress of the task in real time during the task execution process. When a task deviation is detected, the deviation data is obtained, and the task execution plan is adjusted according to the deviation data. At the same time, the task status is fed back to the user through the display interface.
[0075] Optionally, the model analysis module includes: The feature extraction submodule is used to extract feature points from visual image data using a multimodal feature extraction algorithm to obtain an extracted feature point data set, where the extracted feature point data set includes edge features, texture features, depth boundaries, and thermal sensing areas; The matching and fusion submodule is used to match and fuse the data in the extracted feature point data set through a multi-layer fusion network to obtain the feature point data of the visual image data.
[0076] Optionally, the recognition and analysis module includes: The 3D modeling submodule is used to convert the depth information of objects and obstacles in the robot's environment into a 3D model based on the spatial perception data and through a 3D point cloud generation algorithm; The semantic segmentation submodule is used to perform semantic segmentation on the 3D model to obtain semantic segmentation data; The analysis result submodule is used to analyze the semantic segmentation data and obtain the recognition analysis results.
[0077] Optionally, the interactive solution module includes: A priority determination submodule is used to determine the priority of the interaction strategy based on the recognition analysis results, which include the spatial distribution of the interaction objects, obstacles and the surrounding environment; The interaction plan generation submodule is used to generate the interaction plan based on the priority of the interaction strategy and using the real-time path planning algorithm.
[0078] For the specific definition of an embodied robot visual interaction system, please refer to the definition of an embodied robot visual interaction method above, which will not be repeated here. Each module in the above-mentioned embodied robot visual interaction system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0079] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an embodied robot visual interaction method is implemented.
[0080] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Acquire visual image data of the robot, the visual image data including RGB image, depth image and infrared image; Input the visual image data into a pre-trained image processing model for processing to obtain feature point data of the visual image data; Use a multimodal feature fusion algorithm to deeply fuse the feature point data of visual image data to generate spatial perception data of the robot's surrounding environment; Based on the spatial perception data, three-dimensional reconstruction and scene analysis are performed to identify key interactive objects and obstacles in the robot's environment and obtain identification and analysis results; Generate the robot's interaction plan based on the recognition and analysis results. The interaction plan includes path planning, interaction actions with objects, and voice intonation. Adjust the robot's behavior strategy according to the robot's interaction plan.
[0081] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Acquire visual image data of the robot, the visual image data including RGB image, depth image and infrared image; Input the visual image data into a pre-trained image processing model for processing to obtain feature point data of the visual image data; Use a multimodal feature fusion algorithm to deeply fuse the feature point data of visual image data to generate spatial perception data of the robot's surrounding environment; Based on the spatial perception data, three-dimensional reconstruction and scene analysis are performed to identify key interactive objects and obstacles in the robot's environment and obtain identification and analysis results; Generate the robot's interaction plan based on the recognition and analysis results. The interaction plan includes path planning, interaction actions with objects, and voice intonation. Adjust the robot's behavior strategy according to the robot's interaction plan.
[0082] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0083] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.
[0084] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for embodied robot visual interaction, characterized in that: The embodied robot visual interaction method comprises: Acquire visual image data of the robot, wherein the visual image data includes an RGB image, a depth image, and an infrared image; Inputting the visual image data into a pre-trained image processing model for processing to obtain feature point data of the visual image data; Using a multimodal feature fusion algorithm to deeply fuse the feature point data of the visual image data to generate spatial perception data of the robot's surrounding environment; Based on the spatial perception data, three-dimensional reconstruction and scene analysis are performed to identify key interactive objects and obstacles in the environment where the robot is located, and obtain identification and analysis results; Generate an interaction plan for the robot according to the recognition and analysis results, wherein the interaction plan includes path planning, interaction actions with objects, and voice intonation; According to the interaction scheme of the robot, the behavior strategy of the robot is adjusted.
2. The method for visual interaction of an embodied robot according to claim 1, characterized in that: The obtaining of the robot's visual image data comprises: Acquire external environmental conditions, and based on the external environmental conditions, use a visual sensor to collect data to obtain sensor data; The sensor data are synchronously calibrated and aligned to obtain visual image data of the robot.
3. The method for visual interaction of an embodied robot according to claim 1, characterized in that: Before inputting the visual image data into a pre-trained image processing model for processing to obtain the feature point data of the visual image data, the embodied robot visual interaction method further includes: Collecting multimodal data, preprocessing and labeling the multimodal data to obtain a training set; The training set is used to perform forward propagation and back propagation training on the image processing model built based on the convolutional neural network and the self-attention mechanism, and the transfer learning technology is applied to optimize the image processing model after the forward propagation and back propagation training to obtain the pre-trained image processing model.
4. The method for visual interaction of an embodied robot according to claim 1, characterized in that: The step of inputting the visual image data into a pre-trained image processing model for processing to obtain the feature point data of the visual image data comprises: Extracting feature points from the visual image data using a multimodal feature extraction algorithm to obtain an extracted feature point data set, wherein the extracted feature point data set includes edge features, texture features, depth boundaries, and thermal sensing areas; The data in the extracted feature point data set is matched and fused through a multi-layer fusion network to obtain the feature point data of the visual image data.
5. The embodied robot visual interaction method according to claim 1, characterized in that: The three-dimensional reconstruction and scene analysis are performed based on the spatial perception data to identify key interactive objects and obstacles in the environment where the robot is located, and the identification and analysis results obtained include: Based on the spatial perception data, the depth information of objects and obstacles in the environment where the robot is located is converted into a three-dimensional model through a three-dimensional point cloud generation algorithm; Performing semantic segmentation on the three-dimensional model to obtain semantic segmentation data; The semantic segmentation data is analyzed to obtain the recognition analysis result.
6. The embodied robot visual interaction method according to claim 1, characterized in that: Generating the robot's interaction scheme according to the recognition and analysis result includes: Determining the priority of the interaction strategy according to the recognition and analysis results, wherein the recognition and analysis results include the spatial distribution of the interaction object, obstacles, and the surrounding environment; Based on the priority of the interaction strategy, the interaction plan is calculated and generated using a real-time path planning algorithm.
7. The embodied robot visual interaction method according to claim 1, characterized in that: The embodied robot visual interaction method further comprises: Acquire interactive feedback information, and adjust and optimize the behavior strategy of the robot in real time according to the interactive feedback information; When executing user instructions, obtaining the user's task requirements and generating a task execution plan according to the task requirements; During the task execution process, the task progress is monitored in real time. When a task deviation is detected, the deviation data is obtained, and the task execution plan is adjusted according to the deviation data. At the same time, the task status is fed back to the user through the display interface.
8. An embodied robot visual interaction system, characterized in that: The embodied robot visual interaction system comprises: An image acquisition module is used to acquire visual image data of the robot, wherein the visual image data includes RGB images, depth images and infrared images; A model analysis module, used for inputting the visual image data into a pre-trained image processing model for processing to obtain feature point data of the visual image data; A fusion module, used to perform deep fusion of feature point data of the visual image data using a multimodal feature fusion algorithm to generate spatial perception data of the robot's surrounding environment; A recognition and analysis module, used to perform three-dimensional reconstruction and scene analysis based on the spatial perception data, identify key interactive objects and obstacles in the environment where the robot is located, and obtain recognition and analysis results; Obtaining an interaction scheme module, for generating an interaction scheme of the robot according to the recognition and analysis results, wherein the interaction scheme includes path planning, interaction actions with objects, and voice intonation; An adjustment module is used to adjust the behavior strategy of the robot according to the interaction scheme of the robot.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the embodied robot visual interaction method as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the embodied robot visual interaction method as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Active interaction method and system for intelligent robot
CN106502382A
Three-coordinate machine intelligent positioning detection method and system based on stereoscopic vision
CN118261985A
Method and system for identifying target unmanned aerial vehicle through multi-channel information fusion
CN118351463A
Industrial vision robot interaction method and system
CN118596135A
Indoor and outdoor mobile robot and vehicle body dual-purpose apparatus, and management system and method
WO2022161111A1
Cited By
Multimedia interaction analysis method for digital scene recognition
CN119668409A
Multi-modal data fusion method, flexible exoskeleton robot and storage medium
CN119760626A
Unstructured environment supervised personal task data processing method and device
CN120080326A
Body-equipped robot, environment sensing method and device thereof, storage medium and computer program product
CN120326637A
Embodied robot and its environment perception method, device, storage medium, and computer program product
CN120326637B
