Robot visual language navigation method, device and equipment based on key point guidance

Abstract

The invention provides a robot visual language navigation method, device and equipment based on key point guidance. The method comprises the following steps: extracting a keyword of a voice instruction as a target text; based on a multi-modal pre-training knowledge base, determining knowledge prototype features corresponding to the target text, and determining a target similarity with the maximum similarity from the similarity between the knowledge prototype features and a plurality of regional features corresponding to the RGB image; and under the condition that the target similarity is greater than a preset threshold value, determining a target object in the region feature in the matching pair corresponding to the target similarity, determining a target coordinate of the target object based on the depth information of the target object, and determining a navigation path of the robot based on the target coordinate and a robot moving model. According to the method, through vision-language cross-modal matching, the similarity between a target text and an image region is calculated to realize navigation supervision; efficient fusion of multi-modal information can be realized in a complex scene, and the precision and robustness of robot navigation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual language navigation, and particularly to a robot visual language navigation method, device and equipment based on key point guidance. Background Art

[0002] With the continuous progress of artificial intelligence technology, significant progress has been made in robot navigation technology. However, robots still face many challenges in navigating in complex, dynamic and uncertain environments, especially in how to effectively integrate and cooperate using multiple perception modalities (such as vision, language, position, etc.). Most existing robot navigation methods rely on a single perception modality, such as vision or lidar, while ignoring the potential of language information. Especially in environments with complex backgrounds, semantic occlusions or poor lighting conditions, navigation methods based on a single perception modality often perform poorly.

[0003] Traditional visual navigation methods rely on the visual perception system to obtain environmental information, usually guiding the robot to navigate through image feature extraction and visual target recognition. These methods may perform well in simple and standardized environments, but in complex, dynamic or environments with large lighting changes, visual information may be affected by factors such as occlusion and lighting changes, resulting in a significant decline in navigation performance. For complex targets in these environments, traditional methods often struggle to provide efficient and accurate navigation support. In addition, existing visual navigation technologies usually lack the understanding and utilization of language information and cannot further optimize the navigation process with the help of semantic information, which limits their adaptability in practical applications.

[0004] In recent years, multi-modal learning methods have become an important research direction in the field of robot navigation. Multi-modal learning enables robots to have stronger understanding and decision-making abilities in complex environments by combining multiple perception information (such as vision, language, depth sensor data, etc.). Visual-Language Navigation (VLN) is one of the important applications in this direction. It aligns visual and language information and uses natural language instructions to guide the robot to perform navigation tasks. This task not only requires the robot to understand natural language instructions but also to be able to accurately pair these instructions with visual inputs to achieve a more efficient and robust navigation effect.

[0005] Currently, many vision-language navigation models rely on large-scale pre-trained models to learn the joint representation of vision and language using paired image and text data. These methods have achieved good results on some standard tasks, but there are still many challenges when facing complex dynamic environments. First, existing methods usually rely on coarse-grained global feature matching, ignoring the precise alignment of fine-grained semantics between vision and language. Especially when dealing with multiple landmarks or complex instructions, the model often has difficulty accurately understanding the priority, position relationship, and attribute information in the language. Second, many methods do not fully consider factors such as semantic occlusion, lighting changes, and background interference in complex scenes, which seriously affect the performance of the model in practical applications. In addition, existing models often lack the ability of real-time interaction and update when processing multi-modal information and are difficult to adjust the navigation strategy in real time in a dynamic environment. Summary of the Invention

[0006] The present invention provides a key-point-guided robot vision-language navigation method, device, and equipment to solve the problems in the prior art that many vision-language navigation models usually rely on coarse-grained global feature matching, ignoring the precise alignment of fine-grained semantics between vision and language. Especially when dealing with multiple landmarks or complex instructions, the model often has difficulty accurately understanding the priority, position relationship, and attribute information in the language. Second, many methods do not fully consider factors such as semantic occlusion, lighting changes, and background interference in complex scenes, which seriously affect the performance of the model in practical applications. In addition, existing models often lack the ability of real-time interaction and update when processing multi-modal information and are difficult to adjust the navigation strategy in real time in a dynamic environment.

[0007] The present invention provides a key-point-guided robot vision-language navigation method, including the following steps: Obtain the RGB image to be recognized and the user's voice instruction, and extract the keyword of the voice instruction as the target text; Based on the multi-modal pre-trained knowledge base, determine the knowledge prototype feature corresponding to the target text, and determine the maximum target similarity from the similarity between the knowledge prototype feature and multiple region features corresponding to the RGB image; When the target similarity is greater than the preset threshold, determine the target object in the region feature of the matching pair corresponding to the target similarity, determine the target coordinates of the target object based on the depth information of the target object, and determine the navigation path of the robot based on the target coordinates and the robot movement model.

[0008] According to the key-point-guided robot vision-language navigation method provided by the present invention, determining the target coordinates of the target object based on the depth information of the target object includes: Based on the depth information and the pixel coordinates of the target object in the RGB image, determine the target camera coordinates of the target object in the camera coordinate system; Based on the current position coordinates of the robot and the target camera coordinates, determine the target coordinates of the target object in the world coordinate system.

[0009] According to a robot vision-language navigation method based on key-point guidance provided by the present invention, the determination step of the similarity between the knowledge prototype feature and multiple region features corresponding to the RGB image includes: Based on the following formula, determine the similarity between the knowledge prototype feature and multiple region features corresponding to the RGB image: where represents the knowledge prototype feature of the target text and multiple region features the cosine similarity between them.

[0010] According to a robot vision-language navigation method based on key-point guidance provided by the present invention, the construction step of the robot movement model includes: Establish a chassis coordinate system according to the plane where the chassis of the robot movement model is located. The chassis coordinate system takes the center of the chassis as the coordinate origin, the direction parallel to the vehicle body width direction of the robot as the X-axis, and the direction parallel to the vehicle body length direction of the robot as the Y-axis; Based on the chassis coordinate system, determine the speed components of the robot; the first speed components of the robot include the first speed of the robot along the X-axis, the second speed along the Y-axis, and the rotational angular velocity; Based on the first speed, the second speed, and the rotational angular velocity, determine the second speed components of each steering wheel of the robot; the second speed components include the first steering wheel speed, the second steering wheel speed, and the third steering wheel speed.

[0011] According to a robot vision-language navigation method based on key-point guidance provided by the present invention, the method further includes: Based on the second speed components of each steering wheel, determine the third speed components of the robot; the third speed components of the robot include the first robot speed of the robot along the X-axis, the second robot speed along the Y-axis, and the third robot speed along the Z-axis.

[0012] A robot vision-language navigation method based on key-point guidance provided by the present invention, wherein the multi-modal pre-trained knowledge base includes text categories of paired semantic concepts and knowledge prototype features corresponding to the text categories; The knowledge prototype features are obtained by averaging region representations related to the text categories.

[0013] A robot vision-language navigation method based on key-point guidance provided by the present invention, determining the navigation path of the robot based on the target coordinates and the robot movement model, includes: In the environmental map, based on the target coordinates and the robot movement model, perform path planning, and determine the navigation path of the robot based on the target path obtained from the path planning; The environmental map is determined based on the Gmapping method, two-dimensional laser scan data, and RGB image data; the two-dimensional laser scan data is obtained based on a sensor; the RGB image data is obtained based on a camera.

[0014] The present invention also provides a robot vision-language navigation device based on key-point guidance, including the following units: An acquisition unit, configured to acquire an RGB image to be recognized and a voice command of a user, and extract a keyword of the voice command as a target text; A first determination unit, configured to determine, based on the multi-modal pre-trained knowledge base, knowledge prototype features corresponding to the target text, and determine a target similarity with the largest similarity from the similarities between the knowledge prototype features and multiple region features corresponding to the RGB image; A second determination unit, configured to, when the target similarity is greater than a preset threshold, determine a target object in the region features of the matching pair corresponding to the target similarity, determine target coordinates of the target object based on depth information of the target object, and determine the navigation path of the robot based on the target coordinates and the robot movement model.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the robot vision-language navigation method based on key-point guidance as described in any one of the above when executing the program.

[0016] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and the computer program implements the robot vision-language navigation method based on key-point guidance as described in any one of the above when being executed by a processor.

[0017] The present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the key-point-guided robot visual language navigation method as described in any one of the above.

[0018] For the key-point-guided robot visual language navigation method, device and equipment provided by the present invention, on the one hand, the regional features corresponding to the RGB image are finely aligned with the knowledge prototype features corresponding to the target text, that is, the fine-grained semantic alignment between vision and language is realized. Especially when multiple landmarks or complex instructions are involved, the model can accurately understand the priority, positional relationship and attribute information in the language. On the other hand, navigation in a dynamic environment is realized by combining the RGB image and voice instructions. In this process, the robot uses the image captured by the Kinect v2.0 camera and the target text provided by the voice instructions to calculate the similarity between the target text and the image region. When the target similarity reaches the preset threshold, the robot can determine that the target matches the image region successfully, and adjust the navigation strategy according to this information. This process supports real-time interaction and update, and can dynamically adjust the navigation path of the robot according to environmental changes. In addition, the multi-modal pre-trained knowledge base fully considers factors such as semantic occlusion, illumination change and background interference in complex scenes, and determines the knowledge prototype features corresponding to the target text based on the multi-modal pre-trained knowledge base, improving the accuracy and reliability of the knowledge prototype features. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 is one of the flow diagrams of the key-point-guided robot visual language navigation method provided by the present invention.

[0021] Figure 2 is one of the diagrams for calculating the position of the target object in the world coordinate system based on depth information provided by the present invention.

[0022] Figure 3 is the second diagram for calculating the position of the target object in the world coordinate system based on depth information provided by the present invention.

[0023] Figure 4 is the diagram of the operation decomposition method based on the chassis center provided by the present invention.

[0024] Figure 5It is the second flow diagram of the robot vision-language navigation method guided by key points provided by the present invention.

[0025] Figure 6 It is a schematic diagram of key point screening and optimal layout provided by the present invention.

[0026] Figure 7 It is a schematic diagram of parameter settings of the robot in adjacent navigable areas provided by the present invention.

[0027] Figure 8 It is a schematic diagram of the path of the robot moving between three key points provided by the present invention.

[0028] Figure 9 It is a schematic diagram of the navigation performance of the robot in the real environment provided by the present invention.

[0029] Figure 10 It is a schematic diagram of the structure of the robot vision-language navigation device guided by key points provided by the present invention.

[0030] Figure 11 It is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed implementation manners

[0031] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without making creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0032] Terms such as "first" and "second" in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. generally belong to the same category.

[0033] In related technologies, more and more research has begun to focus on how to introduce more semantic information in vision-language navigation and improve the generalization ability of the model. For example, some methods attempt to optimize the understanding of the relationship between the target and the landmark through more refined vision-language alignment techniques, or improve the adaptability of the robot in dynamic and complex environments through reinforcement learning and memory enhancement methods.

[0034] However, despite the progress made by these methods, how to achieve efficient and accurate navigation in more complex environments, especially when multi-modal information (such as language, vision, and location) is fully integrated, remains an urgent challenge to be solved.

[0035] Based on the above problems, the present invention provides a key-point-guided robot visual language navigation method. Figure 1 It is one of the schematic flowcharts of the key-point-guided robot visual language navigation method provided by the present invention, as Figure 1 shown. This method includes step 110, step 120, and step 130.

[0036] Step 110: Obtain the RGB image to be recognized and the user's voice command, and extract the keyword of the voice command as the target text.

[0037] Specifically, the RGB image to be recognized and the user's voice command can be obtained. Here, the RGB image can be captured by a Kinect v2.0 camera, and the user's voice command can be a voice command directly spoken by the user.

[0038] After obtaining the user's voice command, the voice command can be converted into text through the iFlytek speech recognition system, and the keyword can be extracted using NLTK (Natural Language Toolkit) to obtain the target text.

[0039] In addition, in the embodiments of the present invention, multiple navigation points are randomly generated on the environmental map, and then these navigation points are optimized according to the key positions in the environment (such as intersections or complex visual areas), and the most representative key points are selected for subsequent navigation tasks, as follows: 1) Randomly generate navigation points First, multiple navigation points are randomly generated on the pre-constructed 2D grid map. Each navigation point is represented by its spatial position and orientation information, and the formula is as follows: Among them, represents the coordinates and direction of the th navigation point, , is the coordinate of the th navigation point in the map, is the orientation angle of this point.

[0040] 2) Remove invalid navigation points By analyzing the navigable areas in the environment (such as walls, obstacles), the points located in non-navigable areas are removed, and the remaining valid navigation points are used as key points.

[0041] 3) Optimization of key points For navigation points near intersections, entrances, or visually complex areas, their positions are adjusted through optimization strategies to ensure that they accurately represent key positions. The optimized key points are selected through the following optimization function: where is an optimization function that selects the optimal navigation points based on navigation efficiency and environmental characteristics, ensuring that the robot selects the most favorable path when performing navigation tasks.

[0042] Step 120: Based on the multi-modal pre-trained knowledge base, determine the knowledge prototype features corresponding to the target text, and determine the target similarity with the largest similarity from among the similarities between the knowledge prototype features and the multiple region features corresponding to the RGB image.

[0043] Specifically, after obtaining the target text, the knowledge prototype features corresponding to the target text can be determined based on the multi-modal pre-trained knowledge base.

[0044] Here, the multi-modal pre-trained knowledge base includes text categories of paired semantic concepts and the knowledge prototype features corresponding to the text categories. For example, the Open Image V7 database is used to collect and construct indoor environment object concept words and their corresponding image regions. This knowledge base consists of paired semantic concepts and consists of, where represents the text category of the th semantic concept, represents the visual prototype region representation corresponding to the text category. The total number of semantic concepts is represented by K , and the construction of these concepts provides an important basis for subsequent vision-language alignment.

[0045] The knowledge prototype features are obtained by averaging the region representations related to the text category, specifically as follows: For each text category , its knowledge prototype representation is obtained by averaging all relevant region representations , and the formula is as follows: where is the number of regions associated with the text category , is the region representation obtained by inputting the bounding box and the image into a pre-trained object detection model (such as Faster-RCNN), is the visual feature of each region, is the dimension of the feature. The knowledge prototype representation of the category is obtained by solving the average of the feature representations of all relevant regions .

[0046] Then, the target similarity with the maximum similarity is determined from the similarity between the knowledge prototype feature and the multiple regional features corresponding to the RGB image

[0047] Here, the RGB image obtained by using the Kinect v2.0 camera is used to extract multiple regional features through a bottom-up attention mechanism Each feature corresponds to a specific region or salient object in the image. The formula is as follows where is the dimension of each regional feature, representing the length of the visual feature is the number of regions extracted from the image, usually 100. Each regional feature captures the semantic information of different parts of the image

[0048] Finally, based on the following formula, the similarity between the knowledge prototype feature and the multiple regional features corresponding to the RGB image is determined where represents the knowledge prototype feature of the target text and the multiple regional features is the cosine similarity between them

[0049] It can be understood that the cosine similarity is used to measure the closeness of the knowledge prototype feature and the multiple regional features in the feature space. The larger the value, the more similar they are

[0050] Calculate the maximum similarity for all regional features , and the formula is where is the maximum value among all similarity values, that is, the target similarity. The target similarity represents the best visual region matching the target text, and the larger the value, the better the match

[0051] Step 130, when the target similarity is greater than a preset threshold, determine the target object in the regional features of the matching pair corresponding to the target similarity. Based on the depth information of the target object, determine the target coordinates of the target object, and based on the target coordinates and the robot movement model, determine the navigation path of the robot

[0052] Specifically, when the target similarity is greater than the preset threshold, determine the target object in the regional features of the matching pair corresponding to the target similarity.

[0053] For example, if the maximum similarity (target similarity) is higher than the preset threshold , then the target successfully matches the current image area, and the navigation system will predict the best position and orientation for the robot to approach the target. The matching judgment formula is as follows: where represents whether the match is successful, represents a successful match, represents a failed match, is the set similarity threshold. If is greater than or equal to , it is considered a successful match.

[0054] Furthermore, based on the depth information of the target object, determine the target coordinates of the target object, and based on the target coordinates and the robot movement model, determine the navigation path of the robot.

[0055] After successfully matching the target, use the depth information to determine the precise position of the target. The depth pixel value is calculated by the following formula: where is the depth value of the th pixel in the image, is the processing function for obtaining depth information, is the pixel set of the target area.

[0056] Among them, the robot movement model can include an omnidirectional wheel movement model, an inverse kinematics model, and a forward kinematics model.

[0057] It can be understood that through the cross-modal alignment of visual and language information, the navigation ability of the robot in complex environments is improved. Especially in the context of multi-modal information fusion, the robot's understanding and response capabilities to visual information and language instructions are enhanced.

[0058] The method provided by the embodiment of the present invention, on the one hand, finely aligns the regional features corresponding to the RGB image with the knowledge prototype features corresponding to the target text, that is, precisely aligns the fine-grained semantics between vision and language. Especially when involving multiple landmarks or complex instructions, the model can accurately understand the priority, positional relationship, and attribute information in the language. On the other hand, it realizes navigation in a dynamic environment by combining the RGB image and voice instructions. In this process, the robot uses the image captured by the Kinect v2.0 camera and the target text provided by the voice instructions to calculate the similarity between the target text and the image region. When the target similarity reaches the preset threshold, the robot can determine that the target matches the image region successfully and adjust the navigation strategy according to this information. This process supports real-time interaction and update and can dynamically adjust the navigation path of the robot according to environmental changes. In addition, the multi-modal pre-trained knowledge base fully considers factors such as semantic occlusion, illumination change, and background interference in complex scenes. Based on the multi-modal pre-trained knowledge base, the knowledge prototype features corresponding to the target text are determined, improving the accuracy and reliability of the knowledge prototype features.

[0059] The method provided by the embodiment of the present invention proposes a vision-language navigation method based on multi-modal pre-trained knowledge, which combines vision and language information through cross-modal alignment to enhance the navigation robustness and adaptability of the robot in complex environments. Through refined vision-language alignment, real-time environment perception, and semantic modeling, the present invention can effectively improve the navigation ability of the robot in a dynamic environment and show better robustness when solving problems such as semantic occlusion and complex background interference. Experimental results show that the technology of the present invention can significantly improve the performance of the robot in complex navigation tasks, especially in environments involving multiple landmarks, complex backgrounds, and dynamic changes, demonstrating excellent performance.

[0060] Based on the above embodiment, in step 130, determining the target coordinates of the target object based on the depth information of the target object includes: Step 131, based on the depth information and the pixel coordinates of the target object in the RGB image, determine the target camera coordinates of the target object in the camera coordinate system; Step 132, based on the current position coordinates of the robot and the target camera coordinates, determine the target coordinates of the target object in the world coordinate system.

[0061] Specifically, Figure 2 is one of the schematic diagrams of calculating the position of the target object in the world coordinate system based on the depth information provided by the present invention, Figure 3 is the second schematic diagram of calculating the position of the target object in the world coordinate system based on the depth information provided by the present invention, as Figure 2 、Figure 3 As shown, by measuring the depth distance between the target and the camera and combining the internal and external parameters of the camera, the algorithm can accurately map the coordinates of the target from the image plane to the world coordinate system. This method plays an important role in navigation and target positioning and helps to achieve accurate determination of the target position.

[0062] For example, based on the depth information and the pixel coordinates of the target object in the RGB image, determine the target camera coordinates of the target object in the camera coordinate system, where the target camera coordinates coordinates are given by the following formula: where is the coordinate of the target in the camera coordinate system, is the pixel coordinate of the target object, is the coordinate offset of the image center, is the ratio of the focal length of the camera to the physical size of the pixel.

[0063] Similarly, the coordinates in the target camera coordinates are calculated by the formula: where is the coordinate of the target in the camera coordinate system, is the pixel coordinate of the target object, is the coordinate offset of the image center, is the ratio of the focal length of the camera to the physical size of the pixel.

[0064] Finally, is the depth value: .

[0065] Furthermore, based on the current position coordinates of the robot and the target camera coordinates, determine the target coordinates of the target object in the world coordinate system. The formula is as follows: Use the odometer data to convert the target camera coordinates of the target object from the camera coordinate system to the world coordinate system. The conversion formula is as follows: where is the coordinate of the target object, and are respectively the origin and the direction vector of the coordinate transformation, is the orientation angle of the robot.

[0066] The position of the final target coordinates in the world coordinate system is: Among them, 、 are the coordinates of the target object in the world coordinate system, 、 are the coordinates of the current position of the robot, and are the displacements between the target and the current position of the robot.

[0067] Based on the above embodiments, the steps for constructing the robot movement model include: Step 210, establish a chassis coordinate system according to the plane where the chassis of the robot movement model is located. The chassis coordinate system takes the center of the chassis as the coordinate origin, the direction parallel to the body width direction of the robot as the X-axis, and the direction parallel to the body length direction of the robot as the Y-axis; Step 220, determine the speed components of the robot based on the chassis coordinate system; the first speed components of the robot include the first speed of the robot along the X-axis direction, the second speed along the Y-axis direction, and the rotational angular velocity; Step 230, determine the second speed components of each steering wheel of the robot based on the first speed, the second speed, and the rotational angular velocity; the second speed components include the first steering wheel speed, the second steering wheel speed, and the third steering wheel speed.

[0068] Specifically, Figure 4 is a schematic diagram of the operation decomposition method based on the center of the chassis provided by the present invention. As Figure 4 shown, by defining a coordinate system centered on the chassis, the linear motion of the robot in any direction can be decomposed into horizontal and vertical two components. Combining the geometric parameters of the chassis and the wheel layout, calculate the movement speed of each wheel. In addition, and respectively describe the relevant angles between the speed direction and the wheel axis, determine the distance parameters of the geometric distribution. This motion decomposition simplifies the complex motion control of the omnidirectional chassis, enabling the robot to accurately convert the overall motion requirements into the driving tasks of independent wheels, thereby achieving smooth and efficient omnidirectional movement. For clarity, the present invention embodiment also details the motion decomposition process of the first wheel.

[0069] Specifically, according to the plane where the chassis of the robot movement model is located, a chassis coordinate system is established. The origin of the chassis coordinate system is the center of the chassis, the X-axis is in the direction parallel to the width direction of the robot body, and the Y-axis is in the direction parallel to the length direction of the robot body.

[0070] Based on the chassis coordinate system, the velocity components of the robot are determined. Among them, the first velocity component of the robot includes the first velocity of the robot in the X-axis direction , the second velocity in the Y-axis direction , and the rotational angular velocity .

[0071] Finally, based on the first velocity, the second velocity, and the rotational angular velocity, the second velocity components of each steering wheel of the robot are determined; the second velocity components include the first steering wheel velocity, the second steering wheel velocity, and the third steering wheel velocity.

[0072] Among them, the formula for the second velocity components of each steering wheel of the robot is as follows: Among them, the second velocity component is , , , represents the first velocity of the robot in the X-axis direction, represents the second velocity in the Y-axis direction, represents the rotational angular velocity, represents the wheel angle, represents the radius of the wheel, represents the installation angle of each wheel relative to the x-axis.

[0073] The embodiment of the present invention adopts an omnidirectional wheel chassis design, and each wheel is evenly distributed at a 120° angle, supporting linear motion and rotational motion.

[0074] In addition, an inverse kinematics model can be used to calculate the wheel speed to achieve precise control of the robot movement. The formula for the second velocity component can also be: Among them, the second velocity component is , , , represents the first velocity of the robot in the X-axis direction, represents the second velocity in the Y-axis direction, represents the rotational angular velocity, represents the radius of the wheel.

[0075] Based on the above embodiments, the method further includes: Determining a third velocity component of the robot based on the second velocity components of the respective steering wheels; the third velocity component includes a first robot velocity of the robot in the X-axis direction, a second robot velocity in the Y-axis direction, and a third robot velocity in the Z-axis direction.

[0076] Specifically, a forward kinematic model is used to ensure that the actual motion data of the robot is consistent with the preset motion. Correspondingly, based on the second velocity components of the respective steering wheels, a third velocity component of the robot is determined, where the third velocity component includes a first robot velocity of the robot in the X-axis direction, a second robot velocity in the Y-axis direction, and a third robot velocity in the Z-axis direction. The formula is as follows: Where the third velocity component , , includes a first robot velocity of the robot in the X-axis direction, a second robot velocity in the Y-axis direction, and the second velocity component is , , .

[0077] Based on the above embodiments, in step 130, determining the navigation path of the robot based on the target coordinates and the robot motion model includes: In the environmental map, based on the target coordinates and the robot motion model, perform path planning, and based on the target path obtained from the path planning, determine the navigation path of the robot; The environmental map is determined based on the Gmapping method, two-dimensional laser scan data, and RGB image data; the two-dimensional laser scan data is obtained based on a sensor; the RGB image data is obtained based on a camera.

[0078] Specifically, in the environmental map, based on the target coordinates and the robot motion model, perform path planning, and based on the target path obtained from the path planning, determine the navigation path of the robot.

[0079] Among them, the environmental map is determined based on the Gmapping method, two-dimensional laser scan data, and RGB image data. Among them, the two-dimensional laser scan data is obtained based on a sensor, and the RGB image data is obtained based on a camera. Here, the sensor can be a 2D LiDAR sensor, and the camera can be a Kinect v2.0 camera. The embodiments of the present invention do not make specific limitations on this.

[0080] It should be noted that the platform of the embodiment of the present invention is equipped with a Kinect v2.0 camera and a 2D LiDAR sensor for environmental perception, map construction, and navigation positioning.

[0081] The Gmapping method is used for map construction, and path planning is performed through the Move Base package in the ROS framework to ensure the precise positioning and navigation of the robot.

[0082] Based on any of the above embodiments, Figure 5 is the second flowchart of the robot vision-language navigation method guided by key points provided by the present invention. As Figure 5 shown, the method includes: The embodiment of the present invention mainly includes the following five main components: the construction of a multi-modal pre-trained knowledge base, using NLTK to extract keywords to obtain the target text, cross-modal alignment of vision and language for navigation supervision, prediction of the best position and direction, and map-based target navigation path planning.

[0083] Among them, the multi-modal pre-trained knowledge base includes text categories of paired semantic concepts and knowledge prototype features corresponding to the text categories. The knowledge prototype features are obtained by averaging the region representations related to the text categories. That is, visual prototype learning is performed on limited annotations to obtain prior knowledge.

[0084] Then, the RGB image to be recognized and the user's voice command are obtained, and the keyword of the voice command is extracted as the target text.

[0085] Furthermore, based on the multi-modal pre-trained knowledge base, the knowledge prototype features corresponding to the target text are determined, and the maximum target similarity is determined from the similarity between the knowledge prototype features and multiple region features corresponding to the RGB image. Here, based on the maximum similarity in cross-modal similarity, the target position in the image, that is, the target object, can be determined, and the target position in the image can be used for target navigable position prediction.

[0086] When the target similarity is greater than the preset threshold, the target object in the region features of the matching pair corresponding to the target similarity is determined. Based on the depth information of the target object, the target coordinates of the target object are determined, and based on the target coordinates and the robot movement model, the navigation path of the robot is determined.

[0087] It should be noted that path planning can be carried out based on the target coordinates and the robot movement model, including object navigation using multiple packages such as A* alg., AMCL alg., and Move_base alg. In addition, 2D maps can be combined for key point marking and optimization to further accurately predict the target navigable positions. The robot first obtains environmental information through sensors such as LiDAR and generates global and local cost maps to represent the traversal costs of the navigation area. Combining the pre-constructed SLAM map and the position and odometry information provided by AMCL, the system uses a global planner to plan the best path from the starting point to the target point. At the same time, the local planner dynamically adjusts according to real-time environmental data to avoid obstacles and optimize the navigation path. This way of combining global and local planning improves the accuracy and robustness of robot navigation and ensures its efficient operation in complex environments.

[0088] Figure 6 is a schematic diagram of key point screening and optimized layout provided by the present invention, as Figure 6 shown, Figure 6 shows the waypoint generation process on a 2D grid map. The map is pre-constructed by the Gmapping algorithm, where waypoints are randomly generated with different densities. Waypoints within non-navigable areas (identified in gray) are removed, and the remaining waypoints are defined as key points. However, during the actual navigation process, the system may encounter difficulties in key scenarios such as intersections, entrances, and visually complex areas. These areas tend to pose risks of deviating from the planned trajectory, and randomly generated waypoints may not fully cover these important positions, thus having a certain impact on navigation accuracy.

[0089] Figure 7 is a schematic diagram of parameter settings of the robot in adjacent navigable areas, as Figure 7 shown, the radius of the large circle R is 1 meter, representing the maximum operating range of the robotic arm to ensure that the robotic arm can cover the surrounding working area. The radius of the small circle r is 0.2 meter, indicating the basic space required for the robot chassis during movement and positioning to avoid collisions with surrounding obstacles. The setting of these parameters ensures the safety and efficiency of the robot during navigation and operation, and at the same time provides clear constraint conditions for path planning.

[0090] It should be noted that the entire platform consists of seven core components, which work together to support the robot's autonomous navigation and task execution. The Kinect v2.0 RGB-D camera (1) captures the visual and depth information of the environment and is the core of the perception system; the support rod (2) and the lifting platform (3) optimize the height for data collection and manipulator operation; the high-performance Lenovo 9000P computer (4) provides strong support for data processing and model operation; the two-degree-of-freedom gripper (5) endows the robot with flexible operation capabilities; the special bracket (6) ensures the safe storage of the manipulator; and the mobile chassis (7) provides the mobility of the platform. These components together constitute a solid hardware foundation for the robot navigation platform and provide support for achieving complex tasks.

[0091] Figure 8 It is a schematic diagram of the path for the robot provided by the present invention to move between three key points. As Figure 8 shown, the robot travels smoothly at a speed of 0.5 m / s, with a total time of about 40 seconds. During the navigation process, the robot not only completes the path tracking task but also simultaneously executes multiple parallel tasks, including voice interaction to receive and respond to user instructions, visual feature extraction to identify key objects in the surrounding environment, and cross-modal target search to accurately locate the target by combining multi-modal data. This experiment verifies the coordination ability of the robot in a multi-task environment and the efficiency of path planning.

[0092] Figure 9 It is a schematic diagram of the navigation performance of the robot provided by the present invention in a real environment. As Figure 9 shown, the left picture in the two key pictures is the initial view of the robot camera at the start of navigation, showing the preliminary positioning of the target object in the environment. The right picture shows the final result of the robot successfully approaching the target object (a bottle of cola) after path planning and adjustment. The camera always maintains an accurate alignment with the target throughout the process. This result proves the effectiveness and applicability of this method in navigation tasks in complex real scenarios and provides a reliable basis for further promoting and optimizing robot navigation technology.

[0093] In summary, through innovative technologies such as multi-modal information fusion, key point optimization, and combining depth information with SLAM (Simultaneous Localization and Mapping) maps, the present invention significantly improves the navigation accuracy, efficiency, and robustness of robots in complex environments. Compared with traditional methods, the present invention not only enhances the alignment ability of visual and language information but also can handle scenes with complex backgrounds, occlusions, and lighting changes. In addition, through the omnidirectional wheel chassis and multi-task learning strategy, the motion control and path planning of the robot are optimized, further improving the adaptability and real-time performance of the robot in dynamic environments. This technical solution can effectively solve the deficiencies of existing navigation methods in aspects such as multi-modal information fusion, path planning accuracy, and robustness, and has high practical application value and broad application prospects.

[0094] The robot visual language navigation device based on key point guidance provided by the present invention will be described below. The robot visual language navigation device based on key point guidance described below can be correspondingly referred to the robot visual language navigation method based on key point guidance described above.

[0095] Based on any of the above embodiments, the present invention provides a robot visual language navigation device based on key point guidance, Figure 10 which is a schematic structural diagram of the robot visual language navigation device based on key point guidance provided by the present invention. As Figure 10 shown, the device includes: An acquisition unit 910, configured to acquire an RGB image to be recognized and a voice command of a user, and extract a keyword of the voice command as a target text; A first determination unit 920, configured to determine a knowledge prototype feature corresponding to the target text based on a multi-modal pre-training knowledge base, and determine a maximum target similarity from the similarity between the knowledge prototype feature and a plurality of region features corresponding to the RGB image; A second determination unit 930, configured to, when the target similarity is greater than a preset threshold, determine a target object in the region features of a matching pair corresponding to the target similarity, determine a target coordinate of the target object based on depth information of the target object, and determine a navigation path of the robot based on the target coordinate and a robot movement model.

[0096] The device provided by the embodiment of the present invention, on the one hand, finely aligns the regional features corresponding to the RGB image with the knowledge prototype features corresponding to the target text, that is, precisely aligns the fine-grained semantics between vision and language. Especially when multiple landmarks or complex instructions are involved, the model can accurately understand the priority, positional relationship, and attribute information in the language. On the other hand, it realizes navigation in a dynamic environment by combining the RGB image and voice instructions. In this process, the robot uses the image captured by the Kinect v2.0 camera and the target text provided by the voice instructions to calculate the similarity between the target text and the image region. When the target similarity reaches the preset threshold, the robot can determine that the target matches the image region successfully and adjust the navigation strategy based on this information. This process supports real-time interaction and update and can dynamically adjust the navigation path of the robot according to environmental changes. In addition, the multi-modal pre-trained knowledge base fully considers factors such as semantic occlusion, illumination change, and background interference in complex scenarios, and determines the knowledge prototype features corresponding to the target text based on the multi-modal pre-trained knowledge base, improving the accuracy and reliability of the knowledge prototype features.

[0097] Based on any of the above embodiments, the second determination unit 930 is specifically configured to: Based on the depth information and the pixel coordinates of the target object in the RGB image, determine the target camera coordinates of the target object in the camera coordinate system; Based on the current position coordinates of the robot and the target camera coordinates, determine the target coordinates of the target object in the world coordinate system.

[0098] Based on any of the above embodiments, the similarity determination unit is specifically configured to: Based on the following formula, determine the similarity between the knowledge prototype features and multiple regional features corresponding to the RGB image: Wherein, represents the knowledge prototype features of the target text and multiple regional features The cosine similarity between them.

[0099] Based on any of the above embodiments, it further includes a robot movement model construction unit, and the robot movement model construction unit is specifically configured to: Establish a chassis coordinate system according to the plane where the chassis of the robot movement model is located. The chassis coordinate system takes the center of the chassis as the coordinate origin, the direction parallel to the body width direction of the robot as the X axis, and the direction parallel to the body length direction of the robot as the Y axis; Based on the chassis coordinate system, determine the speed components of the robot; the first speed components of the robot include the first speed of the robot in the X-axis direction, the second speed in the Y-axis direction, and the rotational angular velocity; Based on the first speed, the second speed, and the rotational angular velocity, determine the second speed components of the respective steering wheels of the robot; the second speed components include the first steering wheel speed, the second steering wheel speed, and the third steering wheel speed.

[0100] Based on any of the above embodiments, it further includes a second model construction unit, and the second model construction unit is specifically configured to: Based on the second speed components of the respective steering wheels, determine the third speed components of the robot; the third speed components include the first robot speed of the robot in the X-axis direction, the second robot speed in the Y-axis direction, and the third robot speed in the Z-axis direction.

[0101] Based on any of the above embodiments, the multimodal pre-training knowledge base includes text categories of paired semantic concepts, and knowledge prototype features corresponding to the text categories; The knowledge prototype features are obtained by performing an averaging operation on region representations related to the text categories.

[0102] Based on any of the above embodiments, the second determination unit 930 is specifically configured to: In the environmental map, based on the target coordinates and the robot movement model, perform path planning, and based on the target path obtained from the path planning, determine the navigation path of the robot; The environmental map is determined based on the Gmapping method, two-dimensional laser scan data, and RGB image data; the two-dimensional laser scan data is obtained based on a sensor; the RGB image data is obtained based on a camera.

[0103] Figure 11 It is a schematic structural diagram of the electronic device provided by the present invention, as Figure 11As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040. Among them, the processor 1010, the communications interface 1020, and the memory 1030 complete communication with each other through the communication bus 1040. The processor 1010 may call the logical instructions in the memory 1030 to execute a robot vision-language navigation method guided by key points. The method includes: obtaining an RGB image to be recognized and a voice instruction of a user, and extracting a keyword of the voice instruction as a target text; determining a knowledge prototype feature corresponding to the target text based on a multi-modal pre-trained knowledge base, and determining a target similarity with the greatest similarity from among the similarities between the knowledge prototype feature and multiple region features corresponding to the RGB image; in a case where the target similarity is greater than a preset threshold, determining a target object in the region features of a matching pair corresponding to the target similarity, determining a target coordinate of the target object based on depth information of the target object, and determining a navigation path of the robot based on the target coordinate and a robot movement model.

[0104] In addition, when the logical instructions in the foregoing memory 1030 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0105] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot vision-language navigation method guided by key points provided by the above-mentioned various methods. The method includes: obtaining an RGB image to be recognized and a voice instruction of a user, and extracting a keyword of the voice instruction as a target text; based on a multi-modal pre-trained knowledge base, determining a knowledge prototype feature corresponding to the target text, and determining a target similarity with the largest similarity from the similarities between the knowledge prototype feature and multiple region features corresponding to the RGB image; in the case where the target similarity is greater than a preset threshold, determining a target object in the region features of the matching pair corresponding to the target similarity, determining a target coordinate of the target object based on the depth information of the target object, and determining a navigation path of the robot based on the target coordinate and a robot movement model.

[0106] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the robot vision-language navigation method guided by key points provided by the above-mentioned various methods. The method includes: obtaining an RGB image to be recognized and a voice instruction of a user, and extracting a keyword of the voice instruction as a target text; based on a multi-modal pre-trained knowledge base, determining a knowledge prototype feature corresponding to the target text, and determining a target similarity with the largest similarity from the similarities between the knowledge prototype feature and multiple region features corresponding to the RGB image; in the case where the target similarity is greater than a preset threshold, determining a target object in the region features of the matching pair corresponding to the target similarity, determining a target coordinate of the target object based on the depth information of the target object, and determining a navigation path of the robot based on the target coordinate and a robot movement model.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0108] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot vision-language navigation method based on key-point guidance, characterized in that Including: Obtain the RGB image to be recognized and the user's voice instruction, and extract the keyword of the voice instruction as the target text; Based on the multi-modal pre-trained knowledge base, determine the knowledge prototype features corresponding to the target text, and determine the maximum target similarity from the similarities between the knowledge prototype features and multiple region features corresponding to the RGB image; When the target similarity is greater than a preset threshold, determine the target object in the region features of the matching pair corresponding to the target similarity, determine the target coordinates of the target object based on the depth information of the target object, and determine the navigation path of the robot based on the target coordinates and the robot movement model.

2. The robot vision-language navigation method based on key-point guidance according to claim 1, wherein The determining the target coordinates of the target object based on the depth information of the target object includes: Based on the depth information and the pixel coordinates of the target object in the RGB image, determine the target camera coordinates of the target object in the camera coordinate system; Based on the current position coordinates of the robot and the target camera coordinates, determine the target coordinates of the target object in the world coordinate system.

3. The method for robot vision-language navigation based on key-point guidance according to claim 1, wherein The determining step of the similarity between the knowledge prototype features and multiple region features corresponding to the RGB image includes: Based on the following formula, determine the similarity between the knowledge prototype features and multiple region features corresponding to the RGB image: Among them, represents the knowledge prototype features of the target text and multiple regional features the cosine similarity between them.

4. The method for robot vision-language navigation guided by key points according to any one of claims 1 to 3, characterized in that, The constructing step of the robot movement model includes: Establish a chassis coordinate system according to the plane where the chassis of the robot movement model is located. The chassis coordinate system takes the center of the chassis as the coordinate origin, the direction parallel to the body width direction of the robot as the X-axis, and the direction parallel to the body length direction of the robot as the Y-axis; Based on the chassis coordinate system, determine the speed components of the robot; the first speed components of the robot include the first speed of the robot along the X-axis, the second speed along the Y-axis, and the rotational angular velocity; Based on the first speed, the second speed, and the rotational angular velocity, determine the second speed components of each steering wheel of the robot; the second speed components include the first steering wheel speed, the second steering wheel speed, and the third steering wheel speed.

5. The robot vision-language navigation method based on key-point guidance according to claim 4, wherein, The method further includes: Based on the second speed components of each steering wheel, determine the third speed components of the robot; the third speed components include the first robot speed of the robot along the X-axis, the second robot speed along the Y-axis, and the third robot speed along the Z-axis.

6. The method for robot vision-language navigation guided by key points according to any one of claims 1 to 3, characterized in that The multi-modal pre-trained knowledge base includes the text categories of paired semantic concepts and the knowledge prototype features corresponding to the text categories; The knowledge prototype features are obtained by averaging the region representations related to the text categories.

7. The method for robot vision-language navigation based on key-point guidance according to any one of claims 1 to 3, characterized in that The determining the navigation path of the robot based on the target coordinates and the robot movement model includes: In the environmental map, based on the target coordinates and the robot movement model, perform path planning, and determine the navigation path of the robot based on the target path obtained from the path planning; The environmental map is determined based on the Gmapping method, two-dimensional laser scan data, and RGB image data; the two-dimensional laser scan data is acquired based on a sensor; and the RGB image data is acquired based on a camera.

8. A robot vision-language navigation device based on key-point guidance, characterized in that, It includes: An acquisition unit, configured to acquire an RGB image to be recognized and a voice command of a user, and extract a keyword of the voice command as a target text; A first determination unit, configured to determine a knowledge prototype feature corresponding to the target text based on a multi-modal pre-trained knowledge base, and determine a maximum target similarity from the similarity between the knowledge prototype feature and multiple region features corresponding to the RGB image; A second determination unit, configured to, when the target similarity is greater than a preset threshold, determine a target object in the region features of a matching pair corresponding to the target similarity, determine a target coordinate of the target object based on depth information of the target object, and determine a navigation path of the robot based on the target coordinate and a robot movement model.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the key-point-guided robot visual language navigation method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the key-point-guided robot visual language navigation method according to any one of claims 1 to 7.

Images