A robot navigation method, apparatus and electronic device

Abstract

Embodiments of the present application provide a robot navigation method and device and electronic equipment, and relate to the technical field of robot navigation. The electronic equipment can obtain a description text and a map. In the process of the robot traveling, a scene image collected at a current traveling position is obtained. The map, the description text and the scene image are taken as inputs of a pre-trained navigation model. The navigation model determines a map position of the current traveling position in the map according to visual semantics represented by the scene image, and generates a plurality of trajectory points conforming to a traveling direction of the map position described by the description text in the scene image. The robot is controlled to travel according to a ground point corresponding to the trajectory point. The step of obtaining the scene image collected at the current traveling position is returned until the robot reaches a navigation end point. In this way, robot navigation can be realized by combining multi-modal information through a model framework.

Description

Technical Field

[0001] This application relates to the field of robot navigation technology, and in particular to a robot navigation method, apparatus and electronic device. Background Technology

[0002] As robots demonstrate increasing stability and maneuverability in complex environments, they are showing tremendous application potential in scenarios such as outdoor inspection, security patrol, disaster relief, and logistics delivery. In these scenarios, robots need to navigate in large-scale, open, and uncertain environments. However, the complex and varied road conditions in outdoor environments demand strong adaptability from robots; therefore, a robot navigation method is currently needed. Summary of the Invention

[0003] The purpose of this application is to provide a robot navigation method, device, and electronic device, which achieve robot navigation by combining textual and visual information through a unified model framework. The specific technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a robot navigation method, the method comprising:

[0005] Obtain descriptive text and a map, wherein the descriptive text is used to describe the robot's direction of travel, and the direction of travel conforms to the trajectory from the navigation start point to the navigation end point;

[0006] During the robot's movement, scene images are acquired at the current movement position;

[0007] The map, the descriptive text, and the scene image are used as inputs to a pre-trained navigation model. The navigation model determines the current travel position on the map based on the visual semantics represented by the scene image, and generates multiple trajectory points in the scene image that correspond to the travel direction at the map position described in the descriptive text.

[0008] The robot is controlled to move along the ground points corresponding to the trajectory points, and then returns to the step of acquiring the scene image collected at the current moving position, until the robot reaches the navigation endpoint.

[0009] Secondly, embodiments of this application provide a robot navigation device, the device comprising:

[0010] The information acquisition module is used to acquire descriptive text and a map, wherein the descriptive text is used to describe the robot's direction of travel, and the direction of travel conforms to the trajectory from the navigation starting point to the navigation ending point;

[0011] The image acquisition module is used to acquire scene images at the current travel position during the robot's movement.

[0012] The trajectory point generation module is used to take the map, the descriptive text, and the scene image as input to a pre-trained navigation model, use the navigation model to determine the current travel position on the map based on the visual semantics represented by the scene image, and generate multiple trajectory points in the scene image that conform to the travel direction at the map position described by the descriptive text.

[0013] The robot control module is used to control the robot to move along the ground points corresponding to the trajectory points, trigger the image acquisition module, until the robot reaches the navigation endpoint.

[0014] Thirdly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0015] Memory, used to store computer programs;

[0016] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the steps described in the first aspect above.

[0018] Beneficial effects of the embodiments in this application:

[0019] In the solution provided in this application, during the robot's movement, the navigation model onboard the robot can determine its map position based on the visual semantics represented by the scene images acquired in real time. Then, according to the direction of travel described in the descriptive text, it generates multiple trajectory points in the scene image that conform to that direction. Subsequently, it controls the robot to move according to the ground points corresponding to the determined trajectory points. The navigation model can process the input map, scene images, and descriptive text simultaneously. Since the descriptive text can describe the robot's direction of travel on the map, and this direction of travel is consistent with the direction of the trajectory from the navigation start point to the navigation end point, the navigation model can use the direction of travel described in the descriptive text as a basis when generating trajectory points from the scene image. Thus, the trajectory traversed by the robot moving according to the ground points corresponding to the trajectory points output by the navigation model is relatively close to the trajectory from the navigation start point to the navigation end point. During robot navigation, the navigation model can process multimodal information such as descriptive text, map, and scene images, thus enabling robot navigation through a single model framework combined with multimodal information. Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0021] Figure 1 A flowchart illustrating a robot navigation method provided in an embodiment of this application;

[0022] Figure 2(a) shows the results based on Figure 1 A schematic diagram of a scene image in the embodiment shown;

[0023] Figure 2(b) is based on Figure 1 A schematic diagram of an alternative point in a scene image of the illustrated embodiment;

[0024] Figure 2(c) is based on Figure 1 A schematic diagram of trajectory points in a scene image of the illustrated embodiment;

[0025] Figure 2(d) is based on Figure 1 Another schematic diagram of trajectory points in the scene image of the illustrated embodiment;

[0026] Figure 2(e) is based on Figure 1 Another schematic diagram of trajectory points in the scene image of the illustrated embodiment;

[0027] Figure 3 Based on Figure 1 The illustrated embodiment includes a schematic diagram of an obstacle region in a scene image;

[0028] Figure 4 Based on Figure 1 A schematic diagram of a map in the illustrated embodiment;

[0029] Figure 5 Based on Figure 1 A schematic diagram of a robot navigation method in the embodiment shown;

[0030] Figure 6 This is a schematic diagram of the structure of a robot navigation device provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0033] To achieve robot navigation by combining multimodal information such as text, maps, and visual information within a model framework, embodiments of this application provide a robot navigation method, apparatus, electronic device, computer-readable storage medium, and computer program product. The robot navigation method provided in this application embodiment is described below.

[0034] The robot navigation method provided in this application can be applied to any electronic device that requires robot navigation, such as a robot control device or processor, etc., without specific limitations. For clarity, it will be referred to as an electronic device herein. The robot mentioned above can be any robot capable of movement, such as a bipedal robot, a quadrupedal robot, etc., again without specific limitations.

[0035] like Figure 1 As shown, this is a robot navigation method, including S101 to S104.

[0036] S101, retrieve the description text and map.

[0037] The descriptive text describes the robot's direction of travel, which follows the trajectory from the navigation start point to the navigation end point.

[0038] S102: During the robot's movement, acquire scene images collected at the current movement position.

[0039] S103 uses the map, descriptive text, and scene image as a pre-trained navigation model, so that the navigation model can determine the map position of the current travel position in the map based on the visual semantics represented by the scene image, and generate multiple trajectory points in the scene image that match the travel direction at the map position described in the descriptive text.

[0040] S104, control the robot to move according to the ground point corresponding to the trajectory point, return to step S102, until the robot reaches the navigation destination.

[0041] As can be seen, in this embodiment, during the robot's movement, the navigation model onboard the robot can determine its map position based on the visual semantics represented by the scene images acquired in real time. Then, according to the direction of travel described in the descriptive text from the navigation start point to the navigation end point, it generates multiple trajectory points in the scene image that conform to this direction. Subsequently, it controls the robot to move according to the ground points corresponding to the determined trajectory points. The navigation model can process the input map, scene images, and descriptive text simultaneously. Since the descriptive text can describe the robot's direction of travel on the map, and this direction of travel is consistent with the direction of the trajectory from the navigation start point to the navigation end point, the navigation model can use the direction of travel described in the descriptive text as a basis when generating trajectory points from the scene image. Thus, the trajectory traversed by the robot moving according to the ground points corresponding to the trajectory points output by the navigation model is relatively close to the trajectory from the navigation start point to the navigation end point. During robot navigation, the navigation model can process multimodal information such as descriptive text, map, and scene images, thus enabling robot navigation through a single model framework that combines multimodal information.

[0042] During robot navigation, the electronic devices on the robot can acquire descriptive text and a map, i.e., execute step S101. The map can be a map formed by visual semantic modeling of the robot navigation scene, and can be a global map or a local map. The map coverage area is at least between the navigation start point and the navigation end point.

[0043] The above descriptive text can be used to describe the robot's future direction of travel, such as: go straight to the intersection and turn left, continue straight, turn right at the large tree at the end of the road, etc. The direction of travel described in the descriptive text is consistent with the direction of the trajectory from the navigation start point to the navigation end point. The trajectory from the navigation start point to the navigation end point can be a trajectory obtained based on path planning of the navigation start point and the navigation end point, representing the trajectory the robot will travel from the navigation start point to the navigation end point without encountering obstacles.

[0044] Specifically, from a global perspective of the navigation start point to the navigation end point, the direction of travel described in the descriptive text is consistent with the direction of the trajectory from the navigation start point to the navigation end point. For example, in the overall direction of travel, if the trajectory from the navigation start point to the navigation end point is straight → left turn → right turn, then the direction of travel described in the descriptive text is also straight → left turn → right turn.

[0045] From a local perspective, if the robot needs to perform temporary obstacle avoidance, such as avoiding obstacles like pedestrians, vehicles, or puddles in front of it, then during the obstacle avoidance process, the robot's actual direction of travel will deviate to some extent from the direction of the trajectory from the navigation start point to the navigation end point. However, the overall direction of travel will still be consistent with the direction of the trajectory from the navigation start point to the navigation end point.

[0046] During the robot's movement, in order to determine the road conditions ahead of the robot and the robot's current position in real time, the electronic equipment can acquire scene images collected at the robot's current position, i.e., execute step S102. The scene images can be acquired by the robot's onboard image acquisition device at preset acquisition intervals. The acquisition interval can be set according to actual navigation needs, such as 20ms, 50ms, 100ms, etc., and is not specifically limited here.

[0047] At the start of navigation, the electronic device can input a map and descriptive text into a pre-trained navigation model. Subsequently, each time a newly acquired scene image is obtained, the electronic device can input that scene image into the navigation model, i.e., execute step S103. The navigation model is obtained by post-training a multimodal large language model (MLLM) with rich pre-trained knowledge using an open-source autonomous driving dataset; that is, through model training, the multimodal language model acquires navigation capabilities.

[0048] The navigation model can determine the robot's current position on the map based on the visual semantics represented by the scene image. Specifically, the navigation model can identify visual semantic information that matches the visual semantics represented by the scene image from the visual semantic information corresponding to each location on the map, and use the position corresponding to the identified visual semantic information as the robot's map position.

[0049] The map location can be based on a global coordinate system, and the origin of the global coordinate system can be a preset location in the robot navigation scenario. Preferably, the preset location can be the navigation starting point.

[0050] After determining the robot's map location, the navigation model can determine the direction of travel for that location as described in the descriptive text, and then generate multiple trajectory points in the scene image that conform to that direction of travel. In other words, the direction of the trajectory formed by the multiple trajectory points in the scene image is consistent with the direction of travel.

[0051] Specifically, assuming there are no obstacle areas in the scene image, the direction of the trajectory formed by multiple trajectory points in the scene image is the same as the direction of travel. If there are obstacle areas in the scene image, the overall trend of the direction of the trajectory formed by multiple trajectory points is the same as the direction of travel. For example, if the description text indicates a straight direction of travel, and the scene image includes puddles, the trajectory formed by multiple trajectory points generated by the navigation model will bypass the puddles. To avoid the puddles, although the direction of the trajectory formed by multiple trajectory points is not strictly a straight direction, the overall trend is still a straight direction.

[0052] After determining multiple trajectory points, the navigation model can transform these trajectory points from the pixel coordinate system to the camera coordinate system based on the intrinsic parameters of the image acquisition device. Then, using the extrinsic parameters of the image acquisition device, the trajectory points in the camera coordinate system can be transformed to the world coordinate system to obtain the ground points corresponding to the trajectory points.

[0053] Next, the navigation model can call the robot's motion sequence generation interface to generate a sequence of motions that allows the robot to move along ground points. This motion sequence can include the robot's joint data at each moment. Electronic devices can then control the robot to move according to the motion sequence generated by the navigation model, enabling the robot to travel along ground points.

[0054] Robot navigation is a cyclical process. When the image acquisition device acquires scene images for the next time, the electronic device can return to execute steps S102 to S104, which means planning trajectory points in the latest acquired scene image and controlling the robot to move according to the ground points corresponding to the trajectory points until the robot reaches the navigation endpoint and ends the above cycle.

[0055] It is evident that by unifying trajectory understanding, language interaction, visual perception, and motion control from the navigation starting point to the navigation endpoint into a single navigation model, and deploying this model on a robot, the robot can better adapt to the complexities of the real world. This significantly enhances its autonomous decision-making ability, operational flexibility, and environmental adaptability in outdoor tasks, forming a crucial foundational capability for future intelligent mobile platforms. Compared to traditional navigation methods that execute global and local planning through different models, the robot navigation method provided in this application uses a single model framework, eliminating the need to distinguish between global and local information. It takes into account both global and local information, avoiding information fragmentation between multiple models.

[0056] As one embodiment of this application, the step of generating multiple trajectory points in the scene image that conform to the direction of travel at the map location described in the descriptive text may include:

[0057] Visual semantic analysis is performed on the scene image, and the robot-accessible area included in the scene image is determined based on the obtained visual semantic analysis. In the robot-accessible area, multiple candidate points are generated according to preset height direction intervals and preset width direction intervals. According to the relationship between the travel direction described by the sample description text and the calibration trajectory points included in the sample scene image, multiple trajectory points that match the travel direction at the map location described in the description text are selected from the candidate points.

[0058] When determining trajectory points in a scene image, the navigation model can generate multiple candidate points within the robot-reachable area included in the scene image. This robot-reachable area can be a pre-defined image region. For example, since the lower half of a scene image typically represents the road surface, while the upper half typically represents walls, sky, etc., the robot-reachable area can be the lower half of the scene image.

[0059] Next, the navigation model can select multiple trajectory points from the candidate points that match the travel direction described in the sample descriptive text, based on the pre-learned relationship between the travel direction described in the sample descriptive text and the calibration trajectory points included in the sample scene image. The calibration trajectory points are multiple trajectory points marked in the sample scene image based on the travel direction described in the sample descriptive text. The aforementioned relationship can be used to reflect which sample descriptive text corresponds to which calibration trajectory point.

[0060] For example, a schematic diagram of a scene image can be shown in Figure 2(a). The lower half of Figure 2(a) is the robot's reachable area, so multiple candidate points can be generated in the lower half of Figure 2(a) to obtain Figure 2(b). The hollow circles in Figure 2(b) represent candidate points, and each candidate point has corresponding coordinates above it, such as (13,5), (14,5), etc.

[0061] Assuming the described direction of travel is straight, the candidate points with coordinates (13,5), (14,5), (15,5), (16,5), (17,5), and (18,5) can be identified as trajectory points, resulting in Figure 2(c). The solid circles in Figure 2(c) represent trajectory points, and the trajectory formed by these points is in the straight-line direction.

[0062] Assuming the description text indicates a left turn, the candidate points with coordinates (15,0), (15,1), (15,2), (16,4), and (18,5) can be identified as trajectory points, resulting in Figure 2(d). The solid circles in Figure 2(d) represent trajectory points, and the trajectory formed by these points indicates a left turn.

[0063] In one implementation, the number of selected trajectory points is a preset number, and the trajectory length corresponding to the preset number of trajectory points in the world coordinate system is at least the distance traveled by the robot during the scene image acquisition interval.

[0064] Specifically, scene images are acquired at certain intervals. In each loop, the robot moves according to the ground points corresponding to the trajectory points included in the currently acquired scene image. If the number of trajectory points selected from the scene image each time is small, the robot may have already completed the movement according to the trajectory points, but the next scene image has not yet been acquired, causing the robot to be unable to proceed to the next step and its movement to be interrupted.

[0065] To avoid the above situation, the number of trajectory points selected each time can be controlled to a preset number. The trajectory length corresponding to the preset number of trajectory points in the world coordinate system is at least the distance traveled by the robot during the scene image acquisition interval. In other words, the trajectory length corresponding to the selected trajectory points in the world coordinate system at least covers the distance traveled by the robot before the next scene image acquisition. For example, assuming the scene image acquisition interval is 200ms, and the robot travels 0.3 meters in 200ms, if the trajectory length corresponding to 2 trajectory points in the world coordinate system is 0.3 meters, then the preset number is 2, meaning at least 2 trajectory points are selected from the scene image each time.

[0066] In one implementation, if the aforementioned preset number is set too large, it's possible that the robot may have not yet traveled the predetermined number of trajectory points corresponding to the world coordinate system's trajectory length before acquiring the next frame of scene image and determining a new preset number of trajectory points. In this case, the robot can continue to travel according to the ground points corresponding to the new preset number of trajectory points, discarding the remaining trajectory points that have not yet been traveled. Since the new preset number of trajectory points determined based on the currently acquired image, i.e., the aforementioned next frame of scene image, are trajectory points that match the robot's current location, traveling according to these new preset number of trajectory points better conforms to the trajectory from the navigation start point to the navigation end point, ensuring the accuracy of the robot's travel direction.

[0067] For example, as shown in Figure 2(e), the preset number is 4. If the robot has already output the latest 4 trajectory points when it walks to the third trajectory point 201, then the robot can no longer follow the ground point corresponding to the fourth trajectory point 202, but follow the ground point corresponding to the new 4 trajectory points, that is, follow the ground point corresponding to trajectory point 203.

[0068] As one implementation method, when the navigation model determines the robot's reachable area from the scene image, it can perform visual semantic analysis on the scene image to identify the visual entities included in each image region of the scene image. Then, based on the physical characteristics of the visual entities, it can determine whether the robot can reach the visual entity. For example, suppose the scene image includes regions 1 to 3, and the visual entities included are, in order: the sky, buildings, and the road surface. Since the robot can move on the road surface, the robot's reachable area in the scene image is region 3.

[0069] Next, the navigation model can generate multiple candidate points within the robot's reachable area according to pre-set selection point intervals. These pre-set selection point intervals can include a first preset interval corresponding to the image height direction and a second preset interval corresponding to the image width direction. The first and second preset intervals can be set according to actual needs, such as the first preset interval being 10 pixels and the second preset interval being 12 pixels, etc., without specific limitations here. For example, as shown in Figure 2(b), the candidate points are generated according to the preset intervals corresponding to the image height and image width directions.

[0070] As can be seen, in this embodiment, when determining the robot's reachable area in the scene image, the navigation model can perform visual semantic analysis on the scene image to determine the visual entities corresponding to each image region of the scene image, thereby accurately generating the robot's reachable area. When generating candidate points, the navigation model can generate multiple candidate points in the robot's reachable area according to a pre-set selection point interval, which can improve the efficiency of candidate point generation. Since the trajectory points need to be reachable by the robot and cannot be points in areas such as the sky or walls in the scene image, the navigation model can generate multiple candidate points in the robot's reachable area included in the scene image. During the model training phase, the navigation model has pre-learned the relationship between the travel direction described in the sample description text and the calibration trajectory points included in the sample scene image. Thus, during the model application phase, the navigation model can quickly and accurately select trajectory points from the candidate points according to the above relationship.

[0071] As one implementation of this application, the step of selecting multiple trajectory points from the candidate points that match the direction of travel at the map location described in the descriptive text may include:

[0072] In response to the navigation model detecting obstacle regions from the scene image, multiple trajectory points that match the travel direction at the map location described in the descriptive text are selected from the candidate points included in the non-obstacle regions.

[0073] To achieve obstacle avoidance during robot navigation, the navigation model can determine whether there are obstacle regions in the scene image based on the visual semantic analysis results. Since the robot can theoretically only reach the ground points corresponding to the robot's reachable area, the robot only needs to detour around obstacle regions within the robot's reachable area. To reduce computational load, the navigation model can determine only whether the robot's reachable area includes obstacle regions, without needing to determine areas that the robot cannot reach.

[0074] If the navigation model detects obstacle areas from the scene image, then in order to achieve active obstacle avoidance, the navigation model can select trajectory points from the candidate points included in the non-obstacle areas to achieve detour around the obstacle areas.

[0075] For example, Figure 3 This is a schematic diagram of a scene image including an obstacle area, where the obstacle area is a puddle area 301. In this case, the navigation model can select trajectory points from candidate points other than those with coordinates (15,5) and (16,5). Specifically, the navigation model can select candidate points with coordinates (18,5), (17,6), (16,6), (15,6), (14,5), and (13,5) as trajectory points. When the robot moves along the trajectory formed by these trajectory points, it can avoid the puddle.

[0076] As can be seen, in this embodiment, when an obstacle region is detected from the scene image, the navigation model can select trajectory points from candidate points included in the non-obstacle region. Thus, when the robot moves according to the ground points corresponding to the selected trajectory points, it can actively avoid obstacles in the obstacle region, thereby improving the safety of robot navigation. Compared to traditional navigation methods that execute global and local planning through different models, the robot navigation method provided in this embodiment, through a unified model framework, allows the robot to combine local and global information to make globally consistent re-decision when facing unexpected situations, such as the appearance of obstacles or terrain changes.

[0077] As one embodiment of this application, the direction of travel described in the above descriptive text can be referenced to a landmark, for example: walk straight until you reach a tree and then turn right. In this case, before the step of selecting multiple trajectory points from the candidate points that match the direction of travel at the map location described in the descriptive text, the method may further include:

[0078] The system performs target recognition on the scene image to obtain the target recognition result; in response to the target recognition result including a reference object, it determines the map position of the reference object in the map based on the visual semantics represented by the reference object in the scene image; it determines the relative positional relationship between the reference object and the robot based on the map position of the reference object and the robot's map position; in response to the relative positional relationship meeting the preset positional relationship conditions, it performs the step of selecting multiple trajectory points from the candidate points that match the travel direction at the map position described in the descriptive text.

[0079] To determine whether the currently acquired scene image includes a reference object from the descriptive text, the navigation model can perform target recognition on the scene image to obtain the target recognition result. If the target recognition result includes a reference object from the descriptive text, the navigation model can determine the reference object's location on the map based on the visual semantics represented by the reference object in the scene image, combined with the visual semantic information corresponding to various locations included in the map.

[0080] Next, the navigation model can determine the relative positional relationship between the reference object and the robot based on the map position of the reference object and the robot's map position, and then determine whether the relative positional relationship meets the preset positional relationship conditions, wherein the preset positional relationship conditions indicate that the distance between the robot and the reference object is less than the preset distance.

[0081] The relative positional relationship between the reference object and the robot can be represented in various ways, such as the distance between the reference object and the robot, whether the reference object is located within a certain range around the robot, or whether the robot is located within a certain range around the reference object, etc. No specific limitation is made here.

[0082] If the relative positional relationship meets the preset positional relationship conditions, then the distance between the robot and the reference object is less than the preset distance, indicating that the robot has reached a position very close to the reference object. In this case, the navigation model can perform the step of selecting multiple trajectory points from the candidate points that match the direction of travel to the map location described in the descriptive text. The preset distance can be set according to actual needs, such as 0.5 meters, 1 meter, 1.5 meters, etc., and is not specifically limited here.

[0083] For example, assuming the description text is "Go straight to the streetlight and turn right," the robot's map position is determined to be (100, 50) based on the latest captured scene image, and the streetlight's map position is (101, 50). If the unit length in the map coordinate system represents 1 meter in the world coordinate system, then the distance between the robot and the streetlight can be determined to be 1 meter. Assuming the preset distance is 1 meter, this means the robot has reached the streetlight, and at this point, multiple trajectory points that can form a right-turn trajectory can be selected from the scene image.

[0084] If the above relative positional relationship does not meet the preset positional relationship conditions, then the distance between the robot and the reference object is not less than the preset distance, indicating that the robot has not yet reached a position very close to the reference object. At this time, the robot can continue to move until the relative positional relationship between the robot and the reference object meets the preset positional relationship conditions, and then execute the above steps of selecting multiple trajectory points that match the direction of travel at the map location described in the descriptive text.

[0085] As can be seen, in the embodiments of this application, for descriptive text that describes the direction of travel with reference to a reference object, the navigation model can understand the sequential relationship in this language structure and perform spatial reasoning. In this way, the navigation model can improve its flexibility and its ability to understand descriptive text.

[0086] As one embodiment of this application, during the robot's movement, the method may further include:

[0087] If the angular deviation between the direction of the robot's historical trajectory and the direction of travel described in the descriptive text is greater than a preset angular deviation, the robot is controlled to stop moving; the stop position is taken as the current moving position, and the process of acquiring the scene image collected at the current moving position is returned.

[0088] During the robot's movement, the navigation model can determine the robot's historical trajectory based on its historical positions. These historical positions can be positions established a preset number of moments prior to the current moment. The preset number of moments can be set according to actual needs, such as 4, 5, or 6 moments prior to the current moment, etc., without specific limitations.

[0089] To determine whether the robot has veered off course, the navigation model can determine whether the angular deviation between the direction of the historical trajectory and the direction of travel described in the descriptive text is greater than a preset angular deviation. The deviation between the direction of the historical trajectory and the direction of travel described in the descriptive text can be represented by a deviation angle. The preset angular deviation can be set according to actual needs, such as 10°, 15°, 20°, etc., without specific limitations here.

[0090] Furthermore, even when the described direction of travel is straight, the navigation model can determine whether yaw has occurred based on the coordinate deviations of historical trajectories. For example, assuming the described direction of travel is straight along the Y-axis of the global coordinate system, but the difference between the X-axis coordinates of multiple historical trajectories is greater than a preset difference, then it can be determined that the robot has yawed.

[0091] Of course, when determining whether a robot has veered off course, it's necessary to exclude situations where the robot is actively avoiding obstacles. This is because when a robot is actively avoiding obstacles, its actual direction of travel may deviate from the direction described in the descriptive text. In such cases, determining whether the robot has veered off course is highly likely to result in a misjudgment.

[0092] If the angular deviation between the direction of the historical trajectory and the direction of travel described in the descriptive text is greater than the preset angular deviation, it indicates that the robot has veered off course. In order to correct the error in time, the electronic device can control the robot to stop moving and take the position where it stops as the current position, and return to step S102, that is, to re-plan the path from the position where it stops.

[0093] In the above embodiments, the navigation model possesses the ability to determine whether the robot has veered off course because yaw sample data is introduced during the training phase of the navigation model. Yaw sample data can include sample description text and sample travel positions, and the angular deviation between the sample trajectory formed by the sample travel positions and the travel direction described in the sample description text is greater than a preset angular deviation. This allows the navigation model to learn a method for determining whether the robot has veered off course based on its historical travel positions.

[0094] As can be seen, in this embodiment, the navigation model can determine whether the robot has deviated from its course. If so, it can promptly control the robot to stop and replan its path. This improves the stability of robot navigation. Especially in complex, dynamic, and open outdoor environments, traditional rule-based or local planning-based navigation systems struggle to cope with dynamic road structures, weakly structured areas, and changes in multi-source task instructions. The robot navigation method provided in this embodiment utilizes the trajectory from the navigation starting point to the navigation endpoint, along with natural language instructions, to drive the robot's onboard navigation model, significantly enhancing the robot's autonomy and task understanding in real-world scenarios.

[0095] As one implementation of this application, the above-described step of obtaining the descriptive text may include:

[0096] Obtain the description text corresponding to the trajectory; and / or, obtain the text navigation instructions; parse the text navigation instructions to obtain the description text.

[0097] The descriptive text in this application embodiment exists in two forms, which will be described separately below. In the first embodiment, the electronic device can perform path planning based on the map, the robot's navigation starting point, and the navigation ending point to obtain the trajectory from the navigation starting point to the navigation ending point on the map. The trajectory from the navigation starting point to the navigation ending point can be a GPS (Global Positioning System) path, a sequence of target points, etc., and is not specifically limited here.

[0098] Next, the electronic device can convert the map locations and corresponding directions of travel along the route from the navigation start point to the navigation end point into descriptive text. For example, assuming the map is as follows: Figure 4 As shown, by planning the path based on the robot's navigation start and end points, the trajectory 401 from the navigation start point to the navigation end point can be obtained. If the trajectory 401 from the navigation start point to the navigation end point is converted into descriptive text, it can be obtained as follows: Start from the navigation start point and go straight along the road. Turn left at the end of the road and continue going straight along the road. Turn left next to the tree at the end of the road and continue going straight along the road to reach the navigation end point.

[0099] In the second implementation, the electronic device can acquire navigation instructions, which can be voice commands input by the user, such as "keep moving forward, avoid pedestrians" or "keep moving forward, avoid puddles," thereby assisting the navigation model in better local path planning. Navigation instructions can also be text commands input by the user through typing.

[0100] Upon receiving a navigation command, the electronic device can parse the command to obtain descriptive text. For example, if the navigation text is a voice command, the electronic device can perform speech-to-text processing to obtain descriptive text; if the navigation text is a text command, the electronic device can directly parse the text command to obtain descriptive text.

[0101] In the third implementation, the electronic device can acquire descriptive text obtained by converting the trajectory from the navigation start point to the navigation end point when the robot starts navigation, and acquire navigation instructions during the robot navigation process.

[0102] As can be seen, in this embodiment, the electronic device can obtain the descriptive text through the above three implementation methods. In the first implementation method, the descriptive text is obtained by converting the trajectory from the navigation start point to the navigation end point. The descriptive text is input into the navigation model, which can automatically parse the global target. In the second implementation method, the descriptive text is obtained by parsing the navigation instructions input by the user. This can assist the robot in judging the road environment and avoiding obstacles in a timely manner during navigation. In security patrol and outdoor monitoring tasks, the navigation model can flexibly adjust the patrol route and area of ​​interest according to the navigation instructions, actively avoiding obstacles, bypassing crowds, or moving to new viewpoints, thereby reducing the cost of manual intervention. In the third implementation method, the descriptive text corresponding to the trajectory from the navigation start point to the navigation end point, along with the navigation instructions, are input into the navigation model. The navigation model can understand the trajectory from the navigation start point to the navigation end point and the user's natural language instructions, and integrate the environmental visual information included in the scene image to output gait instructions that conform to the semantics of the descriptive text and safety constraints. This allows the robot to stably traverse various terrains such as sidewalks, grass, steps, and slopes, and can be widely applied in large-scale inspection scenarios such as urban roads, parks, and campuses.

[0103] As one implementation method of this application, the training method of the above navigation model may include:

[0104] Obtain training data pairs; use the training data pairs as input to the initial navigation model; use the initial navigation model to generate predicted trajectory points in the sample scene image according to the current model parameters and the travel direction described in the sample description text; adjust the model parameters of the initial navigation model according to the difference between the predicted trajectory points and the corresponding calibration trajectory points in the sample scene image until the initial navigation model meets the convergence condition, and obtain the trained navigation model.

[0105] When training a navigation model, training data pairs can be obtained, which may include sample scene images and corresponding sample descriptive text. Next, the training data pairs can be input into the initial navigation model, which can then generate predicted trajectory points in the sample scene images based on the current model parameters and the travel direction described in the sample descriptive text.

[0106] Since the predicted trajectory points are obtained by the initial navigation model based on the current model parameters, and the calibrated trajectory points are the actual trajectory points corresponding to the travel direction described in the descriptive text, in order to evaluate the accuracy of the predicted trajectory points and thus adjust the model parameters of the initial navigation model, the model parameters of the initial navigation model can be adjusted according to the difference between the predicted trajectory points and the calibrated trajectory points corresponding to the sample scene images, until the initial navigation model meets the convergence condition and the trained navigation model is obtained.

[0107] In the above process, when determining the difference between the predicted trajectory points and the calibrated trajectory points, a loss function in the form of language modeling cross-entropy can be used: ,in, This represents the model parameters of the initial navigation model. Let x represent the calibration trajectory point at time t, and x represent the predicted trajectory point. This represents the probability of predicting the current time's calibration trajectory point based on the predicted trajectory points and calibration trajectory points at historical moments. T represents the length of the time series, and log represents the logarithmic function, which can use the natural logarithm as the base.

[0108] The following analysis examines the aforementioned loss function, because... ,so ,and and Positively correlated. If The smaller, then The larger the value, the less accurate the predicted trajectory points output by the initial navigation model are, and the larger the value of the loss function. Therefore, when adjusting the model parameters, the model parameters of the initial navigation model can be adjusted in the direction that reduces the value of the loss function.

[0109] The above convergence conditions can be, for example, the number of times the model parameters are adjusted reaches a preset number, or the function value of the loss function converges, etc., and no specific limitations are made here.

[0110] As can be seen, in this embodiment of the application, by training the initial navigation model with training data pairs, the ability of the initial navigation model to generate trajectory points according to the direction of travel described in the descriptive text can be continuously optimized, thereby enabling the trained navigation model to have the ability to follow text commands.

[0111] In one embodiment of this application, when the robot's navigation scenario is an outdoor scene, the aforementioned sample scene images may include sample scene images collected in a preset outdoor scene. The visibility of the preset outdoor scene may be lower than a preset visibility threshold, enabling the trained navigation model to handle scene images collected in outdoor scenes with visibility lower than the preset visibility threshold. In other words, during the training phase of the navigation model, sample scene images collected in extreme weather conditions such as rain, snow, fog, and haze can be used, thereby improving the navigation model's stability in extreme weather. The preset visibility threshold can be set according to actual needs, such as 500 meters, 200 meters, 100 meters, etc., and is not specifically limited here.

[0112] In this case, the steps of controlling the robot to move according to the ground point corresponding to the trajectory point may include: controlling the robot to move in an outdoor scene where the visibility is lower than a preset visibility threshold according to the ground point corresponding to the trajectory point.

[0113] Because the navigation model can process scene images acquired in outdoor environments with visibility below a preset threshold, the robot can navigate in such conditions. By processing the acquired scene images using the navigation model, it can output accurate trajectory points. Consequently, the robot can then travel in outdoor environments with low visibility, following the ground points corresponding to these trajectory points. This enables the robot to navigate in low-visibility weather conditions such as rain, snow, and fog.

[0114] After obtaining the trained navigation model, it can be accelerated and then deployed to the robot. Acceleration techniques such as model pruning, knowledge distillation, and model parameter quantization are all reasonable.

[0115] As can be seen, in this embodiment of the application, the navigation model can process scene images collected in outdoor scenes where the visibility is lower than a preset visibility threshold. Using this navigation model, the robot can travel in outdoor scenes where the visibility is lower than a preset visibility threshold according to the ground points corresponding to the trajectory points, so that the robot can travel in weather with low visibility such as rain, snow, fog and haze.

[0116] As one embodiment of this application, a schematic diagram of a robot navigation method can be shown as follows: Figure 5As shown, at the start of robot navigation, the electronic device can input map 501 and descriptive text into the navigation model. The descriptive text can include: go straight along the road; turn left at the end of the road; turn left next to the tree; ..., etc. Subsequently, during the robot's movement, the electronic device can input scene images 502 captured by the robot's onboard image acquisition device into the navigation model. The navigation model can select trajectory points from the candidate points included in scene image 502 based on the direction of travel described in the descriptive text. Hollow circles in scene image 502 represent candidate points, each displaying its coordinates, such as (13,0), (13,1), etc. Solid circles in scene image 502 represent trajectory points, with coordinates of (13,2), (14,2), and (15,2). After obtaining the trajectory points, the navigation model can call the robot's action sequence generation interface to generate the action sequence required for movement according to the ground points corresponding to the trajectory points. The electronic device can then control the robot's movement according to the action sequence. This enables the robot to have outdoor navigation capabilities.

[0117] Corresponding to the robot navigation method described above, this application also provides a robot navigation device. The robot navigation device provided in this application is described below.

[0118] like Figure 6 The diagram shown is a structural schematic of a robot navigation device, which includes:

[0119] The information acquisition module 601 is used to acquire descriptive text and a map, wherein the descriptive text is used to describe the robot's direction of travel, and the direction of travel conforms to the trajectory from the navigation starting point to the navigation ending point;

[0120] The image acquisition module 602 is used to acquire scene images collected at the current travel position during the robot's movement.

[0121] The trajectory point generation module 603 is used to take the map, the descriptive text, and the scene image as input to a pre-trained navigation model, use the navigation model to determine the map position of the current travel position in the map based on the visual semantics represented by the scene image, and generate multiple trajectory points in the scene image that conform to the travel direction at the map position described by the descriptive text.

[0122] The robot control module 604 is used to control the robot to move along the ground points corresponding to the trajectory points, trigger the image acquisition module 602, until the robot reaches the navigation destination.

[0123] As can be seen, in this embodiment, during the robot's movement, the navigation model onboard the robot can determine its map position based on the visual semantics represented by the scene images acquired in real time. Then, according to the direction of travel described in the descriptive text from the navigation start point to the navigation end point, it generates multiple trajectory points in the scene image that conform to this direction. Subsequently, it controls the robot to move according to the ground points corresponding to the determined trajectory points. The navigation model can process the input map, scene images, and descriptive text simultaneously. Since the descriptive text can describe the robot's direction of travel on the map, and this direction of travel is consistent with the direction of the trajectory from the navigation start point to the navigation end point, the navigation model can use the direction of travel described in the descriptive text as a basis when generating trajectory points from the scene image. Thus, the trajectory traversed by the robot moving according to the ground points corresponding to the trajectory points output by the navigation model is relatively close to the trajectory from the navigation start point to the navigation end point. During robot navigation, the navigation model can process multimodal information such as descriptive text, map, and scene images, thus enabling robot navigation through a single model framework that combines multimodal information.

[0124] As one embodiment of this application, the trajectory point generation module 603 may include:

[0125] The reachable area determination submodule is used to perform visual semantic analysis on the scene image and determine the robot reachable area included in the scene image based on the visual semantic analysis.

[0126] The candidate point generation submodule is used to generate multiple candidate points in the robot's reachable area according to preset height direction intervals and preset width direction intervals;

[0127] The trajectory point selection submodule is used to select multiple trajectory points from the candidate points that match the travel direction described in the description text on the map, based on the relationship between the travel direction described in the pre-learned sample description text and the calibration trajectory points included in the sample scene image.

[0128] The number of the plurality of trajectory points is a preset number, and the trajectory length of the preset number of trajectory points in the world coordinate system is at least the distance traveled by the robot during the acquisition interval of the scene image.

[0129] As one embodiment of this application, the above-mentioned trajectory point selection submodule may include:

[0130] The trajectory point selection unit is used to select multiple trajectory points from the candidate points included in the non-obstacle area from the scene image in response to the navigation model detecting an obstacle area. These trajectory points correspond to the direction of travel at the map location described in the descriptive text.

[0131] As one embodiment of this application, the direction of travel described in the above descriptive text is referenced to a reference object. In this case, the above-described device may further include:

[0132] The target recognition module is used to perform target recognition on the scene image before selecting multiple trajectory points from the candidate points that match the direction of travel at the map location described in the descriptive text, and to obtain the target recognition result.

[0133] A location determination module is used to determine the map position of the reference object in the map based on the visual semantics represented by the scene image at the reference object in response to the target recognition result including the reference object;

[0134] The distance determination module is used to determine the relative positional relationship between the reference object and the robot based on the map position of the reference object and the map position of the robot;

[0135] The first triggering module is used to trigger the trajectory point selection submodule in response to the relative positional relationship meeting the preset positional relationship conditions.

[0136] The preset positional relationship condition indicates that the distance between the robot and the reference object is less than a preset distance.

[0137] As one embodiment of this application, the number of the above-mentioned multiple trajectory points is a preset number, and the trajectory length corresponding to the preset number of trajectory points in the world coordinate system is at least the distance traveled by the robot during the scene image acquisition interval.

[0138] As one embodiment of this application, the above-described apparatus may further include:

[0139] The stop-movement module is used to control the robot to stop moving in response to the fact that the angular deviation between the direction of the robot's historical trajectory and the direction of movement described in the descriptive text is greater than a preset angular deviation.

[0140] The second triggering module is used to trigger the image acquisition module 602 with the position where the movement has stopped as the current movement position.

[0141] As one embodiment of this application, the information acquisition module 601 described above may include:

[0142] The description text acquisition submodule is used to acquire the description text corresponding to the trajectory, wherein the trajectory is obtained by path planning based on the map, the navigation starting point and the navigation ending point, and the description text is obtained by converting the map location and the corresponding direction of travel included in the trajectory;

[0143] The navigation instruction acquisition submodule is used to acquire navigation instructions and parse the navigation instructions to obtain description text.

[0144] As one embodiment of this application, the above-described apparatus may further include:

[0145] The data pair acquisition module is used to acquire training data pairs, wherein the training data pairs include sample scene images and corresponding sample description text;

[0146] The trajectory point prediction module is used to take the training data pair as input to the initial navigation model, and use the initial navigation model to generate predicted trajectory points in the sample scene image according to the current model parameters and the direction of travel described by the sample description text.

[0147] The parameter adjustment module is used to adjust the model parameters of the initial navigation model according to the difference between the predicted trajectory points and the calibration trajectory points corresponding to the sample scene image, until the initial navigation model meets the convergence condition and the trained navigation model is obtained.

[0148] As one embodiment of this application, the navigation scenario of the robot is an outdoor scenario, and the sample scenario images include sample scenario images collected in a preset outdoor scenario. The visibility of the preset outdoor scenario is lower than a preset visibility threshold, which enables the trained navigation model to process scenario images collected in outdoor scenarios where the visibility is lower than the preset visibility threshold.

[0149] The robot control module 604 mentioned above may include:

[0150] The robot control submodule is used to control the robot to move in an outdoor scene where the visibility is lower than a preset visibility threshold, according to the ground points corresponding to the trajectory points.

[0151] like Figure 7 The diagram shown is a structural schematic of an electronic device, including:

[0152] The memory 701 is used to store computer programs; the processor 702 is used to execute the program stored in the memory 701 to implement the robot navigation method steps described in any of the above embodiments. Furthermore, the electronic device may also include a communication bus and / or a communication interface, and the processor 702, the communication interface, and the memory 701 communicate with each other through the communication bus.

[0153] As can be seen, in this embodiment, during the robot's movement, the navigation model onboard the robot can determine its map position based on the visual semantics represented by the scene images acquired in real time. Then, according to the direction of travel described in the descriptive text from the navigation start point to the navigation end point, it generates multiple trajectory points in the scene image that conform to this direction. Subsequently, it controls the robot to move according to the ground points corresponding to the determined trajectory points. The navigation model can process the input map, scene images, and descriptive text simultaneously. Since the descriptive text can describe the robot's direction of travel on the map, and this direction of travel is consistent with the direction of the trajectory from the navigation start point to the navigation end point, the navigation model can use the direction of travel described in the descriptive text as a basis when generating trajectory points from the scene image. Thus, the trajectory traversed by the robot moving according to the ground points corresponding to the trajectory points output by the navigation model is relatively close to the trajectory from the navigation start point to the navigation end point. During robot navigation, the navigation model can process multimodal information such as descriptive text, map, and scene images, thus enabling robot navigation through a single model framework that combines multimodal information.

[0154] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0155] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0156] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0157] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0158] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the robot navigation methods described above.

[0159] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the robot navigation methods described above.

[0160] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0161] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0162] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0163] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A robot navigation method, characterized in that, The method includes: Obtain descriptive text and a map, wherein the descriptive text is used to describe the robot's direction of travel, and the direction of travel conforms to the trajectory from the navigation start point to the navigation end point; During the robot's movement, scene images are acquired at the current movement position; The map, the descriptive text, and the scene image are used as inputs to a pre-trained navigation model. The navigation model determines the current travel position on the map based on the visual semantics represented by the scene image, and generates multiple trajectory points in the scene image that correspond to the travel direction at the map position described in the descriptive text. The robot is controlled to move along the ground points corresponding to the trajectory points, and then returns to the step of acquiring the scene image collected at the current moving position, until the robot reaches the navigation endpoint.

2. The method according to claim 1, characterized in that, The step of generating multiple trajectory points in the scene image that conform to the direction of travel at the map location described in the descriptive text includes: Visual semantic analysis is performed on the scene image, and the robot-accessible area included in the scene image is determined based on the obtained visual semantic analysis. In the robot's reachable area, multiple candidate points are generated according to preset height and width intervals; Based on the relationship between the travel direction described in the pre-learned sample description text and the calibration trajectory points included in the sample scene image, multiple trajectory points that conform to the travel direction at the map location described in the description text are selected from the candidate points. The number of multiple trajectory points is a preset number, and the trajectory length of the preset number of trajectory points in the world coordinate system is at least the distance traveled by the robot during the acquisition interval of the scene image.

3. The method according to claim 2, characterized in that, The step of selecting multiple trajectory points from the candidate points that match the direction of travel at the map location described in the descriptive text includes: In response to the navigation model detecting an obstacle region from the scene image, multiple trajectory points that match the direction of travel at the map location described in the descriptive text are selected from candidate points included in the non-obstacle region.

4. The method according to claim 2 or 3, characterized in that, The direction of travel described in the descriptive text is based on a reference point; Prior to the step of selecting multiple trajectory points from the candidate points that match the direction of travel at the map location described in the descriptive text, the method further includes: Target recognition is performed on the scene image to obtain the target recognition result; In response to the target recognition result including the reference object, the map position of the reference object in the map is determined according to the visual semantics represented by the scene image at the reference object; Based on the map positions of the reference object and the robot, determine the relative positional relationship between the reference object and the robot; In response to the relative positional relationship meeting a preset positional relationship condition, the step of selecting multiple trajectory points from the candidate points that conform to the direction of travel at the map location described in the descriptive text is executed, wherein the preset positional relationship condition indicates that the distance between the robot and the reference object is less than a preset distance.

5. The method according to any one of claims 1 to 4, characterized in that, During the robot's movement, the method further includes: If the angular deviation between the direction of the robot's historical trajectory and the direction of travel described in the descriptive text is greater than a preset angular deviation, the robot is controlled to stop moving. Take the position where you stop moving as the current moving position, and return to the step of acquiring the scene image captured at the current moving position.

6. The method according to any one of claims 1 to 5, characterized in that, The step of obtaining the description text includes: Obtain the description text corresponding to the trajectory, wherein the trajectory is obtained by path planning based on the map, the navigation starting point and the navigation ending point, and the description text is obtained by converting the map location and the corresponding direction of travel included in the trajectory; And / or, Obtain navigation instructions; parse the navigation instructions to obtain description text.

7. The method according to any one of claims 1 to 6, characterized in that, The training methods for the navigation model include: Acquire training data pairs, wherein the training data pairs include sample scene images and corresponding sample description text; The training data pairs are used as input to the initial navigation model. The initial navigation model generates predicted trajectory points in the sample scene image according to the current model parameters and the travel direction described in the sample description text. Based on the difference between the predicted trajectory points and the calibration trajectory points corresponding to the sample scene image, the model parameters of the initial navigation model are adjusted until the initial navigation model meets the convergence condition, thus obtaining the trained navigation model.

8. The method according to claim 7, characterized in that, The robot's navigation scenario is an outdoor scenario, and the sample scenario images include sample scenario images collected in a preset outdoor scenario. The visibility of the preset outdoor scenario is lower than a preset visibility threshold, which enables the trained navigation model to process scenario images collected in outdoor scenarios where the visibility is lower than the preset visibility threshold. The step of controlling the robot to move according to the ground points corresponding to the trajectory points includes: The robot is controlled to move in an outdoor scene where visibility is lower than a preset visibility threshold, following the ground points corresponding to the trajectory points.

9. A robot navigation device, characterized in that, The device includes: The information acquisition module is used to acquire descriptive text and a map, wherein the descriptive text is used to describe the robot's direction of travel, and the direction of travel conforms to the trajectory from the navigation starting point to the navigation ending point; The image acquisition module is used to acquire scene images at the current travel position during the robot's movement. The trajectory point generation module is used to take the map, the descriptive text, and the scene image as input to a pre-trained navigation model, use the navigation model to determine the current travel position on the map based on the visual semantics represented by the scene image, and generate multiple trajectory points in the scene image that conform to the travel direction at the map position described by the descriptive text. The robot control module is used to control the robot to move along the ground points corresponding to the trajectory points, trigger the image acquisition module, and continue until the robot reaches the navigation endpoint.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method according to any one of claims 1 to 8.

Images