Global positioning method of robot, robot, medium, device and program product

Abstract

The present application provides a global positioning method for a robot, a robot, a medium, a device, and a program product. The method includes: after receiving a control instruction, using a large language model to identify the control instruction; when it is identified that the control instruction is used to instruct the robot to perform a repositioning operation, using the large language model to determine a historical image that is most similar to a first target image from multiple historical images as a first reference image; based on the first reference posture corresponding to the first reference image, matching the local point cloud map where the robot is currently located with the global map to determine the global posture of the robot. The present application utilizes the large language model's ability to parse control instructions to accurately identify repositioning requirements, and through comparison of historical images, quickly locks the first reference image that best matches the current environment, ensuring the robot's stable positioning effect in complex scenarios.

Description

Technical Field

[0001] The present application relates to the technical field of robot positioning algorithms, and in particular to a global positioning method, robot, medium, device and program product of a robot. Background Art

[0002] With the rapid development of artificial intelligence and robotics, robots are increasingly being used in various fields, including industry, services, and homes. Accurate positioning is crucial for robots to operate autonomously in complex and dynamic environments. Related robot localization methods determine the robot's position within its workspace by constructing and matching an environmental map.

[0003] Large language models, using deep learning technology, can extract complex semantic information from large amounts of data. However, the technology has yet to integrate large language models with robot localization algorithms.

[0004] Based on this, the present application provides a global positioning method of a robot, a robot, a medium, a device and a program product to improve related technologies. Summary of the Invention

[0005] The purpose of this application is to provide a global positioning method, robot, medium, device and program product for a robot, which combines a large language model with a robot positioning algorithm to achieve high-precision global positioning of the robot.

[0006] The purpose of this application is achieved by the following technical solutions:

[0007] In a first aspect, the present application provides a global positioning method for a robot, which is applied to a robot working in a target area, and the method comprises:

[0008] After receiving the control instruction, the control instruction is recognized by using the large language model;

[0009] When it is recognized that the control instruction is for instructing the robot to perform a relocation operation, the large language model is used to determine a historical image that is most similar to a first target image from a plurality of historical images as a first reference image; the first target image is obtained using a corresponding image sensor of the robot;

[0010] Based on a first reference pose corresponding to the first reference image, the local point cloud map where the robot is currently located is matched with a global map to determine the global pose of the robot.

[0011] In some embodiments, the control instruction is in the form of a voice message;

[0012] After receiving the control instruction, the control instruction is identified by using the large language model, including:

[0013] After receiving the control instruction, converting the control instruction into text information;

[0014] The text information is input into the large language model so that the large language model recognizes whether the control instruction is used to instruct the robot to perform a repositioning operation.

[0015] In some embodiments, upon recognizing that the control instruction is used to instruct the robot to perform a relocation operation, determining, using the large language model, from a plurality of historical images a historical image that is most similar to the first target image as the first reference image, includes:

[0016] When it is recognized that the control instruction is used to instruct the robot to perform a relocation operation, the first target image is compared with a plurality of historical images using the large language model to obtain a corresponding similarity of each historical image;

[0017] The historical image with the highest similarity is determined as the first reference image.

[0018] In some embodiments, the method further comprises:

[0019] In a case where it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, dividing the area in front of the robot into a plurality of sub-areas;

[0020] Measuring the distances to the nearest obstacles in the plurality of sub-areas by using a ranging sensor;

[0021] When the distances to the nearest obstacles in all sub-areas are greater than the distances to the target obstacles, controlling the robot to move forward;

[0022] During the movement of the robot, the first target image is acquired by using the corresponding image sensor of the robot.

[0023] In some embodiments, the method further comprises:

[0024] When the closest obstacle distance in at least one sub-area is not greater than the target obstacle distance, the robot is controlled to turn until the closest obstacle distances in the re-divided sub-areas are all greater than the target obstacle distance.

[0025] In some embodiments, the front area of ​​the robot is divided into a first sub-area located in the left front, a second sub-area located in the front, and a third sub-area located in the right front;

[0026] When the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, controlling the robot to turn until the nearest obstacle distances in the re-divided multiple sub-areas are all greater than the target obstacle distance, includes:

[0027] If the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance:

[0028] If the distance to the nearest obstacle in the first sub-area is greater than the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn left on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all greater than the target obstacle distance; or,

[0029] If the distance to the nearest obstacle in the first sub-area is smaller than the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn right on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all larger than the target obstacle distance; or,

[0030] If the distance to the nearest obstacle in the first sub-area is equal to the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn left or right on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all greater than the target obstacle distance.

[0031] In some embodiments, the process of forming the local point cloud map includes:

[0032] During the movement of the robot, a first key frame is acquired using a laser sensor corresponding to the robot, and the first key frame is used to initialize the local point cloud map and the local grid map;

[0033] When the movement distance of the robot is greater than the target distance and / or the movement angle of the robot is greater than the target angle, using the corresponding laser sensor of the robot to obtain a k-th key frame, and using the k-th key frame to update the local point cloud map; k is an integer greater than 1;

[0034] Using the initial pose of the k-th key frame, matching the k-th key frame with the local point cloud map;

[0035] Inserting the laser point cloud data between the k-1th key frame and the kth key frame into the local grid map;

[0036] The updated local grid map is used to filter the local point cloud map so that the local point cloud map only retains laser points that meet the target occupancy condition.

[0037] In some embodiments, matching the local point cloud map where the robot is currently located with a global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot includes:

[0038] Initializing particles based on the first reference pose to obtain a plurality of particles, and initializing a cumulative matching score and a cumulative prior score of each particle; each particle corresponds to a different global pose;

[0039] During the movement of the robot, the position and posture of each particle is updated according to the corresponding odometer data of the robot;

[0040] For at least one particle, based on the particle's position and posture, match the local point cloud map with the global map to obtain a current matching score between the local point cloud map and the global map. The score is used to update the particle's cumulative matching score and total score. The total score of the particle is the sum of the particle's cumulative matching score and the cumulative prior score.

[0041] Using the global pose of the particle with the highest total score, match the local point cloud map and the global map to obtain a current matching score of the local point cloud map and the global map, which is used as the current matching score of the particle with the highest total score;

[0042] When the current matching score of the particle with the highest total score is greater than the target matching score, the global pose corresponding to the particle with the highest total score is determined as the global pose of the robot.

[0043] In some embodiments, during the movement of the robot, updating the position and posture of each particle according to the corresponding odometer data of the robot includes:

[0044] During the movement of the robot, the position and posture of each particle is updated according to the corresponding odometer data and noise data of the robot.

[0045] In some embodiments, matching the local point cloud map where the robot is currently located with a global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot further includes:

[0046] Upon receiving the j-th reference pose of the robot:

[0047] If there is no particle within the target range where the j-th reference pose is located, performing particle initialization based on the j-th reference pose to obtain multiple particles, and initializing the cumulative matching score and the cumulative prior score of each particle obtained by performing particle initialization based on the j-th reference pose; or

[0048] If there is a particle within the target range where the j-th reference pose is located, the current prior score of the particle within the target range is calculated to update the cumulative prior score and total score of the corresponding particle;

[0049] Wherein, j is an integer greater than 1.

[0050] In some embodiments, the jth reference pose corresponds to a jth reference image, and a process of determining the jth reference image includes:

[0051] When the jth target image is received, the large language model is used to determine a historical image that is most similar to the jth target image from multiple historical images as the jth reference image; the jth target image is obtained using the corresponding image sensor of the robot.

[0052] In some embodiments, the first target image is in the form of a panoramic image, and the historical image is in the form of a panoramic image.

[0053] In a second aspect, the present application provides a robot comprising a control module and an image sensor, wherein the control module is used to execute any one of the above methods to determine the global posture of the robot; and the image sensor is used to acquire a first target image.

[0054] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0055] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above methods when executing the computer program.

[0056] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, implements any of the above methods.

[0057] The present application provides a global positioning method, robot, medium, device and program product for a robot, which realizes accurate robot positioning by combining a large language model (LLM) and point cloud map matching technology. Specifically, when the robot receives a control instruction, it first parses the control instruction through a large language model to identify whether a repositioning operation is required. If a repositioning operation is required, a historical image that is most similar to the first target image is selected from multiple historical images as a first reference image. Next, based on the first reference pose corresponding to the first reference image, the current local point cloud map of the robot is matched with the global map, and the global pose of the robot is determined by the matching result. Through the above method, the robot can quickly and accurately perform global positioning after receiving the control instruction, significantly improving the accuracy and efficiency of positioning. By utilizing the large language model's ability to parse the control instruction, the repositioning requirement can be accurately identified, and by comparing the historical images, the first reference image that best matches the current environment can be quickly locked. By matching the local point cloud map with the global map, the stable positioning effect of the robot in complex scenes is ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present application is further described below with reference to the accompanying drawings and specific implementation methods.

[0059] Figure 1 This is a characteristic point schematic diagram provided in an embodiment of the present application.

[0060] Figure 2 This is a grid map provided in an embodiment of the present application.

[0061] Figure 3 This is a flow chart of a global positioning method for a robot provided in an embodiment of the present application.

[0062] Figure 4 This is a schematic diagram of a 360-degree imaging technology for a car provided in an embodiment of the present application.

[0063] Figure 5 This is a schematic diagram of robot camera distribution provided in an embodiment of the present application.

[0064] Figure 6 This is a flowchart of calling a large language model provided in an embodiment of the present application.

[0065] Figure 7 This is a schematic diagram of dividing three sub-areas provided in an embodiment of the present application.

[0066] Figure 8 This is a flow chart of forming a local point cloud map and a local grid map provided in an embodiment of the present application.

[0067] Figure 9This is a schematic diagram of particle initialization provided in an embodiment of the present application.

[0068] Figure 10 Schematic diagram of a grid-based particle scoring strategy provided in an embodiment of the present application.

[0069] Figure 11 This is a structural block diagram of a robot provided in an embodiment of the present application.

[0070] Figure 12 This is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making any creative work are within the scope of protection of this application.

[0072] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly indicate the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0073] Currently, the robot performs global positioning by comparing the detected images and laser frames with a pre-established feature map or raster map. Here, taking a monocular camera and a single-line laser as an example, the relevant positioning process is shown below. The robot establishes a visual feature map and a laser raster map in the work area in advance. When positioning, the robot obtains the current position of the robot by comparing the captured image or the current laser frame with the bag-of-words feature point (refer to ORB-SLAM2) or the branch-and-bound method to obtain the optimal laser matching score (refer to cartographer). Among them, ORB-SLAM2 is a real-time visual SLAM tool library. Cartographer is a real-time indoor mapping algorithm that can generate raster maps.

[0074] See also Figure 1 and Figure 2 , Figure 1 This is a characteristic point diagram provided in an embodiment of the present application. Figure 2 This is a grid map provided in an embodiment of the present application.

[0075] like Figure 1As shown in , feature points can only represent the key features in the image, and at the same time, many other features will be lost, and all image pixel information cannot be used. Therefore, the reliability is often poor. Figure 2 As shown in the figure, the scheme based on the grid map also has obvious disadvantages. This is mainly because there are a large number of similar scenes in the map. By matching the current laser frame with the grid map, it is easy to generate multiple candidate results with a high degree of matching, and the reliability is often difficult to meet engineering requirements.

[0076] The applicant has proposed a precise global positioning solution for robots based on a large language model (LLM). This solution utilizes an LLM coarse positioning module and a laser high-precision positioning module. This solution allows robots to achieve global positioning at any location using only their cameras and lidar, even when lacking global positioning sensors (e.g., GPS or UWB).

[0077] As an example, the working process of the LLM coarse positioning module is shown below.

[0078] 1. Obtain 360-degree panoramic images: The robot uses multiple cameras on its own to record a 360-degree panoramic image database at the target workplace (similar to the 360-degree panoramic view function of a car).

[0079] 2. Data collection: The robot collects a panoramic image and corresponding pose coordinates (x, y, theta) every 2 meters in the working area to form an image pose sequence dataset.

[0080] 3. Voice Interaction: When the robot needs to be relocated, you can directly communicate with the robot to instruct it to relocate. The robot calls the LLM model, determines that the dialogue instruction is a relocation instruction, and then calls the relocation program to perform the relocation operation.

[0081] 4. Coarse relocalization: After the robot relocalization program is triggered, the roaming command is triggered, and the robot enters roaming mode (walking aimlessly). At the same time, the currently captured panoramic image is compared with the historical images (also panoramic images) in the database, and the most similar historical image is found. The pose coordinates of the most similar historical image are output. This comparison process can be implemented using LLM.

[0082] The high-precision laser positioning module performs secondary positioning based on the coarse positioning results of the LLM coarse positioning module, improving positioning accuracy and reliability. As an example, its working process is shown below.

[0083] 1. Point cloud SLAM: After the robot enters the roaming mode, it also turns on the SLAM (Simultaneous Localization and Mapping) mapping function to create a point cloud map fused with multiple laser frames.

[0084] 2. Fusion positioning: Based on the coarse positioning prior information and the posterior fusion of point cloud map matching, the current high-precision posture of the robot is determined to complete the positioning function.

[0085] The implementation methods of this application will be described in detail below.

[0086] See also Figure 3 , Figure 3 This is a flow chart of a global positioning method for a robot provided in an embodiment of the present application.

[0087] In order to improve the relevant technology, an embodiment of the present application provides a global positioning method of a robot, which is applied to a robot working in a target area. The method includes steps S101 to S103.

[0088] Step S101: After receiving a control instruction, the control instruction is recognized using a large language model.

[0089] Step S102: When it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, the large language model is used to determine a historical image that is most similar to a first target image from multiple historical images as a first reference image; the first target image is obtained using the corresponding image sensor of the robot.

[0090] Step S103: Based on the first reference pose corresponding to the first reference image, the local point cloud map where the robot is currently located is matched with the global map to determine the global pose of the robot.

[0091] Historical images are, for example, images collected and stored by the robot during its previous operation. Historical images are associated with corresponding pose information (which can be called historical poses, including position and orientation) to assist the robot in finding the scene most similar to the current environment during the relocalization process. The first reference image is an image selected from multiple historical images that is most similar to the current environment in the first target image. The pose information corresponding to the first reference image is called the first reference pose (including position and orientation), which is used to help the robot determine the current global pose. The local point cloud map is three-dimensional point cloud data collected by a laser sensor in the robot's current environment, which represents the environmental structure near the robot at the current moment. The global map is an overall three-dimensional environment model constructed by the robot in the target area through multiple three-dimensional point cloud data, which is used for global positioning and path planning. The global pose refers to the exact position and orientation of the robot in the global map, which can be regarded as a precise pose. By matching the local point cloud map with the global map, the specific position of the robot in the entire environment can be determined.

[0092] Related robot global positioning methods usually rely on global positioning sensors. The above embodiments are used for robots that lack global positioning sensors (e.g., GPS or UWB) and only use their own image sensors (e.g., cameras) and laser sensors (e.g., lidar) to achieve global positioning capabilities at any location through a large language model. The above embodiments achieve an efficient and reliable robot global positioning method by combining a large language model with point cloud map matching technology. Specifically, when the robot receives a control instruction, it first uses a large language model to parse the control instruction and determine the repositioning requirements. Then, the image that is most similar to the current image (i.e., the first target image) is selected from the previously stored historical images as the first reference image, and based on the pose information corresponding to the first reference image (i.e., the first reference pose), the local point cloud map is matched with the global map. Through this matching process, the global pose of the robot can be accurately determined, so that the robot can maintain accurate positioning even in complex environments.

[0093] Through the method provided by the above embodiment, the robot can achieve accurate global positioning by relying on its own image sensors and laser sensors without the need for global positioning sensors (such as GPS or UWB). By utilizing the natural language processing capabilities of the large language model, repositioning instructions can be efficiently identified and parsed, and the most matching first reference image can be quickly found from historical images, significantly improving the accuracy and reliability of positioning. In addition, the matching process of the local point cloud map and the global map ensures the accuracy of positioning, and a stable repositioning effect can be achieved even in similar scenes or dynamic environments, thereby improving the adaptability of the robot positioning algorithm in complex and changing environments.

[0094] See also Figure 4 and Figure 5 , Figure 4 This is a schematic diagram of a 360-degree car imaging technology provided by an embodiment of the present application. Figure 5 This is a schematic diagram of robot camera distribution provided in an embodiment of the present application.

[0095] In some embodiments, the first target image may be in the form of a panoramic image, and the historical image may be in the form of a panoramic image.

[0096] Among them, the panoramic image is a 360-degree panoramic image. The method of collecting panoramic images is similar to the 360-degree imaging method of automobiles. It is a mature technology and will not be described in detail in this article. It should be noted that the robot camera can be set to face upward instead of downward. This is because when the robot works indoors, the ceiling and surrounding environment information are better reference than the ground information. Figure 4 As shown, the robot cameras are distributed on the top, front, back, left, and right of the robot. For example, they are all monocular cameras with fields of view ABCDE respectively. The top corresponds to field of view E, and the front, back, left, and right correspond to fields of view ABCD. Among them, A corresponds to C, D corresponds to B, and E is located at the top. Figure 5 D is not shown in FIG. Through the image stitching algorithm, it is possible to stitch multiple camera images to form a 360-degree panoramic image.

[0097] See also Figure 6 , Figure 6 This is a flowchart of calling a large language model provided in an embodiment of the present application.

[0098] In some embodiments, the control instruction may be in the form of a voice message.

[0099] In some embodiments, after receiving the control instruction, using the large language model to identify the control instruction may include: after receiving the control instruction, converting the control instruction into text information; inputting the text information into the large language model so that the large language model can identify whether the control instruction is used to instruct the robot to perform a repositioning operation.

[0100] In some embodiments, when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, using the large language model to determine a historical image that is most similar to the first target image from multiple historical images as a first reference image may include: when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, using the large language model to compare the first target image with multiple historical images to obtain the corresponding similarity of each historical image; and determining the historical image with the highest similarity as the first reference image.

[0101] In the above embodiment, the robot receives the user's voice through a microphone, converts it into text information, and recognizes the relocation instruction through the large language model to trigger the relocation function. As an example, the large language model can adopt chatgpt-4, which can receive both text input and image input. The specific execution process is as follows Figure 6 shown.

[0102] 1. Use the voice-to-text module to convert the voice information received by the robot into text information.

[0103] 2. Input the text information as the promt (i.e., prompt word) for calling the large model into the large model calling module.

[0104] 3. The large model call module is implemented through the callback function of the LangChain framework. LangChain is a programming framework that helps use large language models (LLMs) in applications.

[0105] For example, the relevant promt can be set as: "Suppose you are a robot that can realize the self-positioning function. If you find that the user information contains positioning instructions, please reposition it."

[0106] 4. setPoseCallBack uses the LLM to compare the input first target image with the panoramic image database to obtain the most similar historical image and the corresponding similarity, with the similarity value ranging from [0, 1]. This process is also implemented using the LangChain framework. The panoramic image database includes multiple historical panoramic images.

[0107] For example, the relevant promt can be set as: "Now you are given an image to be queried and an image dataset. Please loop through the images in the image dataset, find the image that is most similar to the query image, and output the image number and similarity. The similarity range is 0 to 1, 1 means almost the same, and 0 means very different. Output in JSON format."

[0108] In the above embodiment, when the robot receives a control instruction in the form of voice, it first converts the voice information into text information, and then inputs the text information into the large language model, which then interprets the intent of the control instruction. If the control instruction involves a relocation operation, the large language model can be used to find the historical image that best matches the current environment from multiple historical images as the first reference image. Through the above method, the robot can efficiently and accurately recognize voice control instructions and quickly perform relocation operations. This method not only improves the accuracy of voice instruction recognition, but also reduces the complexity of relocation operations and the consumption of computing resources, enabling the robot to maintain stable positioning performance in dynamic and complex environments. In addition, the combination of the large language model and the LangChain framework enables efficient processing of voice information and image data, enabling the robot to reliably perform relocation under various environmental conditions, thereby improving the overall system's intelligence level and application scope.

[0109] See also Figure 7 , Figure 7 This is a schematic diagram of dividing three sub-areas provided in an embodiment of the present application.

[0110] In some embodiments, the method may further include: when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, dividing the area in front of the robot into multiple sub-areas; measuring the distance to the nearest obstacle in multiple sub-areas by a ranging sensor; when the distance to the nearest obstacle in all sub-areas is greater than the target obstacle distance, controlling the robot to move forward; and during the movement of the robot, using the corresponding image sensor of the robot to obtain the first target image.

[0111] In some embodiments, the method may further include: when the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, controlling the robot to turn until the nearest obstacle distances in multiple re-divided sub-areas are all greater than the target obstacle distance.

[0112] In some embodiments, the front area of the robot can be divided into a first sub-area located in the front left, a second sub-area located directly in front, and a third sub-area located in the front right. When the distance to the nearest obstacle in at least one sub-area is not greater than the target obstacle distance, controlling the robot to turn until the distance to the nearest obstacle in each of the re-divided sub-areas is greater than the target obstacle distance includes: when the distance to the nearest obstacle in at least one sub-area is not greater than the target obstacle distance: if the distance to the nearest obstacle in the first sub-area is greater than the distance to the nearest obstacle in the second sub-area, controlling the robot to turn left in place until the distance to the nearest obstacle in each of the 3 re-divided sub-areas is greater than the target obstacle distance; or, if the distance to the nearest obstacle in the first sub-area is less than the distance to the nearest obstacle in the second sub-area, controlling the robot to turn right in place until the distance to the nearest obstacle in each of the 3 re-divided sub-areas is greater than the target obstacle distance; or, if the distance to the nearest obstacle in the first sub-area is equal to the distance to the nearest obstacle in the second sub-area, controlling the robot to turn left or right in place until the distance to the nearest obstacle in each of the 3 re-divided sub-areas is greater than the target obstacle distance.

[0113] The above embodiments propose a simple and reliable roaming scheme. The purpose of robot roaming is to make the robot move so as to see more environmental features, thereby improving the stability of positioning. The roaming principle is as Figure 6 shown. The front area of the robot is divided into multiple sub-areas. Assuming the number of sub-areas is 3, the front area can be divided into three sub-areas: left, middle, and right. The distances to the nearest obstacles in the three sub-areas are measured respectively by a ranging sensor (e.g., lidar, depth camera, ultrasonic sensor, etc.), and are denoted as L, M, and R respectively. Among them, the distance to the nearest obstacle in a sub-area refers to the minimum value of the distances between all obstacles in the sub-area and the robot.

[0114] Assuming the target obstacle distance is 2m, when L, M, and R are all greater than 2m, the robot moves forward. When any one or more of L, M, and R are less than 2m, compare the magnitudes of L and R: when L > R, the robot turns left in place until L, M, and R are all greater than 2m, and then continues to move forward; similarly, when L < R, the robot turns right in place until L, M, and R are all greater than 2m, and then continues to move forward.

[0115] Through the above simple measurements, the robot can achieve active exploration and operation in unknown areas, and has a smooth obstacle avoidance function, which can meet the requirements of repositioning movement.

[0116] In the above embodiment, a sub-area refers to a small area within the robot's field of view. The area in front of the robot can be divided into multiple sub-areas (e.g., left, center, and right), which are used to independently measure the distance to obstacles in each sub-area to achieve accurate obstacle avoidance and path planning. Distance measuring sensors include, for example, lidar, depth cameras, ultrasonic sensors, etc., which are used to measure the distance between the robot and obstacles. The target obstacle distance is, for example, a pre-set distance threshold, such as 1m, 2m, 3m, etc. Roaming refers to the robot moving randomly or according to certain rules in the environment to detect more environmental features and improve its positioning accuracy. In the above embodiment, the roaming mode is used to achieve partitioned measurement and obstacle avoidance of the area in front of the robot, enabling active exploration of unknown areas. L, M, and R represent the distance to the nearest obstacle in the left, center, and right sub-areas in front of the robot, respectively, and are used to determine the safety of the robot's progress and decide whether the robot should turn or continue forward.

[0117] When avoiding obstacles, related robots often lack sufficient detection and response speed to obstacles, which can easily lead to collisions or stagnation in complex environments. Related robot positioning and obstacle avoidance algorithms often operate independently, preventing the robot from fully utilizing obstacle avoidance information during relocalization operations, thereby affecting positioning accuracy and efficiency. Related roaming solutions involve complex path planning and algorithm calculations, making it difficult to maintain stable and reliable operation in dynamic or unknown environments. The above-described embodiment provides a simple and reliable roaming and obstacle avoidance solution, combining the collaborative operation of ranging sensors and image sensors to achieve efficient robot movement and positioning during relocalization. Specifically, when the robot receives a relocalization command, it first divides the area in front of the robot into multiple subareas and uses ranging sensors to measure the distance to the nearest obstacle in each subarea. If the distance to the nearest obstacle in all subareas is greater than the target obstacle distance, the robot continues forward; otherwise, the robot performs steering operations based on the distance to the nearest obstacle in each subarea until the distance to the nearest obstacle in all subareas is greater than the target obstacle distance. During the robot's movement, it continuously uses image sensors to acquire environmental data and combines this with historical imagery for localization, enabling active exploration of unknown areas and stable global positioning. Furthermore, this solution simplifies roaming strategies, enabling the robot to achieve smooth exploration and localization in dynamic or unknown environments.

[0118] See also Figure 8 , Figure 8 This is a flow chart of forming a local point cloud map and a local grid map provided by an embodiment of the present application. Figure 8 In the figure, the left picture is a schematic diagram of the first key frame, the middle picture is a schematic diagram of initializing the local point cloud map and the local grid map, and the right picture is the final local point cloud map and the local grid map.

[0119] In some embodiments, the process of forming the local point cloud map may include:

[0120] During the movement of the robot, a first key frame is acquired using a laser sensor corresponding to the robot, and the first key frame is used to initialize the local point cloud map and the local grid map;

[0121] When the movement distance of the robot is greater than the target distance and / or the movement angle of the robot is greater than the target angle, using the corresponding laser sensor of the robot to obtain a k-th key frame, and using the k-th key frame to update the local point cloud map; k is an integer greater than 1;

[0122] Using the initial pose of the k-th key frame, matching the k-th key frame with the local point cloud map;

[0123] Inserting the laser point cloud data between the k-1th key frame and the kth key frame into the local grid map;

[0124] The updated local grid map is used to filter the local point cloud map so that the local point cloud map only retains laser points that meet the target occupancy condition.

[0125] Here, the target distance is, for example, 9 cm, 10 cm, 11 cm, etc., and the target angle is, for example, 4 degrees, 5 degrees, 6 degrees, etc. Each laser point in the point cloud corresponds to a grid, and each grid can be divided into an obstacle point, a traversable area, and an unexplored area based on the number of occupancy and / or penetration times. The process of determining whether a laser point meets the target occupancy condition can be, for example, when the grid corresponding to the laser point is an obstacle point, the laser point is deemed to meet the target occupancy condition; when the grid corresponding to the laser point is a traversable area or an unexplored area, the laser point is deemed not to meet the target occupancy condition.

[0126] Taking a single-line laser sensor as an example, the process of building a local point cloud map based on SLAM is introduced as follows.

[0127] S1. Use the first laser frame as the first key frame (or initial key frame) to initialize the local point cloud map and the local grid map.

[0128] S2. When the robot moves a certain distance (e.g., 10 cm) or angle (e.g., 5 degrees), insert the kth keyframe into the local point cloud map and calculate the initial pose of the kth keyframe based on the odometry data. Based on the initial pose of the kth keyframe, perform ICP matching between the kth keyframe and the local point cloud map.

[0129] S3. Insert the laser point cloud data between the kth key frame and the k-1th key frame into the local grid map. The local grid map generation method can refer to the slam-gmapping program, for example.

[0130] S4. Filter the local point cloud map through the local grid map, for example, only retaining the laser points corresponding to the obstacle points.

[0131] S5. Repeat steps S2-S4 to form a local point cloud map.

[0132] In the above embodiment, the local grid map is, for example, a two-dimensional grid map generated based on laser point cloud data. Each grid represents a small area in the environment, indicating whether the area is occupied (e.g., an obstacle) or passable. A keyframe refers to a representative laser frame selected during the robot's motion. Keyframes can be used to update the local point cloud map and accurately determine the robot's local pose in the local point cloud map through registration (e.g., ICP matching). ICP (Iterative Closest Point) matching is an algorithm for point cloud data alignment that, through continuous iteration, finds the optimal matching position between two point clouds to minimize error. In the above embodiment, ICP matching is used to align a new keyframe with the existing local point cloud map, thereby updating the local point cloud map and correcting the robot's local pose in the local point cloud map. The target occupancy condition is, for example, a criterion used to determine whether a laser point should be retained in the local point cloud map. In the above embodiment, if the grid corresponding to the laser point is an obstacle point, the laser point meets the target occupancy condition and is retained; otherwise, the laser point is filtered out. SLAM (Simultaneous Localization and Mapping) is a technology that allows a robot to build a map in real time while simultaneously determining its own position in an unknown environment. In the above embodiment, SLAM technology is used to construct a local point cloud map and a grid map using point cloud data acquired by a laser sensor, and to update the robot's position in real time.

[0133] The above-described embodiment combines the construction and updating of local point cloud maps and local grid maps to provide an efficient and accurate robot positioning and navigation method. Specifically, during robot motion, the local point cloud map and local grid map are initialized using the first keyframe captured by the laser sensor. When the robot moves beyond a preset target distance or angle, it captures a new keyframe and aligns it with the existing local point cloud map using a matching method such as the ICP matching algorithm to correct the robot's position. Next, the laser point cloud data between adjacent keyframes is inserted into the local grid map. The local grid map is then filtered to retain only those laser points that meet the target occupancy criteria. This process repeats as the robot continues to move, continuously updating and optimizing the local point cloud map. Through this method, the robot can generate high-precision local point cloud maps in complex and dynamic environments, significantly improving the accuracy and detail of map construction. By introducing the target occupancy criteria and the filtering mechanism of the grid map, the impact of noise and redundant data is effectively reduced, ensuring the practicality of the local point cloud map.

[0134] See also Figure 9 and Figure 10 , Figure 9 This is a schematic diagram of particle initialization provided in an embodiment of the present application. Figure 10 Schematic diagram of a grid-based particle scoring strategy provided in an embodiment of the present application.

[0135] In some embodiments, the matching of the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot may include: initializing particles based on the first reference pose to obtain multiple particles, and initializing the cumulative matching score and the cumulative prior score of each particle; each particle corresponds to a different global pose; during the movement of the robot, the pose of each particle is updated according to the corresponding odometer data of the robot; for at least one particle, based on the pose of the particle, matching the local point cloud map with the global map to obtain The current matching score of the local point cloud map and the global map is used as the current matching score of the particle, and is used to update the cumulative matching score and the total score of the particle; the total score of the particle is the sum of the cumulative matching score and the cumulative prior score of the particle; the global pose of the particle with the highest total score is used to match the local point cloud map and the global map to obtain the current matching score of the local point cloud map and the global map as the current matching score of the particle with the highest total score; when the current matching score of the particle with the highest total score is greater than the target matching score, the corresponding global pose of the particle with the highest total score is determined as the global pose of the robot.

[0136] In order to reduce the amount of calculation and take into account the robot positioning accuracy and operation efficiency, in some embodiments, for at least one particle, based on the posture of the particle, the local point cloud map and the global map are matched to obtain the current matching score of the local point cloud map and the global map, which is used as the current matching score of the particle to update the cumulative matching score and the total score of the particle. It can include: for multiple particles corresponding to the target grid, based on the posture of any particle corresponding to the target grid, the local point cloud map and the global map are matched to obtain the current matching score of the local point cloud map and the global map, which is used as the current matching score of the particle to update the cumulative matching score and the total score of the particle.

[0137] In some embodiments, updating the position and posture of each particle according to the corresponding odometer data of the robot during the movement of the robot may include: updating the position and posture of each particle according to the corresponding odometer data and noise data of the robot during the movement of the robot.

[0138] In some embodiments, the matching of the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot may also include: upon receiving the j-th reference pose of the robot: if there are no particles within the target range where the j-th reference pose is located, initializing the particles based on the j-th reference pose to obtain multiple particles, and initializing the cumulative matching score and the cumulative prior score of each particle obtained by initializing the particles based on the j-th reference pose; or, if there are particles within the target range where the j-th reference pose is located, calculating the current prior score of the particles within the target range to update the cumulative prior score and the total score of the corresponding particles; wherein j is an integer greater than 1.

[0139] In some embodiments, the jth reference pose corresponds to the jth reference image, and the process of determining the jth reference image may include: when the jth target image is received, using the large language model to determine a historical image that is most similar to the jth target image from multiple historical images as the jth reference image; the jth target image is obtained using the corresponding image sensor of the robot.

[0140] Since the coarse positioning result has low accuracy (for example, the error is about 2m) and there is a certain probability of error, the coarse positioning result can be further verified by point cloud matching to improve the positioning accuracy. The above embodiment designs a coarse and fine positioning fusion strategy. The basic idea is as follows: the coarse positioning result is used as prior information, the matching score of the local point cloud map and the global map is used as posterior information, and the final result is obtained through particle filter fusion. The specific steps are as follows:

[0141] R1. Initialization. When the first coarse positioning result (e.g., the first reference pose) is received, the particles are initialized using this pose as the starting point. As an example, the particle position (x, y) and angle (theta) are Gaussian distributed, with a variance of 0.02 and a particle number of 120. Initialize the cumulative matching score M of each particle. i is 0, and the cumulative prior score P i is 0, and the prior score is calculated according to the following formula:

[0142] P i =P i +pj

[0143] p j =h j max(0.0,(2.0-d i ) / 2.0)

[0144] Among them, h j is the similarity between the jth target image and the jth reference image, P i is the cumulative prior score of the i-th particle, p j is the jth matching score, d i Represents the distance of the particle from the jth reference pose. Assuming this is the jth time, the matching score for this time is the jth matching score. It should be noted that the prefixes "first" and "jth" in the "first target image" and "jth target image" only serve to distinguish them, and their essence is both target images. Similarly, the prefixes "first" and "jth" in the "first reference image" and "jth reference image" only serve to distinguish them, and their essence is both reference images (i.e., one of the historical images). Similarly, the prefixes "first" and "jth" in the "first reference pose" and "jth reference pose" only serve to distinguish them, and their essence is both reference poses (i.e., the historical poses of the corresponding historical images).

[0145] R2. Use the motion model to add motion noise to each particle during the robot's motion. For example, add noise every 10 cm or every 5 degrees.

[0146] x=x last +T·Δx·(1+α)+σ

[0147] y=y last +T·Δy·(1+α)+σ

[0148] δ=δ last +T·Δδ·(1+β)+E

[0149] Among them, x, y, δ are the robot posture and angle, x last ,y last , δ last is the robot pose at the time of the last calculation, Δx, Δy, and Δδ are the coordinate changes in the robot's odometry coordinates, T is the transformation matrix from the odometry coordinate system to the global coordinate system, σ and E are Gaussian noise with variances of, for example, 0.01 and 0.02, respectively. α and β are error coefficients, with α ranging from, for example, -0.2 to 0.2, with an interval of 0.04, for example, for a total of 11 values. β ranges from, for example, -0.1 to 0.1, with an interval of 0.02, for example, for a total of 11 values. α and β form 121 combinations, for example, for each of the 121 particles initialized according to the first reference pose.

[0150] R3. Calculate the matching score. As an example, calculate the position coordinate range [X min , X max ],[Y min , Y max ], using the above pose coordinate range as the boundary, and setting a grid with a resolution of 0.2. If a particle exists within the grid, use the pose of any particle within the grid as the initial value, perform NICP matching on the local point cloud map and the global map, and use the resulting match score as the current match score for all particles within the grid. As an example, the current match score is the proportion of points whose nearest neighbor is less than 0.05m after point cloud matching, and the value range is [0, 1]. The cumulative match score for the particle is calculated by summing up the scores of each match.

[0151] M; = ∑m,

[0152] Among them, M i is the cumulative matching score of the i-th particle, m j is the j-th matching score of the i-th particle.

[0153] Update the total score of each particle.

[0154] S i =M i +P i

[0155] R4. Integrate the latest coarse positioning information. When new coarse positioning information (e.g., the jth reference pose) is obtained, determine whether there are particles near the pose (e.g., within a 2m diameter range centered on the particle). If no particles exist, add particles according to step R1. That is, initialize the particles based on the pose and initialize the particles' cumulative matching scores and cumulative prior scores. If particles exist, calculate the current prior scores for these particles according to step R1.

[0156] R5. Get the particle S with the highest total score max and with the particle (ie, S max ) is used as the initial value, and the local point cloud map and the global map are matched to obtain the current matching score. If the current matching score is greater than the target matching score (for example, 0.6), the relocalization is successful.

[0157] In the above embodiment, particle initialization is a step in the particle filter algorithm. Initially, a group of particles is generated based on known prior information (such as coarse positioning results). These particles represent the possible global poses of the robot. In the above embodiment, particle initialization exemplarily uses a Gaussian distribution to generate multiple particles, each particle corresponding to a different global pose. The cumulative matching score refers to the score accumulated through multiple matching calculations, which is used to evaluate the degree of matching of the corresponding particle in the global map. The higher the cumulative matching score, the more consistent the pose represented by the particle is with the actual situation. The prior score is a preliminary evaluation of the particle based on prior information (such as the similarity between the corresponding reference image and the target image). The cumulative prior score is used together with the cumulative matching score to calculate the total score of the particle, thereby helping to determine the most likely global pose of the robot. The motion model describes the changes in the position and posture of the particles during the robot's motion. For example, the particles are updated through odometry data and a noise model. In the above embodiment, every time the robot moves a certain distance or angle, motion noise can be added to simulate the uncertainty of actual motion. NICP (Normalized Iterative Processing) e Closest Point (NICP) matching is an improved point cloud matching algorithm that aligns a local point cloud map with a global map and calculates a matching score between the two. In the above embodiment, NICP matching is used to precisely align the poses represented by different particles and evaluate the accuracy of each particle. The total score of a particle is the sum of its cumulative matching score and its prior score, representing the overall confidence of the particle in the current positioning process. The particle with the highest total score is selected as the most likely global pose of the robot.

[0158] Related robot localization methods typically require processing a large number of particles and data when performing point cloud matching, resulting in a significant computational effort and impacting the robot's real-time positioning performance and efficiency. The above-described embodiment utilizes a coarse-fine localization fusion strategy to achieve an efficient robot global localization method. After receiving the coarse localization results, the system first uses these results as prior information to initialize particles, generating multiple particles, each representing a possible global pose. During robot motion, the pose of each particle is dynamically updated based on odometry data and a noise model. The NICP matching algorithm calculates the particle's matching score between the local point cloud map and the global map. Through successive iterations, the matching scores and prior scores of each particle are accumulated, ultimately selecting the particle with the highest total score as the robot's global pose. This process effectively combines coarse localization information with precise point cloud matching, gradually approximating the robot's true position. Through this method, the robot can achieve high-precision global localization in complex environments, significantly improving positioning accuracy. The improved particle filtering algorithm reduces the number of particles involved in matching while ensuring the stability and reliability of the positioning results. In addition, the fusion strategy of cumulative matching score and cumulative prior score is used to reduce the dependence of the positioning process on the initial pose. Even when there are errors in the coarse positioning results, the accurate global pose can be gradually corrected and obtained. This not only improves the operating efficiency of the algorithm, but also enhances the adaptability and positioning performance of the robot in dynamic environments.

[0159] As can be seen, the above embodiment not only implements voice interaction relocalization based on LLM, but also uses panoramic imagery for robot positioning. A simple and reliable robot roaming solution enables forward and steering control during robot motion. It also employs a matching method based on a local point cloud map, a matching scoring mechanism combining global map particle filtering and ICP, and an efficient particle grid search mechanism. These embodiments specifically improve the robot's global positioning problem, enabling high-precision and efficient robot positioning.

[0160] An embodiment of the present application also provides a global positioning device for a robot, which is applied to a robot working in a target area. The device includes an LLM coarse positioning module and a laser high-precision positioning module.

[0161] The LLM coarse positioning module is used to use a large language model to identify the control instruction after receiving the control instruction; when it is identified that the control instruction is used to instruct the robot to perform a repositioning operation, the large language model is used to determine a historical image that is most similar to a first target image from multiple historical images as a first reference image; the first target image is obtained using the corresponding image sensor of the robot.

[0162] The laser high-precision positioning module is used to match the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot.

[0163] In some embodiments, the control instruction can be in the form of voice information; the LLM coarse positioning module can use the large language model to identify the control instruction after receiving the control instruction in the following manner: after receiving the control instruction, convert the control instruction into text information; input the text information into the large language model so that the large language model can identify whether the control instruction is used to instruct the robot to perform a repositioning operation.

[0164] In some embodiments, the LLM coarse positioning module can use the following method to determine a historical image that is most similar to the first target image from multiple historical images using the large language model when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation: when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, use the large language model to compare the first target image with multiple historical images to obtain the corresponding similarity of each historical image; and determine the historical image with the highest similarity as the first reference image.

[0165] In some embodiments, the device may further include a roaming module, which is used to divide the area in front of the robot into multiple sub-areas when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation; measure the distance to the nearest obstacle in multiple sub-areas through a ranging sensor; control the robot to move forward when the distance to the nearest obstacle in all sub-areas is greater than the target obstacle distance; and obtain the first target image using the corresponding image sensor of the robot during the movement of the robot.

[0166] In some embodiments, the roaming module can also be used to control the robot to turn when the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, until the nearest obstacle distances in multiple re-divided sub-areas are all greater than the target obstacle distance.

[0167] In some embodiments, the area in front of the robot can be divided into a first sub-area located in the left front, a second sub-area located in the front, and a third sub-area located in the right front; the roaming module adopts the following method to control the robot to turn when the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, until the nearest obstacle distances in the re-divided multiple sub-areas are all greater than the target obstacle distance: when the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance: if the nearest obstacle distance in the first sub-area is greater than the nearest obstacle distance in the second sub-area, then The robot is controlled to turn left on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all greater than the target obstacle distance; or, if the distance to the nearest obstacle in the first sub-area is less than the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn right on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all greater than the target obstacle distance; or, if the distance to the nearest obstacle in the first sub-area is equal to the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn left or right on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all greater than the target obstacle distance.

[0168] In some embodiments, the device may further include a point cloud map forming module, which is used to obtain a first key frame using a corresponding laser sensor of the robot during the movement of the robot, and use the first key frame to initialize the local point cloud map and the local grid map; when the movement distance of the robot is greater than the target distance and / or the movement angle of the robot is greater than the target angle, use the corresponding laser sensor of the robot to obtain the kth key frame, and use the kth key frame to update the local point cloud map; k is an integer greater than 1; use the initial pose of the kth key frame to match the kth key frame with the local point cloud map; insert the laser point cloud data between the k-1th key frame and the kth key frame into the local grid map; use the updated local grid map to filter the local point cloud map so that the local point cloud map only retains laser points that meet the target occupancy conditions.

[0169] In some embodiments, the laser high-precision positioning module can match the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image in the following manner to determine the global pose of the robot: perform particle initialization based on the first reference pose to obtain multiple particles, and initialize the cumulative matching score and the cumulative prior score of each particle; each particle corresponds to a different global pose; during the movement of the robot, update the pose of each particle according to the corresponding odometer data of the robot; for at least one particle, match the local point cloud map with the global map based on the pose of the particle to obtain The current matching score of the local point cloud map and the global map is used as the current matching score of the particle, and the current matching score is used to update the cumulative matching score and the total score of the particle; the total score of the particle is the sum of the cumulative matching score and the cumulative prior score of the particle; the global pose of the particle with the highest total score is used to match the local point cloud map and the global map to obtain the current matching score of the local point cloud map and the global map as the current matching score of the particle with the highest total score; when the current matching score of the particle with the highest total score is greater than the target matching score, the corresponding global pose of the particle with the highest total score is determined as the global pose of the robot.

[0170] In some embodiments, the laser high-precision positioning module can update the position and posture of each particle according to the corresponding odometer data of the robot during the movement of the robot in the following manner: during the movement of the robot, the position and posture of each particle is updated according to the corresponding odometer data and noise data of the robot.

[0171] In some embodiments, the laser high-precision positioning module can also be used to, when receiving the j-th reference pose of the robot: if there is no particle within the target range where the j-th reference pose is located, perform particle initialization based on the j-th reference pose to obtain multiple particles, and initialize the cumulative matching score and cumulative prior score of each particle obtained by particle initialization based on the j-th reference pose; or, if there is a particle within the target range where the j-th reference pose is located, calculate the current prior score of the particle within the target range, and use the current prior score to update the cumulative prior score and total score of the corresponding particle; wherein j is an integer greater than 1.

[0172] In some embodiments, the jth reference pose corresponds to the jth reference image, and the laser high-precision positioning module can determine the jth reference image in the following manner: when the jth target image is received, the large language model is used to determine a historical image that is most similar to the jth target image from multiple historical images as the jth reference image; the jth target image is obtained using the corresponding image sensor of the robot.

[0173] In some embodiments, the first target image may be in the form of a panoramic image, and the historical image may be in the form of a panoramic image.

[0174] See also Figure 11 , Figure 11 This is a structural block diagram of a robot provided in an embodiment of the present application.

[0175] An embodiment of the present application also provides a robot, comprising a control module and an image sensor, wherein the control module is used to execute any one of the above methods to determine the global posture of the robot; and the image sensor is used to capture a first target image.

[0176] In some embodiments, the robot may further include a laser sensor. In some embodiments, the laser sensor may include a 2D laser sensor and / or a 3D laser sensor.

[0177] In some embodiments, the robot may also include a microphone.

[0178] In some embodiments, the robot may also include an odometry and / or an IMU.

[0179] In some embodiments, the robot may further include one or more of an angle encoder, a torque sensor, and a PIR sensor.

[0180] In some embodiments, the robot may be a multi-jointed robot.

[0181] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0182] An embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, implements any of the above methods.

[0183] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the present application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0184] An embodiment of the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0185] See also Figure 12 , Figure 12 This is a structural block diagram of a computer device provided in an embodiment of the present application.

[0186] The embodiments of the present application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0187] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected via an internal connection path.

[0188] The memory 110 is used to store computer programs. In some implementations, the computer programs may include codes for implementing the methods of the embodiments of the present application.

[0189] The processor 120 is configured to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output data such as operation results. In some implementations, when the solutions of the embodiments of the present application are implemented through software or firmware, the computer program for implementing the solutions of the embodiments of the present application may be stored in the processor 120 and executed by the processor 120.

[0190] The memory 110 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes a random access memory (RAM), a cache memory and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120 so that the processor 120 implements the steps of any of the above methods.

[0191] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or the processor 120 may be any conventional processor.

[0192] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor 120 or by instructions in the form of software. The method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor 120. The software module can be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0193] In some implementations, in addition to the hardware units described above, the computer device may also include software modules, where the software modules may be, for example, an operating system, a basic input and output system (BIOS), application software, etc.

[0194] An operating system manages the hardware and / or software resources of a computer device and is the core and cornerstone of the computer. It handles basic tasks such as managing and allocating memory, prioritizing the supply and demand of system resources, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide an interface for users to interact with the system.

[0195] The BIOS is used to run hardware initialization during the power-on boot phase and provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display the processor temperature and execute functions such as adjusting temperature protection strategies.

[0196] Application software, also known as an application program, is software written for a specific user purpose. It is a major category of computer software. For example, application software might be a program used for power control, temperature management, and other purposes.

[0197] It should be noted that although some embodiments of this application take mobile robots as an example, this application can be applied to other bionic robots, such as AGVs, drones, etc., and this application is not limited to this.

[0198] It should be understood that the specific examples in this specification are only intended to help those skilled in the art better understand the implementation methods of the present application, rather than to limit the scope of protection of the present application.

[0199] It can be understood that in the various implementations of this specification, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0200] It can be understood that the various implementation methods described in this specification can be implemented individually or in combination, and this application is not limited to this.

[0201] Unless otherwise indicated, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art in the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more of the relevant listed items. The singular forms "a", "above", and "the" used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0202] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0203] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described embodiments may refer to the corresponding processes in other embodiments and will not be repeated here.

[0204] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0205] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the objectives of the technical solutions of this application.

[0206] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0207] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0208] The above are only specific embodiments of this specification, but the scope of protection of this application is not limited to them. Any changes or substitutions that can be easily conceived by any person skilled in the art within the technical scope disclosed in this specification should be included in the scope of protection of this specification. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A global positioning method for a robot, characterized in that: Applied to a robot operating in a target area, the method comprises: After receiving the control instruction, the control instruction is recognized by using the large language model; When it is recognized that the control instruction is for instructing the robot to perform a relocation operation, the large language model is used to determine a historical image that is most similar to a first target image from a plurality of historical images as a first reference image; the first target image is obtained using a corresponding image sensor of the robot; Based on a first reference pose corresponding to the first reference image, matching a local point cloud map where the robot is currently located with a global map to determine a global pose of the robot; The matching of the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot includes: Initializing particles based on the first reference pose to obtain a plurality of particles, and initializing a cumulative matching score and a cumulative prior score of each particle; each particle corresponds to a different global pose; During the movement of the robot, the position and posture of each particle is updated according to the corresponding odometer data of the robot; For at least one particle, based on the particle's position and posture, match the local point cloud map with the global map to obtain a current matching score between the local point cloud map and the global map. The score is used to update the particle's cumulative matching score and total score. The total score of the particle is the sum of the particle's cumulative matching score and the cumulative prior score. Using the global pose of the particle with the highest total score, match the local point cloud map and the global map to obtain a current matching score of the local point cloud map and the global map, which is used as the current matching score of the particle with the highest total score; When the current matching score of the particle with the highest total score is greater than the target matching score, the global pose corresponding to the particle with the highest total score is determined as the global pose of the robot.

2. The global positioning method of a robot according to claim 1, characterized in that: The control instruction is in the form of voice information; After receiving the control instruction, the control instruction is identified by using the large language model, including: After receiving the control instruction, converting the control instruction into text information; The text information is input into the large language model so that the large language model recognizes whether the control instruction is used to instruct the robot to perform a repositioning operation.

3. The global positioning method of a robot according to claim 1, characterized in that: The method of, when recognizing that the control instruction is used to instruct the robot to perform a relocation operation, using the large language model to determine, from a plurality of historical images, a historical image that is most similar to the first target image as a first reference image, comprises: When it is recognized that the control instruction is used to instruct the robot to perform a relocation operation, the first target image is compared with a plurality of historical images using the large language model to obtain a corresponding similarity of each historical image; The historical image with the highest similarity is determined as the first reference image.

4. The global positioning method of a robot according to claim 1, characterized in that: The method further comprises: In a case where it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, dividing the area in front of the robot into a plurality of sub-areas; Measuring the distances to the nearest obstacles in the plurality of sub-areas by using a ranging sensor; When the distances to the nearest obstacles in all sub-areas are greater than the distances to the target obstacles, controlling the robot to move forward; During the movement of the robot, the first target image is acquired by using the corresponding image sensor of the robot.

5. The global positioning method of a robot according to claim 4, characterized in that: The method further comprises: When the closest obstacle distance in at least one sub-area is not greater than the target obstacle distance, the robot is controlled to turn until the closest obstacle distances in the re-divided sub-areas are all greater than the target obstacle distance.

6. The global positioning method of a robot according to claim 5, characterized in that: The front area of ​​the robot is divided into a first sub-area located in the left front, a second sub-area located in the front, and a third sub-area located in the right front; When the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, controlling the robot to turn until the nearest obstacle distances in the re-divided multiple sub-areas are all greater than the target obstacle distance, includes: If the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance: If the distance to the nearest obstacle in the first sub-area is greater than the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn left on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all greater than the target obstacle distance; or, If the distance to the nearest obstacle in the first sub-area is smaller than the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn right on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all larger than the target obstacle distance; or, If the distance to the nearest obstacle in the first sub-area is equal to the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn left or right on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all greater than the target obstacle distance.

7. The global positioning method of a robot according to claim 1, characterized in that: The formation process of the local point cloud map includes: During the movement of the robot, a first key frame is acquired using a laser sensor corresponding to the robot, and the first key frame is used to initialize the local point cloud map and the local grid map; When the movement distance of the robot is greater than the target distance and / or the movement angle of the robot is greater than the target angle, using the corresponding laser sensor of the robot to obtain a k-th key frame, and using the k-th key frame to update the local point cloud map; k is an integer greater than 1; Using the initial pose of the k-th key frame, matching the k-th key frame with the local point cloud map; Inserting the laser point cloud data between the k-1th key frame and the kth key frame into the local grid map; The updated local grid map is used to filter the local point cloud map so that the local point cloud map only retains laser points that meet the target occupancy condition.

8. The global positioning method of a robot according to claim 1, characterized in that: During the movement of the robot, updating the position and posture of each particle according to the corresponding odometer data of the robot includes: During the movement of the robot, the position and posture of each particle is updated according to the corresponding odometer data and noise data of the robot.

9. The global positioning method of a robot according to claim 1, characterized in that: The matching of the local point cloud map where the robot is currently located with a global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot further includes: Upon receiving the j-th reference pose of the robot: If there is no particle within the target range where the j-th reference pose is located, performing particle initialization based on the j-th reference pose to obtain multiple particles, and initializing the cumulative matching score and the cumulative prior score of each particle obtained by performing particle initialization based on the j-th reference pose; or If there is a particle within the target range where the j-th reference pose is located, the current prior score of the particle within the target range is calculated to update the cumulative prior score and total score of the corresponding particle; Wherein, j is an integer greater than 1.

10. The global positioning method of a robot according to claim 9, characterized in that: The j-th reference pose corresponds to the j-th reference image, and the process of determining the j-th reference image includes: When the jth target image is received, the large language model is used to determine a historical image that is most similar to the jth target image from multiple historical images as the jth reference image; the jth target image is obtained using the corresponding image sensor of the robot.

11. A robot, characterized in that: The robot includes a control module and an image sensor, the control module is used to execute the method according to any one of claims 1 to 10 to determine the global pose of the robot; the image sensor is used to acquire a first target image.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 10 when executing the computer program.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images