Global positioning method of robot, robot, medium, equipment and program product

Abstract

The invention provides a robot global positioning method, a robot, a medium, equipment and a program product, and the method comprises the steps: carrying out the recognition of a control instruction through a large language model after the control instruction is received; under the condition that the control instruction is identified to be used for indicating the robot to perform relocation operation, determining a historical image most similar to a first target image from a plurality of historical images by using the large language model as a first reference image; and based on a first reference pose corresponding to the first reference image, matching a local point cloud map where the robot is currently located with a global map to determine the global pose of the robot. According to the method and the device, the analysis capability of the large language model on the control instruction is utilized, the repositioning requirement can be accurately identified, the first reference image which is most matched with the current environment is quickly locked through comparison of historical images, and the stable positioning effect of the robot in a complex scene is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of robot positioning algorithms, and in particular to a global positioning method of a robot, a robot, a medium, a device and a program product. Background Art

[0002] With the rapid development of artificial intelligence and robotics, robots are increasingly used in various fields such as industry, service industry and home. In order to enable robots to operate autonomously in complex and dynamic environments, accurate positioning capabilities are essential. Related robot positioning methods determine the position of the robot in the working area by building and matching environmental maps.

[0003] Large language models can extract complex semantic information from large amounts of data through deep learning technology. However, related technologies have not yet achieved the integration of large language models with robot positioning algorithms.

[0004] Based on this, the present application provides a global positioning method of a robot, a robot, a medium, a device and a program product to improve the relevant technology. Summary of the invention

[0005] The purpose of this application is to provide a global positioning method, robot, medium, device and program product for a robot, combining a large language model with a robot positioning algorithm to achieve high-precision global positioning of the robot.

[0006] The purpose of this application is achieved by the following technical solutions:

[0007] In a first aspect, the present application provides a global positioning method of a robot, which is applied to a robot working in a target area, and the method comprises:

[0008] After receiving the control instruction, the control instruction is recognized by using the large language model;

[0009] In the case where it is recognized that the control instruction is used to instruct the robot to perform a relocation operation, the large language model is used to determine a historical image most similar to a first target image from a plurality of historical images as a first reference image; the first target image is obtained using an image sensor corresponding to the robot;

[0010] Based on the first reference pose corresponding to the first reference image, the local point cloud map where the robot is currently located is matched with the global map to determine the global pose of the robot.

[0011] In some embodiments, the control instruction is in the form of a voice message;

[0012] After receiving the control instruction, using the large language model to identify the control instruction includes:

[0013] After receiving the control instruction, converting the control instruction into text information;

[0014] The text information is input into the large language model so that the large language model recognizes whether the control instruction is used to instruct the robot to perform a repositioning operation.

[0015] In some embodiments, when recognizing that the control instruction is used to instruct the robot to perform a relocation operation, determining, using the large language model, from a plurality of historical images a historical image that is most similar to the first target image as the first reference image, includes:

[0016] When it is recognized that the control instruction is used to instruct the robot to perform a relocation operation, the first target image is compared with a plurality of historical images using the large language model to obtain a corresponding similarity of each historical image;

[0017] A historical image with the highest similarity is determined as the first reference image.

[0018] In some embodiments, the method further comprises:

[0019] In a case where it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, dividing the front area of ​​the robot into a plurality of sub-areas;

[0020] Measuring the distances of the nearest obstacles in the plurality of sub-areas by using a ranging sensor;

[0021] When the nearest obstacle distances in all sub-areas are greater than the target obstacle distance, controlling the robot to move forward;

[0022] During the movement of the robot, the first target image is acquired using the image sensor corresponding to the robot.

[0023] In some embodiments, the method further comprises:

[0024] When the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, the robot is controlled to turn until the nearest obstacle distances in the re-divided multiple sub-areas are all greater than the target obstacle distance.

[0025] In some embodiments, the front area of ​​the robot is divided into a first sub-area located in the left front, a second sub-area located in the front, and a third sub-area located in the right front;

[0026] In the case where the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, controlling the robot to turn until the nearest obstacle distances in the re-divided multiple sub-areas are all greater than the target obstacle distance, comprising:

[0027] When the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance:

[0028] If the distance to the nearest obstacle in the first sub-area is greater than the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn left on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all greater than the target obstacle distance; or,

[0029] If the distance to the nearest obstacle in the first sub-area is smaller than the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn right on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all larger than the target obstacle distance; or,

[0030] If the nearest obstacle distance in the first sub-area is equal to the nearest obstacle distance in the second sub-area, the robot is controlled to turn left or right on the spot until the nearest obstacle distances in the three re-divided sub-areas are all greater than the target obstacle distance.

[0031] In some embodiments, the process of forming the local point cloud map includes:

[0032] During the movement of the robot, a first key frame is acquired using a laser sensor corresponding to the robot, and the first key frame is used to initialize the local point cloud map and the local grid map;

[0033] When the movement distance of the robot is greater than the target distance and / or the movement angle of the robot is greater than the target angle, a k-th key frame is acquired by using a laser sensor corresponding to the robot, and the local point cloud map is updated by using the k-th key frame; k is an integer greater than 1;

[0034] Using the initial pose of the k-th key frame, matching the k-th key frame with the local point cloud map;

[0035] Inserting the laser point cloud data between the k-1th key frame and the kth key frame into the local grid map;

[0036] The updated local grid map is used to filter the local point cloud map so that the local point cloud map only retains laser points that meet the target occupancy condition.

[0037] In some embodiments, matching the local point cloud map where the robot is currently located with a global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot includes:

[0038] Initializing particles based on the first reference pose to obtain a plurality of particles, and initializing the cumulative matching score and the cumulative prior score of each particle; each particle corresponds to a different global pose;

[0039] During the movement of the robot, the position and posture of each particle is updated according to the corresponding odometer data of the robot;

[0040] For at least one particle, based on the position and posture of the particle, the local point cloud map and the global map are matched to obtain a current matching score of the local point cloud map and the global map, which is used as the current matching score of the particle to update the cumulative matching score and the total score of the particle; the total score of the particle is the sum of the cumulative matching score and the cumulative prior score of the particle;

[0041] Using the global pose of the particle with the highest total score, matching the local point cloud map with the global map to obtain a current matching score of the local point cloud map and the global map as the current matching score of the particle with the highest total score;

[0042] When the current matching score of the particle with the highest total score is greater than the target matching score, the global posture corresponding to the particle with the highest total score is determined as the global posture of the robot.

[0043] In some embodiments, during the movement of the robot, updating the position and posture of each particle according to the odometer data corresponding to the robot includes:

[0044] During the movement of the robot, the position and posture of each particle is updated according to the corresponding odometer data and noise data of the robot.

[0045] In some embodiments, matching the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot further includes:

[0046] Upon receiving the jth reference pose of the robot:

[0047] If there is no particle within the target range where the j-th reference pose is located, particle initialization is performed based on the j-th reference pose to obtain multiple particles, and the cumulative matching score and the cumulative prior score of each particle obtained by performing particle initialization based on the j-th reference pose are initialized; or,

[0048] If there is a particle within the target range where the j-th reference posture is located, the current prior score of the particle within the target range is calculated to update the cumulative prior score and the total score of the corresponding particle;

[0049] Here, j is an integer greater than 1.

[0050] In some embodiments, the jth reference pose corresponds to a jth reference image, and the process of determining the jth reference image includes:

[0051] When the jth target image is received, the large language model is used to determine a historical image that is most similar to the jth target image from multiple historical images as the jth reference image; the jth target image is obtained using the corresponding image sensor of the robot.

[0052] In some embodiments, the first target image is in the form of a panoramic image, and the historical image is in the form of a panoramic image.

[0053] In a second aspect, the present application provides a robot comprising a control module and an image sensor, wherein the control module is used to execute any one of the above methods to determine the global position of the robot; and the image sensor is used to acquire a first target image.

[0054] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements any of the above methods when executed by a processor.

[0055] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the above methods when executing the computer program.

[0056] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program implements any one of the above methods when executed by a processor.

[0057] The present application provides a global positioning method, robot, medium, device and program product of a robot, which realizes accurate robot positioning by combining a large language model (LLM) and a point cloud map matching technology. Specifically, when the robot receives a control instruction, the control instruction is first parsed by a large language model to identify whether a repositioning operation is required. If a repositioning operation is required, a historical image that is most similar to the first target image is selected from multiple historical images as the first reference image. Next, based on the first reference pose corresponding to the first reference image, the current local point cloud map of the robot is matched with the global map, and the global pose of the robot is determined by the matching result. Through the above method, the robot can quickly and accurately perform global positioning after receiving the control instruction, which significantly improves the accuracy and efficiency of positioning. By using the large language model's ability to parse the control instruction, the repositioning requirement can be accurately identified, and the first reference image that best matches the current environment can be quickly locked by comparing the historical images. By matching the local point cloud map with the global map, the stable positioning effect of the robot in complex scenes is ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present application is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0059] Figure 1 It is a characteristic point schematic diagram provided in an embodiment of the present application.

[0060] Figure 2 It is a grid map provided in an embodiment of the present application.

[0061] Figure 3 It is a flow chart of a global positioning method of a robot provided in an embodiment of the present application.

[0062] Figure 4 It is a schematic diagram of a car 360 imaging technology provided in an embodiment of the present application.

[0063] Figure 5 This is a schematic diagram of robot camera distribution provided in an embodiment of the present application.

[0064] Figure 6 It is a flowchart of calling a large language model provided in an embodiment of the present application.

[0065] Figure 7 This is a schematic diagram of dividing into three sub-areas provided in an embodiment of the present application.

[0066] Figure 8 It is a schematic diagram of a process for forming a local point cloud map and a local grid map provided in an embodiment of the present application.

[0067] Fig. 9This is a schematic diagram of particle initialization provided in an embodiment of the present application.

[0068] Fig.10 It is a schematic diagram of a grid-based particle scoring strategy provided in an embodiment of the present application.

[0069] Fig.11 It is a structural block diagram of a robot provided in an embodiment of the present application.

[0070] Fig.12 It is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in the implementation mode of this application. Obviously, the described implementation mode is only a part of the implementation mode of this application, not all the implementation modes. Based on the implementation mode in this application, all other implementation modes obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0072] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0073] Currently, the robot performs global positioning by comparing the detected images and laser frames with the pre-established feature maps or grid maps. Here, taking a monocular camera and a single-line laser as an example, the relevant positioning process is as follows. The robot establishes a visual feature map and a laser grid map in the work area in advance. When positioning, the robot obtains the current posture of the robot through the collected image or current laser frame, the bag-of-words feature point comparison (refer to ORB-SLAM2) or the branch-and-bound method to obtain the laser optimal matching score (refer to cartographer). Among them, ORB-SLAM2 is a real-time visual SLAM tool library. Cartographer is a real-time indoor mapping algorithm that can generate a grid map.

[0074] See also Figure 1 and Figure 2 , Figure 1 is a characteristic point schematic diagram provided in an embodiment of the present application, Figure 2 It is a grid map provided in an embodiment of the present application.

[0075] like Figure 1As shown in , feature points can only represent the key features in the image, and at the same time, many other features will be lost, and all image pixel information cannot be used. Therefore, the reliability is often poor. Figure 2 As shown, the scheme based on the grid map also has obvious disadvantages. This is mainly because there are a large number of similar scenes in the map. By matching the current laser frame with the grid map, it is easy to generate multiple candidate results with a high degree of matching, and the reliability is often difficult to meet engineering requirements.

[0076] The applicant proposes a robot precise global positioning solution based on a large language model (LLM). The specific implementation is mainly through the LLM coarse positioning module and the laser high-precision positioning module, which is used for the robot to achieve global positioning capability at any position based on the camera and laser radar carried by the robot alone in the absence of a global positioning sensor (for example, GPS or UWB).

[0077] As an example, the working process of the LLM coarse positioning module is shown below.

[0078] 1. Obtain 360-degree panoramic images: The robot carries multiple cameras on its own to record a 360-degree panoramic image database at the target workplace (similar to the 360-degree panoramic view function of a car).

[0079] 2. Data collection: The robot collects a panoramic image and corresponding pose coordinates (x, y, theta) every 2 meters in the working area to form an image pose sequence data set.

[0080] 3. Voice interaction: When the robot needs to be relocated, you can talk to the robot directly and tell it to relocate. The robot calls the LLM model, determines that the dialogue command is a relocation command, and then calls the relocation program to perform the relocation operation.

[0081] 4. Rough repositioning: After the robot repositioning program is triggered, the roaming command is triggered and the robot enters the roaming mode (walking aimlessly). At the same time, the currently acquired panoramic image is compared with the historical image (also a panoramic image) in the database, and the most similar historical image is found, and the pose coordinates of the most similar historical image are output. This comparison process can be achieved through LLM.

[0082] The laser high-precision positioning module performs secondary positioning based on the coarse positioning results of the LLM coarse positioning module to improve positioning accuracy and reliability. As an example, its working process is shown below.

[0083] 1. Point cloud SLAM: After the robot enters the roaming mode, the robot also turns on the SLAM (Simultaneous Localization and Mapping) mapping function to build a point cloud map fused with multiple laser frames.

[0084] 2. Fusion positioning: Based on the coarse positioning prior information and the posterior fusion of point cloud map matching, the current high-precision posture of the robot is determined to complete the positioning function.

[0085] The implementation methods of the present application will be described in detail below.

[0086] See also Figure 3 , Figure 3 It is a flow chart of a global positioning method of a robot provided in an embodiment of the present application.

[0087] In order to improve the relevant technology, an embodiment of the present application provides a global positioning method of a robot, which is applied to a robot working in a target area. The method includes steps S101 to S103.

[0088] Step S101: after receiving a control instruction, the control instruction is identified using a large language model.

[0089] Step S102: When it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, the large language model is used to determine a historical image that is most similar to a first target image from multiple historical images as a first reference image; the first target image is obtained using the corresponding image sensor of the robot.

[0090] Step S103: Based on the first reference pose corresponding to the first reference image, the local point cloud map where the robot is currently located is matched with the global map to determine the global pose of the robot.

[0091] For example, a historical image is an image collected and stored by the robot during the previous operation. The historical image is associated with the corresponding pose information (which can be called historical pose, including position and direction), which is used to assist the robot in finding the scene most similar to the current environment during the relocation process. The first reference image is an image selected from multiple historical images and is most similar to the current environment in the first target image. The pose information corresponding to the first reference image is called the first reference pose (including position and direction), which is used to help the robot determine the current global pose. The local point cloud map is the three-dimensional point cloud data collected by the laser sensor in the current environment of the robot, which represents the environmental structure near the robot at the current moment. The global map is an overall three-dimensional environment model constructed by the robot in the target area through multiple three-dimensional point cloud data, which is used for global positioning and path planning. The global pose refers to the exact position and direction of the robot in the global map, which can be regarded as an accurate pose. By matching the local point cloud map with the global map, the specific position of the robot in the entire environment can be determined.

[0092] The related robot global positioning method usually relies on the global positioning sensor. The above embodiment is used for the robot to achieve the global positioning capability of any position through the large language model based only on the image sensor (e.g., camera) and laser sensor (e.g., lidar) carried by the robot in the absence of a global positioning sensor (e.g., GPS or UWB). The above embodiment realizes an efficient and reliable robot global positioning method by combining the large language model and the point cloud map matching technology. Specifically, when the robot receives the control instruction, the large language model is first used to parse the control instruction and determine the repositioning requirement. Then, the image most similar to the current image (i.e., the first target image) is selected from the previously stored historical images as the first reference image, and the local point cloud map is matched with the global map based on the posture information corresponding to the first reference image (i.e., the first reference posture). Through this matching process, the global posture of the robot can be accurately determined, so that the robot can maintain accurate positioning even in a complex environment.

[0093] Through the method provided by the above embodiment, the robot can achieve accurate global positioning by relying on its own image sensor and laser sensor without the need for a global positioning sensor (such as GPS or UWB). By utilizing the natural language processing capabilities of the large language model, repositioning instructions can be efficiently identified and parsed, and the most matching first reference image can be quickly found from historical images, significantly improving the accuracy and reliability of positioning. In addition, the matching process of the local point cloud map and the global map ensures the accuracy of positioning, and a stable repositioning effect can be achieved even in similar scenes or dynamic environments, thereby improving the adaptability of the robot positioning algorithm in complex and changing environments.

[0094] See also Figure 4 and Figure 5 , Figure 4 is a schematic diagram of a 360-degree imaging technology for a car provided in an embodiment of the present application. Figure 5 This is a schematic diagram of robot camera distribution provided in an embodiment of the present application.

[0095] In some embodiments, the first target image may be in the form of a panoramic image, and the historical image may be in the form of a panoramic image.

[0096] The panoramic image is a 360-degree panoramic image. The method of collecting panoramic images is similar to the 360-degree imaging method of automobiles. It is a mature technology and will not be described in detail in this article. It should be noted that the robot camera can be set to face upward instead of downward. This is because when the robot works indoors, the ceiling and surrounding environment information are better reference than the ground information. Figure 4 As shown, the robot cameras are distributed on the top, front, back, left, and right of the robot. For example, they are all monocular cameras with fields of view ABCDE respectively. The top corresponds to field of view E, and the front, back, left, and right correspond to field of view ABCD. Among them, A corresponds to C, D corresponds to B, and E is located at the top. Figure 5 D is not shown in the figure. Through the image stitching algorithm, the stitching of multi-camera images can be realized to form a 360-degree panoramic image.

[0097] See also Figure 6 , Figure 6 It is a flowchart of calling a large language model provided in an embodiment of the present application.

[0098] In some embodiments, the control instruction may be in the form of a voice message.

[0099] In some embodiments, after receiving the control instruction, using the large language model to identify the control instruction may include: after receiving the control instruction, converting the control instruction into text information; inputting the text information into the large language model so that the large language model can identify whether the control instruction is used to instruct the robot to perform a repositioning operation.

[0100] In some embodiments, when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, using the large language model to determine a historical image that is most similar to the first target image from multiple historical images as a first reference image may include: when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, using the large language model to compare the first target image with multiple historical images to obtain the corresponding similarity of each historical image; and determining the historical image with the highest similarity as the first reference image.

[0101] In the above embodiment, the robot receives the user's voice through a microphone, converts it into text information, and recognizes the relocation instruction through a large language model to trigger the relocation function. As an example, the large language model can use chatgpt-4, which can receive both text input and image input. The specific execution process is as follows Figure 6 shown.

[0102] 1. Convert the voice information received by the robot into text information through the voice-to-text module.

[0103] 2. Input the text information into the large model calling module as the promt (i.e., prompt word) for calling the large model.

[0104] 3. The large model calling module is implemented through the callback function of the LangChain framework. LangChain is a programming framework that helps use large language models (LLM) in applications.

[0105] For example, the relevant promt can be set as: "Assuming that you are a robot that can realize the self-positioning function, if you find that the user information contains positioning instructions, please perform the repositioning operation."

[0106] 4.setPoseCallBack compares the input first target image with the panoramic image database through LLM to obtain the most similar historical image and the corresponding similarity, where the value range of the similarity is, for example, [0,1]. This process is also implemented through the LangChain framework. The panoramic image database includes multiple historical panoramic images.

[0107] For example, the relevant promt can be set as: "Now you are given an image to be queried and an image dataset. Please loop through the images in the image dataset, find the image that is most similar to the image to be queried, and output the image number and similarity. The similarity range is 0 to 1, 1 means almost the same, and 0 means very different, and output in json format."

[0108] In the above embodiment, when the robot receives a control instruction in the form of voice, the voice information is first converted into text information, and then the text information is input into the large language model, and the large language model parses the intention of the control instruction. If the control instruction involves a relocation operation, the large language model can be used to find the historical image that best matches the current environment from multiple historical images as the first reference image. Through the above method, the robot can efficiently and accurately recognize voice control instructions and quickly perform relocation operations. This method not only improves the accuracy of voice instruction recognition, but also reduces the complexity of relocation operations and the consumption of computing resources, so that the robot can maintain stable positioning performance in dynamic and complex environments. In addition, by combining the large language model and the LangChain framework, efficient processing of voice information and image data is achieved, so that the robot can reliably perform relocation under various environmental conditions, improving the intelligence level and application scope of the overall system.

[0109] See also Figure 7 , Figure 7 This is a schematic diagram of dividing into three sub-areas provided in an embodiment of the present application.

[0110] In some embodiments, the method may further include: when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, dividing the area in front of the robot into multiple sub-areas; measuring the distances to the nearest obstacles in the multiple sub-areas by using a ranging sensor; when the distances to the nearest obstacles in all sub-areas are greater than the target obstacle distance, controlling the robot to move forward; and during the movement of the robot, acquiring the first target image using the corresponding image sensor of the robot.

[0111] In some embodiments, the method may further include: when the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, controlling the robot to turn until the nearest obstacle distances in multiple re-divided sub-areas are all greater than the target obstacle distance.

[0112] In some embodiments, the front area of the robot can be divided into a first sub-area located at the front left, a second sub-area located directly in front, and a third sub-area located at the front right. When the distance to the nearest obstacle in at least one sub-area is not greater than the target obstacle distance, controlling the robot to turn until the distance to the nearest obstacle in each of the re-divided sub-areas is greater than the target obstacle distance includes: when the distance to the nearest obstacle in at least one sub-area is not greater than the target obstacle distance: if the distance to the nearest obstacle in the first sub-area is greater than the distance to the nearest obstacle in the second sub-area, controlling the robot to turn left in place until the distance to the nearest obstacle in each of the 3 re-divided sub-areas is greater than the target obstacle distance; or, if the distance to the nearest obstacle in the first sub-area is less than the distance to the nearest obstacle in the second sub-area, controlling the robot to turn right in place until the distance to the nearest obstacle in each of the 3 re-divided sub-areas is greater than the target obstacle distance; or, if the distance to the nearest obstacle in the first sub-area is equal to the distance to the nearest obstacle in the second sub-area, controlling the robot to turn left or right in place until the distance to the nearest obstacle in each of the 3 re-divided sub-areas is greater than the target obstacle distance.

[0113] The above embodiments propose a simple and reliable roaming scheme. The purpose of the robot's roaming is to make the robot move so as to see more environmental features, thereby improving the stability of positioning. The roaming principle is as Figure 6 shown. The front area of the robot is divided into multiple sub-areas. Assuming the number of sub-areas is 3, the front area can be divided into three sub-areas: left, middle, and right. The distances to the nearest obstacles in the three sub-areas are measured respectively by a distance measuring sensor (such as a lidar, a depth camera, an ultrasonic sensor, etc.), and are denoted as L, M, and R respectively. Among them, the distance to the nearest obstacle in a sub-area refers to the minimum value of the distances between all obstacles in the sub-area and the robot.

[0114] Assuming the target obstacle distance is 2m, when L, M, and R are all greater than 2m, the robot moves forward. When any one or more of L, M, and R are less than 2m, compare the magnitudes of L and R: when L > R, the robot turns left in place until L, M, and R are all greater than 2m, and then continues to move forward; similarly, when L < R, the robot turns right in place until L, M, and R are all greater than 2m, and then continues to move forward.

[0115] Through the above simple measurements, the robot can achieve active exploration and operation in unknown areas and has a smooth obstacle avoidance function, which can meet the requirements of repositioning movement.

[0116] In the above embodiment, the sub-area refers to a small area divided within the field of vision in front of the robot. The area in front of the robot can be divided into multiple sub-areas (e.g., left, middle, and right) to independently measure the distance of obstacles in each sub-area to achieve accurate obstacle avoidance and path planning. Distance measuring sensors include, for example, laser radars, depth cameras, ultrasonic sensors, etc., which are used to measure the distance between the robot and the obstacle. The target obstacle distance is, for example, a pre-set distance threshold, such as 1m, 2m, 3m, etc. Roaming refers to the robot moving randomly or according to certain rules in the environment to detect more environmental features and improve its own positioning accuracy. In the above embodiment, the roaming mode is used to realize the partition measurement and obstacle avoidance of the area in front of the robot, and realize the active exploration of unknown areas. L, M, and R represent the nearest obstacle distances in the three sub-areas on the left, middle, and right in front of the robot, respectively, which are used to judge the safety of the robot's advancement and decide whether the robot turns or continues to move forward.

[0117] When the relevant robot avoids obstacles, the detection and response speed of obstacles is not fast enough, which easily causes the robot to collide or stagnate in a complex environment. The relevant robot positioning algorithm and obstacle avoidance algorithm often run independently, resulting in the robot being unable to fully utilize obstacle avoidance information when performing relocation operations, thereby affecting the accuracy and efficiency of positioning. The relevant roaming scheme involves complex path planning and algorithm calculations, and it is difficult to maintain stable and reliable operation in a dynamic or unknown environment. The above embodiment provides a simple and reliable roaming and obstacle avoidance scheme, which combines the collaborative work of ranging sensors and image sensors to achieve efficient movement and positioning of the robot during the relocation process. Specifically, when the robot receives a relocation instruction, it first divides the area in front of the robot into multiple sub-areas, and measures the distance to the nearest obstacle in each sub-area through a ranging sensor. If the distance to the nearest obstacle in all sub-areas is greater than the target obstacle distance, the robot will continue to move forward; otherwise, the robot will perform a steering operation according to the distance to the nearest obstacle in each sub-area until the distance to the nearest obstacle in all sub-areas is greater than the target obstacle distance. During the robot's movement, the robot continuously uses image sensors to obtain environmental data and combines historical images for positioning, thereby achieving active exploration of unknown areas and stable global positioning. In addition, this solution simplifies the roaming strategy, enabling the robot to achieve smooth exploration and positioning in dynamic or unknown environments.

[0118] See also Figure 8 , Figure 8 1 is a schematic diagram of a process for forming a local point cloud map and a local grid map provided by an embodiment of the present application. Figure 8 In the figure, the left figure is a schematic diagram of the first key frame, the middle figure is a schematic diagram of initializing the local point cloud map and the local grid map, and the right figure is the final local point cloud map and the local grid map.

[0119] In some embodiments, the process of forming the local point cloud map may include:

[0120] During the movement of the robot, a first key frame is acquired using a laser sensor corresponding to the robot, and the first key frame is used to initialize the local point cloud map and the local grid map;

[0121] When the movement distance of the robot is greater than the target distance and / or the movement angle of the robot is greater than the target angle, a k-th key frame is acquired by using a laser sensor corresponding to the robot, and the local point cloud map is updated by using the k-th key frame; k is an integer greater than 1;

[0122] Using the initial pose of the k-th key frame, matching the k-th key frame with the local point cloud map;

[0123] Inserting the laser point cloud data between the k-1th key frame and the kth key frame into the local grid map;

[0124] The updated local grid map is used to filter the local point cloud map so that the local point cloud map only retains laser points that meet the target occupancy condition.

[0125] The target distance is, for example, 9 cm, 10 cm, 11 cm, etc., and the target angle is, for example, 4 degrees, 5 degrees, 6 degrees, etc. Each laser point in the point cloud corresponds to a grid, and each grid can be divided into an obstacle point, a passable area, and an unexplored area according to the number of occupancy and / or penetration times. The process of determining whether a laser point meets the target occupancy condition can be, for example, when the grid corresponding to the laser point is an obstacle point, it is determined that the laser point meets the target occupancy condition; when the grid corresponding to the laser point is a passable area or an unexplored area, it is determined that the laser point does not meet the target occupancy condition.

[0126] Taking the laser sensor as a single-line laser sensor as an example, the mapping process of the local point cloud map based on SLAM is introduced as follows.

[0127] S1. Using the first laser frame as the first key frame (or initial key frame) to initialize the local point cloud map and the local grid map.

[0128] S2. When the robot moves a certain distance (e.g., 10 cm) or angle (e.g., 5 degrees), insert the kth keyframe into the local point cloud map and calculate the initial pose of the kth keyframe based on the odometer data. Perform ICP matching between the kth keyframe and the local point cloud map with the initial pose of the kth keyframe.

[0129] S3. Insert the laser point cloud data between the kth key frame and the k-1th key frame into the local grid map. The local grid map generation method may refer to the slam-gmapping program, for example.

[0130] S4. Filter the local point cloud map through the local grid map, for example, only retain the laser points corresponding to the obstacle points.

[0131] S5. Repeat steps S2-S4 to form a local point cloud map.

[0132] In the above embodiment, the local grid map is, for example, a two-dimensional grid map generated based on laser point cloud data, each grid representing a small area in the environment, marking whether the area is occupied (such as an obstacle) or passable. A key frame refers to a representative laser frame selected by the robot during movement. The key frame can be used to update the local point cloud map and accurately determine the local pose of the robot in the local point cloud map through registration (such as ICP matching). ICP (Iterative Closest Point) matching is an algorithm for point cloud data alignment. Through continuous iteration, the best matching position between two point clouds is found to minimize the error. In the above embodiment, ICP matching is used to align a new key frame with an existing local point cloud map, thereby updating the local point cloud map and correcting the local pose of the robot in the local point cloud map. The target occupation condition is, for example, a standard for determining whether a laser point should be retained in the local point cloud map. In the above embodiment, if the grid corresponding to the laser point is an obstacle point, the laser point meets the target occupation condition and is retained; otherwise, the laser point will be filtered out. SLAM (Simultaneous Localization and Mapping) is a technology used to allow a robot to build a map in real time in an unknown environment and determine its own position at the same time. In the above embodiment, SLAM technology is used to build a local point cloud map and a grid map through point cloud data obtained by a laser sensor, and update the robot's position in real time.

[0133] The above embodiment combines the construction and update of the local point cloud map and the local grid map to provide an efficient and accurate robot positioning and navigation method. Specifically, during the robot movement, the local point cloud map and the local grid map are first initialized using the first key frame collected by the laser sensor. When the distance moved by the robot exceeds the preset target distance or the angle exceeds the target angle, the robot collects a new key frame and aligns it with the existing local point cloud map through a matching method such as an ICP matching algorithm to correct the robot posture. Then, the laser point cloud data between adjacent key frames is inserted into the local grid map, and the local point cloud map is filtered through the local grid map to retain only those laser points that meet the target occupancy conditions. As the robot continues to move, the process is repeated, thereby continuously updating and optimizing the local point cloud map. Through the above method, the robot can generate a high-precision local point cloud map in a complex dynamic environment, significantly improving the accuracy of map construction and the ability to express details. By introducing the filtering mechanism of the target occupancy condition and the grid map, the influence of noise and redundant data is effectively reduced, ensuring the practicality of the local point cloud map.

[0134] See also Fig. 9 and Fig.10 , Fig. 9 is a schematic diagram of performing particle initialization provided in an embodiment of the present application, Fig.10 It is a schematic diagram of a grid-based particle scoring strategy provided in an embodiment of the present application.

[0135] In some embodiments, the matching of the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot may include: initializing particles based on the first reference pose to obtain multiple particles, and initializing the cumulative matching score and the cumulative prior score of each particle; each particle corresponds to a different global pose; during the movement of the robot, updating the pose of each particle according to the corresponding odometer data of the robot; for at least one particle, matching the local point cloud map with the global map based on the pose of the particle to obtain The current matching score of the local point cloud map and the global map is used as the current matching score of the particle, and is used to update the cumulative matching score and the total score of the particle; the total score of the particle is the sum of the cumulative matching score of the particle and the cumulative prior score; the global pose of the particle with the highest total score is used to match the local point cloud map and the global map to obtain the current matching score of the local point cloud map and the global map as the current matching score of the particle with the highest total score; when the current matching score of the particle with the highest total score is greater than the target matching score, the corresponding global pose of the particle with the highest total score is determined as the global pose of the robot.

[0136] In order to reduce the amount of calculation and take into account the positioning accuracy and operation efficiency of the robot, in some embodiments, for at least one particle, based on the position and posture of the particle, the local point cloud map and the global map are matched to obtain the current matching score of the local point cloud map and the global map, which is used as the current matching score of the particle to update the cumulative matching score and the total score of the particle. It can include: for multiple particles corresponding to the target grid, based on the position and posture of any particle corresponding to the target grid, the local point cloud map and the global map are matched to obtain the current matching score of the local point cloud map and the global map, which is used as the current matching score of the particle to update the cumulative matching score and the total score of the particle.

[0137] In some embodiments, updating the position and posture of each particle according to the corresponding odometer data of the robot during the movement of the robot may include: updating the position and posture of each particle according to the corresponding odometer data and noise data of the robot during the movement of the robot.

[0138] In some embodiments, the matching of the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot may also include: upon receiving the jth reference pose of the robot: if there are no particles within the target range where the jth reference pose is located, initializing the particles based on the jth reference pose to obtain multiple particles, and initializing the cumulative matching score and the cumulative prior score of each particle obtained by initializing the particles based on the jth reference pose; or, if there are particles within the target range where the jth reference pose is located, calculating the current prior score of the particles within the target range to update the cumulative prior score and the total score of the corresponding particles; wherein j is an integer greater than 1.

[0139] In some embodiments, the jth reference pose corresponds to the jth reference image, and the process of determining the jth reference image may include: when receiving the jth target image, using the large language model to determine a historical image that is most similar to the jth target image from multiple historical images as the jth reference image; the jth target image is obtained using the corresponding image sensor of the robot.

[0140] Since the coarse positioning result has a low accuracy (for example, the error is about 2m) and there is a certain error probability, the coarse positioning result can be further verified through point cloud matching to improve the positioning accuracy. The above embodiment designs a coarse and fine positioning fusion strategy. The basic idea is as follows: the coarse positioning result is used as prior information, the matching score of the local point cloud map and the global map is used as a posteriori information, and the final result is obtained through particle filter fusion. The specific steps are as follows:

[0141] R1. Initialization. When the first rough positioning result (e.g., the first reference pose) is received, the pose is used as the starting point to initialize the particles. As an example, the particle position (x, y) and angle (theta) are both Gaussian distributed, with a variance of 0.02 and a particle number of 120. Initialize the cumulative matching score M of each particle i is 0, and the cumulative prior score P i is 0, and the prior score is calculated according to the following formula:

[0142] P i =P i +pj

[0143] p j =h j ·max(0.0,(2.0-d i ) / 2.0)

[0144] Among them, h j is the similarity between the jth target image and the jth reference image, P i is the cumulative prior score of the ith particle, p j is the jth matching score, d i Represents the distance of the particle from the jth reference pose. Assuming that this is the jth time, the matching score of this time is the jth matching score. It should be noted that the prefixes "first" and "jth" in the "first target image" and "jth target image" only serve to distinguish, and their essence is the target image. Similarly, the prefixes "first" and "jth" in the "first reference image" and "jth reference image" only serve to distinguish, and their essence is the reference image (that is, one of the historical images). Similarly, the prefixes "first" and "jth" in the "first reference pose" and "jth reference pose" only serve to distinguish, and their essence is the reference pose (that is, the historical pose of the corresponding historical image).

[0145] R2. Use the motion model to add motion noise to each particle during the robot's motion. For example, add noise every 10 cm or 5 degrees.

[0146] x=x last +T·Δx·(1+α)+σ

[0147] y=y last +T·Δy·(1+α)+σ

[0148] δ=δ last +T·Δδ·(1+β)+E

[0149] Among them, x, y, δ are the robot posture and angle, x last ,y last , δ last is the robot pose at the last calculation, Δx, Δy, Δδ are the coordinate change information under the robot odometer coordinates, T is the transformation matrix from the odometer coordinate system to the global coordinates, σ and E are Gaussian noise, and the variances are, for example, 0.01 and 0.02 respectively. α and β are error coefficients, and the value range of α is, for example, [-0.2: 0.2], and the interval is, for example, 0.04, for example, a total of 11 values. The value range of β is, for example, [-0.1: 0.1], and the interval is, for example, 0.02, for example, a total of 11 values. α and β, for example, form a total of 121 combinations, which are used for 121 particles initialized according to the first reference pose.

[0150] R3. Calculate the matching score. As an example, calculate the position coordinate range [X min , X max ],[Y min , Y max ], with the above-mentioned pose coordinate range as the boundary, set the grid with a resolution of 0.2. If there are particles in the grid, take the pose of any particle in the grid as the initial value, perform NICP matching on the local point cloud map and the global map, and the obtained matching score is used as the current matching score of all particles in the grid. As an example, the current matching score is the proportion of points whose distance to the nearest point is less than 0.05m after point cloud matching, and the value range is [0, 1]. Add up the scores of each matching to get the cumulative matching score of the particle.

[0151] M; = ∑m,

[0152] Among them, M i is the cumulative matching score of the i-th particle, m j is the j-th matching score of the ith particle.

[0153] Update the total score for each particle.

[0154] S i =M i +P i

[0155] R4, integrate the latest rough positioning information. When new rough positioning information (e.g., the jth reference pose) is obtained, determine whether there is a particle near the pose (e.g., within a diameter of 2m with the particle as the center). If there is no particle, add the particle according to step R1, that is, initialize the particle based on the pose, and initialize the cumulative matching score and cumulative prior score of the particle. If there are particles, calculate the current prior scores of these particles according to step R1.

[0156] R5. Get the particle S corresponding to the highest total score max , and the particle (i.e., S max ) is used as the initial value, and the local point cloud map and the global map are matched to obtain the current matching score. If the current matching score is greater than the target matching score (for example, 0.6), the relocalization is successful.

[0157] In the above embodiment, particle initialization is a step in the particle filter algorithm. Initially, a group of particles is generated based on known prior information (such as coarse positioning results), and these particles represent the possible global posture of the robot. In the above embodiment, particle initialization exemplarily uses Gaussian distribution to generate multiple particles, each particle corresponding to a different global posture. The cumulative matching score refers to the score accumulated through multiple matching calculations, which is used to evaluate the matching degree of the corresponding particle in the global map. The higher the cumulative matching score, the more consistent the posture represented by the particle is with the actual situation. The prior score is a preliminary evaluation of the particle based on prior information (such as the similarity between the corresponding reference image and the target image). The cumulative prior score is used together with the cumulative matching score to calculate the total score of the particle, thereby helping to determine the most likely global posture of the robot. The motion model describes the changes in the position and posture of the particles during the movement of the robot, for example, by updating the particles through odometer data and noise model. In the above embodiment, whenever the robot moves a certain distance or angle, motion noise can be added to simulate the uncertainty of the actual movement. NICP (Normalized Iterative Particle Detection) e Closest Point) matching is an improved point cloud matching algorithm that can align the local point cloud map with the global map and calculate the matching score of the two. In the above embodiment, NICP matching is used to accurately align the poses represented by different particles and evaluate the accuracy of each particle. The total score of the particle is the sum of the cumulative matching score and the prior score of the particle, which represents the total credibility of the particle in the current positioning process. The particle with the highest total score is selected as the most likely global pose of the robot at present.

[0158] When performing point cloud matching, the related robot positioning method usually needs to process a large number of particles and data, resulting in a huge amount of calculation, which affects the real-time positioning performance and efficiency of the robot. The above embodiment uses a coarse-fine positioning fusion strategy to realize an efficient robot global positioning method. After receiving the coarse positioning result, the coarse positioning result is first used as prior information to initialize the particles and generate multiple particles, each of which represents a possible global pose. During the robot movement, the pose of each particle is dynamically updated based on the odometer data and the noise model, and the particle's current matching score between the local point cloud map and the global map is calculated by the NICP matching algorithm. Through successive iterations, the matching score and prior score of each particle are accumulated, and finally the particle with the highest total score is selected as the global pose of the robot. This process effectively combines coarse positioning information and precise point cloud matching, and gradually approaches the real position of the robot. Through the above method, the robot can achieve high-precision global positioning in a complex environment, significantly improving the accuracy of positioning. The improved particle filter algorithm is adopted to reduce the number of particles involved in matching while ensuring the stability and reliability of the positioning result. In addition, the fusion strategy of cumulative matching score and cumulative prior score reduces the dependence of the positioning process on the initial pose. Even when there are errors in the coarse positioning results, the accurate global pose can be gradually corrected and obtained. This not only improves the operating efficiency of the algorithm, but also enhances the adaptability and positioning performance of the robot in dynamic environments.

[0159] It can be seen that the above embodiment not only realizes the voice interaction relocation function based on LLM, but also performs robot positioning through panoramic images, realizes forward movement and steering control during robot movement through a simple and reliable robot roaming solution, applies a matching method based on a local point cloud map, adopts a matching scoring mechanism combining global map particle filtering and ICP, and adopts an efficient particle grid search mechanism. The above embodiment specifically improves the global positioning problem of the robot and can achieve high-precision and high-efficiency robot positioning.

[0160] An embodiment of the present application also provides a global positioning device for a robot, which is applied to a robot working in a target area. The device includes an LLM coarse positioning module and a laser high-precision positioning module.

[0161] The LLM coarse positioning module is used to use a large language model to identify the control instruction after receiving the control instruction; when it is identified that the control instruction is used to instruct the robot to perform a repositioning operation, the large language model is used to determine a historical image that is most similar to a first target image from multiple historical images as a first reference image; the first target image is obtained using the corresponding image sensor of the robot.

[0162] The laser high-precision positioning module is used to match the local point cloud map where the robot is currently located with the global map based on the first reference posture corresponding to the first reference image, so as to determine the global posture of the robot.

[0163] In some embodiments, the control instruction may be in the form of voice information; the LLM coarse positioning module may use the following method to identify the control instruction using a large language model after receiving the control instruction: after receiving the control instruction, convert the control instruction into text information; input the text information into the large language model so that the large language model can identify whether the control instruction is used to instruct the robot to perform a repositioning operation.

[0164] In some embodiments, the LLM coarse positioning module can use the following method to determine a historical image that is most similar to the first target image from multiple historical images using the large language model when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation: when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, use the large language model to compare the first target image with multiple historical images to obtain the corresponding similarity of each historical image; determine the historical image with the highest similarity as the first reference image.

[0165] In some embodiments, the device may also include a roaming module, which is used to divide the area in front of the robot into multiple sub-areas when it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation; measure the distances to the nearest obstacles in multiple sub-areas by using a ranging sensor; control the robot to move forward when the distances to the nearest obstacles in all sub-areas are greater than the target obstacle distance; and obtain the first target image using the corresponding image sensor of the robot during the movement of the robot.

[0166] In some embodiments, the roaming module can also be used to control the robot to turn when the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, until the nearest obstacle distances in multiple re-divided sub-areas are all greater than the target obstacle distance.

[0167] In some embodiments, the front area of ​​the robot can be divided into a first sub-area located in the left front, a second sub-area located in the front, and a third sub-area located in the right front; the roaming module adopts the following method to control the robot to turn when the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, until the nearest obstacle distances in the re-divided multiple sub-areas are all greater than the target obstacle distance: when the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance: if the nearest obstacle distance in the first sub-area is greater than the nearest obstacle distance in the second sub-area, then The robot is controlled to turn left on the spot until the nearest obstacle distances in the three re-divided sub-areas are all greater than the target obstacle distance; or, if the nearest obstacle distance in the first sub-area is less than the nearest obstacle distance in the second sub-area, the robot is controlled to turn right on the spot until the nearest obstacle distances in the three re-divided sub-areas are all greater than the target obstacle distance; or, if the nearest obstacle distance in the first sub-area is equal to the nearest obstacle distance in the second sub-area, the robot is controlled to turn left or right on the spot until the nearest obstacle distances in the three re-divided sub-areas are all greater than the target obstacle distance.

[0168] In some embodiments, the device may also include a point cloud map forming module, which is used to obtain a first key frame using a corresponding laser sensor of the robot during the movement of the robot, and use the first key frame to initialize the local point cloud map and the local grid map; when the movement distance of the robot is greater than the target distance and / or the movement angle of the robot is greater than the target angle, use the corresponding laser sensor of the robot to obtain the kth key frame, and use the kth key frame to update the local point cloud map; k is an integer greater than 1; use the initial pose of the kth key frame to match the kth key frame with the local point cloud map; insert the laser point cloud data between the k-1th key frame and the kth key frame into the local grid map; use the updated local grid map to filter the local point cloud map so that the local point cloud map only retains laser points that meet the target occupancy conditions.

[0169] In some embodiments, the laser high-precision positioning module can match the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image in the following manner to determine the global pose of the robot: perform particle initialization based on the first reference pose to obtain multiple particles, and initialize the cumulative matching score and the cumulative prior score of each particle; each particle corresponds to a different global pose; during the movement of the robot, update the pose of each particle according to the corresponding odometer data of the robot; for at least one particle, match the local point cloud map with the global map based on the pose of the particle to obtain The current matching score of the local point cloud map and the global map is used as the current matching score of the particle, and the current matching score is used to update the cumulative matching score and the total score of the particle; the total score of the particle is the sum of the cumulative matching score and the cumulative prior score of the particle; the global pose of the particle with the highest total score is used to match the local point cloud map and the global map to obtain the current matching score of the local point cloud map and the global map as the current matching score of the particle with the highest total score; when the current matching score of the particle with the highest total score is greater than the target matching score, the corresponding global pose of the particle with the highest total score is determined as the global pose of the robot.

[0170] In some embodiments, the laser high-precision positioning module can update the position and posture of each particle according to the corresponding odometer data of the robot during the movement of the robot in the following manner: during the movement of the robot, update the position and posture of each particle according to the corresponding odometer data and noise data of the robot.

[0171] In some embodiments, the laser high-precision positioning module can also be used to, when the jth reference posture of the robot is received: if there are no particles within the target range where the jth reference posture is located, then the particles are initialized based on the jth reference posture to obtain multiple particles, and the cumulative matching score and cumulative prior score of each particle obtained by initializing the particles based on the jth reference posture are initialized; or, if there are particles within the target range where the jth reference posture is located, the current prior score of the particle within the target range is calculated, and the current prior score is used to update the cumulative prior score and total score of the corresponding particle; wherein j is an integer greater than 1.

[0172] In some embodiments, the jth reference pose corresponds to the jth reference image, and the laser high-precision positioning module can determine the jth reference image in the following manner: when the jth target image is received, the large language model is used to determine a historical image that is most similar to the jth target image from multiple historical images as the jth reference image; the jth target image is obtained using the corresponding image sensor of the robot.

[0173] In some embodiments, the first target image may be in the form of a panoramic image, and the historical image may be in the form of a panoramic image.

[0174] See also Fig.11 , Fig.11 It is a structural block diagram of a robot provided in an embodiment of the present application.

[0175] An embodiment of the present application also provides a robot, comprising a control module and an image sensor, wherein the control module is used to execute any one of the above methods to determine the global position and posture of the robot; and the image sensor is used to acquire a first target image.

[0176] In some embodiments, the robot may further include a laser sensor. In some embodiments, the laser sensor may include a 2D laser sensor and / or a 3D laser sensor.

[0177] In some embodiments, the robot may also include a microphone.

[0178] In some embodiments, the robot may also include an odometer and / or an IMU.

[0179] In some embodiments, the robot may further include one or more of an angle encoder, a torque sensor, and a PIR sensor.

[0180] In some embodiments, the robot may be a multi-jointed robot.

[0181] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0182] An embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0183] The computer program product may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the computer program product of the present application is not limited thereto, and the computer program product may be any combination of one or more computer readable media.

[0184] An embodiment of the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0185] See also Fig.12 , Fig.12 It is a structural block diagram of a computer device provided in an embodiment of the present application.

[0186] The embodiments of the present application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0187] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected via an internal connection path.

[0188] The memory 110 is used to store computer programs. In some implementations, the computer programs may include codes for implementing the methods of the embodiments of the present application.

[0189] The processor 120 is used to execute the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information and output data such as operation results. In some implementations, when the solution of the embodiment of the present application is implemented by software or firmware, the computer program for implementing the solution of the embodiment of the present application can be stored in the processor 120 and executed by the processor 120.

[0190] The memory 110 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes a random access memory (RAM), a cache memory and a read-only memory (ROM). Among them, the memory 110 stores a computer program, and the computer program can be executed by the processor 120 so that the processor 120 implements the steps of any of the above methods.

[0191] The processor 120 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor 120 may be any conventional processor, etc.

[0192] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 120 or an instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor 120 for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0193] In some implementations, in addition to the hardware units described above, the computer device may also include software modules, where the software modules may be, for example, an operating system, a basic input output system (BIOS), application software, etc.

[0194] The operating system is used to manage the hardware and / or software resources of computer devices and is the core and cornerstone of computer devices. The operating system needs to handle basic tasks such as managing and configuring memory, determining the priority of system resource supply and demand, controlling input and output devices, operating networks, and managing file systems. In order to facilitate user operations, most operating systems will provide an operating interface for users to interact with the system.

[0195] BIOS is used to run hardware initialization during the power-on boot phase and provide runtime services for the operating system and application programs. In some implementations, BIOS can also monitor and display processor temperature and execute functions such as adjusting temperature protection strategies.

[0196] Application software is also called application program, which can be understood as software written for a specific application purpose of the user. It is one of the main categories of computer software. For example, application software can be a program used to achieve power control, temperature management, etc.

[0197] It should be noted that although some embodiments of the present application take mobile robots as examples, the present application can be applied to other bionic robots, such as AGVs, drones, etc., and the present application is not limited to this.

[0198] It should be understood that the specific examples in this specification are only intended to help those skilled in the art to better understand the implementation methods of the present application, rather than to limit the scope of protection of the present application.

[0199] It is understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0200] It can be understood that the various implementation modes described in this specification can be implemented individually or in combination, and this application is not limited to this.

[0201] Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0202] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0203] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each embodiment described above can refer to the corresponding process in other embodiments and will not be repeated here.

[0204] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0205] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the technical solution of the present application.

[0206] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0207] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0208] The above are only specific implementations of this specification, but the protection scope of this application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in this specification, which should be included in the protection scope of this specification. Therefore, the protection scope of this application should be based on the protection scope of the claims.

Claims

1. A global positioning method for a robot, characterized in that: Applied to a robot working in a target area, the method comprises: After receiving the control instruction, the control instruction is recognized by using the large language model; In the case where it is recognized that the control instruction is used to instruct the robot to perform a relocation operation, the large language model is used to determine a historical image most similar to a first target image from a plurality of historical images as a first reference image; the first target image is obtained using an image sensor corresponding to the robot; Based on the first reference pose corresponding to the first reference image, the local point cloud map where the robot is currently located is matched with the global map to determine the global pose of the robot.

2. The global positioning method of a robot according to claim 1, characterized in that: The control instruction is in the form of voice information; After receiving the control instruction, using the large language model to identify the control instruction includes: After receiving the control instruction, converting the control instruction into text information; The text information is input into the large language model so that the large language model recognizes whether the control instruction is used to instruct the robot to perform a repositioning operation.

3. The global positioning method of a robot according to claim 1, characterized in that: The method of, in the case of recognizing that the control instruction is used to instruct the robot to perform a relocation operation, using the large language model to determine a historical image most similar to the first target image from a plurality of historical images as a first reference image, comprises: When it is recognized that the control instruction is used to instruct the robot to perform a relocation operation, the first target image is compared with a plurality of historical images using the large language model to obtain a corresponding similarity of each historical image; A historical image with the highest similarity is determined as the first reference image.

4. The global positioning method of a robot according to claim 1, characterized in that: The method further comprises: In a case where it is recognized that the control instruction is used to instruct the robot to perform a repositioning operation, dividing the front area of ​​the robot into a plurality of sub-areas; Measuring the distances of the nearest obstacles in the plurality of sub-areas by using a ranging sensor; When the nearest obstacle distances in all sub-areas are greater than the target obstacle distance, controlling the robot to move forward; During the movement of the robot, the first target image is acquired using the image sensor corresponding to the robot.

5. The global positioning method of a robot according to claim 4, characterized in that: The method further comprises: When the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, the robot is controlled to turn until the nearest obstacle distances in the re-divided multiple sub-areas are all greater than the target obstacle distance.

6. The global positioning method of a robot according to claim 5, characterized in that: The front area of ​​the robot is divided into a first sub-area located at the left front, a second sub-area located directly in front, and a third sub-area located at the right front; In the case where the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance, controlling the robot to turn until the nearest obstacle distances in the re-divided multiple sub-areas are all greater than the target obstacle distance, comprising: In the case where the nearest obstacle distance in at least one sub-area is not greater than the target obstacle distance: If the distance to the nearest obstacle in the first sub-area is greater than the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn left on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all greater than the target obstacle distance; or, If the distance to the nearest obstacle in the first sub-area is smaller than the distance to the nearest obstacle in the second sub-area, the robot is controlled to turn right on the spot until the distances to the nearest obstacles in the three re-divided sub-areas are all larger than the target obstacle distance; or, If the nearest obstacle distance in the first sub-area is equal to the nearest obstacle distance in the second sub-area, the robot is controlled to turn left or right on the spot until the nearest obstacle distances in the three re-divided sub-areas are all greater than the target obstacle distance.

7. The global positioning method of a robot according to claim 1, characterized in that: The formation process of the local point cloud map includes: During the movement of the robot, a first key frame is acquired using a laser sensor corresponding to the robot, and the first key frame is used to initialize the local point cloud map and the local grid map; When the movement distance of the robot is greater than the target distance and / or the movement angle of the robot is greater than the target angle, a k-th key frame is acquired by using a laser sensor corresponding to the robot, and the local point cloud map is updated by using the k-th key frame; k is an integer greater than 1; Using the initial pose of the k-th key frame, matching the k-th key frame with the local point cloud map; Inserting the laser point cloud data between the k-1th key frame and the kth key frame into the local grid map; The updated local grid map is used to filter the local point cloud map so that the local point cloud map only retains laser points that meet the target occupancy condition.

8. The global positioning method of a robot according to claim 1, characterized in that: The matching of the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot includes: Initializing particles based on the first reference pose to obtain a plurality of particles, and initializing the cumulative matching score and the cumulative prior score of each particle; each particle corresponds to a different global pose; During the movement of the robot, the position and posture of each particle is updated according to the corresponding odometer data of the robot; For at least one particle, based on the position and posture of the particle, the local point cloud map and the global map are matched to obtain a current matching score of the local point cloud map and the global map, which is used as the current matching score of the particle to update the cumulative matching score and the total score of the particle; the total score of the particle is the sum of the cumulative matching score and the cumulative prior score of the particle; Using the global pose of the particle with the highest total score, matching the local point cloud map with the global map to obtain a current matching score of the local point cloud map and the global map as the current matching score of the particle with the highest total score; When the current matching score of the particle with the highest total score is greater than the target matching score, the global posture corresponding to the particle with the highest total score is determined as the global posture of the robot.

9. The global positioning method of a robot according to claim 8, characterized in that: During the movement of the robot, updating the position and posture of each particle according to the corresponding odometer data of the robot includes: During the movement of the robot, the position and posture of each particle is updated according to the corresponding odometer data and noise data of the robot.

10. The global positioning method of a robot according to claim 8, characterized in that: The matching of the local point cloud map where the robot is currently located with the global map based on the first reference pose corresponding to the first reference image to determine the global pose of the robot also includes: Upon receiving the jth reference pose of the robot: If there is no particle within the target range where the j-th reference pose is located, particle initialization is performed based on the j-th reference pose to obtain multiple particles, and the cumulative matching score and the cumulative prior score of each particle obtained by performing particle initialization based on the j-th reference pose are initialized; or, If there is a particle within the target range where the j-th reference posture is located, the current prior score of the particle within the target range is calculated to update the cumulative prior score and the total score of the corresponding particle; Here, j is an integer greater than 1.

11. The global positioning method of a robot according to claim 10, characterized in that: The j-th reference pose corresponds to the j-th reference image, and the process of determining the j-th reference image includes: When the jth target image is received, the large language model is used to determine a historical image that is most similar to the jth target image from multiple historical images as the jth reference image; the jth target image is obtained using the corresponding image sensor of the robot.

12. A robot, characterized in that: The robot comprises a control module and an image sensor, wherein the control module is used to execute the method described in any one of claims 1 to 11 to determine the global position and posture of the robot; and the image sensor is used to acquire a first target image.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer device, characterized in that: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 11 when executing the computer program.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images