Mobile robot positioning apparatus, method, device and storage medium

Abstract

The application relates to the technical field of mobile robots, and discloses a mobile robot positioning device, a method, equipment and a storage medium, the device comprising an infrared sensor, an infrared light supplementing light source device, a calculation processing module, a power module and a communication module; the infrared sensor collects a top view image of a mobile robot; the infrared light supplementing light source device emits infrared light to supplement light for the infrared sensor; the calculation processing module positions the mobile robot based on the top view image to obtain positioning data of the mobile robot; the power module supplies power for the infrared light supplementing light source device and the calculation processing module; and the communication module sends the positioning data of the mobile robot to the mobile robot. The application combines various parts of the mobile robot positioning device, improves the quality of the top view image, determines positioning based on the high-quality top view image, can stably position under different light conditions, reduces the influence of light, and improves the accuracy and robustness of the positioning of the mobile robot.

Description

Technical Field

[0001] This invention relates to the field of mobile robot technology, and more specifically to mobile robot positioning devices, methods, equipment, and storage media. Background Technology

[0002] Mobile robots are robots capable of moving automatically and completing specific tasks without direct human intervention, and they are used in all aspects of production and daily life. Precise positioning of mobile robots is a crucial foundation for task completion. This includes ensuring the robot is in the correct location to perform tasks, avoiding task failures or errors due to positioning errors, making path planning more efficient, and improving work efficiency. Therefore, how to achieve precise positioning of mobile robots is a problem that needs to be solved.

[0003] Currently, existing mobile robot positioning devices may suffer from optical problems such as overexposure and glare in the images captured by the camera due to changes in lighting conditions. This can lead to the inability to clearly capture feature points. If positioning is determined based on such images, the accuracy and precision of the mobile robot positioning will be low. Summary of the Invention

[0004] In view of this, the present invention provides a mobile robot positioning device, method, equipment and storage medium to solve the problems of low accuracy and low precision in mobile robot positioning.

[0005] In a first aspect, the present invention provides a mobile robot positioning device, which includes an infrared sensor, an infrared supplementary light source device, a computing processing module, a power supply module, and a communication module;

[0006] Infrared sensors are used to capture top-view images of the mobile robot;

[0007] An infrared supplementary lighting source device is used to emit infrared light to supplement the light of an infrared sensor;

[0008] The calculation and processing module is used to locate the mobile robot based on the top view image obtained by the infrared sensor, and obtain the positioning data of the mobile robot.

[0009] The power module is used to power the infrared supplementary light source device and the computing processing module;

[0010] The communication module is used to send the mobile robot's positioning data to the mobile robot.

[0011] The mobile robot positioning device provided in this invention, through its infrared supplementary light source, computing processing module, power supply module, and communication module, can improve the quality of the acquired top-view image, thereby enabling positioning based on the high-quality top-view image and further improving the accuracy of mobile robot positioning. Simultaneously, by sending the positioning data to the mobile robot, the robot can perform subsequent operations based on the accurate positioning data. Compared to existing mobile robot positioning devices, this device can achieve stable positioning under different lighting conditions, reducing the influence of light during the positioning process, thus obtaining accurate mobile robot positioning data and improving the accuracy and robustness of mobile robot positioning.

[0012] In one optional implementation, the computational processing module includes an image processing unit, an arithmetic processing unit, and a neural network processing unit.

[0013] The image processing unit is used to optimize the image quality of the top-view image;

[0014] The neural network processing unit is used to detect the optimized top view image using a neural network to obtain all semantic features in the optimized top view image. Each semantic feature includes the category to which the semantic feature belongs and the two-dimensional pixel position corresponding to the semantic feature.

[0015] The computational processing unit is used to locate the mobile robot based on all semantic features of the optimized top view image, thereby obtaining the mobile robot's localization data.

[0016] The mobile robot positioning device provided in this embodiment of the invention optimizes the top view image through an image processing unit, a neural network processing unit, and a computational processing unit, and then performs semantic analysis. The semantic features obtained from the semantic analysis are used to obtain positioning data, thereby enhancing the positioning accuracy of the mobile robot in complex environments.

[0017] In a second aspect, the present invention provides a mobile robot positioning method, applied to a mobile robot positioning device according to the first aspect above or any corresponding embodiment thereof, the method comprising:

[0018] Obtain a first top-view image of the mobile robot at the current moment;

[0019] Based on the first top view image, the target positioning data of the mobile robot at the current moment is obtained;

[0020] Send the target location data of the mobile robot at the current moment to the mobile robot.

[0021] The mobile robot positioning method provided in this invention improves the accuracy and reliability of positioning by acquiring a first top view image of the mobile robot at the current moment, avoiding the collection of highly dynamic environmental information in the surrounding low-lying area, analyzing the first top view image for positioning, and directly sending the positioning data to the mobile robot to ensure that the mobile robot can obtain its own position information in real time.

[0022] In one alternative implementation, before obtaining the target positioning data of the mobile robot at the current moment based on the first top view image, the method further includes:

[0023] Load a visual map of the mobile robot's work area.

[0024] The mobile robot localization method provided in this embodiment of the invention supports subsequent mobile robot localization by loading a visual map of the mobile robot's working area.

[0025] In one optional implementation, positioning is performed based on the first top view image to obtain the target positioning data of the mobile robot at the current moment, including:

[0026] Acquire the mobile robot's location data from the previous moment, the mobile robot's odometry data at the current moment, and the target timestamp of the first top view image;

[0027] Based on the mobile robot's positioning data from the previous moment, the mobile robot's odometer data at the current moment, and the target timestamp of the first top view image, the first positioning data of the mobile robot corresponding to the target timestamp is determined.

[0028] Based on the first top view image and visual map, the first positioning data is optimized to obtain the second positioning data of the mobile robot at the target timestamp;

[0029] Based on the odometry data of the mobile robot at the current moment, the odometry interpolation algorithm is used to update the second positioning data to obtain the target positioning data of the mobile robot at the current moment.

[0030] The mobile robot localization method provided in this invention combines the localization data from the previous moment with the odometry data from the current moment to obtain a preliminary estimate of the first localization data for the current moment. This preliminary estimate can reduce the matching range of subsequent map semantic features and improve the speed of localization calculation. Then, it is optimized by using a first top view image and a visual map, and the first localization data is corrected by using detailed features of the environment, which improves the accuracy and robustness of localization. By considering the target timestamp of the first top view image, the time consistency between visual data and motion data is ensured, avoiding localization errors caused by time deviation. At the same time, the odometry interpolation algorithm can fine-tune the localization results according to the robot's real-time movement state, maintaining the real-time performance and accuracy of localization.

[0031] In one alternative implementation, the visual map includes all map semantic features present in the working area of ​​the mobile robot, and each map semantic feature includes the category to which the map semantic feature belongs and the spatial three-dimensional location corresponding to the map semantic feature.

[0032] Based on the first top-view image and visual map, the first localization data is optimized to obtain the second localization data of the mobile robot at the target timestamp, including:

[0033] Optimize the image quality of the first top view image to obtain the second top view image;

[0034] A neural network is used to detect the second top view image to obtain a first semantic feature set. The first semantic feature set includes all detected semantic features in the second top view image. Each detected semantic feature includes the category to which the detected semantic feature belongs and the two-dimensional pixel position corresponding to the detected semantic feature.

[0035] For any map semantic feature in the visual map, based on the first positioning data, the corresponding three-dimensional spatial position in the map semantic feature is reprojected to the two-dimensional image space to obtain the two-dimensional pixel position corresponding to the map semantic feature.

[0036] After reprojecting the visual map, a second semantic feature set is determined, which includes all map semantic features located within a preset range of the first positioning data.

[0037] The first semantic feature set and the second semantic feature set are matched to obtain the third semantic feature set that is successfully matched;

[0038] Based on the third semantic feature set, an optimization algorithm is used to optimize the first positioning data to obtain the second positioning data.

[0039] The mobile robot localization method provided in this invention optimizes a first top view image and uses neural network detection to accurately extract semantic features to obtain a first semantic feature set. Even under conditions of changing ambient lighting, occlusion, or interference from dynamic objects, the neural network can still stably extract semantic features, thereby improving the accuracy and robustness of the localization data obtained based on semantic features. The first semantic feature set is matched with a second semantic feature set selected from the visual map to obtain a third semantic feature set that has been successfully matched. The optimization algorithm specifically processes the successfully matched semantic feature set to optimize the localization data and improve its accuracy.

[0040] In one optional implementation, the first semantic feature set and the second semantic feature set are matched to obtain a third semantic feature set that matches successfully, including:

[0041] For any detected semantic feature in the first semantic feature set, traverse all map semantic features in the second semantic feature set;

[0042] If any map semantic feature exists, and the distance between the two-dimensional pixel position corresponding to the map semantic feature and the two-dimensional pixel position corresponding to the detected semantic feature is less than a preset threshold, and the category to which the map semantic feature belongs is the same as the category to which the detected semantic feature belongs, then the detected semantic feature and the map semantic feature are successfully matched.

[0043] The successfully matched detection semantic features and map semantic features are added to the third semantic feature set.

[0044] The mobile robot localization method provided in this invention effectively filters out irrelevant feature interference by setting a preset threshold for distance and matching based on the condition that the semantic features are of the same category. This ensures the high relevance and accuracy of the matching results, reduces the possibility of false matching, and serves as the basis for subsequent localization optimization, thereby improving the reliability and robustness of localization.

[0045] In one optional implementation, based on a third semantic feature set, an optimization algorithm is used to optimize the first location data to obtain second location data, including:

[0046] For any map semantic feature in the third semantic feature set, obtain the detection semantic feature that successfully matches the map semantic feature from the third semantic feature set;

[0047] Determine the pixel difference between the two-dimensional pixel position in the detected semantic features and the two-dimensional pixel position in the map semantic features;

[0048] The pixel difference is optimized with the goal of minimizing the pixel difference. Based on the first positioning data, the second positioning data is obtained.

[0049] The mobile robot localization method provided in this invention corrects the localization data by minimizing the pixel-level difference between the two-dimensional pixel position after reprojection using map semantic features and the two-dimensional pixel position of detected semantic features, thereby ensuring the accuracy and reliability of the localization data.

[0050] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the mobile robot positioning method of the second aspect above or any corresponding embodiment thereof.

[0051] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the mobile robot positioning method of the second aspect or any corresponding embodiment described above.

[0052] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the mobile robot positioning method of the second aspect above or any corresponding embodiment thereof. Attached Figure Description

[0053] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of a mobile robot positioning device according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram showing the placement of a mobile robot positioning device according to an embodiment of the present invention;

[0056] Figure 3 This is a flowchart of a mobile robot positioning method according to an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Precise positioning of mobile robots is crucial for completing various tasks, thus, how to achieve accurate positioning is a problem that needs to be solved. Existing mobile robot positioning devices cannot address the impact of changing lighting conditions on positioning, resulting in low accuracy and precision. The mobile robot positioning device provided in this invention, through the combination of its components, improves the quality of the acquired top-view image, thereby enhancing positioning accuracy and robustness based on this high-quality image. Furthermore, by sending the positioning data to the mobile robot, it enables subsequent operations based on accurate positioning data.

[0060] This embodiment provides a mobile robot positioning device, such as Figure 1 As shown, the device includes an infrared sensor 1, an infrared supplementary light source device 2, a computing processing module 3, a power supply module 4, and a communication module 5. The infrared sensor 1 is used to acquire a top view image of the mobile robot; the infrared supplementary light source device 2 is used to emit infrared light to supplement the infrared sensor 1; the computing processing module 3 is used to locate the mobile robot based on the top view image acquired by the infrared sensor 1 to obtain the positioning data of the mobile robot; the power supply module 4 is used to supply power to the infrared supplementary light source device 2 and the computing processing module 3; and the communication module 5 is used to send the positioning data of the mobile robot to the mobile robot.

[0061] Specifically, the infrared sensor 1 is equipped with an infrared bandpass filter, which can effectively filter visible light, thus reducing the impact of changes in lighting conditions on positioning compared to using a regular camera to capture images. Furthermore, the infrared sensor 1 has an automatic exposure function, automatically adjusting the exposure time based on the brightness of the captured image to ensure stable brightness and further reduce the influence of lighting conditions. The infrared supplementary lighting source device 2 is located on both sides of the infrared sensor 1, and the infrared light emitted by it can provide supplementary illumination for the infrared sensor 1. Optionally, this embodiment of the invention uses two sets of infrared supplementary lighting source devices 2 as an example; in practice, only one set of infrared supplementary lighting source devices 2 can be installed on one side of the infrared sensor 1. The infrared supplementary lighting source device 2 uses an instantaneous supplementary lighting method, emitting infrared light from a supplementary lighting lamp source. The wavelength of the supplementary lighting lamp source can be selected from infrared bands such as 850nm, 905nm, or 940nm. The supplementary lighting lamp source can be an LED (Light Emitting Diode), a laser source, or other light sources; this embodiment of the invention does not impose any limitations on this. The laser source can be a VCSEL (Vertical-cavity surface-emitting laser). The illumination time of the supplementary light source is synchronized with the shutter exposure time of the infrared sensor 1. Based on the infrared supplementary light source device 2, the mobile robot positioning device has the ability to locate in completely dark or dimly lit working environments. Combined with the automatic exposure function of the infrared sensor 1, the brightness of the top-view image acquired during day / night and when the light is on / off can be kept constant, which not only reduces the difficulty of subsequent image processing but also improves the reliability of positioning. The calculation and processing module 3 is the main functional module for determining the positioning of the mobile robot positioning device, used to obtain the precise positioning of the mobile robot. The power supply module 4 supplies power to both the infrared supplementary light source device 2 and the calculation and processing module 3 after voltage stabilization. Since the infrared supplementary light source device 2 provides instantaneous illumination, its instantaneous current and power are relatively large. Therefore, the power supply module 4 needs to provide sufficient power output and ensure power isolation to prevent fluctuations in the instantaneous illumination current and voltage from affecting the stable operation of the calculation and processing module 3. Communication module 5 uses a gigabit Ethernet communication interface to communicate with the mobile robot, obtain information such as the mobile robot's odometer data, and send the positioning data determined by the mobile robot's positioning device to the mobile robot. Optionally, it can send the data to the mobile robot's navigation module so that the mobile robot knows its current location for path planning and task scheduling.By combining multiple components in the mobile robot positioning device, the quality of the acquired top-view image can be improved. Based on the high-quality top-view image, the positioning can be determined, further improving the accuracy of the mobile robot's positioning. At the same time, by sending the positioning data to the mobile robot, the mobile robot can perform subsequent operations based on the accurate positioning data.

[0062] In some optional implementations, corresponding host computer software can be developed for communication module 5. After the host computer software successfully connects with communication module 5, communication module 5 can perform functions such as loading visual maps, updating the weights of the neural network, monitoring odometer data, obtaining positioning data calculated by the computing and processing module 3, and obtaining the operation logs of the mobile robot positioning device, so as to facilitate monitoring and debugging by engineers.

[0063] Figure 2 This is a schematic diagram showing the placement of a mobile robot positioning device according to an embodiment of the present invention, such as... Figure 2 As shown, Figure 2 In the figure, 'a' represents a roller conveying robot. The mobile robot positioning device is installed at the top position shown in Figure 6, which is the top of the front pole. Figure 2 In the figure, 'b' represents an automated forklift robot. The mobile robot positioning device is installed at the top position shown in reference numeral 7, that is, at the top of the forklift's front steering frame. During installation, the infrared sensor 1 is mounted vertically upwards, enabling the acquisition of reliable and fixed feature information from the robot's upper field of view during movement. This avoids interference from the surrounding environment and further improves the reliability and accuracy of positioning. It should be noted that... Figure 2 The two mobile robots shown are merely examples. The mobile robot positioning device can also be installed on the top of any mobile robot that needs to be positioned, such as a sweeping robot or a tractor robot. This embodiment of the invention does not limit this.

[0064] In some optional implementations, the computational processing module 3 includes an image processing unit, an arithmetic processing unit, and a neural network processing unit; the image processing unit is used to optimize the image quality of the top view image; the neural network processing unit is used to use a neural network to detect the optimized top view image and obtain all semantic features in the optimized top view image, wherein each semantic feature includes the category to which the semantic feature belongs and the two-dimensional pixel position corresponding to the semantic feature; the arithmetic processing unit is used to locate the mobile robot based on all the semantic features of the optimized top view image and obtain the positioning data of the mobile robot.

[0065] Specifically, the mobile robot positioning device uses an ARM (Advanced RISC Machine) embedded platform as its computing processing module. The ARM embedded platform includes a CPU (Central Processing Unit) and an NPU (Neural Processing Unit) in its hardware. The image processing unit is a dedicated image processing program running on the CPU, the computational processing unit is a positioning program running on the CPU, and the neural network processing unit is a deep learning program running on the NPU. The image signal processing unit is responsible for processing the raw top-view image acquired by the infrared sensor 1, such as linear correction, noise removal, bad pixel removal, interpolation, white balance, and automatic exposure control, to optimize the image quality of the top-view image. The neural network processing unit has deep learning capabilities and therefore requires a computing power of at least 1 TOPS. The input to the neural network processing unit is a two-dimensional image. In this embodiment of the invention, the optimized top-view image is input to the neural network processing unit. A convolutional layer performs a convolution operation on the optimized top-view image to find specific edges, lines, or texture patterns. Then, an activation function is applied to the convolution result to learn more complex features in the image. A pooling layer then reduces the spatial dimension of the image while preserving important features, resulting in multiple semantic feature categories included in the optimized top-view image. Pixels representing the same semantic feature category are aggregated. Furthermore, post-processing techniques, such as conditional random fields or more advanced instance segmentation algorithms, are used to separate semantic features belonging to the same semantic feature category but located at different positions in the top-view image, further improving the accuracy of the final output and ensuring accurate separation of different instances. The final output is all the semantic features in the optimized top-view image. For example, the neural network processing unit detects three semantic features in any optimized top-view image, and the categories of these three semantic features are: fluorescent light, smoke alarm, and air conditioner. For "fluorescent lamp," all pixels in the optimized top-view image whose semantic feature category is "fluorescent lamp" are aggregated. The center pixel of at least one of the aggregated pixels, or any corner point on the bounding box formed by at least one of the aggregated pixels, is taken as the two-dimensional pixel position corresponding to "fluorescent lamp." Optionally, other pixels can also be selected as the two-dimensional pixel positions corresponding to the semantic features; this embodiment of the invention does not impose any restrictions on this. By determining the two-dimensional pixel position for each semantic feature's category, the two-dimensional pixel positions corresponding to the semantic features "fluorescent lamp," "smoke alarm," and "air conditioner" in the top-view image can be obtained.The computational processing unit locates the image based on the semantic features detected by the neural network processing unit, thereby obtaining accurate location data.

[0066] The mobile robot positioning device provided in this invention, through its infrared supplementary light source, computing processing module, power supply module, and communication module, can improve the quality of the acquired top-view image, thereby enabling positioning based on the high-quality top-view image and further improving the accuracy of mobile robot positioning. Simultaneously, by sending the positioning data to the mobile robot, the robot can perform subsequent operations based on the accurate positioning data. Compared to existing mobile robot positioning devices, this device can achieve stable positioning under different lighting conditions, reducing the influence of light during the positioning process, thus obtaining accurate mobile robot positioning data and improving the accuracy and robustness of mobile robot positioning.

[0067] According to an embodiment of the present invention, a method for locating a mobile robot is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0068] This embodiment provides a mobile robot positioning method, which can be used in the aforementioned mobile robot positioning device. Figure 3 This is a flowchart of a mobile robot localization method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0069] Step S301: Obtain the first top view image of the mobile robot at the current moment. Specifically, based on the infrared sensor in the mobile robot positioning device, the first top view image of the mobile robot at the current moment is acquired. This avoids acquiring highly dynamic environmental information from the surrounding low-lying areas, ensuring the reliability of the first top view image and providing support for subsequent determination of the mobile robot's current location.

[0070] Step S302: Based on the first top view image, localization is performed to obtain the target localization data of the mobile robot at the current moment. Specifically, since the first top view image includes various feature information, by analyzing the first top view image and performing localization based on the first top view image through the calculation and processing module of the mobile robot localization device, more accurate target localization data can be obtained.

[0071] Step S303: The target positioning data of the mobile robot at the current moment is sent to the mobile robot. Specifically, the target positioning data includes the position information and posture information of the mobile robot at the current moment. The target positioning data of the mobile robot is sent to the mobile robot through the communication module in the mobile robot positioning device, so that the mobile robot can know its own posture, which facilitates subsequent operations such as path planning and path scheduling.

[0072] The mobile robot positioning method provided in this invention improves the accuracy and reliability of positioning by acquiring a first top view image of the mobile robot at the current moment, avoiding the collection of highly dynamic environmental information in the surrounding low-lying area, analyzing the first top view image for positioning, and directly sending the positioning data to the mobile robot to ensure that the mobile robot can obtain its own position information in real time.

[0073] This embodiment provides a mobile robot positioning method, which can be used in the aforementioned mobile robot positioning device. The method specifically includes the following steps:

[0074] Step S401: Obtain the first top-view image of the mobile robot at the current moment. See details below. Figure 3 Step S301 of the illustrated embodiment will not be described again here.

[0075] In some optional implementations, prior to step S401 above, the method further includes:

[0076] Step a1: Load the visual map of the mobile robot's working area. Specifically, before performing mobile robot localization, the visual map can be pre-loaded into the computational processing unit of the mobile robot localization device for localization. The specific process of obtaining the visual map is as follows: the mobile robot is manually controlled to move around its working area, collecting image information within the working area. Visual SLAM technology is used to generate a visual map of visual feature points throughout the working area based on the image information. Taking the starting position of the mobile robot's movement as the origin of the visual map, the spatial three-dimensional coordinates of all visual feature points relative to the origin can be generated. Each visual feature point also carries a descriptor for feature matching. The camera used to collect images can be a general camera or an infrared sensor in the mobile robot localization device used in this embodiment of the invention; this embodiment of the invention does not impose any limitations on this. The visual map includes at least one map semantic feature, and each map semantic feature includes the category to which the map semantic feature belongs and the spatial three-dimensional position corresponding to the map semantic feature. By loading the visual map of the mobile robot's working area, support is provided for subsequent mobile robot localization.

[0077] In some optional implementations, before performing mobile robot localization, the weights of the neural network used by the neural network processing unit in the computational processing module of the mobile robot localization device can be obtained. These neural network weights can be obtained through pre-training. For example, during the process of acquiring the visual map, the image information of the mobile robot's working area can be manually labeled with semantic features. These manually labeled semantic features can then be used as a training set to train the neural network weights using existing techniques. Alternatively, a large amount of image information of the working areas of various mobile robots can be collected, and the neural network weights can be trained using existing techniques. Simultaneously, image information of the mobile robot's current working area can be used as incremental training to improve the neural network's perception capabilities, resulting in the final neural network weights. Using the pre-trained neural network weights during the mobile robot localization process can improve the efficiency and accuracy of localization determination.

[0078] Step S402: Based on the first top view image, perform localization to obtain the target localization data of the mobile robot at the current moment.

[0079] Specifically, step S402 includes:

[0080] Step S4021: Obtain the mobile robot's positioning data from the previous moment, the mobile robot's odometry data at the current moment, and the target timestamp of the first top view image. Specifically, based on the communication module of the mobile robot's positioning device, obtain the positioning data from the previous moment and the odometry data at the current moment from the mobile robot's navigation module. The odometry data reflects the changes in distance and direction of the mobile robot as it moves from its initial position. The target timestamp of the first top view image can be obtained based on an infrared sensor and is used to identify the precise time when the first top view image was acquired.

[0081] Step S4022: Based on the mobile robot's positioning data from the previous moment, the mobile robot's odometry data from the current moment, and the target timestamp of the first top view image, determine the first positioning data corresponding to the target timestamp. Specifically, the first positioning data includes position information and attitude information. Typically, the motion model of the mobile robot, such as the kinematic equations of a wheeled robot, is used to map the positioning data and odometry data from the previous moment to the position under the target timestamp, achieving temporal alignment and positioning estimation to obtain preliminary estimated first positioning data. Optionally, other methods can be used to determine the first positioning data, such as integration or filtering algorithms, etc., and this embodiment of the invention does not limit this. By preliminarily estimating the first positioning data, the current positioning of the mobile robot can be roughly understood, providing a basis for subsequent precise positioning.

[0082] Step S4023: Based on the first top view image and the visual map, optimize the first positioning data to obtain the second positioning data of the mobile robot at the target timestamp. The visual map includes all map semantic features existing in the working area of ​​the mobile robot. Each map semantic feature includes the category to which the map semantic feature belongs and the spatial three-dimensional position corresponding to the map semantic feature.

[0083] Specifically, step S4023 includes:

[0084] Step b1: Optimize the image quality of the first top view image to obtain the second top view image. Specifically, the image processing unit of the computational processing module of the mobile robot positioning device optimizes the first top view image to improve its image quality and obtain the second top view image, thereby improving the accuracy of subsequent feature detection.

[0085] Step b2 involves using a neural network to detect the second top view image, obtaining a first semantic feature set. This first semantic feature set includes all detected semantic features in the second top view image. Each detected semantic feature includes its category and the corresponding two-dimensional pixel position. Specifically, a neural network processing unit, based on pre-trained neural network weights and referring to the detection process of the neural network processing unit, detects the second top view image to obtain all detected semantic features included in the second top view image, thus forming the first semantic feature set.

[0086] Step b3: For any map semantic feature in the visual map, based on the first positioning data, the corresponding three-dimensional spatial position of the map semantic feature is reprojected to the two-dimensional image space to obtain the two-dimensional pixel position corresponding to the map semantic feature. Specifically, for any map semantic feature in the visual map, the map semantic feature has been identified and located during the acquisition of the visual map and has three-dimensional spatial coordinates. In order to perform subsequent positioning operations, the three-dimensional spatial position corresponding to the map semantic feature needs to be first transformed into a coordinate system using the first positioning data, and then the coordinates corresponding to the coordinate system transformation are transformed into two-dimensional pixel coordinates in the two-dimensional image space through projection transformation, such as pinhole model projection, that is, reprojection is performed to obtain the two-dimensional pixel position in the two-dimensional image space after reprojection.

[0087] Step b4: After reprojecting the visual map, a second semantic feature set is determined. This second semantic feature set includes all map semantic features located within a preset range of the first positioning data. Specifically, the map semantic features within the preset range refer to the map semantic features on the two-dimensional plane corresponding to the visual map. A specific radius or bounding box is drawn around the initially estimated first positioning data as the center. This bounding box is used to filter whether any two-dimensional pixel positions corresponding to map semantic features fall within this spatial region. All map semantic features falling within this spatial region are included in the second semantic feature set. By determining the second semantic feature set, the analysis is ensured to focus on local areas closely related to the mobile robot's current position, thereby improving the effectiveness and accuracy of the positioning.

[0088] Step b5: Match the first semantic feature set and the second semantic feature set to obtain the third semantic feature set that has been successfully matched.

[0089] Specifically, step b5 above includes:

[0090] Step c1: For any detected semantic feature in the first semantic feature set, traverse all map semantic features in the second semantic feature set. Specifically, match the detected first semantic feature set with the second semantic feature set obtained based on the visual map one by one. Through semantic feature matching, a more accurate location of the mobile robot on the known visual map can be identified.

[0091] Step c2: If any map semantic feature exists, and the distance between the two-dimensional pixel position corresponding to the map semantic feature and the two-dimensional pixel position corresponding to the detected semantic feature is less than a preset threshold, and the category to which the map semantic feature belongs is the same as the category to which the detected semantic feature belongs, then the detected semantic feature and the map semantic feature are successfully matched. Specifically, there are two conditions for semantic feature matching: first, the distance between the two two-dimensional pixel positions is less than the preset threshold, which ensures that only sufficiently close features are considered as candidate matches; second, the semantic feature categories are consistent, meaning that they represent the same type of object or environmental feature. If both matching conditions are met, then the two are considered to be successfully matched. This distance can be Euclidean distance, or other distances can be used. This embodiment of the invention does not impose any limitations on this. By performing detailed matching between the detected semantic feature and the pre-constructed map semantic feature, the accuracy of the mobile robot in determining its own location can be significantly improved. At the same time, by setting conditions to filter valid matches, false matches can be filtered out to a certain extent, enhancing the stability and adaptability of location determination in complex and changing environments.

[0092] Step c3 involves adding the successfully matched detection semantic features and map semantic features to the third semantic feature set. Specifically, each successfully matched pair of detection semantic features and map semantic features is included in the third semantic feature set. By determining the third semantic feature set and rigorously matching the actually detected detection semantic features with the map semantic features in the visual map, the accuracy of the mobile robot in determining its own localization can be significantly improved.

[0093] Step b6: Based on the third semantic feature set, the first positioning data is optimized using an optimization algorithm to obtain the second positioning data.

[0094] Specifically, step b6 above includes:

[0095] Step d1: For any map semantic feature in the third semantic feature set, obtain the detection semantic feature that successfully matches the map semantic feature from the third semantic feature set. Specifically, the third semantic feature set contains one-to-one corresponding detection semantic features and map semantic features.

[0096] Step d2 involves determining the pixel difference between the two-dimensional pixel positions in the detected semantic features and the two-dimensional pixel positions in the map semantic features. Specifically, the two-dimensional pixel positions in the detected semantic features are the predicted positions, while the two-dimensional pixel positions in the map semantic features are the actual positions obtained through the visual map. The difference between the two two-dimensional pixel positions reflects the accuracy of the prediction, thereby better guiding the localization process based on this difference, and improving the accuracy and reliability of localization.

[0097] Step d3 involves optimizing the pixel difference. The optimization objective is to minimize the pixel difference. Based on the first positioning data, optimization is performed to obtain the second positioning data. Specifically, optimization is performed with the minimum pixel difference as the optimization objective. This is achieved by adjusting the parameters corresponding to the first positioning data. Based on each adjusted first positioning data, the map semantic features are reprojected to obtain new two-dimensional pixel positions corresponding to the map semantic features. This process is repeated using iterative or filtering methods to minimize the pixel difference, thus aligning the two pixel positions as closely as possible, resulting in more accurate second positioning data and improving the accuracy and reliability of the positioning.

[0098] Step S4024: Based on the mobile robot's odometry data at the current moment, an odometry interpolation algorithm is used to update the second positioning data, obtaining the mobile robot's target positioning data at the current moment. Specifically, after obtaining the second positioning data, the odometry data of the mobile robot acquired by the communication module of the mobile robot positioning device is fused with the second positioning data using an odometry interpolation algorithm, such as Kalman filtering or particle filtering. This not only utilizes the real-time acquired images but also incorporates the mobile robot's dynamic model, improving the stability and accuracy of positioning, ultimately obtaining the target positioning data.

[0099] Step S403: Send the target positioning data of the mobile robot at the current moment to the mobile robot. For details, please refer to [link to relevant documentation]. Figure 3 Step S303 of the illustrated embodiment will not be described again here.

[0100] The mobile robot positioning method provided in this invention improves the accuracy and reliability of positioning by acquiring a first top view image of the mobile robot at the current moment, avoiding the collection of highly dynamic environmental information in the surrounding low-lying area, analyzing the first top view image for positioning, and directly sending the positioning data to the mobile robot to ensure that the mobile robot can obtain its own position information in real time.

[0101] This invention also provides a computer device; please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 4 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 4 Take a processor 10 as an example.

[0102] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0103] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0104] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0105] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0106] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0107] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0108] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0109] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A mobile robot positioning device, characterized in that, The device includes an infrared sensor, an infrared supplementary light source, a computing processing module, a power supply module, and a communication module; The infrared sensor is equipped with an infrared bandpass filter and uses automatic exposure to acquire a first top view image of the mobile robot at the current moment. The infrared supplementary light source device uses instantaneous supplementary light to emit infrared light to supplement the infrared sensor. The lighting time of the infrared supplementary light source device is synchronized with the shutter exposure time of the infrared sensor. The computing and processing module is used to load a visual map of the working area of ​​the mobile robot, perform positioning based on the first top view image, and obtain the target positioning data of the mobile robot at the current moment. The visual map includes all map semantic features existing in the working area of ​​the mobile robot. Each map semantic feature includes the category to which the map semantic feature belongs and the spatial three-dimensional position corresponding to the map semantic feature. The power module is used to supply power to the infrared supplementary light source device and the computing processing module; The communication module is used to acquire the positioning data of the mobile robot at the previous moment, the odometer data of the mobile robot at the current moment, and the target timestamp of the first top view image, and send the target positioning data of the mobile robot at the current moment to the mobile robot. The computational processing module includes an image processing unit, an arithmetic processing unit, and a neural network processing unit. The image processing unit is used to optimize the image quality of the first top view image; The neural network processing unit is used to detect the optimized first top view image using a neural network to obtain all semantic features in the optimized first top view image. Each semantic feature includes the category to which the semantic feature belongs and the two-dimensional pixel position corresponding to the semantic feature. The processing unit is used to locate the mobile robot based on all semantic features of the optimized first top view image, and obtain the target location data of the mobile robot at the current moment. The computation processing module is specifically used for: Based on the mobile robot's positioning data from the previous moment, the mobile robot's odometer data at the current moment, and the target timestamp of the first top view image, the first positioning data of the mobile robot corresponding to the target timestamp is determined. Based on the first top view image and the visual map, the first positioning data is optimized to obtain the second positioning data of the mobile robot corresponding to the target timestamp. Based on the odometry data of the mobile robot at the current moment, the second positioning data is updated using an odometry interpolation algorithm to obtain the target positioning data of the mobile robot at the current moment; The computation processing module is also specifically used for: Optimize the image quality of the first top view image to obtain the second top view image; A neural network is used to detect the second top view image to obtain a first semantic feature set. The first semantic feature set includes all detected semantic features in the second top view image. Each detected semantic feature includes the category to which the detected semantic feature belongs and the two-dimensional pixel position corresponding to the detected semantic feature. For any map semantic feature in the visual map, based on the first positioning data, the spatial three-dimensional position corresponding to the map semantic feature is reprojected to the two-dimensional image space to obtain the two-dimensional pixel position corresponding to the map semantic feature. After the visual map is reprojected, a second semantic feature set is determined. The second semantic feature set includes all map semantic features located within a preset range of the first positioning data. The first semantic feature set and the second semantic feature set are matched to obtain the third semantic feature set that is successfully matched; Based on the third semantic feature set, the first positioning data is optimized using an optimization algorithm to obtain the second positioning data.

2. A mobile robot localization method, characterized in that, The method, applied to the mobile robot positioning device as described in claim 1, comprises: Obtain a first top-view image of the mobile robot at the current moment; Load a visual map of the working area of ​​the mobile robot. The visual map includes all map semantic features existing in the working area of ​​the mobile robot. Each map semantic feature includes the category to which the map semantic feature belongs and the spatial three-dimensional position corresponding to the map semantic feature. Based on the first top view image, the target positioning data of the mobile robot at the current moment is obtained; Send the target location data of the mobile robot at the current moment to the mobile robot; The step of obtaining the target positioning data of the mobile robot at the current moment based on the first top view image includes: Obtain the positioning data of the mobile robot at the previous time, the odometer data of the mobile robot at the current time, and the target timestamp of the first top view image; Based on the mobile robot's positioning data from the previous moment, the mobile robot's odometer data at the current moment, and the target timestamp of the first top view image, the first positioning data of the mobile robot corresponding to the target timestamp is determined. Based on the first top view image and the visual map, the first positioning data is optimized to obtain the second positioning data of the mobile robot corresponding to the target timestamp. Based on the odometry data of the mobile robot at the current moment, the second positioning data is updated using an odometry interpolation algorithm to obtain the target positioning data of the mobile robot at the current moment; The step of optimizing the first positioning data based on the first top view image and the visual map to obtain the second positioning data of the mobile robot corresponding to the target timestamp includes: Optimize the image quality of the first top view image to obtain the second top view image; A neural network is used to detect the second top view image to obtain a first semantic feature set. The first semantic feature set includes all detected semantic features in the second top view image. Each detected semantic feature includes the category to which the detected semantic feature belongs and the two-dimensional pixel position corresponding to the detected semantic feature. For any map semantic feature in the visual map, based on the first positioning data, the spatial three-dimensional position corresponding to the map semantic feature is reprojected to the two-dimensional image space to obtain the two-dimensional pixel position corresponding to the map semantic feature. After the visual map is reprojected, a second semantic feature set is determined. The second semantic feature set includes all map semantic features located within a preset range of the first positioning data. The first semantic feature set and the second semantic feature set are matched to obtain the third semantic feature set that is successfully matched; Based on the third semantic feature set, the first positioning data is optimized using an optimization algorithm to obtain the second positioning data.

3. The method according to claim 2, characterized in that, The step of matching the first semantic feature set and the second semantic feature set to obtain a successfully matched third semantic feature set includes: For any detected semantic feature in the first semantic feature set, traverse all map semantic features in the second semantic feature set; If any map semantic feature exists, and the distance between the two-dimensional pixel position corresponding to the map semantic feature and the two-dimensional pixel position corresponding to the detected semantic feature is less than a preset threshold, and the category to which the map semantic feature belongs is the same as the category to which the detected semantic feature belongs, then the detected semantic feature and the map semantic feature are successfully matched. The successfully matched detection semantic features and map semantic features are added to the third semantic feature set.

4. The method according to claim 2, characterized in that, The second location data is obtained by optimizing the first location data based on the third semantic feature set using an optimization algorithm, including: For any map semantic feature in the third semantic feature set, obtain the detection semantic feature that successfully matches the map semantic feature from the third semantic feature set; Determine the pixel difference between the two-dimensional pixel position in the detected semantic features and the two-dimensional pixel position in the map semantic features; The pixel difference is optimized with minimizing the pixel difference as the optimization objective. Based on the first positioning data, the second positioning data is obtained.

5. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory storing computer instructions, and the processor executing the computer instructions to perform the mobile robot positioning method according to any one of claims 2 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the mobile robot positioning method according to any one of claims 2 to 4.

Images