Mobile-robot positioning apparatus and method, and device and storage medium

Abstract

The present application relates to the technical field of mobile robots. Disclosed are a mobile-robot positioning apparatus and method, and a device and a storage medium. The apparatus comprises an infrared sensor, an infrared fill light source apparatus, a computing and processing module, a power supply module and a communication module, wherein the infrared sensor collects a top-view image of a mobile robot; the infrared fill light source apparatus emits infrared light to provide fill light for the infrared sensor; the computing and processing module positions the mobile robot on the basis of the top-view image, so as to obtain positioning data of the mobile robot; the power supply module supplies power to the infrared fill light source apparatus and the computing and processing module; and the communication module sends the positioning data of the mobile robot to the mobile robot. The present application improves the quality of a top-view image by means of combining various parts of the mobile-robot positioning apparatus, determines positioning on the basis of a high-quality top-view image, and can perform stable positioning under different lighting conditions, thereby reducing the impact of light, and improving the accuracy and robustness of mobile-robot positioning.

Description

Mobile robot positioning apparatus, method, device and storage medium

[0001] Cross-reference to related applications

[0002] This application claims priority to the Chinese patent application No. 2024110990158, filed on August 9, 2024, and entitled "Mobile robot positioning apparatus, method, device and storage medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of mobile robots, in particular to a mobile robot positioning apparatus, method, device and storage medium. BACKGROUND

[0004] A mobile robot refers to a robot capable of automatically moving without direct human operation to complete a specific task, and has been applied in various aspects of production and life. Among them, accurate positioning of the mobile robot is an important basis for completing the task, such as making the mobile robot located at the correct position to perform the task, avoiding task failure or error caused by position error, making the mobile robot path planning more efficient, and improving work efficiency, etc. Therefore, how to accurately position the mobile robot is a problem to be solved.

[0005] At present, the existing mobile robot positioning apparatus may cause the image captured by the camera to have optical problems such as overexposure and underexposure under the influence of changes in lighting conditions, resulting in the inability to clearly capture feature points, and if the positioning is determined based on the image, the accuracy of the mobile robot positioning is low and the precision is low.

[0006] SUMMARY

[0007] Therefore, the present application provides a mobile robot positioning apparatus, method, device and storage medium to solve the problem of low accuracy and precision of mobile robot positioning.

[0008] In a first aspect, the present application provides a mobile robot positioning apparatus, which comprises an infrared sensor, an infrared light supplementing light source apparatus, a computing processing module, a power module and a communication module.

[0009] The infrared sensor is used to collect a top view image of the mobile robot.

[0010] The infrared light supplementing light source apparatus is used to emit infrared light to supplement light for the infrared sensor.

[0011] The computing processing module is used to position the mobile robot based on the top view image obtained by the infrared sensor, and obtain positioning data of the mobile robot.

[0012] The power module is configured to supply power for the infrared light supplement light source device and the computing processing module.

[0013] The communication module is configured to send the positioning data of the mobile robot to the mobile robot.

[0014] The mobile robot positioning device provided by the embodiments of the present application can improve the quality of the top view image collected, and then perform positioning based on the high-quality top view image, thereby further improving the positioning accuracy of the mobile robot. Meanwhile, by sending the positioning data to the mobile robot, the mobile robot can perform subsequent operations based on the accurate positioning data. Compared with the existing mobile robot positioning device, the mobile robot can be stably positioned under different lighting conditions, the influence of light on the positioning process is reduced, thereby obtaining accurate mobile robot positioning data, and the accuracy and robustness of the mobile robot positioning are improved.

[0015] In an optional embodiment, the computing processing module includes an image processing unit, an operation processing unit, and a neural network processing unit.

[0016] The image processing unit is configured to optimize the image quality of the top view image.

[0017] The neural network processing unit is configured to detect the optimized top view image by using a neural network to obtain all semantic features in the optimized top view image. Any semantic feature includes a category to which the semantic feature belongs and a two-dimensional pixel position corresponding to the semantic feature.

[0018] The operation processing unit is configured to perform positioning on the mobile robot based on all semantic features of the optimized top view image to obtain the positioning data of the mobile robot.

[0019] The mobile robot positioning device provided by the embodiments of the present application can optimize the top view image by using the image processing unit, the neural network processing unit, and the operation processing unit, and then perform semantic analysis. The semantic features obtained by the semantic analysis are used to obtain the positioning data, thereby enhancing the positioning accuracy of the mobile robot in a complex environment.

[0020] In a second aspect, the present application provides a mobile robot positioning method applied to the mobile robot positioning device of the first aspect or any of the corresponding embodiments thereof. The method includes the following steps.

[0021] Obtaining a first top view image of the mobile robot at a current time;

[0022] Performing positioning based on the first top view image to obtain target positioning data of the mobile robot at the current time;

[0023] The target positioning data of the mobile robot at the current time is sent to the mobile robot.

[0024] The mobile robot positioning method provided by the embodiment of the application avoids collecting the environment information of the lower area with high dynamic change around, analyzes the first top view image for positioning, improves the accuracy and reliability of the positioning, and directly sends the positioning data to the mobile robot, so that the mobile robot can obtain the position information in time.

[0025] In an optional implementation, before the positioning based on the first top view image to obtain the target positioning data of the mobile robot at the current time, the method further comprises:

[0026] The visual map of the working area of the mobile robot is loaded.

[0027] The mobile robot positioning method provided by the embodiment of the application loads the visual map of the working area of the mobile robot to provide support for subsequent mobile robot positioning.

[0028] In an optional implementation, the positioning based on the first top view image to obtain the target positioning data of the mobile robot at the current time comprises:

[0029] The positioning data of the mobile robot at the previous time of the current time, the odometer data of the mobile robot at the current time and the target time stamp of the first top view image are obtained;

[0030] Based on the positioning data of the mobile robot at the previous time of the current time, the odometer data of the mobile robot at the current time and the target time stamp of the first top view image, the first positioning data of the mobile robot corresponding to the target time stamp is determined.

[0031] Based on the first top view image and the visual map, the first positioning data is optimized to obtain the second positioning data of the mobile robot corresponding to the target time stamp.

[0032] Based on the odometer data of the mobile robot at the current time, the odometer interpolation algorithm is used to update the second positioning data to obtain the target positioning data of the mobile robot at the current time.

[0033] The mobile robot positioning method provided by the embodiments of the present application can obtain first positioning data of a current time through combining positioning data of a previous time and odometer data of the current time, the preliminary estimation can reduce the matching range of subsequent map semantic features, and the speed of positioning calculation is improved, then the first top view image is optimized with a visual map, the first positioning data is corrected by using detailed features of the environment, the accuracy and robustness of positioning are improved, the time consistency between visual data and motion data is ensured by considering the target timestamp of the first top view image, positioning errors caused by time deviation are avoided, and the positioning real-time and accuracy are maintained by using an odometer interpolation algorithm to fine-tune the positioning result according to the real-time movement state of the robot.

[0034] In an optional implementation, the visual map includes all map semantic features existing in a working area of the mobile robot, and any map semantic feature includes a category to which the map semantic feature belongs and a spatial three-dimensional position corresponding to the map semantic feature.

[0035] The first positioning data is optimized based on the first top view image and the visual map to obtain second positioning data of the mobile robot corresponding to the target timestamp, and the method includes:

[0036] The image quality of the first top view image is optimized to obtain a second top view image.

[0037] The second top view image is detected by using a neural network to obtain a first semantic feature set, the first semantic feature set includes all detected semantic features in the second top view image, and any detected semantic feature includes a category to which the detected semantic feature belongs and a two-dimensional pixel position corresponding to the detected semantic feature.

[0038] For any map semantic feature in the visual map, the spatial three-dimensional position corresponding to the map semantic feature is re-projected to a two-dimensional image space based on the first positioning data to obtain a two-dimensional pixel position corresponding to the map semantic feature.

[0039] After the re-projection of the visual map, a second semantic feature set is determined, and the second semantic feature set includes all map semantic features located in a preset range of the first positioning data.

[0040] The first semantic feature set and the second semantic feature set are matched to obtain a third semantic feature set with successful matching.

[0041] The first positioning data is optimized based on the third semantic feature set by using an optimization algorithm to obtain the second positioning data.

[0042] The mobile robot positioning method provided in the embodiments of the present application can accurately extract semantic features to obtain a first semantic feature set through optimization of a first top view image and use of a neural network detection. Even in the case of changes in environmental illumination, occlusion or interference of dynamic objects, the neural network can still stably extract semantic features, thereby improving the accuracy and robustness of positioning data obtained based on the semantic features. The first semantic feature set is matched with a second semantic feature set obtained by screening in a visual map, a third semantic feature set with successful matching is obtained, and an optimization algorithm is used to process the semantic feature set with successful matching, so as to optimize the positioning data and improve the accuracy of the positioning data.

[0043] In an optional implementation, matching the first semantic feature set and the second semantic feature set to obtain the third semantic feature set with successful matching includes:

[0044] For any detection semantic feature in the first semantic feature set, all map semantic features in the second semantic feature set are traversed;

[0045] If there is any map semantic feature, the distance between the two-dimensional pixel position corresponding to the map semantic feature and the two-dimensional pixel position corresponding to the detection semantic feature is less than a preset threshold, and the category to which the map semantic feature belongs is the same as the category to which the detection semantic feature belongs, the detection semantic feature and the map semantic feature are matched successfully;

[0046] The detection semantic feature and the map semantic feature with successful matching are added to the third semantic feature set.

[0047] The mobile robot positioning method provided in the embodiments of the present application matches by setting the distance preset threshold and the condition that the semantic feature categories are the same, effectively filters irrelevant feature interference, ensures the high relevance and accuracy of the matching result, reduces the possibility of false matching, and improves the reliability and robustness of positioning as a basis for subsequent positioning optimization.

[0048] In an optional implementation, based on the third semantic feature set, an optimization algorithm is used to optimize the first positioning data to obtain second positioning data, including:

[0049] For any map semantic feature in the third semantic feature set, a detection semantic feature with successful matching to the map semantic feature is obtained from the third semantic feature set;

[0050] A pixel difference value between the two-dimensional pixel position in the detection semantic feature and the two-dimensional pixel position in the map semantic feature is determined;

[0051] The pixel difference value is optimized, and the optimization is performed based on the first positioning data to obtain the second positioning data, with the minimization of the pixel difference value as the optimization target.

[0052] The mobile robot positioning method provided by the embodiments of the present application uses the difference between the two-dimensional pixel position after re-projection by using the map semantic feature and the two-dimensional pixel position of the detected semantic feature, takes the pixel-level difference minimization as the optimization target, realizes the correction of the positioning data, and thus guarantees the accuracy and reliability of the positioning data.

[0053] In a third aspect, the present application provides a computer device, comprising a memory and a processor, which are connected with each other in communication, and the memory stores computer instructions, and the processor executes the computer instructions to perform the mobile robot positioning method of the second aspect or any of the corresponding embodiments thereof.

[0054] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the mobile robot positioning method of the second aspect or any of the corresponding embodiments thereof.

[0055] In a fifth aspect, the present application provides a computer program product, which comprises computer instructions, and the computer instructions are used to make a computer execute the mobile robot positioning method of the second aspect or any of the corresponding embodiments thereof. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0057] FIG. 1 is a schematic diagram of a mobile robot positioning device according to an embodiment of the present application;

[0058] FIG. 2 is a schematic diagram of the placement position of a mobile robot positioning device according to an embodiment of the present application;

[0059] FIG. 3 is a flowchart of a mobile robot positioning method according to an embodiment of the present application;

[0060] FIG. 4 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0062] Since accurate positioning of the mobile robot is an important basis for completing various tasks, how to accurately position the mobile robot is a problem to be solved. The mobile robot positioning device in the related art cannot solve the influence of changes in light conditions on determining positioning, resulting in low accuracy and low precision of the mobile robot positioning. The mobile robot positioning device provided in the embodiments of the present application can improve the quality of the top view image collected through the combination of each part of the mobile robot positioning device, and then position based on the high-quality top view image, further improving the accuracy and robustness of the mobile robot positioning. At the same time, by sending the positioning data to the mobile robot, the mobile robot performs subsequent operations based on accurate positioning data.

[0063] In the present embodiment, a mobile robot positioning device is provided, as shown in FIG. 1, which comprises an infrared sensor 1, an infrared light supplement light source device 2, a computing processing module 3, a power module 4 and a communication module 5; the infrared sensor 1 is used to collect the top view image of the mobile robot; the infrared light supplement light source device 2 is used to emit infrared light to supplement light for the infrared sensor 1; the computing processing module 3 is used to position the mobile robot based on the top view image obtained by the infrared sensor 1, and obtain the positioning data of the mobile robot; the power module 4 is used to supply power for the infrared light supplement light source device 2 and the computing processing module 3; and the communication module 5 is used to send the positioning data of the mobile robot to the mobile robot.

[0064] Specifically, the infrared sensor 1 is installed with an infrared band-pass filter, which can effectively filter the visible light in the light, and compared with using a common camera to collect images, the influence of the change of light conditions on determining the positioning can be effectively reduced. Moreover, the infrared sensor 1 has an automatic exposure function, that is, the exposure time is automatically adjusted according to the brightness of the collected image, so as to ensure that the collected image has stable brightness, and further reduce the influence of the light condition. The infrared light source device 2 is arranged on both sides of the infrared sensor 1, and the infrared light emitted by the infrared light source device 2 can realize light compensation for the infrared sensor 1. Optionally, two groups of infrared light source devices 2 are used in the embodiment of the application as an example for description, and in actual situation, only one group of infrared light source device 2 can be installed on one side of the infrared sensor 1. The infrared light source device 2 adopts a transient light compensation mode, and the infrared light is emitted by the light compensation lamp source for light compensation. The wave band of the light compensation lamp source can be selected from 850nm, 905nm or 940nm infrared wave band. The light compensation lamp source can be a LED (Light Emitting Diode, Light Emitting Diode), a laser light source or other light source, and the embodiment of the application does not limit this. The laser light source can be a VCSEL (Vertical-cavity surface-emitting laser, Vertical-cavity surface-emitting laser). The lighting time of the light compensation lamp source is synchronized with the shutter exposure time of the infrared sensor 1. Based on the infrared light source device 2, the mobile robot positioning device has the ability to position in a completely dark or relatively dark working environment, and combined with the automatic exposure function of the infrared sensor 1, the brightness of the top view image collected in the on / off light and day / night can be kept at a constant brightness, which not only reduces the difficulty of subsequent image processing, but also improves the reliability of positioning. The computing processing module 3 is the main functional module of the mobile robot positioning device for determining the positioning of the mobile robot. The power module 4 supplies power to the infrared light source device 2 and the computing processing module 3 after power stabilization. Since the infrared light source device 2 is a transient light compensation, the instantaneous current and power are large, so the power module 4 needs to provide sufficient power output, and needs to do a good job in power isolation to prevent the influence of transient light compensation current and voltage fluctuation on the stable work of the computing processing module 3. The communication module 5 adopts a gigabit Ethernet communication interface, communicates with the mobile robot through the interface, obtains the odometer data and other information of the mobile robot, and sends the positioning data determined by the mobile robot positioning device to the mobile robot. Optionally, the positioning data can be sent to the navigation module of the mobile robot, so that the mobile robot can clearly know the current position, so as to perform path planning and task scheduling.Through the combination of multiple parts in the mobile robot positioning device, the quality of the collected top view image can be improved, and then the positioning is determined based on the high-quality top view image, further improving the accuracy of the mobile robot positioning. Meanwhile, by sending the positioning data to the mobile robot, the mobile robot can perform subsequent operations based on the accurate positioning data.

[0065] In some optional embodiments, for the communication module 5, corresponding host computer software can be developed. After the host computer software is successfully connected with the communication module 5, the communication module 5 has the functions of loading the visual map, updating the weight of the neural network, monitoring the odometer data, obtaining the positioning data calculated by the calculation processing module 3, obtaining the running log of the mobile robot positioning device, and the like, so as to facilitate the monitoring and debugging of the engineers.

[0066] FIG. 2 is a schematic diagram of the placement position of the mobile robot positioning device according to the embodiment of the present application. As shown in FIG. 2, a is a roller conveying type robot, and the mobile robot positioning device is installed at the top position shown by the reference numeral 6, that is, the top of the front vertical rod. b is an automatic forklift robot, and the mobile robot positioning device is installed at the top position shown by the reference numeral 7, that is, the top of the front truck head frame. When the mobile robot positioning device is installed, the infrared sensor 1 is installed vertically upward, so that reliable and fixed feature information in the upper field of view of the mobile robot during movement can be collected, avoiding the interference of the surrounding environment, and further improving the reliability and accuracy of the positioning. It should be noted that the two types of mobile robots shown in FIG. 2 are only examples, and the mobile robot positioning device can be installed on the top of any mobile robot that needs to determine the positioning, such as a sweeping robot, a towing robot, and the like, and the embodiment of the present application does not limit this.

[0067] In some optional embodiments, the calculation processing module 3 includes an image processing unit, an operation processing unit, and a neural network processing unit; the image processing unit is used to optimize the image quality of the top view image; the neural network processing unit is used to detect the optimized top view image by using a neural network to obtain all semantic features in the optimized top view image, any semantic feature including a category to which the semantic feature belongs and a two-dimensional pixel position corresponding to the semantic feature; and the operation processing unit is used to position the mobile robot based on all semantic features of the optimized top view image to obtain the positioning data of the mobile robot.

[0068] Specifically, the mobile robot positioning device selects an ARM (Advanced RISC Machine, ARM processor) embedded platform as a computing processing module. The ARM embedded platform includes a CPU (Central Processing Unit) and an NPU (Neural Processing Unit) in hardware. The image processing unit is a special image processing program running on the CPU, the operation processing unit is a positioning program running on the CPU, and the neural network processing unit is a deep learning program running on the NPU. The image signal processing unit is responsible for processing the original top view image collected by the infrared sensor 1, such as linear correction, noise removal, bad point removal, interpolation, white balance, automatic exposure control, etc., to realize the image quality optimization of the top view image. The neural network processing unit has deep learning capability, so it needs to have at least more than 1TOPS of computing power. The input of the neural network processing unit is a two-dimensional image. In the embodiment of the present application, the optimized top view image is input into the neural network processing unit, the optimized top view image is convolved by the convolution layer, specific edges, lines or texture patterns are found, and then the activation function is applied to the results of convolution to learn more complex features in the image. The pooling operation of the pooling layer reduces the spatial dimension of the image while keeping important features, obtains a plurality of semantic feature categories included in the optimized top view image, and aggregates the pixels representing the same semantic feature category. In addition, post-processing techniques such as conditional random field or more advanced instance segmentation algorithms are used to separate semantic features belonging to the same semantic feature category but having different positions in the top view image, further improve the accuracy of the final output, ensure the accurate separation of different instances, and finally obtain all semantic features in the optimized top view image. For example, the neural network processing unit detects any optimized top view image, detects that the top view image includes three semantic features, and the categories to which the three semantic features belong are: fluorescent lamp, smoke alarm, and air conditioner. For the "fluorescent lamp", all pixel points in the optimized top view image whose semantic feature category is "fluorescent lamp" are aggregated, and the center pixel point in the at least one pixel point or any corner point on the bounding box formed by the at least one pixel point is selected as the two-dimensional pixel position corresponding to the "fluorescent lamp". Optionally, other pixel points can also be selected as the two-dimensional pixel position corresponding to the semantic feature, and the embodiment of the present application does not limit this. By determining the two-dimensional pixel position corresponding to the category to which each semantic feature belongs, the two-dimensional pixel positions corresponding to the semantic features "fluorescent lamp", "smoke alarm" and "air conditioner" in the top view image can be obtained.The operation processing unit performs positioning based on the semantic features of the top view image detected by the neural network processing unit, and obtains accurate positioning data.

[0069] The mobile robot positioning device provided by the embodiment of the present application can improve the quality of the collected top view image through the infrared light supplementing light source device, the calculation processing module, the power module and the communication module in the mobile robot positioning device, and then positioning is performed based on the high-quality top view image, thereby further improving the accuracy of the positioning of the mobile robot. Meanwhile, the positioning data is sent to the mobile robot, so that the mobile robot performs subsequent operations based on the accurate positioning data. Compared with the existing mobile robot positioning device, stable positioning can be performed under different lighting conditions, the influence of light on the positioning process is reduced, thereby obtaining accurate mobile robot positioning data, and the accuracy and robustness of the positioning of the mobile robot are improved.

[0070] According to the embodiment of the present application, a mobile robot positioning method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0071] In the embodiment, a mobile robot positioning method is provided, which can be used in the mobile robot positioning device described above. FIG. 3 is a flowchart of the mobile robot positioning method according to the embodiment of the present application. As shown in FIG. 3, the flow includes the following steps:

[0072] In step S301, a first top view image of the mobile robot at the current time is obtained. Specifically, based on the infrared sensor in the mobile robot positioning device, the first top view image of the mobile robot at the current time is collected, the environment information of the lower area with high dynamic change around is avoided, the reliability of the first top view image is ensured, and support is provided for subsequent determination of the positioning of the mobile robot at the current time.

[0073] In step S302, positioning is performed based on the first top view image to obtain target positioning data of the mobile robot at the current time. Specifically, since the first top view image includes multi-aspect feature information, the calculation processing module of the mobile robot positioning device analyzes the first top view image and performs positioning based on the first top view image, so that more accurate target positioning data can be obtained.

[0074] In step S303, the target positioning data of the mobile robot at the current time is sent to the mobile robot. Specifically, the target positioning data includes the position information and the attitude information of the mobile robot at the current time, and the target positioning data of the mobile robot is sent to the mobile robot through a communication module in the mobile robot positioning device, so that the mobile robot can determine its own pose, and subsequent operations such as path planning and path scheduling are facilitated.

[0075] The mobile robot positioning method provided in the embodiments of the present application can avoid collecting environmental information of a lower area around the mobile robot with high dynamic changes, and can improve the accuracy and reliability of positioning by analyzing the first top view image for positioning, and can ensure that the mobile robot can obtain its own position information in time by directly sending the positioning data to the mobile robot.

[0076] In the embodiments of the present application, a mobile robot positioning method is provided, which can be used in the mobile robot positioning device described above, and the method specifically includes the following steps:

[0077] In step S401, a first top view image of the mobile robot at the current time is obtained. For details, refer to step S301 of the embodiment shown in FIG. 3, which will not be repeated here.

[0078] In some optional embodiments, before step S401, the method further includes:

[0079] In step a1, a visual map of the working area of the mobile robot is loaded. Specifically, before positioning the mobile robot, the visual map can be loaded in advance to the operation processing unit of the mobile robot positioning device, so that the operation processing unit can perform positioning. The specific process of obtaining the visual map is as follows: manually control the mobile robot to move around in its working area, collect image information in the working area, generate a visual map of visual feature points in the entire working area based on the image information using visual SLAM technology, and take the starting position of the mobile robot as the origin of the visual map. Then, the spatial three-dimensional coordinates of all visual feature points relative to the origin can be generated, and each visual feature point has a descriptor for feature matching. The camera used for collecting images can be a general camera or an infrared sensor in the mobile robot positioning device used in the embodiments of the present application, and the embodiments of the present application do not limit the camera. The visual map includes at least one map semantic feature, and any map semantic feature includes a category to which the map semantic feature belongs and a spatial three-dimensional position corresponding to the map semantic feature. By loading the visual map of the working area of the mobile robot, support is provided for subsequent positioning of the mobile robot.

[0080] In some optional embodiments, the weight of the neural network used by the neural network processing unit in the computing processing module of the mobile robot positioning device can also be obtained before the positioning of the mobile robot. The weight of the neural network can be obtained by pre-training, for example, manually labeling the semantic features in the image information of the working area of the mobile robot collected from the above process of obtaining the visual map, taking the manually labeled semantic features as a training set, and training the weight of the neural network by using the existing technology. Alternatively, a large number of image information of the working area of various mobile robots is collected, the weight of the neural network is trained by using the existing technology, and at the same time, the image information of the working area where the mobile robot is currently located is taken as an incremental training to improve the perception ability of the neural network, so as to obtain the final weight of the neural network. In the process of determining the positioning of the mobile robot, the pre-trained weight of the neural network can improve the efficiency and accuracy of the determination of the positioning.

[0081] In step S402, the positioning is performed based on the first top view image to obtain target positioning data of the mobile robot at the current time.

[0082] Specifically, the above step S402 includes:

[0083] In step S4021, the positioning data of the mobile robot at the previous time of the current time, the odometer data of the mobile robot at the current time, and the target timestamp of the first top view image are obtained. Specifically, based on the communication module of the mobile robot positioning device, the positioning data at the previous time of the current time and the odometer data at the current time are obtained from the navigation module of the mobile robot. The odometer data reflects the change of distance and direction of the mobile robot during the movement from the initial position. The target timestamp of the first top view image can be obtained based on the infrared sensor, and the target timestamp is used to identify the accurate time of collecting the first top view image.

[0084] In step S4022, the first positioning data corresponding to the target timestamp of the mobile robot is determined based on the positioning data of the mobile robot at the previous time of the current time, the odometer data of the mobile robot at the current time, and the target timestamp of the first top view image. Specifically, the first positioning data includes position information and attitude information. Generally, the motion model of the mobile robot is used, for example, the kinematics equation of the wheeled robot, to map the positioning data at the previous time and the odometer data to the position at the target timestamp, realize the alignment and positioning estimation in time, and obtain the preliminary estimated first positioning data. Optionally, other methods such as integral or filtering algorithm can also be used to determine the first positioning data, and the embodiments of the present application do not limit this. Through the preliminary estimation of the first positioning data, the current positioning of the mobile robot can be roughly understood, which provides a basis for subsequent accurate positioning.

[0085] In step S4023, the first positioning data is optimized based on the first top-view image and the visual map to obtain second positioning data of the mobile robot corresponding to the target timestamp, the visual map including all map semantic features existing in a working area of the mobile robot, and any map semantic feature including a category to which the map semantic feature belongs and a spatial three-dimensional position corresponding to the map semantic feature.

[0086] Specifically, the step S4023 includes:

[0087] In step b1, the image quality of the first top-view image is optimized to obtain a second top-view image. Specifically, the image processing unit of the computing processing module of the mobile robot positioning device is used to optimize the first top-view image to improve the image quality of the first top-view image, thereby improving the accuracy of subsequent feature detection.

[0088] In step b2, a neural network is used to detect the second top-view image to obtain a first semantic feature set, the first semantic feature set including all detected semantic features in the second top-view image, and any detected semantic feature including a category to which the detected semantic feature belongs and a two-dimensional pixel position corresponding to the detected semantic feature. Specifically, the neural network of the neural network processing unit is used to detect the second top-view image based on the weights of the pre-trained neural network and the detection process of the neural network processing unit, thereby obtaining all detected semantic features in the second top-view image and forming the first semantic feature set.

[0089] In step b3, for any map semantic feature in the visual map, the spatial three-dimensional position corresponding to the map semantic feature is re-projected to a two-dimensional image space based on the first positioning data to obtain a two-dimensional pixel position corresponding to the map semantic feature. Specifically, for any map semantic feature in the visual map, the map semantic feature has been identified and positioned in the process of obtaining the visual map and has a three-dimensional spatial coordinate. In order to perform subsequent positioning operations, the spatial three-dimensional position corresponding to the map semantic feature needs to be first converted in the coordinate system through the first positioning data, and then the coordinate corresponding to the coordinate system after the conversion is converted into a two-dimensional pixel coordinate in the two-dimensional image space through a projection transformation, such as a pinhole model projection, that is, re-projection, to obtain a two-dimensional pixel position in the two-dimensional image space after re-projection.

[0090] Step b4, after the re-projection on the visual map, a second semantic feature set is determined, which includes all map semantic features located within a preset range of the first positioning data. Specifically, the map semantic features within the preset range refer to a specific radius or bounding box centered on the preliminary estimated first positioning data on the two-dimensional plane corresponding to the visual map. Whether the two-dimensional pixel positions corresponding to the map semantic features fall within this spatial region is screened, and all map semantic features falling within the spatial region are taken as the second semantic feature set. By determining the second semantic feature set, it is ensured that the analysis is focused on the local area closely related to the current position of the mobile robot, thereby improving the effectiveness and accuracy of positioning.

[0091] Step b5, the first semantic feature set and the second semantic feature set are matched to obtain a third semantic feature set with successful matching.

[0092] Specifically, the above step b5 includes:

[0093] Step c1, for any detection semantic feature in the first semantic feature set, all map semantic features in the second semantic feature set are traversed. Specifically, the first semantic feature set detected is matched one by one with the second semantic feature set based on the visual map. Through semantic feature matching, more accurate positioning of the mobile robot on the known visual map can be identified.

[0094] Step c2, if there is any map semantic feature, the distance between the two-dimensional pixel position corresponding to the map semantic feature and the two-dimensional pixel position corresponding to the detection semantic feature is less than a preset threshold, and the category to which the map semantic feature belongs is the same as the category to which the detection semantic feature belongs, the detection semantic feature and the map semantic feature are successfully matched. Specifically, there are two conditions for semantic feature matching: one is that the distance between the two-dimensional pixel positions of the two is less than the preset threshold, which ensures that only features close enough are considered as candidate matching items; the second is that the semantic feature categories are consistent, which means that they represent the same type of object or environmental feature. If both matching conditions are met, it is considered that the two are successfully matched. The distance can be Euclidean distance, and other distances can also be used. The present application embodiment does not limit this. By carefully matching the detection semantic feature with the pre-constructed map semantic feature, the accuracy of the mobile robot in determining its own position can be significantly improved. At the same time, by setting conditions to screen effective matches, it can filter out mis-matches to some extent, and enhance the stability and adaptability of determining the position in complex and variable environments.

[0095] Step c3, adding the matched detection semantic feature and the map semantic feature to the third semantic feature set. Specifically, adding each pair of matched detection semantic feature and map semantic feature to the third semantic feature set. By determining the third semantic feature set, strictly matching the actual detected detection semantic feature and the map semantic feature in the visual map, the accuracy of the mobile robot in determining its own position can be significantly improved.

[0096] Step b6, based on the third semantic feature set, using an optimization algorithm to optimize the first positioning data to obtain the second positioning data.

[0097] Specifically, the above step b6 includes:

[0098] Step d1, for any map semantic feature in the third semantic feature set, obtaining the detection semantic feature matched with the map semantic feature from the third semantic feature set. Specifically, the third semantic feature set is a one-to-one correspondence between the detection semantic feature and the map semantic feature.

[0099] Step d2, determining the pixel difference between the two-dimensional pixel position in the detection semantic feature and the two-dimensional pixel position in the map semantic feature. Specifically, the two-dimensional pixel position of the detection semantic feature is the predicted position, and the two-dimensional pixel position of the map semantic feature is the actual position obtained by the visual map. The difference between the two two-dimensional pixel positions can reflect the accuracy of the prediction, so as to better guide the process of determining the position based on the difference, so as to improve the accuracy and reliability of the positioning.

[0100] Step d3, optimizing the pixel difference, taking the minimization of the pixel difference as the optimization target, based on the first positioning data to obtain the second positioning data. Specifically, taking the minimization of the pixel difference as the optimization target, by adjusting the parameters corresponding to the first positioning data, based on the first positioning data after each adjustment, the map semantic feature is re-projected to obtain the new two-dimensional pixel position corresponding to the map semantic feature. Using the iteration or filtering method to repeat the above process, the minimization of the pixel difference is realized, that is, the two pixel positions are aligned as much as possible, so as to obtain more accurate second positioning data, and improve the accuracy and reliability of the positioning.

[0101] At step S4024, based on the odometer data of the mobile robot at the current time, the second positioning data is updated by using an odometer interpolation algorithm to obtain target positioning data of the mobile robot at the current time. Specifically, after obtaining the second positioning data, the odometer data of the mobile robot obtained by the communication module of the mobile robot positioning device is fused with the second positioning data by using an odometer interpolation algorithm such as Kalman filtering or particle filtering. Not only the image collected in real time is used, but also the dynamics model of the mobile robot is integrated to improve the stability and accuracy of positioning, and finally the target positioning data is obtained.

[0102] At step S403, the target positioning data of the mobile robot at the current time is sent to the mobile robot. For details, please refer to step S303 of the embodiment shown in FIG. 3, which will not be repeated here.

[0103] The mobile robot positioning method provided by the embodiment of the present application obtains the first top view image of the mobile robot at the current time, avoids collecting the environmental information of the surrounding low area with high dynamic change, analyzes the first top view image for positioning, improves the accuracy and reliability of positioning, and directly sends the positioning data to the mobile robot to ensure that the mobile robot can obtain the position information in real time.

[0104] The embodiment of the present application also provides a computer device. Please refer to FIG. 4, which is a structural schematic diagram of a computer device according to an optional embodiment of the present application. As shown in FIG. 4, the computer device comprises one or more processors 10, a memory 20, and an interface for connecting various components, including a high-speed interface and a low-speed interface. Various components are communicatively connected to each other by using different buses, and can be installed on a common mainboard or in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in the memory or on the memory to display graphical information of a GUI on an external input / output device such as a display device coupled to the interface. In some optional embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple storage devices, if necessary. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). One processor 10 is taken as an example in FIG. 4.

[0105] The processor 10 can be a central processor, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic gate array, a general array logic, or any combination thereof.

[0106] The memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform the methods illustrated by the above embodiments.

[0107] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function, and the like. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely from the processor 10, which can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0108] The memory 20 can include a volatile memory such as a random access memory, and can further include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk, and can further include a combination of the above-mentioned kinds of memories.

[0109] The computer device further includes a communication interface 30 for communication of the computer device with other devices or communication networks.

[0110] The embodiments of the present application also provide a computer readable storage medium. The above-mentioned methods according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or implemented as computer code originally stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded to a local storage medium, so that the methods described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, and the like. Alternatively, the storage medium can further include a combination of the above-mentioned kinds of memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods illustrated by the above embodiments are implemented.

[0111] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, can invoke or provide methods and / or technical solutions according to the present application. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source files, executable files, installation package files and the like, and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0112] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A mobile robot positioning apparatus, characterized by comprising: The device comprises an infrared sensor, an infrared light supplement light source device, a computing processing module, a power module and a communication module. The infrared sensor is used to collect the top view image of the mobile robot. The infrared light supplement light source device is used to emit infrared light to supplement the light for the infrared sensor. The computing processing module is used to position the mobile robot based on the top view image obtained by the infrared sensor to obtain the positioning data of the mobile robot. The power module is used to supply power for the infrared light supplement light source device and the computing processing module. The communication module is used to send the positioning data of the mobile robot to the mobile robot.

2. The apparatus of claim 1, wherein, The computing processing module comprises an image processing unit, an operation processing unit and a neural network processing unit. The image processing unit is used to optimize the image quality of the top view image. The neural network processing unit is used to detect the optimized top view image by using a neural network to obtain all semantic features in the optimized top view image, any semantic feature comprising a category to which the semantic feature belongs and a two-dimensional pixel position corresponding to the semantic feature. The operation processing unit is used to position the mobile robot based on all semantic features of the optimized top view image to obtain the positioning data of the mobile robot.

3. A mobile robot positioning method characterized by, The method applied to the mobile robot positioning device of any one of claims 1-2, the method comprising: obtaining a first top view image of the mobile robot at a current time; positioning based on the first top view image to obtain target positioning data of the mobile robot at the current time; sending the target positioning data of the mobile robot at the current time to the mobile robot.

4. The method of claim 3, wherein, Before the positioning based on the first top view image to obtain the target positioning data of the mobile robot at the current time, the method further comprises: loading a visual map of a working area of the mobile robot.

5. The method of claim 4, wherein, The positioning based on the first top view image to obtain the target positioning data of the mobile robot at the current time comprises: obtaining positioning data of the mobile robot at a previous time of the current time, odometer data of the mobile robot at the current time and a target timestamp of the first top view image; determining first positioning data of the mobile robot corresponding to the target timestamp based on the positioning data of the mobile robot at the previous time of the current time, the odometer data of the mobile robot at the current time and the target timestamp of the first top view image; optimizing the first positioning data based on the first top view image and the visual map to obtain second positioning data of the mobile robot corresponding to the target timestamp; updating the second positioning data based on the odometer data of the mobile robot at the current time by using an odometer interpolation algorithm to obtain the target positioning data of the mobile robot at the current time.

6. The method of claim 5, wherein, The visual map comprises all map semantic features existing in the working area of the mobile robot, any map semantic feature comprising a category to which the map semantic feature belongs and a spatial three-dimensional position corresponding to the map semantic feature. The first positioning data is optimized based on the first top view image and the visual map to obtain second positioning data of the mobile robot at a second time stamp corresponding to the target time stamp, including: optimizing the image quality of the first top view image to obtain a second top view image; using a neural network to detect the second top view image to obtain a first semantic feature set, the first semantic feature set including all detected semantic features in the second top view image, and any detected semantic feature including a category to which the detected semantic feature belongs and a two-dimensional pixel position corresponding to the detected semantic feature; for any map semantic feature in the visual map, based on the first positioning data, the spatial three-dimensional position corresponding to the map semantic feature is re-projected to a two-dimensional image space to obtain a two-dimensional pixel position corresponding to the map semantic feature; after re-projecting the visual map, a second semantic feature set is determined, the second semantic feature set including all map semantic features within a preset range of the first positioning data; matching the first semantic feature set and the second semantic feature set to obtain a third semantic feature set that matches successfully; based on the third semantic feature set, an optimization algorithm is used to optimize the first positioning data to obtain the second positioning data.

7. The method of claim 6, wherein, The first semantic feature set and the second semantic feature set are matched to obtain a third semantic feature set that matches successfully, including: for any detected semantic feature in the first semantic feature set, all map semantic features in the second semantic feature set are traversed; if there is any map semantic feature, the distance between the two-dimensional pixel position corresponding to the map semantic feature and the two-dimensional pixel position corresponding to the detected semantic feature is less than a preset threshold, and the category to which the map semantic feature belongs is the same as the category to which the detected semantic feature belongs, the detected semantic feature and the map semantic feature match successfully; the detected semantic feature and the map semantic feature that match successfully are added to the third semantic feature set.

8. The method of claim 6, wherein, Based on the third semantic feature set, an optimization algorithm is used to optimize the first positioning data to obtain the second positioning data, including: for any map semantic feature in the third semantic feature set, a detected semantic feature that matches successfully with the map semantic feature is obtained from the third semantic feature set; determining the pixel difference between the two-dimensional pixel position in the detected semantic feature and the two-dimensional pixel position in the map semantic feature; optimizing the pixel difference, with the minimization of the pixel difference as the optimization objective, based on the first positioning data to obtain the second positioning data.

9. A computer device, comprising: including: a memory and a processor, the memory and the processor being communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the mobile robot positioning method of any one of claims 3 to 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to execute the mobile robot positioning method of any one of claims 3-8.

Images