Mobile robot positioning device, method and equipment and storage medium

Abstract

The invention relates to the technical field of mobile robots, and discloses a mobile robot positioning device, method and equipment and a storage medium, the device comprises an infrared sensor, an infrared light supplementing light source device, a calculation processing module, a power supply module and a communication module; the infrared sensor collects a top view image of the mobile robot; the infrared light supplementing light source device emits infrared light to supplement light for the infrared sensor; the calculation processing module positions the mobile robot based on the top view image to obtain positioning data of the mobile robot; the power supply module supplies power to the infrared light supplementing light source device and the calculation processing module; and the communication module sends the positioning data of the mobile robot to the mobile robot. Through the combination of all parts of the mobile robot positioning device, the quality of the top view image is improved, positioning is determined based on the high-quality top view image, stable positioning can be carried out under different illumination conditions, the influence of light is reduced, and the positioning accuracy and robustness of the mobile robot are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robots, and in particular to a mobile robot positioning device, method, equipment and storage medium. Background Art

[0002] Mobile robots are robots that can move automatically without human direct operation and complete specific tasks. They have been applied in all aspects of production and life. Among them, the precise positioning of mobile robots is an important basis for completing tasks, such as making mobile robots in the correct position to perform tasks, avoiding task failures or errors caused by position errors, making mobile robot path planning more efficient, and improving work efficiency. Therefore, how to accurately position mobile robots is a problem that needs to be solved.

[0003] At present, the existing mobile robot positioning device may cause optical problems such as overexposure and blur in the images collected by the camera due to changes in lighting conditions, resulting in the inability to clearly capture feature points. If the positioning is determined based on the image, the obtained mobile robot positioning has low precision and accuracy. Summary of the invention

[0004] In view of this, the present invention provides a mobile robot positioning device, method, equipment and storage medium to solve the problem of low precision and low accuracy of mobile robot positioning.

[0005] In a first aspect, the present invention provides a mobile robot positioning device, which includes an infrared sensor, an infrared fill light source device, a computing and processing module, a power module and a communication module;

[0006] The infrared sensor is used to collect the top view image of the mobile robot;

[0007] The infrared fill light source device is used to emit infrared light to fill light for the infrared sensor;

[0008] The computing and processing module is used to locate the mobile robot based on the top view image acquired by the infrared sensor to obtain the positioning data of the mobile robot;

[0009] The power module is used to supply power to the infrared fill light source device and the computing processing module;

[0010] The communication module is used to send the positioning data of the mobile robot to the mobile robot.

[0011] The mobile robot positioning device provided by the embodiment of the present invention can improve the quality of the collected top-view image through the infrared fill light source device, computing and processing module, power supply module and communication module in the mobile robot positioning device, and then perform positioning based on the high-quality top-view image, thereby further improving the accuracy of the mobile robot positioning. At the same time, by sending the positioning data to the mobile robot, the mobile robot performs subsequent operations based on the accurate positioning data. Compared with the existing mobile robot positioning device, it can perform stable positioning under different lighting conditions, reduces the influence of light in the process of determining positioning, thereby obtaining accurate mobile robot positioning data, and improving the accuracy and robustness of mobile robot positioning.

[0012] In an optional embodiment, the computing processing module includes an image processing unit, an operation processing unit and a neural network processing unit;

[0013] An image processing unit is used to optimize the image quality of the top view image;

[0014] The neural network processing unit is used to detect the optimized top view image using a neural network to obtain all semantic features in the optimized top view image, wherein any semantic feature includes a category to which the semantic feature belongs and a two-dimensional pixel position corresponding to the semantic feature;

[0015] The operation processing unit is used to locate the mobile robot based on all semantic features of the optimized top view image to obtain the positioning data of the mobile robot.

[0016] The mobile robot positioning device provided in the embodiment of the present invention optimizes the top view image and then performs semantic analysis through an image processing unit, a neural network processing unit and an operation processing unit, and obtains positioning data using semantic features obtained by the semantic analysis, thereby enhancing the positioning accuracy of the mobile robot in a complex environment.

[0017] In a second aspect, the present invention provides a mobile robot positioning method, which is applied to the mobile robot positioning device of the first aspect or any corresponding embodiment thereof, and the method comprises:

[0018] Obtain the first top view image of the mobile robot at the current moment;

[0019] Perform positioning based on the first top view image to obtain target positioning data of the mobile robot at the current moment;

[0020] The target positioning data of the mobile robot at the current moment is sent to the mobile robot.

[0021] The mobile robot positioning method provided by the embodiment of the present invention obtains the first top view image of the mobile robot at the current moment, avoids collecting highly dynamically changing environmental information of the surrounding lower areas, analyzes the first top view image for positioning, thereby improving the accuracy and reliability of positioning, and sends the positioning data directly to the mobile robot, ensuring that the mobile robot can obtain its own position information in real time.

[0022] In an optional implementation, before performing positioning based on the first top view image to obtain target positioning data of the mobile robot at the current moment, the method further includes:

[0023] Load a visual map of the mobile robot's work area.

[0024] The mobile robot positioning method provided by the embodiment of the present invention provides support for subsequent mobile robot positioning by loading a visual map of the working area of ​​the mobile robot.

[0025] In an optional implementation, performing positioning based on the first top view image to obtain target positioning data of the mobile robot at the current moment includes:

[0026] Obtaining the positioning data of the mobile robot at the previous moment before the current moment, the odometer data of the mobile robot at the current moment, and the target timestamp of the first top view image;

[0027] Determine first positioning data of the mobile robot corresponding to the target timestamp based on the positioning data of the mobile robot at the previous moment of the current moment, the odometer data of the mobile robot at the current moment, and the target timestamp of the first top view image;

[0028] Based on the first top view image and the visual map, the first positioning data is optimized to obtain second positioning data of the mobile robot corresponding to the target timestamp;

[0029] Based on the odometer data of the mobile robot at the current moment, an odometer interpolation algorithm is adopted to update the second positioning data to obtain the target positioning data of the mobile robot at the current moment.

[0030] The mobile robot positioning method provided by the embodiment of the present invention obtains the first positioning data of the current moment with a preliminary estimate by combining the positioning data of the previous moment with the odometer data of the current moment. The preliminary estimate can reduce the matching range of the subsequent map semantic features and improve the speed of the positioning calculation. Then, the first top view image and the visual map are optimized, and the first positioning data is corrected using the detailed features of the environment, thereby improving the accuracy and robustness of the positioning. By considering the target timestamp of the first top view image, the time consistency between the visual data and the motion data is ensured, and the positioning error caused by the time deviation is avoided. At the same time, the odometer interpolation algorithm is adopted, and the positioning result can be fine-tuned according to the robot's immediate mobile state to maintain the real-time and accuracy of the positioning.

[0031] In an optional embodiment, the visual map includes all map semantic features present in the working area of ​​the mobile robot, and any map semantic feature includes a category to which the map semantic feature belongs and a spatial three-dimensional position corresponding to the map semantic feature;

[0032] Based on the first top view image and the visual map, the first positioning data is optimized to obtain second positioning data corresponding to the target timestamp of the mobile robot, including:

[0033] optimizing the image quality of the first top view image to obtain a second top view image;

[0034] Using a neural network, the second top view image is detected to obtain a first semantic feature set, the first semantic feature set includes all detected semantic features in the second top view image, and any detected semantic feature includes a category to which the detected semantic feature belongs and a two-dimensional pixel position corresponding to the detected semantic feature;

[0035] For any map semantic feature in the visual map, based on the first positioning data, reproject the corresponding three-dimensional spatial position in the map semantic feature to the two-dimensional image space to obtain the two-dimensional pixel position corresponding to the map semantic feature;

[0036] After the visual map is reprojected, a second semantic feature set is determined, the second semantic feature set including all map semantic features within a preset range of the first positioning data;

[0037] Matching the first semantic feature set with the second semantic feature set to obtain a third semantic feature set that matches successfully;

[0038] Based on the third semantic feature set, an optimization algorithm is used to optimize the first positioning data to obtain second positioning data.

[0039] The mobile robot positioning method provided by the embodiment of the present invention optimizes the first top view image and uses neural network detection to accurately extract semantic features to obtain a first semantic feature set. Even in the case of changes in ambient lighting, occlusion or interference from dynamic objects, the neural network can still stably extract semantic features, thereby improving the accuracy and robustness of positioning data obtained based on semantic features. The first semantic feature set is matched with the second semantic feature set screened from the visual map to obtain a successfully matched third semantic feature set. The optimization algorithm specifically processes the successfully matched semantic feature set to optimize the positioning data and improve the accuracy of the positioning data.

[0040] In an optional implementation, matching the first semantic feature set with the second semantic feature set to obtain a third semantic feature set that matches successfully includes:

[0041] For any detection semantic feature in the first semantic feature set, traverse all map semantic features in the second semantic feature set;

[0042] If there is any map semantic feature, when the distance between the two-dimensional pixel position corresponding to the map semantic feature and the two-dimensional pixel position corresponding to the detection semantic feature is less than a preset threshold, and the category to which the map semantic feature belongs is the same as the category to which the detection semantic feature belongs, the detection semantic feature is successfully matched with the map semantic feature;

[0043] The successfully matched detection semantic features and map semantic features are added to the third semantic feature set.

[0044] The mobile robot positioning method provided by the embodiment of the present invention performs matching by setting a preset threshold of the distance and the condition of the same semantic feature category, thereby effectively filtering out irrelevant feature interference, ensuring the high relevance and accuracy of the matching results, and reducing the possibility of mismatching. It serves as the basis for achieving subsequent positioning optimization and improves the reliability and robustness of positioning.

[0045] In an optional implementation, based on the third semantic feature set, an optimization algorithm is used to optimize the first positioning data to obtain the second positioning data, including:

[0046] For any map semantic feature in the third semantic feature set, obtaining a detection semantic feature that successfully matches the map semantic feature from the third semantic feature set;

[0047] determining a pixel difference between a two-dimensional pixel position in the detected semantic feature and a two-dimensional pixel position in the map semantic feature;

[0048] The pixel difference is optimized, with minimization of the pixel difference as an optimization goal, and optimization is performed based on the first positioning data to obtain the second positioning data.

[0049] The mobile robot positioning method provided by the embodiment of the present invention corrects the positioning data by utilizing the difference between the two-dimensional pixel position after reprojection of the map semantic features and the two-dimensional pixel position of the detected semantic features, with minimization of the pixel-level difference as the optimization goal, thereby ensuring the accuracy and reliability of the positioning data.

[0050] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the mobile robot positioning method of the second aspect or any corresponding embodiment thereof by executing the computer instructions.

[0051] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the mobile robot positioning method of the second aspect or any corresponding embodiment thereof.

[0052] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the mobile robot positioning method of the second aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 is a schematic diagram of a mobile robot positioning device according to an embodiment of the present invention;

[0055] Figure 2 is a schematic diagram of the placement position of a mobile robot positioning device according to an embodiment of the present invention;

[0056] Figure 3 is a flow chart of a mobile robot positioning method according to an embodiment of the present invention;

[0057] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0059] Since the precise positioning of the mobile robot is an important basis for completing various tasks, how to accurately position the mobile robot is a problem that needs to be solved. The mobile robot positioning device in the related art cannot solve the impact of changes in lighting conditions on the determination of positioning, resulting in low accuracy and precision of the mobile robot positioning. The mobile robot positioning device provided in the embodiment of the present invention can improve the quality of the collected top view image through the combination of various parts in the mobile robot positioning device, and then perform positioning based on the high-quality top view image, further improving the accuracy and robustness of the mobile robot positioning, and at the same time, by sending the positioning data to the mobile robot, the mobile robot performs subsequent operations based on the accurate positioning data.

[0060] In this embodiment, a mobile robot positioning device is provided. Figure 1 As shown, the device includes an infrared sensor 1, an infrared fill light source device 2, a computing and processing module 3, a power module 4 and a communication module 5; the infrared sensor 1 is used to collect the top view image of the mobile robot; the infrared fill light source device 2 is used to emit infrared light to fill light for the infrared sensor 1; the computing and processing module 3 is used to position the mobile robot based on the top view image acquired by the infrared sensor 1 to obtain the positioning data of the mobile robot; the power module 4 is used to supply power to the infrared fill light source device 2 and the computing and processing module 3; the communication module 5 is used to send the positioning data of the mobile robot to the mobile robot.

[0061] Specifically, the infrared sensor 1 is equipped with an infrared bandpass filter, which can effectively filter the visible light in the light. Compared with using an ordinary camera to collect images, it can effectively reduce the influence of changes in lighting conditions on determining positioning. In addition, the infrared sensor 1 has an automatic exposure function, that is, it automatically adjusts the exposure time according to the brightness of the collected image to ensure that the collected image has a stable brightness, further reducing the influence of lighting conditions. The infrared fill light source device 2 is arranged on both sides of the infrared sensor 1, and the infrared light emitted by it can achieve fill light for the infrared sensor 1. Optionally, the embodiment of the present invention uses two groups of infrared fill light source devices 2 as an example for explanation. In actual situations, only one group of infrared fill light source devices 2 can be installed on one side of the infrared sensor 1. The infrared fill light source device 2 adopts an instantaneous fill light mode, and fill light is emitted by a fill light light source. The wavelength of the fill light light source can be selected from infrared bands such as 850nm, 905nm or 940nm. Among them, the fill light light source can be an LED (Light Emitting Diode), a laser light source or other light sources, and the embodiment of the present invention does not limit this. The laser light source can be a VCSEL (Vertical-cavity surface-emitting laser). The lighting time of the fill light source is synchronized with the shutter exposure time of the infrared sensor 1. Based on the infrared fill light source device 2, the mobile robot positioning device has the ability to locate in a completely dark or dark working environment. At the same time, combined with the automatic exposure function of the infrared sensor 1, the brightness of the top view image collected during the on / off light and day / night can be kept at a constant brightness, which can not only reduce the difficulty of subsequent image processing, but also improve the reliability of positioning. The calculation and processing module 3 is the main functional module for determining the positioning of the mobile robot positioning device, which is used to obtain the precise positioning of the mobile robot. The power supply module 4 supplies power to the infrared fill light source device 2 and the calculation and processing module 3 respectively after the power supply is stabilized. Since the infrared fill light source device 2 is instantaneous fill light, its instantaneous current and power are large, so the power supply module 4 needs to provide sufficient power output, and the power supply isolation needs to be done well to prevent the instantaneous fill light current and voltage fluctuations from affecting the stable operation of the calculation and processing module 3. The communication module 5 uses a Gigabit Ethernet communication interface to communicate with the mobile robot, obtain the odometer data and other information of the mobile robot, and send the positioning data determined by the mobile robot positioning device to the mobile robot. Optionally, it can be sent to the navigation module of the mobile robot so that the mobile robot can clearly know the current location, so as to carry out path planning and task scheduling, etc.Through the combination of multiple parts in the mobile robot positioning device, the quality of the collected top view image can be improved, and the positioning can be determined based on the high-quality top view image, thereby further improving the accuracy of the mobile robot positioning. At the same time, by sending the positioning data to the mobile robot, the mobile robot can perform subsequent operations based on the accurate positioning data.

[0062] In some optional implementations, corresponding host computer software can be developed for the communication module 5. After the host computer software is successfully connected to the communication module 5, the communication module 5 has the functions of loading visual maps, updating the weights of the neural network, monitoring odometer data, obtaining the positioning data calculated by the calculation processing module 3, and obtaining the operation log of the mobile robot positioning device, so as to facilitate monitoring and debugging by engineers.

[0063] Figure 2 Schematic diagram of the placement position of the mobile robot positioning device according to an embodiment of the present invention. Figure 2 As shown, Figure 2 Reference numeral a is a roller transport robot, and the mobile robot positioning device is installed at the top position shown by reference numeral 6, i.e., the top of the vehicle front upright pole. Figure 2 The reference numeral b in the figure is an automatic forklift robot, and the mobile robot positioning device is installed at the top position shown by reference numeral 7, that is, the top of the forklift front faucet frame. When the mobile robot positioning device is installed, the infrared sensor 1 is installed vertically upward, so that reliable and fixed feature information in the upper field of view of the mobile robot during movement can be collected to avoid interference from the surrounding environment, and further improve the reliability and accuracy of positioning. It should be noted that Figure 2 The two mobile robots shown are only examples. The mobile robot positioning device can also be installed on the top of any mobile robot that needs to be positioned, such as a sweeping robot, a traction robot, etc., as needed. The embodiment of the present invention is not limited to this.

[0064] In some optional embodiments, the computing and processing module 3 includes an image processing unit, an operation processing unit and a neural network processing unit; the image processing unit is used to optimize the image quality of the top view image; the neural network processing unit is used to detect the optimized top view image using a neural network to obtain all semantic features in the optimized top view image, and any semantic feature includes the category to which the semantic feature belongs and the two-dimensional pixel position corresponding to the semantic feature; the operation processing unit is used to position the mobile robot based on all semantic features of the optimized top view image to obtain the positioning data of the mobile robot.

[0065] Specifically, the mobile robot positioning device uses an ARM (Advanced RISC Machine, ARM processor) embedded platform as a computing and processing module. The ARM embedded platform includes a CPU (Central Processing Unit, central processing unit) and an NPU (Neural Processing Unit, neural network processor) in hardware. Among them, the above-mentioned image processing unit is a dedicated image processing program running on the CPU, the operation processing unit is a positioning program running on the CPU, and the neural network processing unit is a deep learning program running on the NPU. The image signal processing unit is responsible for processing the original top view image collected by the infrared sensor 1, such as linear correction, noise removal, bad pixel removal, interpolation, white balance, automatic exposure control, etc., to achieve image quality optimization of the top view image. The neural network processing unit has deep learning capabilities, so it needs to have a computing power of at least greater than 1TOPS. The input of the neural network processing unit is a two-dimensional image. In the embodiment of the present invention, the optimized top view image is input to the neural network processing unit, and the optimized top view image is convolved by the convolution layer to find specific edges, lines or texture patterns, and then the activation function is applied to the result obtained by the convolution to learn more complex features in the image, and then the spatial dimension of the image is reduced by the pooling operation of the pooling layer, while maintaining important features, to obtain multiple semantic feature categories included in the optimized top view image, and to aggregate pixels representing the same semantic feature category. In addition, post-processing technology, such as conditional random fields or more advanced instance segmentation algorithms, is used to separate semantic features belonging to the same semantic feature category but different positions in the top view image, further improve the accuracy of the final output, ensure the accurate separation of different instances, and the final output is all semantic features in the optimized top view image. For example, the neural network processing unit detects any optimized top view image and detects that the top view image includes three semantic features, and the categories to which the three semantic features belong are: fluorescent lamp, smoke alarm, and air conditioner. For "fluorescent lamp", all pixels with the semantic feature category of "fluorescent lamp" in the optimized top view image are aggregated, and the central pixel of at least one pixel obtained from the aggregation or any corner point on the bounding box formed by at least one pixel is used as the two-dimensional pixel position corresponding to "fluorescent lamp". Optionally, other pixels can also be selected as the two-dimensional pixel position corresponding to the semantic feature, and the embodiment of the present invention does not limit this. By determining the two-dimensional pixel position for the category to which each semantic feature belongs, it can be obtained that the semantic features of the top view image are "fluorescent lamp" and the two-dimensional pixel position corresponding to "fluorescent lamp", "smoke alarm" and the two-dimensional pixel position corresponding to "smoke alarm", and "air conditioning" and the two-dimensional pixel position corresponding to "air conditioning".The calculation processing unit performs positioning based on the semantic features of the top view image detected by the neural network processing unit to obtain accurate positioning data.

[0066] The mobile robot positioning device provided by the embodiment of the present invention can improve the quality of the collected top-view image through the infrared fill light source device, computing and processing module, power supply module and communication module in the mobile robot positioning device, and then perform positioning based on the high-quality top-view image, thereby further improving the accuracy of the mobile robot positioning. At the same time, by sending the positioning data to the mobile robot, the mobile robot performs subsequent operations based on the accurate positioning data. Compared with the existing mobile robot positioning device, it can perform stable positioning under different lighting conditions, reduces the influence of light in the process of determining positioning, thereby obtaining accurate mobile robot positioning data, and improving the accuracy and robustness of mobile robot positioning.

[0067] According to an embodiment of the present invention, an embodiment of a mobile robot positioning method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0068] In this embodiment, a mobile robot positioning method is provided, which can be used for the above-mentioned mobile robot positioning device. Figure 3 is a flow chart of a mobile robot positioning method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0069] Step S301, obtaining a first top view image of the mobile robot at the current moment. Specifically, based on the infrared sensor in the mobile robot positioning device, the first top view image of the mobile robot at the current moment is collected to avoid collecting highly dynamically changing environmental information of the surrounding lower area, ensure the reliability of the first top view image, and provide support for the subsequent determination of the current position of the mobile robot.

[0070] Step S302, positioning is performed based on the first top view image to obtain target positioning data of the mobile robot at the current moment. Specifically, since the first top view image includes various feature information, the first top view image is analyzed and positioning is performed based on the first top view image by the calculation processing module of the mobile robot positioning device, so that more accurate target positioning data can be obtained.

[0071] Step S303, the target positioning data of the mobile robot at the current moment is sent to the mobile robot. Specifically, the target positioning data includes the position information and posture information of the mobile robot at the current moment. The target positioning data of the mobile robot is sent to the mobile robot through the communication module in the mobile robot positioning device, so that the mobile robot can clearly understand its own position and posture, which is convenient for subsequent operations such as path planning and path scheduling.

[0072] The mobile robot positioning method provided by the embodiment of the present invention obtains the first top view image of the mobile robot at the current moment, avoids collecting highly dynamically changing environmental information of the surrounding lower areas, analyzes the first top view image for positioning, thereby improving the accuracy and reliability of positioning, and sends the positioning data directly to the mobile robot, ensuring that the mobile robot can obtain its own position information in real time.

[0073] In this embodiment, a mobile robot positioning method is provided, which can be used for the above-mentioned mobile robot positioning device. The method specifically includes the following steps:

[0074] Step S401, obtain the first top view image of the mobile robot at the current moment. Figure 3 Step S301 of the illustrated embodiment will not be described in detail here.

[0075] In some optional implementations, before the above step S401, the method further includes:

[0076] Step a1, loading a visual map of the working area of ​​the mobile robot. Specifically, before the mobile robot is positioned, the visual map can be loaded in advance into the computing unit of the mobile robot positioning device so that the computing unit can perform positioning. The specific process of obtaining the visual map is: manually controlling the mobile robot to move one circle in its working area, collecting image information in the working area, using visual SLAM technology to generate a visual map of visual feature points in the entire working area based on the image information, and taking the mobile robot's moving starting position as the origin of the visual map, the spatial three-dimensional coordinates of all visual feature points relative to the origin can be generated, and each visual feature point has a descriptor for feature matching. Among them, the camera used to collect images can be a general camera, or an infrared sensor in the mobile robot positioning device used in the embodiment of the present invention, and the embodiment of the present invention does not limit this. The visual map includes at least one map semantic feature, and any map semantic feature includes the category to which the map semantic feature belongs and the spatial three-dimensional position corresponding to the map semantic feature. By loading the visual map of the working area of ​​the mobile robot, support is provided for subsequent mobile robot positioning.

[0077] In some optional embodiments, before performing the positioning of the mobile robot, the weight of the neural network used by the neural network processing unit in the computing and processing module of the mobile robot positioning device can also be obtained. The weight of the neural network can be obtained by pre-training. For example, in the process of obtaining the visual map, the image information of the working area of ​​the mobile robot collected is manually annotated, and the semantic features therein are annotated. The manually annotated semantic features are used as a training set, and the weight of the neural network is obtained by training using existing technologies. Alternatively, a large amount of image information of the working areas of various types of mobile robots is collected, and the weight of the neural network is obtained by training using existing technologies. At the same time, the image information of the working area where the mobile robot is currently located is used as incremental training to enhance the perception ability of the neural network and obtain the final weight of the neural network. In the process of determining the positioning of the mobile robot, the weight of the neural network obtained by pre-training can be used to improve the efficiency and accuracy of determining the positioning.

[0078] Step S402: performing positioning based on the first top view image to obtain target positioning data of the mobile robot at the current moment.

[0079] Specifically, the above step S402 includes:

[0080] Step S4021, obtain the positioning data of the mobile robot at the previous moment before the current moment, the odometer data of the mobile robot at the current moment, and the target timestamp of the first top view image. Specifically, based on the communication module of the mobile robot positioning device, the positioning data of the previous moment before the current moment and the odometer data of the current moment are obtained from the navigation module of the mobile robot. Among them, the odometer data reflects the change of the distance and direction of the mobile robot during the movement from the initial position. The target timestamp of the first top view image can be obtained based on the infrared sensor, and the target timestamp is used to identify the precise time of collecting the first top view image.

[0081] Step S4022, based on the positioning data of the mobile robot at the previous moment before the current moment, the odometer data of the mobile robot at the current moment, and the target timestamp of the first top view image, determine the first positioning data of the mobile robot corresponding to the target timestamp. Specifically, the first positioning data includes position information and posture information. Usually, a motion model of the mobile robot, such as the kinematic equation of a wheeled robot, is used to map the positioning data and odometer data of the previous moment to the position under the target timestamp, and time alignment and positioning estimation are achieved to obtain a preliminary estimated first positioning data. Optionally, other methods may be used to determine the first positioning data, such as integration or filtering algorithms, which are not limited in the embodiments of the present invention. By preliminarily estimating the first positioning data, the current positioning of the mobile robot can be roughly understood, providing a basis for subsequent precise positioning.

[0082] Step S4023, based on the first top view image and the visual map, the first positioning data is optimized to obtain the second positioning data of the mobile robot corresponding to the target timestamp, the visual map includes all map semantic features existing in the working area of ​​the mobile robot, and any map semantic feature includes the category to which the map semantic feature belongs and the spatial three-dimensional position corresponding to the map semantic feature.

[0083] Specifically, the above step S4023 includes:

[0084] Step b1, optimizing the image quality of the first top view image to obtain a second top view image. Specifically, based on the image processing unit of the computing processing module of the mobile robot positioning device, the first top view image is optimized to improve the image quality of the first top view image, and the second top view image is obtained to improve the accuracy of subsequent feature detection.

[0085] Step b2, using a neural network to detect the second top view image to obtain a first semantic feature set, the first semantic feature set including all detected semantic features in the second top view image, any detected semantic feature including the category to which the detected semantic feature belongs and the two-dimensional pixel position corresponding to the detected semantic feature. Specifically, using a neural network of a neural network processing unit, based on the weight of the neural network obtained by pre-training, referring to the detection process of the neural network processing unit, the second top view image is detected to obtain all detected semantic features included in the second top view image, thereby forming a first semantic feature set.

[0086] Step b3, for any map semantic feature in the visual map, based on the first positioning data, the corresponding spatial three-dimensional position in the map semantic feature is reprojected to the two-dimensional image space to obtain the two-dimensional pixel position corresponding to the map semantic feature. Specifically, for any map semantic feature in the visual map, the map semantic feature has been identified and located in the process of acquiring the visual map and has three-dimensional spatial coordinates. In order to perform subsequent positioning operations, it is necessary to first convert the spatial three-dimensional position corresponding to the map semantic feature into a coordinate system through the first positioning data, and then convert the corresponding coordinates after the coordinate system conversion into two-dimensional pixel coordinates in the two-dimensional image space through a projection transformation, such as a pinhole model projection, that is, reprojection is performed to obtain the two-dimensional pixel position in the two-dimensional image space after reprojection.

[0087] Step b4, after the visual map is reprojected, a second semantic feature set is determined, and the second semantic feature set includes all map semantic features within a preset range of the first positioning data. Specifically, the map semantic features within the preset range refer to a map semantic feature on a two-dimensional plane corresponding to the visual map, with the preliminarily estimated first positioning data as the center, defining a specific radius or boundary box to screen whether the two-dimensional pixel position corresponding to the map semantic feature falls within this spatial area, and all map semantic features falling within this spatial area are taken as the second semantic feature set. By determining the second semantic feature set, it is ensured that the analysis is concentrated on a local area closely related to the current position of the mobile robot, thereby improving the effectiveness and accuracy of positioning.

[0088] Step b5, matching the first semantic feature set with the second semantic feature set to obtain a third semantic feature set that matches successfully.

[0089] Specifically, the above step b5 includes:

[0090] Step c1, for any detected semantic feature in the first semantic feature set, traverse all map semantic features in the second semantic feature set. Specifically, the first semantic feature set obtained by detection is matched one by one with the second semantic feature set obtained based on the visual map. Through semantic feature matching, a more accurate positioning of the mobile robot on a known visual map can be identified.

[0091] Step c2: if there is any map semantic feature, when the distance between the two-dimensional pixel position corresponding to the map semantic feature and the two-dimensional pixel position corresponding to the detection semantic feature is less than a preset threshold, and the category to which the map semantic feature belongs is the same as the category to which the detection semantic feature belongs, the detection semantic feature matches the map semantic feature successfully. Specifically, there are two conditions for semantic feature matching: one is that the distance between the two-dimensional pixel positions of the two is less than a preset threshold, which ensures that only features that are close enough will be considered as candidate matches; the other is that the semantic feature categories are consistent, which means that the two represent the same type of object or environmental features. If both matching conditions are met, the two are considered to be matched successfully. Among them, the distance can be a Euclidean distance, and other distances can also be used. The embodiment of the present invention does not limit this. By performing a detailed match between the detection semantic feature and the pre-constructed map semantic feature, the accuracy of the mobile robot in determining its own positioning can be significantly improved. At the same time, by setting conditions to screen effective matches, it is possible to filter out mismatches to a certain extent, thereby enhancing the stability and adaptability of determining positioning in complex and changing environments.

[0092] Step c3, adding the successfully matched detection semantic features and map semantic features to the third semantic feature set. Specifically, each pair of successfully matched detection semantic features and map semantic features is used as the third semantic feature set. By determining the third semantic feature set and strictly matching the actually detected detection semantic features with the map semantic features in the visual map, the accuracy of the mobile robot in determining its own positioning can be significantly improved.

[0093] Step b6: Based on the third semantic feature set, an optimization algorithm is used to optimize the first positioning data to obtain second positioning data.

[0094] Specifically, the above step b6 includes:

[0095] Step d1, for any map semantic feature in the third semantic feature set, obtain a detection semantic feature that successfully matches the map semantic feature from the third semantic feature set. Specifically, the third semantic feature set contains one-to-one corresponding detection semantic features and map semantic features.

[0096] Step d2, determining the pixel difference between the two-dimensional pixel position in the detection semantic feature and the two-dimensional pixel position in the map semantic feature. Specifically, the two-dimensional pixel position of the detection semantic feature is the predicted position, and the two-dimensional pixel position of the map semantic feature is the actual position obtained through the visual map. The difference between the two two-dimensional pixel positions can reflect the accuracy of the prediction, so as to better guide the process of determining the positioning based on the difference, so as to improve the accuracy and reliability of the positioning.

[0097] Step d3, optimize the pixel difference, minimize the pixel difference as the optimization goal, optimize based on the first positioning data, and obtain the second positioning data. Specifically, optimize with the minimum pixel difference as the optimization goal, adjust the parameters corresponding to the first positioning data, and reproject the map semantic features based on the first positioning data after each adjustment to obtain a new two-dimensional pixel position corresponding to the map semantic feature. Repeat the above process using an iterative or filtering method to minimize the pixel difference, that is, to align the two pixel positions as much as possible, thereby obtaining more accurate second positioning data and improving the accuracy and reliability of positioning.

[0098] Step S4024, based on the odometer data of the mobile robot at the current moment, the odometer interpolation algorithm is used to update the second positioning data to obtain the target positioning data of the mobile robot at the current moment. Specifically, after obtaining the second positioning data, the odometer data of the mobile robot obtained by the communication module of the mobile robot positioning device is fused with the second positioning data through an odometer interpolation algorithm, such as a Kalman filter or a particle filter. Not only the images collected in real time are utilized, but also the dynamic model of the mobile robot is integrated, which improves the stability and accuracy of positioning, and finally obtains the target positioning data.

[0099] Step S403, sending the target positioning data of the mobile robot at the current moment to the mobile robot. Figure 3 Step S303 of the illustrated embodiment will not be described in detail here.

[0100] The mobile robot positioning method provided by the embodiment of the present invention obtains the first top view image of the mobile robot at the current moment, avoids collecting highly dynamically changing environmental information of the surrounding lower areas, analyzes the first top view image for positioning, thereby improving the accuracy and reliability of positioning, and sends the positioning data directly to the mobile robot, ensuring that the mobile robot can obtain its own position information in real time.

[0101] The embodiment of the present invention also provides a computer device, see Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0102] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0103] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0104] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0106] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0107] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0108] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0109] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A mobile robot positioning device, characterized in that: The device includes an infrared sensor, an infrared fill light source device, a computing and processing module, a power module and a communication module; The infrared sensor is used to collect the top view image of the mobile robot; The infrared fill light source device is used to emit infrared light to provide fill light for the infrared sensor; The computing and processing module is used to locate the mobile robot based on the top view image acquired by the infrared sensor to obtain the positioning data of the mobile robot; The power supply module is used to supply power to the infrared fill light source device and the calculation processing module; The communication module is used to send the positioning data of the mobile robot to the mobile robot.

2. The device according to claim 1, characterized in that The computing processing module includes an image processing unit, an operation processing unit and a neural network processing unit; The image processing unit is used to optimize the image quality of the top view image; The neural network processing unit is used to detect the optimized top view image using a neural network to obtain all semantic features in the optimized top view image, wherein any semantic feature includes the category to which the semantic feature belongs and the two-dimensional pixel position corresponding to the semantic feature; The operation processing unit is used to locate the mobile robot based on all semantic features of the optimized top view image to obtain the positioning data of the mobile robot.

3. A mobile robot positioning method, characterized in that: Applied to the mobile robot positioning device according to any one of claims 1 to 2, the method comprising: Obtain the first top view image of the mobile robot at the current moment; Performing positioning based on the first top view image to obtain target positioning data of the mobile robot at the current moment; The target positioning data of the mobile robot at the current moment is sent to the mobile robot.

4. The method according to claim 3, characterized in that Before performing positioning based on the first top view image to obtain target positioning data of the mobile robot at the current moment, the method further includes: A visual map of the mobile robot's working area is loaded.

5. The method according to claim 4, characterized in that The performing positioning based on the first top view image to obtain target positioning data of the mobile robot at the current moment includes: Acquire positioning data of the mobile robot at a moment before the current moment, odometer data of the mobile robot at the current moment, and a target timestamp of the first top view image; Determine first positioning data of the mobile robot corresponding to the target timestamp based on the positioning data of the mobile robot at the previous moment before the current moment, the odometer data of the mobile robot at the current moment, and the target timestamp of the first top-view image; Based on the first top view image and the visual map, optimizing the first positioning data to obtain second positioning data of the mobile robot corresponding to the target timestamp; Based on the odometer data of the mobile robot at the current moment, an odometer interpolation algorithm is adopted to update the second positioning data to obtain the target positioning data of the mobile robot at the current moment.

6. The method according to claim 5, characterized in that The visual map includes all map semantic features present in the working area of ​​the mobile robot, and any map semantic feature includes the category to which the map semantic feature belongs and the spatial three-dimensional position corresponding to the map semantic feature; The optimizing the first positioning data based on the first top view image and the visual map to obtain the second positioning data of the mobile robot corresponding to the target timestamp includes: optimizing the image quality of the first top view image to obtain a second top view image; Using a neural network, detecting the second top-view image to obtain a first semantic feature set, wherein the first semantic feature set includes all detected semantic features in the second top-view image, and any detected semantic feature includes a category to which the detected semantic feature belongs and a two-dimensional pixel position corresponding to the detected semantic feature; For any map semantic feature in the visual map, based on the first positioning data, reproject the spatial three-dimensional position corresponding to the map semantic feature to the two-dimensional image space to obtain the two-dimensional pixel position corresponding to the map semantic feature; After the visual map is reprojected, determining a second semantic feature set, the second semantic feature set comprising all map semantic features within a preset range of the first positioning data; Matching the first semantic feature set with the second semantic feature set to obtain a third semantic feature set that matches successfully; Based on the third semantic feature set, an optimization algorithm is used to optimize the first positioning data to obtain the second positioning data.

7. The method according to claim 6, characterized in that The matching of the first semantic feature set and the second semantic feature set to obtain a third semantic feature set with successful matching includes: For any detection semantic feature in the first semantic feature set, traverse all map semantic features in the second semantic feature set; If there is any map semantic feature, the detection semantic feature is successfully matched with the map semantic feature when the distance between the two-dimensional pixel position corresponding to the map semantic feature and the two-dimensional pixel position corresponding to the detection semantic feature is less than a preset threshold, and the category to which the map semantic feature belongs is the same as the category to which the detection semantic feature belongs; The successfully matched detection semantic features and map semantic features are added to the third semantic feature set.

8. The method according to claim 6, characterized in that The step of optimizing the first positioning data based on the third semantic feature set by using an optimization algorithm to obtain the second positioning data includes: For any map semantic feature in the third semantic feature set, obtaining a detection semantic feature that successfully matches the map semantic feature from the third semantic feature set; determining a pixel difference between a two-dimensional pixel position in the detected semantic feature and a two-dimensional pixel position in the map semantic feature; The pixel difference is optimized, with minimization of the pixel difference as an optimization goal, and optimization is performed based on the first positioning data to obtain the second positioning data.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the mobile robot positioning method according to any one of claims 3 to 8 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the mobile robot positioning method according to any one of claims 3 to 8.

Images