Macro-micro composite visual positioning method and system based on mobile robot

Abstract

The invention relates to the technical field of robot positioning, in particular to a macro-micro composite visual positioning method and system based on a mobile robot. The macro-micro composite visual positioning method integrates an overlook global view angle and robot body vision, combines target object sensing data obtained by sensing an environment through a current mobile robot, associates a sensed identification result of a target object with a unique ID tag of an environment object communication area corresponding to the position, and establishes an exact corresponding relation. And environment understanding is realized, so that the mobile robot is endowed with semantic understanding capability on the environment, and decision making of the mobile robot is facilitated. And a multi-source sensing data fusion algorithm is constructed, so that the system implementation complexity is remarkably reduced while the positioning precision is ensured. Through the method, rapid identification and positioning of the indoor environment of the mobile robot are realized, the development process and hardware cost of the mobile robot are reduced, and the data processing amount of a mobile main control end of the mobile robot is reduced.

Description

Technical Field

[0001] This invention relates to the field of robot positioning technology, and in particular to a macro-micro composite visual positioning method and system based on mobile robots. Background Technology

[0002] Existing mobile robots not only possess the ability to move autonomously but also to perceive and analyze their working environment, making real-time responses and decisions. Currently, mobile robots primarily achieve map building and self-localization by fusing data from multiple onboard sensors, such as binocular cameras, inertial measurement units, and wheeled odometry. This localization method, based on the robot's own sensors, heavily relies on a high-performance hardware platform, resulting in high overall system costs, which may be a limiting factor in practical applications. Summary of the Invention

[0003] The main objective of this invention is to provide a macro-micro composite visual positioning method and system for mobile robots. This method integrates a top-down global perspective with the robot's own vision and constructs a multi-source sensor data fusion algorithm, significantly reducing system complexity while maintaining positioning accuracy. This method enables rapid identification and positioning of mobile robots in indoor environments, aiming to reduce the development process and hardware costs of mobile robots, as well as the data processing load on the mobile robot's main control terminal.

[0004] To achieve the above objectives, the present invention proposes a macro-micro composite visual localization method based on mobile robots, comprising the following steps: Based on the global image data collected by macroscopic vision modules deployed in different locations and the tracking image data of the mobile robot by the macroscopic vision modules, an environmental map is generated and the current observation data of the mobile robot is determined; the environmental map contains connected regions of environmental objects with unique ID tags; the environmental map records the pose of the mobile robot at the previous moment. Based on the environmental data collected by the mobile robot, navigation sensing data and target object perception data are obtained; wherein, the target object is an environmental object identified in the local image data collected by the microscopic vision module mounted on the mobile robot; the target object perception data includes the identification result of the target object and the position information between the target object and the mobile robot at the current moment. Based on the mobile robot's previous pose, current observation data, and navigation sensor data, determine the mobile robot's current pose in the environmental map. Based on the current pose of the mobile robot and the position information between the target object and the mobile robot at the current moment, determine the position of the target object on the environmental map; Based on the location of the target object on the environmental map, the identification result of the target object is associated with the unique ID label of the connected area of ​​the environmental object corresponding to the location.

[0005] In the above-mentioned macro-micro composite visual localization method based on mobile robots, generating an environment map based on the global image data includes the following steps: Preprocess each global image data; Edge processing is performed on the preprocessed global image data based on the edge detection algorithm to obtain the edge information of environmental objects in the image. Calculate and determine connected regions in the image based on the edge information of environmental objects; Assign a unique ID label to each of the connected regions; The global images of each connected region with a unique ID label are stitched together to generate a global image that serves as an environment map.

[0006] In the aforementioned macro-micro composite visual localization method based on mobile robots, the preprocessing of each global image data includes the following steps: Gamma correction is applied to the global image data to obtain a brighter image. The image with enhanced brightness is converted to the Lab color space. By comparing the average brightness channels of the original Lab image and the Lab image after global brightness equalization, the shadow areas in the image are detected. Noise suppression processing is applied to the detected shadow areas; The denoised image is processed into a grayscale image using a color channel weighted average method.

[0007] In the aforementioned macro-micro composite vision localization method based on mobile robots, the positional information between the target object and the mobile robot at the current moment is the coordinate of the target object in the coordinate system of the microscopic vision module; determining the position of the target object on the environmental map includes the following steps: Transform the coordinates of the target object in the microscopic vision module coordinate system to the mobile robot coordinate system to obtain the coordinates of the target object in the mobile robot coordinate system; Based on the current pose of the mobile robot in the environmental map coordinate system, the coordinates of the target object in the mobile robot coordinate system are transformed to the environmental map coordinate system, thus generating the coordinates of the target object in the environmental map coordinate system.

[0008] In the aforementioned macro-micro composite visual positioning method based on mobile robots, the navigation sensing data includes attitude angle data and displacement data between the previous moment and the current moment. Determining the current pose of the mobile robot in the environmental map includes the following steps: Starting from the previous pose of the mobile robot, the predicted pose of the mobile robot is determined by combining the attitude angle data and displacement data between the previous and current moments. The predicted pose is corrected by fusing the current observation data with the state estimator, and the current pose of the mobile robot in the environment map is obtained.

[0009] The aforementioned macro-micro composite visual localization method based on mobile robots involves associating the recognition result of the target object with the unique ID label of the connected region of the corresponding environmental object, including the following steps: Match the location with the connected regions of environmental objects corresponding to the location of the target object on the environment map; Associate the unique ID label corresponding to the connected region of the matched environmental object with the recognition result of the target object.

[0010] In the aforementioned macro-micro composite visual localization method based on mobile robots, if no successfully matched connected regions of environmental objects are found, the macro vision module is controlled to re-acquire global view data and update the environmental map; simultaneously, the micro vision module is controlled to update the target object perception data and update the target object's position on the updated environmental map; then, the connected regions of environmental objects corresponding to the target object's position on the environmental map are re-matched; and the unique ID tag corresponding to the matched connected regions of environmental objects is associated with the recognition result of the target object.

[0011] The second aspect of this invention discloses a macro-micro composite visual positioning system based on a mobile robot, including a macro vision module, a mobile robot, and a host computer, wherein the host computer is communicatively connected to the macro vision module and the mobile robot respectively. The macroscopic vision module is configured as follows: Based on the global image data collected by macroscopic vision modules deployed in different locations and the tracking image data of the mobile robot by the macroscopic vision modules, an environmental map is generated and the current observation data of the mobile robot is determined; the environmental map contains connected regions of environmental objects with unique ID tags; the environmental map records the pose of the mobile robot at the previous moment. The mobile robot is configured as follows: Based on the environmental data collected by the mobile robot, navigation sensing data and target object perception data are obtained; wherein, the target object is an environmental object identified in the local image data collected by the microscopic vision module mounted on the mobile robot; the target object perception data includes the identification result of the target object and the position information between the target object and the mobile robot at the current moment. The host computer is configured as follows: Based on the mobile robot's previous pose, current observation data, and navigation sensor data, determine the mobile robot's current pose in the environmental map. Based on the current pose of the mobile robot and the position information between the target object and the mobile robot at the current moment, determine the position of the target object on the environmental map; Based on the location of the target object on the environmental map, the identification result of the target object is associated with the unique ID label of the connected area of ​​the environmental object corresponding to the location.

[0012] A third aspect of the present invention discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1-7.

[0013] A fourth aspect of the present invention discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

[0014] The technical solution provided by this invention may include the following beneficial effects: In the macro-micro composite visual positioning method provided by this invention, an environmental map is constructed by processing global image data collected by macro-visual modules from different perspectives at the same time. This effectively solves the problem of missing local information caused by mutual occlusion between objects, and achieves the fusion of two-dimensional information from multiple perspectives into a seamless and complete panoramic map of the indoor environment, ensuring the integrity and accuracy of the map. Furthermore, each connected region of an environmental object in the environmental map is assigned a unique ID label, providing a foundation for subsequent data association. Compared to relying solely on mobile robot perception to construct a map, this invention provides a faster environmental map construction, simpler processing steps, lower development costs, and the ability to handle complex and changing indoor environments.

[0015] Based on the mobile robot's previous pose, the current observation data from the macroscopic vision module is fused with the robot's navigation sensor data to determine the robot's current pose on the environmental map. This eliminates the positioning errors of the macroscopic vision module and the inherent cumulative errors of the navigation sensors in the mobile robot (such as inertial measurement units and wheeled odometers), effectively improving the accuracy of the mobile robot's positioning.

[0016] By combining the target object perception data obtained by the mobile robot in perceiving the environment, the recognition results of the perceived target object are associated with the unique ID label of the connected area of ​​the corresponding environmental object, establishing an accurate correspondence, realizing environmental understanding, and thus endowing the mobile robot with the semantic understanding ability of the environment, which is beneficial to the mobile robot's decision-making. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the macro-micro composite visual positioning method of the present invention. Figure 2 This is a schematic diagram of the framework of the macro-micro composite visual positioning system of the present invention; Figure 3 This is a structural framework diagram of an electronic device according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0021] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0022] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the word "and / or" throughout the text means including three parallel solutions; taking "A and / or B" as an example, it includes solution A, solution B, or a solution that simultaneously satisfies A and B. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0023] The macro-micro composite visual positioning method based on a mobile robot according to the present invention is applied to a macro-micro composite visual positioning system. The macro-micro composite visual positioning system includes a macro vision module, a mobile robot, and a host computer. The host computer is communicatively connected to the macro vision module and the mobile robot, and can transmit data to each other, for example, through wireless connection such as Bluetooth or WiFi.

[0024] In the macro-micro composite visual localization method based on mobile robots, all steps can be processed by a host computer, with the macro vision module and the mobile robot providing indoor environmental data. Alternatively, some steps can be assigned to each of the three components. For example, the macro vision module could be an intelligent industrial camera with image processing capabilities, not only collecting global indoor data but also processing it to obtain an environmental map. The mobile robot can also have its own control processor, which processes the collected environmental data to obtain navigation sensor data and target object perception data. The host computer receives the processed data from the macro vision module and the mobile robot, performing steps such as localizing the mobile robot and assigning semantic meaning to the environmental map.

[0025] In a specific embodiment of the macro-micro composite vision positioning system, the macro vision module can be a smart industrial camera with image processing capabilities. Multiple such cameras are deployed in the upper space of an indoor space to capture images of the indoor environment and mobile robots from a downward-sloping perspective. Optionally, multiple macro vision modules are deployed at different locations indoors, enabling them to capture images of the environment and mobile robots below from different perspectives at the same height, and then process the acquired global image data.

[0026] The mobile robot is equipped with a microscopic vision module, an inertial measurement unit (IMU), and a wheeled odometry system. Specifically, the microscopic vision module can be a binocular camera. This module captures images of a localized indoor environment from the robot's perspective, providing local image data. The IMU and odometry system collect navigation sensor data; the IMU measures the robot's attitude angles, and the odometry system collects its displacement data. The mobile robot has its own data processor, which calculates the distance to target objects and their 3D data based on parallax principles. It can also identify target objects using deep learning algorithms. Thus, it can determine the type of target object, its position relative to the robot, and its size, among other parameters.

[0027] The host computer is a terminal with data processing capabilities, used to process the data collected by the macro vision module and the mobile robot.

[0028] The following is combined with Figure 1 The present invention describes a macro-micro composite visual localization method based on a mobile robot, comprising the following steps: Step S1: Based on the global image data collected by the macroscopic vision modules deployed in different locations and the tracking image data of the mobile robot by the macroscopic vision modules, generate an environment map and determine the current observation data of the mobile robot; the environment map contains connected regions of environmental objects with unique ID tags; the environment map records the pose of the mobile robot at the previous moment.

[0029] The global image data consists of images collected by macroscopic vision modules deployed at the same height and from multiple directions within the room. For example, multiple macroscopic vision modules can be installed on the ceiling at the same height indoors to capture images of the indoor environment and moving robots from multiple angles and downward perspectives. The images from multiple perspectives are then stitched together to form a complete panoramic view of the indoor environment, serving as an environmental map and effectively solving the problem of mutual occlusion of environmental objects.

[0030] Specifically, the environment map can be displayed as a two-dimensional image. For example, if a chair is an environmental object in the global image data, then the environment map contains connected regions corresponding to the edge of the chair. The unique ID label for this connected region is "Region 1," meaning that in the environment map, there is an environmental object connected region labeled "Region 1." The environment map records the previous pose of the mobile robot; the previous pose refers to the mobile robot's position coordinates and orientation on the environment map at the previous moment.

[0031] Specifically, tracking image data refers to images that can detect a mobile robot. The detected mobile robot in the tracking image data is then back-projected onto the coordinate system of the environment map using camera calibration parameters. The tracking image data can be selected from images acquired by a single macroscopic vision module that can currently track the mobile robot. Alternatively, images acquired by multiple macroscopic vision modules can be used, with the image that best detects the mobile robot being selected as the tracking image data. The current observation data of the mobile robot includes its coordinates projected onto the environment map from the tracking image data, as well as the robot's orientation.

[0032] Step S2: Based on the environmental data collected by the mobile robot, navigation sensing data and target object perception data are obtained. The target object is an environmental object identified from local image data collected by the microscopic vision module mounted on the mobile robot. The target object perception data includes the identification result of the target object and the positional information between the target object and the mobile robot at the current moment. For example, the mobile robot's data processor calculates the distance to the target object and its three-dimensional data from the local image data based on the parallax principle, and simultaneously identifies the target object using a deep learning algorithm. In this way, the type of the target object, its position relative to the mobile robot, and its size, among other relevant parameters, can be determined.

[0033] Specifically, the navigation sensing data includes the mobile robot's attitude angle data and displacement data. The attitude angle data is collected by an inertial measurement unit (IMU) mounted on the mobile robot. The IMU, through its internal accelerometer, gyroscope, and magnetometer, collects real-time data on the object's acceleration, angular velocity, and magnetic field in three orthogonal directions. It then uses a sensor fusion algorithm to filter, reduce noise, and correct errors in the raw signals, thereby obtaining the object's attitude information and angle changes. The displacement data is collected by a wheeled odometer mounted on the mobile robot. The wheeled odometer detects the number of rotations or angle changes of its mounted wheels. Using internal optical, magnetic, or mechanical devices, it senses the rotation of the wheel axle and generates corresponding electrical pulse signals. Finally, by accumulating these pulse counts and combining them with the known wheel circumference, it can accurately calculate the straight-line distance or displacement traveled by the wheel.

[0034] Step S3: Based on the mobile robot's previous pose, current observation data, and navigation sensor data, determine the mobile robot's current pose in the environmental map. This is called localization of the mobile robot. By fusing navigation sensor data and current observation data, complementary effects are achieved, eliminating the inherent cumulative errors of navigation sensors (such as inertial measurement units and wheeled odometers) in the mobile robot, as well as errors present in the macroscopic vision module, effectively improving the accuracy of mobile robot localization.

[0035] Step S4: Determine the position of the target object on the environment map based on the current pose of the mobile robot and the position information between the target object and the mobile robot at the current moment.

[0036] Step S5: Based on the target object's position on the environmental map, associate the target object's recognition result with the unique ID tag of the connected region of the corresponding environmental object. For example, if the mobile robot detects that the environmental object in front of it is a chair, then based on the chair's position, match the connected region of the environmental object that matches the chair's position on the environmental map, associate the chair with the unique ID tag of the connected region of the environmental object, establish an accurate correspondence, realize environmental understanding, and thus endow the mobile robot with the semantic understanding ability of the environment.

[0037] Furthermore, as an optional embodiment, generating an environment map includes the following steps: Step S11: Preprocess each global image data; Step S12: Perform edge processing on the preprocessed global image data based on the edge detection algorithm to obtain the edge information of environmental objects in the image; specifically, the improved Canny algorithm can be used for edge processing.

[0038] Step S13: Calculate and determine the connected regions in the image based on the edge information of the environmental objects.

[0039] Step S14: Assign a unique ID label to each of the connected regions; Step S15: Stitch together the global images of each connected region with a unique ID label to generate a global image that serves as the environment map. Specifically, based on the perspective transformation method, multiple oblique viewpoints are converted to orthographic viewpoints, i.e., from oblique projection to orthographic projection, and then stitched together to obtain the global image. Each connected region of an environmental object in the environment map is assigned a unique ID label, providing a foundation for subsequent data association.

[0040] Furthermore, as an optional embodiment, preprocessing each global image data includes the following steps: Step S111: Perform gamma correction on the global image data to obtain a brighter image; thus achieving the image brightness enhancement effect.

[0041] Step S112: Convert the brightness-enhanced image to the Lab color space. By comparing the average brightness channels of the original Lab image and the globally balanced Lab image, the shadow areas in the image are detected.

[0042] Step S113: Perform noise suppression processing on the detected shadow areas; for example, use Gaussian filtering, median filtering and bilateral filtering methods to suppress noise and effectively remove most of the interference components in the image.

[0043] Step S114: Perform grayscale processing on the denoised image using the color channel weighted average method to obtain a grayscale image. This preprocessing improves image quality, suppresses noise interference from environmental factors, and preserves as much effective feature information as possible.

[0044] Further, as an optional embodiment, the positional information between the target object and the mobile robot at the current moment is the coordinates of the target object in the coordinate system of the microscopic vision module; determining the position of the target object on the environmental map includes the following steps: Step S41: Transform the coordinates of the target object in the microscopic vision module coordinate system to the mobile robot coordinate system to obtain the coordinates of the target object in the mobile robot coordinate system; Step S42: Based on the current pose of the mobile robot in the environment map coordinate system, transform the coordinates of the target object in the mobile robot coordinate system to the environment map coordinate system, and generate the coordinates of the target object in the environment map coordinate system.

[0045] Further, as an optional embodiment, the navigation sensing data includes attitude angle data and displacement data between the previous moment and the current moment; determining the current pose of the mobile robot in the environmental map includes the following steps: Step S31: Starting from the previous pose of the mobile robot, combine the attitude angle data and displacement data between the previous and current moments to determine the predicted pose of the mobile robot. Step S32: Correct the predicted pose by fusing the current observation data through a state estimator to obtain the current pose of the mobile robot in the environment map. Specifically, the state estimator can be a Kalman filter.

[0046] Furthermore, as an optional embodiment, associating the recognition result of the target object with the unique ID label of the connected region of the environmental object corresponding to the location includes the following steps: Step S51: Match the location of the target object with the connected regions of the environment objects corresponding to the location of the target object on the environment map; Step S52: Associate the unique ID label corresponding to the connected region of the matched environmental object with the recognition result of the target object.

[0047] Furthermore, as an optional embodiment, if no successfully matched connected regions of environmental objects are found, the macroscopic vision module is controlled to re-acquire global view data and update the environmental map; simultaneously, the microscopic vision module is controlled to update the target object perception data and update the position of the target object in the updated environmental map; then, the connected regions of environmental objects corresponding to the position of the target object in the environmental map are re-matched; and the unique ID tag corresponding to the matched connected regions of environmental objects is associated with the recognition result of the target object.

[0048] Thus, some environmental objects in the environment map can move. For example, if an environmental object is moved to another area before it is associated with another object, the connected regions of that object in the environment map will not match its actual location. Therefore, when trying to associate the object, a matching connected region will fail. To address this, this embodiment updates both the environment map and the target object's perception data simultaneously, ensuring that the connected regions of the environmental object and the target object are correctly associated in the latest data. When updating the environment map, connected regions of environmental objects are compared. Connected regions of objects whose positions have not changed inherit the labels and associations from the previous environment map. Connected regions of other environmental objects whose connected regions have changed are assigned new unique ID labels, ready to be associated with the target perception data of the mobile robot.

[0049] like Figure 2 As shown, the second aspect of the present invention discloses a macro-micro composite visual positioning system based on a mobile robot, including a macro vision module 201, a mobile robot 202, and a host computer 203; the host computer 203 is communicatively connected to the macro vision module 201 and the mobile robot 202 respectively, and can transmit data to each other, for example, through wireless connection such as Bluetooth or WiFi.

[0050] The macroscopic vision module 201 is configured as follows: Based on the global image data collected by the macroscopic vision modules 201 deployed at different locations and the tracking image data of the mobile robot 202 by the macroscopic vision modules 201, an environmental map is generated and the current observation data of the mobile robot 202 is determined; the environmental map contains connected regions of environmental objects with unique ID tags; the environmental map records the pose of the mobile robot 202 at the previous moment. The mobile robot 202 is configured as follows: Based on the environmental data collected by the mobile robot 202, navigation sensing data and target object perception data are obtained; wherein, the target object is an environmental object identified in the local image data collected by the microscopic vision module mounted on the mobile robot 202; the target object perception data includes the identification result of the target object and the position information between the target object and the mobile robot 202 at the current moment. The host computer 203 is configured as follows: Based on the previous pose of the mobile robot 202, the current observation data, and the navigation sensor data, determine the current pose of the mobile robot 202 in the environmental map. Based on the current pose of the mobile robot 202 and the position information between the target object and the mobile robot 202 at the current moment, the position of the target object on the environmental map is determined; Based on the location of the target object on the environmental map, the identification result of the target object is associated with the unique ID label of the connected area of ​​the environmental object corresponding to the location.

[0051] like Figure 3 As shown, another aspect of the present invention discloses an electronic device 300, including a processor 301 and a memory 302. The processor 301 and the memory 302 are connected, for example, via a bus 303. Further, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of this application. The processor 301 is used in the embodiments of this application to implement a macro-micro composite visual positioning method based on a mobile robot. The processor 301 can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 301 can also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc. The bus 303 may include a path for transmitting information between the above components. The bus 303 may be a PCI bus or an EISA bus, etc. Bus 303 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3The code uses only a single thick line to represent a bus, but this does not imply that there is only one bus or one type of bus. The memory 302 can be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or it can be an EEPROM, CD-ROM or other optical disc storage, optical disk storage (including compressed optical disks, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory 302 is used to store application code that executes the scheme of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 302 to implement the steps of the macro-micro composite visual positioning method based on mobile robots provided by this invention.

[0052] On the other hand, embodiments of the present invention provide a storage medium storing a computer program thereon, which is executed by a processor as described above in the steps of the macro-micro composite visual positioning method based on a mobile robot executed by the server.

[0053] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A macro-micro compound visual positioning method based on a mobile robot, characterized in that: The method comprises the following steps: According to the global image data collected by the macro visual module deployed in different positions and the tracking image data of the mobile robot by the macro visual module, an environment map is generated and the current observation data of the mobile robot is determined; the environment map contains a connected region of environment objects with a unique ID label; the last time pose of the mobile robot is recorded in the environment map; According to the environment data collected by the mobile robot, navigation sensing data and target object perception data are obtained; the target object is an environment object identified in the local image data collected by the micro visual module carried on the mobile robot; the target object perception data includes the identification result of the target object and the position information between the target object and the mobile robot at the current time; According to the last time pose of the mobile robot, the current observation data and the navigation sensing data, the current time pose of the mobile robot in the environment map is determined; According to the current time pose of the mobile robot and the position information between the target object and the mobile robot at the current time, the position of the target object in the environment map is determined; According to the position of the target object in the environment map, the identification result of the target object is associated with the unique ID label of the connected region of the environment object corresponding to the position.

2. The macro-micro hybrid visual positioning method based on mobile robots according to claim 1, characterized in that: Generating the environment map according to the global image data comprises the following steps: Preprocessing each global image data; Performing edge processing on the preprocessed global image data based on an edge detection algorithm to obtain edge information of environment objects in the image; Calculating and determining the connected region in the image according to the edge information of the environment objects; Assigning a unique ID label to each connected region; Splicing the global image data of each connected region with a unique ID label to generate a global image as the environment map.

3. The macro-micro hybrid visual positioning method based on mobile robots according to claim 2, characterized in that: The preprocessing of each global image data comprises the following steps: Performing gamma correction on the global image data to obtain a brightness-enhanced image; Converting the brightness-enhanced image to Lab color space, and detecting the shadow region in the image by comparing the average value of the brightness channel of the original Lab image with that of the Lab image after global brightness equalization; Performing noise suppression processing on the detected shadow region; Performing gray processing on the denoised image based on the color channel weighted average method to obtain a gray image. 4.The macro-micro hybrid visual positioning method based on mobile robot according to claim 1, wherein: The position information between the target object and the mobile robot at the current time is the coordinates of the target object in the micro visual module coordinate system; determining the position of the target object in the environment map comprises the following steps: Converting the coordinates of the target object in the micro visual module coordinate system to the mobile robot coordinate system to obtain the coordinates of the target object in the mobile robot coordinate system; Converting the coordinates of the target object in the mobile robot coordinate system to the environment map coordinate system based on the current time pose of the mobile robot in the environment map coordinate system to generate the coordinates of the target object in the environment map coordinate system.

5. The macro-micro hybrid visual positioning method based on mobile robots according to claim 1, characterized in that: The navigation sensing data includes attitude angle data and displacement data between the last time and the current time; The determination of the current time pose of the mobile robot in the environment map comprises the following steps: The last time pose of the mobile robot is taken as a starting point, and the predicted pose of the mobile robot is determined by combining the attitude angle data and displacement data between the last time and the current time; The predicted pose is corrected by fusing the current observation data through a state estimator to obtain the current time pose of the mobile robot in the environment map. 6.The macro-micro hybrid visual positioning method based on mobile robot according to claim 1, wherein: The steps of associating the recognition result of the target object with the unique ID tag of the environment object connected region corresponding to the position include: Matching the position with the environment object connected region corresponding to the position of the target object in the environment map; Associating the unique ID tag of the matched environment object connected region with the recognition result of the target object.

7. The macro-micro composite visual positioning method based on a mobile robot according to claim 6, characterized in that: If there is no matched environment object connected region, the macro visual module is controlled to reacquire global view data and update the environment map; at the same time, the micro visual module is controlled to update the target object perception data and update the position of the target object in the updated environment map; Then, the position is matched with the environment object connected region corresponding to the position of the target object in the environment map again; The unique ID tag of the matched environment object connected region is associated with the recognition result of the target object.

8. A macro-micro compound vision positioning system based on a mobile robot, characterized in that: The system is applied to the macro-micro composite visual positioning method based on a mobile robot according to any one of claims 1-7; the system comprises a macro visual module, a mobile robot, and an upper computer, and the upper computer is in communication connection with the macro visual module and the mobile robot respectively; The macro visual module is configured to: generate an environment map and determine the current observation data of the mobile robot according to the global image data acquired by the macro visual module deployed at different positions and the tracking image data of the mobile robot by the macro visual module; the environment map comprises environment object connected regions with unique ID tags; and the last time pose of the mobile robot is recorded in the environment map; The mobile robot is configured to: obtain navigation sensing data and target object perception data according to the environment data acquired by the mobile robot; the target object is an environment object recognized in the local image data acquired by the micro visual module carried on the mobile robot; and the target object perception data comprises the recognition result of the target object and the position information between the target object and the mobile robot at the current time; The upper computer is configured to: determine the current time pose of the mobile robot in the environment map according to the last time pose of the mobile robot, the current observation data, and the navigation sensing data; determine the position of the target object in the environment map according to the current time pose of the mobile robot and the position information between the target object and the mobile robot at the current time; associate the recognition result of the target object with the unique ID tag of the environment object connected region corresponding to the position according to the position of the target object in the environment map.

9. An electronic device, comprising: The computer program stored in the memory and executable on the processor comprises the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which, when executed by the processor, implements the method of any one of claims 1-7.

Images