Warehouse storage robot positioning and mapping method, robot and storage medium

Abstract

An embodiment of the present application provides a method for localizing and mapping a warehouse storage robot, a robot, and a storage medium. In this embodiment, a semantic map of the warehouse storage environment is created in advance, the semantic map including multiple objects present in the warehouse storage environment and their semantic information. During the localization process, the warehouse storage robot uses its image sensor to collect image or video data of the surrounding environment (11), recognizes a target object and its semantic information in the image or video data (12), obtains a relative positional relationship between the target object and the warehouse storage robot (13), and determines the warehouse storage robot's position on the semantic map based on the relative positional relationship and the semantic information of the target object (14). This method uses a form based on visual semantic localization, which directly detects clear targets, resulting in fast detection speed, rich semantic information, and resistance to other interference factors. This method eliminates reliance on signs in the warehouse storage environment and provides high-degree of positioning flexibility.

Description

[Technical field]

[0001] This application references a Chinese patent application filed on September 14, 2020, entitled "Warehouse storage robot positioning and mapping method, robot and storage medium" with application number No. 2020109634154, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of warehousing robots, and in particular to a method for localizing and mapping a warehousing robot, a robot, and a storage medium. [Background technology]

[0003] With the development of artificial intelligence technology, logistics warehouse storage robots have emerged. Usually, logistics warehouse storage robots work in warehouse storage environments, which have many shelves and objects stacked on the ground, and the environment is relatively complex. In order to assist the positioning of the warehouse storage robot, signs such as reflectors and QR codes are installed in the warehouse storage environment, and when mapping, the warehouse storage robot traverses the warehouse storage environment, and uses Simultaneous Localization and Mapping (SLAM) technology to create an environmental map and mark the position of the "sign" on the environmental map. When positioning, the warehouse storage robot detects the positional relationship of the sign to the robot body, and combines the position of the sign on the environmental map to estimate the position of the warehouse storage robot in the environmental map. Traditional warehouse storage positioning methods require signs to be pre-placed in the warehouse storage environment and are heavily dependent on the signs, resulting in low positioning freedom. Summary of the Invention [Problem to be solved by the invention]

[0004] Aspects of the present application provide a warehousing robot localization and mapping method, robot, and storage medium to move away from dependency on signs in a warehousing environment. [Means for solving the problem]

[0005] An embodiment of the present application provides a method for positioning a warehouse storage robot, including: collecting image or video data of a surrounding warehouse storage environment using an image sensor in the warehouse storage robot during movement of the warehouse storage robot; performing target detection and semantic recognition on the image or video data of the surrounding warehouse storage environment to obtain target objects existing in the surrounding warehouse storage environment of the warehouse storage robot and their semantic information; calculating a relative positional relationship between the target objects and the warehouse storage robot based on a transformation relationship between the sensor coordinate system and the robot coordinate system; and locating the warehouse storage robot in a semantic map according to the relative positional relationship between the target objects and the warehouse storage robot and the semantic information of the target objects, wherein the semantic map includes a plurality of objects existing in the warehouse storage environment and their semantic information.

[0006] An embodiment of the present application further provides a method for creating a semantic map, including: collecting image or video data of the warehouse storage environment using an image sensor in the warehouse storage robot while the warehouse storage robot traverses the warehouse storage environment; performing target detection and semantic recognition on the image or video data of the warehouse storage environment to obtain a plurality of objects existing in the warehouse storage environment and their semantic information; calculating relative positional relationships between the plurality of objects and the warehouse storage robot based on the transformation relationship between the sensor coordinate system and the robot coordinate system, respectively; and adding the semantic information of the plurality of objects to the base environment map according to the relative positional relationship between the plurality of objects and the warehouse storage robot and the position of the warehouse storage robot in the base environment map when collecting the image or video data of the warehouse storage environment to obtain a semantic map of the warehouse storage environment.

[0007] An embodiment of the present application further provides a warehouse storage robot comprising an equipment body provided with a memory and a processor, wherein the memory is used for storing a computer program, and the processor is coupled to the memory and executes the computer program to collect image or video data of a surrounding warehouse storage environment using an image sensor in the warehouse storage robot during movement of the warehouse storage robot, and perform target detection and semantic recognition on the image or video data of the surrounding warehouse storage environment to obtain target objects and their semantic information existing in the surrounding warehouse storage environment of the warehouse storage robot, and calculate a relative positional relationship between the target object and the warehouse storage robot based on a transformation relationship between the sensor coordinate system and the robot coordinate system, which is used to locate the warehouse storage robot in a semantic map according to the relative positional relationship between the target object and the warehouse storage robot and the semantic information of the target object, and the semantic map includes a plurality of objects and their semantic information existing in the warehouse storage environment.

[0008] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, the computer program causing the processor to implement steps in the warehouse storage robot positioning method and semantic map creation method of any one of the embodiments of the present application when executed by the processor.

[0009] In the embodiment of the present application, a semantic map of the warehousing environment is prepared in advance, the semantic map includes a number of objects existing in the warehousing environment and their semantic information, and in the positioning process, the warehousing robot uses its image sensor to collect image or video data of the surrounding environment, recognizes a target object and its semantic information in the image or video data, obtains the relative positional relationship between the target object and the warehousing robot, and further locates the position of the warehousing robot in the semantic map based on the relative positional relationship and the semantic information of the target object. This method uses a form based on visual semantic positioning, and directly detects a clear target, so the detection speed is fast, the semantic information is rich, and it is not easily affected by other interference factors, which breaks away from the reliance on signs in the warehousing environment and has a high degree of freedom in positioning. [Brief description of the drawings]

[0010] The drawings described herein are used for further understanding of the present application and constitute a part of the present application. The schematic examples and the description of the present application are for interpreting the present application, but are not for limiting the present application. In the drawings, [Figure 1a] 1 is a schematic flowchart of a method for positioning a warehousing robot provided by an exemplary embodiment of the present application. [Figure 1b] FIG. 1 is a schematic diagram of a shelf in a warehousing environment provided by an exemplary embodiment of the present application. [Figure 1c] FIG. 2 is a schematic diagram of a shelf and ground plane intersection in a warehousing environment provided by an exemplary embodiment of the present application. [Figure 1d] FIG. 2 is a schematic diagram of a shelf intersection in a warehousing environment provided by an exemplary embodiment of the present application. [Figure 1e] FIG. 2 is a schematic diagram of a shelf obtained by target detection provided by an exemplary embodiment of the present application. [Diagram 2] FIG. 2 is a schematic diagram of the positional relationship between a pixel plane and a ground plane of a monocular camera provided by an exemplary embodiment of the present application. [Diagram 3]1 is a schematic flowchart of a method for constructing a semantic map provided by an exemplary embodiment of the present application; [Figure 4] FIG. 2 is a structural schematic diagram of a warehouse storage robot provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In order to clarify the purpose, technical solution and advantages of the present application, the technical solution of the present application will be described below clearly and completely with reference to certain embodiments of the present application and corresponding drawings. It goes without saying that the described embodiments are only a part of the embodiments of the present application, and are not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts are all included in the protection scope of the present application.

[0012] In response to the problem of the strong dependency on signs in the prior art and the low degree of freedom in positioning, in the embodiment of the present application, a semantic map of the warehouse storage environment is prepared in advance, the semantic map includes a number of objects existing in the warehouse storage environment and their semantic information, and in the positioning process, the warehouse storage robot uses its own image sensor to collect image or video data of the surrounding environment, recognizes the target object and its semantic information in the image or video data, obtains the relative positional relationship between the target object and the warehouse storage robot, and further locates the warehouse storage robot in the semantic map based on the relative positional relationship and the semantic information of the target object. This method uses a form based on visual semantic positioning, and directly detects a clear target, so the detection speed is fast, the semantic information is rich, and it is not easily affected by other interference factors, breaking away from the dependency on signs in the warehouse storage environment and providing a high degree of freedom in positioning.

[0013] Hereinafter, the technical solutions provided by the embodiments of the present application will be described in detail with reference to the drawings.

[0014] FIG. 1a is a schematic flow chart of a method for positioning a warehouse storage robot provided by an exemplary embodiment of the present application. As shown in FIG. 1a, the method includes: 11: Collecting image or video data of a surrounding warehousing environment using an image sensor in the warehousing robot during movement of the warehousing robot; 12: Perform target detection and semantic recognition on the image or video data of the surrounding warehousing environment to obtain target objects and their semantic information existing in the surrounding warehousing environment of the warehousing robot; 13: Calculating a relative positional relationship between the target object and the warehouse storage robot based on a transformation relationship between the sensor coordinate system and the robot coordinate system; 14: Determining a position of the warehouse storage robot in a semantic map according to a relative positional relationship between the target object and the warehouse storage robot and semantic information of the target object, where the semantic map includes a plurality of objects existing in the warehouse storage environment and their semantic information.

[0015] In this embodiment, the warehousing environment refers to a place where goods are stored, such as a warehouse, a storage room, or a storehouse. The warehousing environment typically includes a number of objects, such as stored goods, a number of shelves storing goods, an intersection line between the shelf and the ground plane, and an intersection formed by the shelves, as shown in Figures 1b to 1d, and may further include at least one object of a shelf leg position, a fire hydrant, and an evacuation sign.

[0016] In this embodiment, the warehousing robot refers to a mechanical device that can work in a warehousing environment and move in space with a high degree of autonomy in the warehousing environment. For example, it is a warehousing collaborative robot, a picking robot, or a transport robot. The warehousing robot includes an equipment body including a control unit and a chassis structure, where the chassis is a load-bearing member of the warehousing robot, and the chassis structure includes a bottom plate, a driving wheel assembly, and a driven wheel assembly. The control unit can control the driving wheel assembly to move the driven wheel assembly and move in the warehousing environment. In addition, the warehousing robot may further include members for human-computer interaction, such as a display screen and / or an audio assembly. Through these members, a user can issue control commands to the warehousing robot or provide the warehousing robot with relevant data such as goods, orders, and transport destinations to assist the warehousing robot in successfully completing warehousing tasks. In addition, the warehousing robot may include an image sensor for collecting image or video data of the surrounding warehousing environment of the warehousing robot. In this embodiment, the image sensor is not limited, and any image sensor capable of collecting image or video data of the surrounding warehousing environment of the warehousing robot is applicable to the embodiment of the present application. For example, the image sensor may be, but is not limited to, a monocular camera, a binocular camera, or a Red Green Blue Depth map (RGBD) camera. In addition, the warehousing robot may have other sensors, such as at least one of a laser sensor, an odometer, and an inertial measurement unit (IMU), for collecting surrounding environmental information, motion data, and / or posture data of the warehousing robot to position the warehousing robot.

[0017] In this embodiment, there is a semantic map corresponding to the warehousing environment where the warehousing robot is located, and the semantic map refers to an environment map containing semantic information of multiple objects in the warehousing environment. The semantic information of an object refers to understanding and interpreting what information such as an object or its category is with the help of natural language, and may include, but is not limited to, the name, shape, location, etc. of the object. Compared with the traditional non-semantic environment map, the semantic map has a high semantic level, which makes it easier for the warehousing robot to position itself, improves the user's experience, and meets people's needs for semantic maps. In this embodiment, the warehousing robot can realize visual semantic positioning based on the semantic map. For the process of creating the semantic map, you can refer to the embodiment shown in Figure 3 below, which will not be described in detail here.

[0018] As shown in FIG. 1b to FIG. 1d, the warehousing robot moves in the warehousing environment, and during the movement, its image sensor can collect image or video data of the surrounding warehousing environment of the warehousing robot. The surrounding warehousing environment refers to a part of the warehousing environment near the warehousing robot, and the range in which the warehousing robot can collect image or video data in the surrounding warehousing environment is related to the field of view of the image sensor, and the larger the field of view of the image sensor, the larger the range of the collected image or video data. After the image or video data of the surrounding warehousing environment is collected, target detection and semantic recognition are performed on the image or video data of the surrounding warehousing environment to obtain objects and their semantic information present in the surrounding warehousing environment of the warehousing robot. Here, taking the image data shown in FIG. 1b as an example, as shown in FIG. 1e, objects contained in the image data are detected, and further, after the objects are recognized, semantic recognition is performed on them to obtain semantic information corresponding to the objects such as shelves. There are multiple objects in the entire warehousing environment, and for ease of distinction and description, the objects present in the surrounding warehousing environment of the warehousing robot collected during the positioning process are called target objects. In this embodiment, the semantic information of the objects is recognized by visual semantic technology. That is, it can directly detect clear targets, has a fast detection speed, rich semantic information, and is less affected by other interference factors, providing a basis for positioning of warehouse storage robots to break away from dependence on signs in warehouse storage environments, and helping to improve the freedom of positioning.

[0019] In this embodiment, the embodiment of performing target detection and semantic recognition on the image and video data of the warehousing environment is not limited, and for example, a target detection model can be used to perform target detection and semantic recognition on the image or video data. The target detection model has both target detection and semantic recognition functions, and can not only recognize objects and their positions included in the warehousing environment, but also perform semantic recognition on the detected objects to obtain visual semantic information of the objects. Any model having both target detection and semantic recognition functions is applied to the target detection model of the embodiment of this application. For example, the target detection model may be a region-based target detection model such as Region-Convolutional Neural Networks (R-CNN) or Fast Region-Convolutional Neural Networks (Fast-R-CNN), or a regression-based target detection model such as a You Only Look Once (YOLO) model or a Single Shot MultiBox Detector (SSD), but is not limited thereto.

[0020] After obtaining the semantic information of the target object, the position of the target object in the semantic map can be determined in combination with the semantic map of the warehousing environment, and the position of the warehousing robot in the semantic map can be determined in combination with the semantic information of the target object if the relative positional relationship between the target object and the warehousing robot is known. Based on this, in this embodiment, after obtaining the image or video data of the surrounding warehousing environment collected by the image sensor, the position coordinates of the target object are calculated, and the relative positional relationship between the target object and the warehousing robot can also be calculated based on the position coordinates of the target object. Here, the image sensor is attached to the warehousing robot, and the image sensor uses a sensor coordinate system. Preferably, when the image sensor is a camera, the sensor coordinate system can be called a camera coordinate system, and the warehousing robot uses a robot coordinate system, and there is a transformation relationship between the two coordinate systems. The image and video data of the surrounding warehousing environment of the warehousing robot collected by the image sensor are located in the sensor coordinate system. That is, the position coordinates of the target object recognized by the target detection model are located in the sensor coordinate system. Based on this, in order to obtain the relative positional relationship between the target object and the warehouse storage robot, the position coordinates of the target object are converted into the robot coordinate system based on the conversion relationship between the sensor coordinate system and the robot coordinate system, thereby calculating the relative positional relationship between the target object and the warehouse storage robot. Specifically, the method includes calculating the three-dimensional coordinates of pixel points on the target object in the sensor coordinate system, converting the three-dimensional coordinates of pixel points on the target object in the sensor coordinate system into the robot coordinate system based on the conversion relationship between the sensor coordinate system and the robot coordinate system, and calculating the relative positional relationship between the target object and the warehouse storage robot according to the three-dimensional coordinates of pixel points on the target object in the robot coordinate system.Here, the image sensor is located on the warehouse storage robot, and the mounting position of the image sensor relative to the warehouse storage robot is fixed and known, so the transformation relationship between the sensor coordinate system and the robot coordinate system can be calculated in advance according to the mounting position of the image sensor relative to the warehouse storage robot.

[0021] Here, the conversion relationship between the sensor coordinate system and the robot coordinate system can be obtained as follows using a rotation matrix R1 and a translation vector T1. JPEG0007627001000001.jpg28170

[0022] JPEG0007627001000002.jpg46170

[0023] Furthermore, after obtaining the relative positional relationship between the target object and the warehouse storage robot, the position of the warehouse storage robot in the semantic map can be identified according to the relative positional relationship between the target object and the warehouse storage robot and the semantic information of the target object.

[0024] In the embodiment of the present application, a semantic map of the warehousing environment is prepared in advance, the semantic map includes a number of objects existing in the warehousing environment and their semantic information, and in the positioning process, the warehousing robot uses its own image sensor to collect image or video data of the surrounding environment, recognizes a target object and its semantic information in the image or video data, obtains the relative positional relationship between the target object and the warehousing robot, and further locates the position of the warehousing robot in the semantic map based on the relative positional relationship and the semantic information of the target object. This method uses a form based on visual semantic positioning, which directly detects a clear target, has a fast detection speed, rich semantic information, is not easily affected by other interference factors, breaks away from the reliance on signs in the warehousing environment, and has a high degree of freedom in positioning.

[0025] In the following embodiment, an embodiment will be described in which the image sensor is a monocular camera, a binocular camera, or an RGBD camera, and the three-dimensional coordinates of a pixel point on a target object are calculated in the sensor coordinate system.

[0026] Embodiment A1: The image sensor used in the warehouse storage robot is a monocular camera. The monocular camera collects image or video data of the surrounding warehouse storage environment of the warehouse storage robot, and the image or video data includes some target objects that intersect with the ground plane, such as the intersection between the shelf and the ground plane, and also includes some target objects located on the ground plane, such as the intersection formed between the shelves on the ground plane. The intersection may be a T-junction or a crossroads intersection. In this case, in combination with the shooting geometry of the monocular camera, the three-dimensional coordinates on the ground plane of each pixel point that intersects with the ground plane of the target object can be calculated, and the three-dimensional coordinates are the position coordinates of the sensor coordinate system.

[0027] As shown in FIG. 2, assume that O is the optical center of the monocular camera, D is the pixel plane of the monocular camera, and Q is the ground plane. In the image or video data collected by the monocular camera, each pixel point p corresponds to one ray from the optical center O through the pixel plane D, and the ray Op intersects with the ground plane Q at point q. The manner of calculating the three-dimensional coordinates on the ground plane of the pixel point intersecting with the ground plane in the target object, that is, the manner of calculating the three-dimensional coordinates of point q, depends on the internal parameters and external parameters of the monocular camera. The internal parameters of the monocular camera are parameters related to the characteristics of the monocular camera itself, such as the focal length and pixel size of the monocular camera, and the external parameters of the monocular camera are the position and rotation direction of the camera, etc.

[0028] JPEG0007627001000003.jpg31170

[0029] Embodiment A2: The image sensor used in the warehouse storage robot is a binocular camera. When the binocular camera collects images of the warehouse storage environment, it can capture two viewpoint images of the same scene, one on the left and one on the right, and apply a binocular stereo matching algorithm to obtain a disparity image of the two viewpoint images, and further obtain a depth image, and obtain the three-dimensional coordinates of the pixel points on the target object in the camera coordinate system according to the depth image.

[0030] Embodiment A3: The image sensor used in the warehouse storage robot is an RGBD camera. When the RGBD camera collects images of the warehouse storage environment, it can capture depth images and RGB images of the same scene, and directly obtain the 3D coordinates of pixel points on the target object in the camera coordinate system by utilizing the matching relationship between the RGB image and the depth image.

[0031] In addition to the image sensor, the warehousing robot may also include a depth sensor. The depth sensor can collect a depth image of the warehousing environment, and the warehousing robot can combine the image information collected by the image sensor with the depth information collected by the depth sensor to obtain three-dimensional coordinates of a target object in the image.

[0032] Regardless of the above embodiment, after obtaining the three-dimensional coordinates of the pixel point on the target object in the sensor coordinate system, the three-dimensional coordinates of the pixel point on the target object in the sensor coordinate system can be transformed into the robot coordinate system based on the transformation relationship between the sensor coordinate system and the robot coordinate system, and the relative positional relationship between the target object and the warehouse storage robot can be calculated according to the three-dimensional coordinates of the pixel point on the target object in the robot coordinate system.

[0033] In an embodiment of the present application, after obtaining the semantic information of the target object and the relative positional relationship between the target object and the warehouse storage robot, the position of the target object in the semantic map can be determined according to the semantic information of the target object, and the position of the warehouse storage robot in the semantic map can be identified according to the relative positional relationship between the target object and the warehouse storage robot and the position of the target object in the semantic map.

[0034] In some application scenarios, there are multiple objects in the entire warehouse storage environment, but the semantic information of the multiple objects is different from each other. Or, in the process of creating a semantic map, each object can have different semantic information by finely distinguishing the semantic information. For example, for the same shelf, the meaning of the shelf can be defined by the characteristics such as the brand, specification, model number, color, etc. of the shelf, so that different shelves have different semantic information. In this way, the semantic map contains multiple objects with different semantic information. In this case, the position of the target object in the semantic map can be directly determined according to the semantic information of the target object.

[0035] Of course, in other application scenarios, there may be many similar shelves in the warehousing environment, and these shelves may all be marked as shelves in the semantic map, so that it is not possible to directly determine which shelf the warehousing robot is near simply based on the semantic information of the shelves. In this case, in order to distinguish which shelf the warehousing robot is near, the warehousing robot may be roughly positioned first, and based on the result of the rough positioning, it may first be determined which shelf the warehousing robot is near, and then the warehousing robot may be finely positioned according to the position of the shelf. Of course, in addition to many similar shelves in the warehousing environment, there may be many other objects such as similar intersections, shelf leg positions, fire hydrants, evacuation signs, etc., and the processing methods are the same or similar. Based on this, in some preferred embodiments, an embodiment of determining the location of the target object in the semantic map according to the semantic information of the target object includes: calculating the initial location of the warehousing robot in the semantic map based on other sensor data of the warehousing robot; and searching for the location of the target object in the semantic map from the periphery of the initial location according to the semantic information of the target object. After determining the location of the target object in the semantic map, the location of the warehousing robot in the semantic map may be determined in combination with the relative positional relationship between the target object and the warehousing robot. Here, the target object around the initial position may be an object having the recognized semantic information that is closest to the initial position, or an object having the recognized semantic information within a certain range from the initial position.

[0036] Preferably, the other sensor data may be at least one of laser sensor data, inertial measurement unit (IMU) data, and odometer data. Some embodiments for calculating the initial position of the warehousing robot in the semantic map based on the other sensor data are described below.

[0037] Embodiment B1: The warehouse storage robot is further equipped with a laser sensor, and during the movement of the warehouse storage robot, the laser sensor can also collect surrounding environment information, and the collected environment information at this time is called laser sensor data. In combination with the laser sensor data, the warehouse storage robot can identify the initial position of the warehouse storage robot in the created base environment map, which is the SLAM initial positioning process. Here, the base environment map can be created in advance by the warehouse storage robot using SLAM technology.

[0038] In embodiment B1, the initial positioning of the warehouse storage robot is first performed using SLAM technology, and then a secondary fine positioning is performed in combination with visual semantics. This prevents the accuracy of the initial position of the warehouse storage robot from being too high, so that a relatively small amount of sensor data can be used in the process of using the laser sensor to identify the initial position of the warehouse storage robot, which helps to improve positioning efficiency and save computing resources. In addition, in combination with visual semantic positioning, the problem of insufficient positioning accuracy due to the lack of features collected in the process of positioning by SLAM or the lack of obvious or sufficient texture information can be overcome to ensure positioning accuracy.

[0039] In the following embodiments B2 and B3, assuming that the position of the warehouse storage robot in the semantic map is known at the start of its movement, other sensors can measure the changes in the pose and orientation angle of the warehouse storage robot, and thus, based on the position of the warehouse storage robot in the semantic map at the start of its movement, in combination with the changes in the pose and orientation angle of the warehouse storage robot, the initial position of the warehouse storage robot in the semantic map is obtained.

[0040] Embodiment B2: The warehouse storage robot is further equipped with an IMU, and during the movement of the warehouse storage robot, the IMU can collect the motion and / or posture data of the warehouse storage robot, referred to as IMU data. The IMU includes a three-axis accelerometer and a three-axis gyroscope inside. Here, the accelerometer is used to output the acceleration information of the warehouse storage robot in the directions of three coordinate axes in the robot coordinate system, and the gyroscope is used to output the angular velocity information of the warehouse storage robot relative to the directions of the three coordinate axes of the robot coordinate system. According to the angular velocity and acceleration information of the warehouse storage robot in the three-dimensional coordinate system of the robot, the change of the pose and orientation angle of the warehouse storage robot can be obtained, and the initial position of the warehouse storage robot in the semantic map can be further obtained.

[0041] Embodiment B3: The warehouse storage robot is further equipped with an odometer, and during the movement of the warehouse storage robot, the odometer can measure data such as the movement distance and angle of the warehouse storage robot, which is referred to as odometer data for short. The odometer mainly calculates the movement distance and the change in the direction angle of the warehouse storage robot relative to the ground based on the change in the pulse of the photoelectric encoder within the sampling period, and calculates the initial position of the warehouse storage robot in the semantic map.

[0042] In the above embodiment, when locating the warehousing robot in the semantic map, it is necessary to use the semantic map of the warehousing environment where the warehousing robot is located. On this basis, the embodiment of the present application also provides a schematic flow chart of a method for constructing a semantic map. As shown in FIG. 3, the method includes: 31: Collecting image or video data of the warehousing environment using an image sensor on the warehousing robot while the warehousing robot traverses the warehousing environment; 32: Performing target detection and semantic recognition on the image or video data of the warehousing environment to obtain a plurality of objects and their semantic information present in the warehousing environment; 33: Calculating relative positional relationships between the plurality of objects and the warehouse storage robot based on a transformation relationship between the sensor coordinate system and the robot coordinate system; 34: Adding semantic information of the multiple objects to the base environment map to obtain a semantic map of the warehouse storage environment according to the relative positional relationship between the multiple objects and the warehouse storage robot and the position of the warehouse storage robot in the base environment map when collecting image or video data of the warehouse storage environment.

[0043] In this embodiment, the warehousing robot needs to traverse the entire warehousing environment to create a semantic map corresponding to the warehousing environment. In the process of traversing the entire warehousing environment, an image sensor can be used to collect image or video data of the entire warehousing environment, and target detection and semantic recognition can be performed on the collected image or video data to obtain a plurality of objects existing in the warehousing environment and their semantic information. Here, the detailed implementation process of obtaining a plurality of objects existing in the warehousing environment and their semantic information can be referred to the above embodiment, and will not be repeated here.

[0044] In a preferred embodiment, in the process of traversing the entire warehousing environment, a base environment map of the warehousing environment may be created using SLAM technology. Furthermore, when image or video data of the warehousing environment is collected, the SLAM technology may be used to identify the position of the warehousing robot in the base environment map. Here, the base environment map refers to the environment map created by the warehousing robot using SLAM technology, which may be, for example, a grid map. Compared with the semantic map, the base environment map is an environment map that does not contain semantic information. Of course, it should be noted that the creation of the base environment map and the semantic map may be completed in the same traversal process, or the creation of the base environment map may be completed in advance in a different traversal process, and is not limited thereto.

[0045] After obtaining the plurality of objects existing in the warehousing environment and their semantic information, if the relative positional relationship between the plurality of objects and the warehousing robot is known, the positions of the plurality of objects in the semantic map can be determined based on the position of the warehousing robot in the base environment map when collecting image or video data of the warehousing environment. On this basis, in this embodiment, after obtaining the image or video data of the warehousing environment collected by the image sensor, the position coordinates of the target object can also be calculated, and the relative positional relationship between the target object and the warehousing robot can also be calculated based on the position coordinates of the target object. For details, please refer to the above embodiment, and will not be further repeated here.

[0046] Furthermore, obtain the positions of the multiple objects in the base environment map according to the relative positional relationship between the multiple objects and the warehousing robot and the position of the warehousing robot in the base environment map when collecting the image or video data of the warehousing environment. Finally, add the multiple objects in the warehousing environment and their semantic information to the base environment map to obtain a semantic map.

[0047] It should be noted that the steps of the method provided in the above embodiment may be executed on the same device or on different devices. For example, steps 11 to 13 may be executed on device A, or, as another example, steps 11 and 12 may be executed on device A and step 13 may be executed on device B, etc.

[0048] In addition, while some of the processes described in the above embodiments and drawings include multiple operations that are represented in a particular order, it should be clearly understood that these operations may not be performed in the order or in parallel represented herein, and that the serial numbers of the operations, such as 11, 12, etc., are used only to distinguish different operations, and the serial numbers themselves do not represent the order of execution. Also, these processes may include more or fewer operations, and these operations may be performed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this specification are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, and do not limit the fact that "first" and "second" are different types.

[0049] 4 is a structural schematic diagram of a warehouse storage robot provided by an exemplary embodiment of the present application. As shown in FIG. 4, the warehouse storage robot includes a machine body 40 in which a memory 44 and a processor 45 are provided.

[0050] The memory 44 is used to store computer programs and may be configured to store various other data to support the operation of the warehousing robot, examples of which include application or method instructions for operating the warehousing robot, contact data, phone book data, messages, photos, videos, etc.

[0051] The memory 44 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0052] The processor 45 is coupled to the memory 44 and executes a computer program in the memory 44 to collect image or video data of the surrounding warehouse storage environment using an image sensor in the warehouse storage robot during the movement of the warehouse storage robot, perform target detection and semantic recognition on the image or video data of the surrounding warehouse storage environment to obtain target objects and their semantic information existing in the surrounding warehouse storage environment of the warehouse storage robot, calculate a relative positional relationship between the target objects and the warehouse storage robot based on the transformation relationship between the sensor coordinate system and the robot coordinate system, and use the relative positional relationship between the target objects and the warehouse storage robot and the semantic information of the target objects to locate the warehouse storage robot in a semantic map according to the relative positional relationship between the target objects and the warehouse storage robot and the semantic information of the target objects, and the semantic map includes a plurality of objects existing in the warehouse storage environment and their semantic information.

[0053] In a preferred embodiment, when the processor 45 calculates the relative positional relationship between the target object and the warehouse storage robot based on the transformation relationship between the sensor coordinate system and the robot coordinate system, it specifically calculates the three-dimensional coordinates of the pixel points on the target object in the sensor coordinate system, and transforms the three-dimensional coordinates of the pixel points on the target object in the sensor coordinate system into the robot coordinate system based on the transformation relationship between the sensor coordinate system and the robot coordinate system, and calculates the relative positional relationship between the target object and the warehouse storage robot according to the three-dimensional coordinates of the pixel points on the target object in the robot coordinate system.

[0054] In a preferred embodiment, when the image sensor is a monocular camera and the target object is an object that intersects with the ground plane or is located on the ground plane, when the processor 45 calculates the three-dimensional coordinates of pixel points on the target object in the sensor coordinate system, specifically, it is used to combine the shooting geometry of the monocular camera to calculate the three-dimensional coordinates of pixel points on the target object that intersect with the ground plane on the ground plane; when the image sensor is a binocular camera, when the processor 45 calculates the three-dimensional coordinates of pixel points on the target object in the sensor coordinate system, specifically, it is used to calculate the three-dimensional coordinates of pixel points on the target object in the sensor coordinate system using a binocular stereo matching algorithm; when the image sensor is an RGBD camera, when the processor 45 calculates the three-dimensional coordinates of pixel points on the target object in the sensor coordinate system, specifically, it is used to calculate the three-dimensional coordinates of pixel points on the target object in the sensor coordinate system from the matching relationship between the collected RGB image and the depth image including the target object.

[0055] In a preferred embodiment, when the processor 45 locates the position of the warehouse storage robot in the semantic map according to the relative positional relationship between the target object and the warehouse storage robot and the semantic information of the target object, specifically, it determines the position of the target object in the semantic map according to the semantic information of the target object, which is used to locate the position of the warehouse storage robot in the semantic map according to the relative positional relationship between the target object and the warehouse storage robot and the position of the target object in the semantic map.

[0056] In a preferred embodiment, when the processor 45 determines the location of the target object in the semantic map according to the semantic information of the target object, specifically, it calculates an initial location of the warehouse storage robot in the semantic map based on other sensor data of the warehouse storage robot, which is used to search for the location of the target object around the initial location in the semantic map according to the semantic information of the target object.

[0057] In a preferred embodiment, when the processor 45 calculates the initial position of the warehouse storage robot in the semantic map based on other sensor data in the warehouse storage robot, specifically, the data is used to calculate the initial position of the warehouse storage robot in the semantic map based on at least one of laser sensor data, inertial measurement unit IMU data, and odometer data in the warehouse storage robot.

[0058] In a preferred embodiment, the processor 45 is further used to collect image or video data of the warehouse storage environment using an image sensor in the warehouse storage robot while the warehouse storage robot traverses the warehouse storage environment, perform target detection and semantic recognition on the image or video data of the warehouse storage environment to obtain multiple objects existing in the warehouse storage environment and their semantic information, calculate the relative positional relationship between the multiple objects and the warehouse storage robot based on the transformation relationship between the sensor coordinate system and the robot coordinate system, respectively, and add the semantic information of the multiple objects to the base environment map according to the relative positional relationship between the multiple objects and the warehouse storage robot and the position of the warehouse storage robot in the base environment map when collecting the image or video data of the warehouse storage environment to obtain a semantic map of the warehouse storage environment.

[0059] In a preferred embodiment, in the process of the processor 45 collecting image or video data of the warehousing environment, SLAM techniques are further used to establish a base environment map of the warehousing environment and to identify the position of the warehousing robot in the base environment map when collecting the image or video data of the warehousing environment.

[0060] In a preferred embodiment, the target object includes at least the shelf, the intersection of the shelf with the ground plane, and the intersection point formed by the shelf.

[0061] In a preferred embodiment, the target objects further include at least one of a shelf leg position, a fire hydrant, and an evacuation sign.

[0062] Furthermore, as shown in Fig. 4, the warehouse storage robot further includes a communication assembly 46, a display 47, a power supply assembly 48, an audio assembly 49, and other components. Fig. 4 only shows some components diagrammatically, and does not mean that the warehouse storage robot includes only the components shown in Fig. 4. Note that the components in the dashed box in Fig. 4 are not specified components but selectable components, and can be specifically determined depending on the product form of the warehouse storage robot.

[0063] Correspondingly, an embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, can realize each step that may be performed by a warehouse storage robot in the above-mentioned embodiment of the warehouse storage robot positioning method and semantic map creation method.

[0064] The communication assembly in FIG. 4 above is configured to facilitate wired or wireless communication between a device in which the communication assembly is disposed and other devices. The device in which the communication assembly is disposed can access a wireless network based on a communication standard such as a mobile communication network such as WiFi, 2G, 3G, 4G / LTE, 5G, or a combination thereof. In an exemplary embodiment, the communication assembly receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication assembly further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0065] The display in FIG. 4 above includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). When the screen includes a touch panel, the screen may be realized as a touch screen that receives input signals from a user. The touch panel includes one or more touch sensors for sensing touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action.

[0066] 4 described above provides power to various components of the equipment in which it is located. The power supply assembly may include a power management system, one or more power sources, and other components associated with the generation, management, and distribution of power to the equipment in which the power supply assembly resides.

[0067] The audio assembly in FIG. 4 described above may be configured to output and / or input audio signals. For example, the audio assembly includes a microphone (MIC) configured to receive an external audio signal when the device in which the audio assembly is located is in an operational mode such as a call mode, a recording mode, a voice recognition mode, etc. The received audio signal may be further stored in a memory or transmitted via a communication assembly. In some embodiments, the audio assembly further includes a speaker for outputting the audio signal.

[0068] It should be understood by those skilled in the art that the embodiments of the present application may be provided as a method, a system, or a computer program product. Thus, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer usable storage media (including, but not limited to, disk storage devices, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0069] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each step and / or block in the flowcharts and / or block diagrams, and combinations of steps and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate an apparatus that implements the function specified in a step or steps of the flowcharts and / or a block or blocks of the block diagrams.

[0070] These computer program instructions may be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner such that the instructions stored in the computer readable memory produce an article of manufacture that includes an instruction apparatus that implements the functions specified in a step or steps of the flowcharts and / or a block or blocks of the block diagrams.

[0071] These computer program instructions may be installed on a computer or other programmable data processing device such that the computer or other programmable device performs a sequence of operational steps to generate a computer-implemented process, whereby the instructions that execute on the computer or other programmable device provide steps for implementing the function specified in a procedure or procedures of the flowcharts and / or a block or blocks of the block diagrams.

[0072] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0073] Memory may include non-persistent memory, such as random access memory (RAM) and / or non-volatile memory in the form of a computer readable medium, for example, read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer readable medium.

[0074] Computer-readable media includes persistent and non-persistent, movable and non-movable media, and may achieve information storage by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), re-programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, cassette magnetic tape, magnetic tape-magnetic disk storage or other magnetic storage or other non-transmission media that may be used to store information accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals or carrier waves.

[0075] It should be noted that the terms "comprising," "including," and other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or device that includes a set of elements includes not only those elements, but also other elements not expressly listed or that are inherent in the process, method, article, or device. In the absence of more limitations, an element qualified by the term "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0076] The above is merely an embodiment of the present application, and is not intended to limit the present application. Those skilled in the art may make various modifications and changes to the present application. All modifications, equivalent replacements, improvements, etc. made within the scope of the spirit and principles of the present application should be included in the scope of the claims of the present application.

Claims

1. 1. A method for positioning a warehouse storage robot, comprising: collecting image or video data of a surrounding warehousing environment using an image sensor on the warehousing robot while the warehousing robot is moving; Performing target detection and semantic recognition on the image or video data of the surrounding warehousing environment to obtain target objects and their semantic information existing in the surrounding warehousing environment of the warehousing robot; creating a semantic map including a plurality of objects present in the surrounding warehousing environment, including the target object, and their semantic information; Calculating a relative positional relationship between the target object and the warehouse storage robot based on a transformation relationship between a sensor coordinate system and a robot coordinate system; and locating the warehousing robot in the semantic map according to the relative position relationship and the semantic information of the target object; The semantic information is information that is understood and interpreted with the help of natural language, the semantic map includes objects of the same type with different semantic information; determining a position of the warehouse storage robot in a semantic map according to a relative positional relationship between the target object and the warehouse storage robot and semantic information of the target object; determining a position of the target object in a semantic map according to semantic information of the target object; determining a position of the warehousing robot in a semantic map according to a relative positional relationship between the target object and the warehousing robot and a position of the target object in the semantic map; The process of collecting image or video data of the surrounding warehousing environment includes: and establishing a base environmental map of a surrounding warehousing environment using SLAM technology and locating the warehousing robot in the base environmental map when collecting image or video data of the surrounding warehousing environment. A method for positioning a warehouse storage robot, comprising:

2. Calculating a relative positional relationship between the target object and the warehouse storage robot based on a transformation relationship between a sensor coordinate system and a robot coordinate system calculating three-dimensional coordinates of pixel points on the target object in a sensor coordinate system; Transforming three-dimensional coordinates of pixel points on the target object in a sensor coordinate system into a robot coordinate system based on a transformation relationship between a sensor coordinate system and a robot coordinate system; 2. The method of claim 1, further comprising: calculating a relative position of the target object and the warehousing robot according to three-dimensional coordinates of pixel points on the target object in a robot coordinate system.

3. When the image sensor is a monocular camera and the target object is an object that intersects with a ground plane or an object located on the ground plane, calculating the three-dimensional coordinates of pixel points on the target object in a sensor coordinate system includes combining the shooting geometry of the monocular camera and calculating the three-dimensional coordinates of pixel points on the target object that intersect with the ground plane on the ground plane; When the image sensor is a binocular camera, calculating the three-dimensional coordinates of the pixel points on the target object in a sensor coordinate system includes calculating the three-dimensional coordinates of the pixel points on the target object in a sensor coordinate system using a binocular stereo matching algorithm; or 3. The method of claim 2, wherein when the image sensor is a red-green-blue depth map RGBD camera, calculating three-dimensional coordinates of pixel points on the target object in a sensor coordinate system includes calculating three-dimensional coordinates of pixel points on the target object in a sensor coordinate system from a matching relationship between a collected RGB image and a depth image including the target object.

4. collecting image or video data of the surrounding warehousing environment using an image sensor on the warehousing robot while the warehousing robot traverses the surrounding warehousing environment; performing target detection and semantic recognition on the image or video data of the surrounding warehousing environment to obtain a plurality of objects present in the surrounding warehousing environment and their semantic information; Calculating a relative positional relationship between each of the plurality of objects and the warehouse storage robot based on a transformation relationship between a sensor coordinate system and a robot coordinate system; The method according to any one of claims 1 to 3, further comprising: adding semantic information of the plurality of objects to the base environment map to obtain a semantic map of the surrounding warehouse storage environment according to a relative positional relationship between the plurality of objects and the warehouse storage robot and a position of the warehouse storage robot in the base environment map when collecting image or video data of the surrounding warehouse storage environment.

5. The process of collecting image or video data of the surrounding warehousing environment includes:

5. The method of claim 4, further comprising establishing a base environmental map of a surrounding warehousing environment using SLAM technology and locating the warehousing robot in the base environmental map when collecting image or video data of the surrounding warehousing environment.

6. The method according to any one of claims 1 to 3 and 5, characterized in that the target objects include at least a shelf, an intersection line between the shelf and the ground plane, and an intersection point formed by the shelf.

7. The method of claim 6 , wherein the target objects further include at least one of a shelf leg position, a fire hydrant, and an evacuation sign.

8. A warehouse storage robot comprising a machine body provided with a memory and a processor, the memory is used to store a computer program; The processor is coupled to the memory and executes the computer program to collect image or video data of a surrounding warehousing environment using an image sensor in the warehousing robot during the movement of the warehousing robot, perform target detection and semantic recognition on the image or video data of the surrounding warehousing environment to obtain target objects and their semantic information existing in the surrounding warehousing environment of the warehousing robot, calculate a relative positional relationship between the target objects and the warehousing robot based on a transformation relationship between a sensor coordinate system and a robot coordinate system, and use the relative positional relationship between the target objects and the warehousing robot and the semantic information of the target objects to locate the warehousing robot in a semantic map according to the relative positional relationship between the target objects and the warehousing robot and the semantic information of the target objects, the semantic map including a plurality of objects existing in the surrounding warehousing environment and their semantic information; The semantic information is information that is understood and interpreted with the help of natural language, the semantic map includes objects of the same type with different semantic information; determining a position of the warehouse storage robot in a semantic map according to a relative positional relationship between the target object and the warehouse storage robot and semantic information of the target object; determining a position of the target object in a semantic map according to semantic information of the target object; determining a position of the warehousing robot in a semantic map according to a relative positional relationship between the target object and the warehousing robot and a position of the target object in the semantic map; The process of collecting image or video data of the surrounding warehousing environment includes: and establishing a base environmental map of a surrounding warehousing environment using SLAM technology and locating the warehousing robot in the base environmental map when collecting image or video data of the surrounding warehousing environment. A warehouse storage robot characterized by:

9. When the processor calculates the relative positional relationship between the target object and the warehouse storage robot based on the transformation relationship between a sensor coordinate system and a robot coordinate system, specifically, Calculating three-dimensional coordinates of pixel points on the target object in a sensor coordinate system; Transforming three-dimensional coordinates of pixel points on the target object in the sensor coordinate system into the robot coordinate system based on a transformation relationship between the sensor coordinate system and the robot coordinate system; The warehouse storage robot according to claim 8, characterized in that it is used to calculate a relative positional relationship between the target object and the warehouse storage robot according to three-dimensional coordinates of pixel points on the target object in a robot coordinate system.

10. When the processor determines the position of the warehouse storage robot in the semantic map according to the relative positional relationship between the target object and the warehouse storage robot and the semantic information of the target object, specifically, determining a position of the target object in a semantic map according to the semantic information of the target object; 9. The warehouse storage robot of claim 8, wherein the robot is used to locate the warehouse storage robot in a semantic map according to a relative positional relationship between the target object and the warehouse storage robot and a position of the target object in the semantic map.

11. The processor further comprises: collecting image or video data of the surrounding warehousing environment using an image sensor on the warehousing robot while the warehousing robot traverses the surrounding warehousing environment; performing target detection and semantic recognition on the image or video data of the surrounding warehousing environment to obtain a plurality of objects present in the surrounding warehousing environment and their semantic information; Calculating a relative positional relationship between each of the plurality of objects and the warehouse storage robot based on a transformation relationship between a sensor coordinate system and a robot coordinate system; A warehouse storage robot as described in any one of claims 8 to 10, characterized in that it is used to add semantic information of the multiple objects to the base environment map to obtain a semantic map of the surrounding warehouse storage environment according to the relative positional relationship between the multiple objects and the warehouse storage robot and the position of the warehouse storage robot in the base environment map when collecting image or video data of the surrounding warehouse storage environment.

12. A computer-readable storage medium having a computer program stored thereon, the computer program causing the processor to implement the steps of the method according to any one of claims 1 to 7 when executed by the processor.

Images