Map updating method and device and mobile robot

Abstract

The invention provides a map updating method and device and a mobile robot. The method comprises the following steps: acquiring an original image and a depth image corresponding to the original image; determining a corresponding current pose of the mobile robot at the current moment based on the original image and the depth image; inputting the original image into a target detection model to obtain an initial position of the semantic target under the current pose; based on a conversion relation between the current pose and a stored initial pose, converting the initial position into a target position of the semantic target under the initial pose; wherein the initial pose is the pose of the mobile robot at the initial moment, and the initial moment is the moment of starting to update the target map; determining a target three-dimensional frame of the semantic target based on the target position; and updating the target map based on the target three-dimensional frame of the semantic target. Through the technical scheme of the invention, target map updating with higher precision can be realized.

Description

Technical Field

[0001] The present application relates to the field of robots, and in particular to a map updating method, device and mobile robot. Background Art

[0002] In recent years, various types of mobile robots have developed rapidly in terms of technology and market. Mobile robots are mechanical devices that automatically perform work and rely on their own power and control capabilities to achieve various functions. Mobile robots can accept human commands, run pre-programmed programs, and act according to strategies formulated by artificial intelligence. For example, users use manual remote controls to control mobile robots to perform related operations. For example, the manual remote control sends operation commands to the mobile robot wirelessly. After receiving the operation command, the mobile robot executes the operation specified by the operation command to complete the related functions.

[0003] With the rapid development of mobile robot technology, mobile robots are becoming more and more commonly used in household, logistics, warehousing, factory production, etc. For example, sweeping robots can perform cleaning functions, and AGV (Automated Guided Vehicle) can transport various types of goods.

[0004] In order for mobile robots to complete related tasks, maps need to be built. SLAM (Simultaneous Localization and Mapping) is a map building method that is widely used in various types of mobile robots. The principle of SLAM map building is that the mobile robot starts moving from an unknown position in an unknown environment, locates itself according to the position and map during the movement, and builds an incremental map based on its own positioning to achieve autonomous positioning and navigation of the mobile robot.

[0005] SLAM algorithms can construct dense maps or sparse maps. When sparse maps are used for navigation tasks, they have insufficient information, while dense maps provide a lot of redundant information. In summary, maps constructed using SLAM algorithms cannot effectively guide the movement of mobile robots. Summary of the invention

[0006] The present application provides a map updating method, which is applied to a mobile robot, and the method comprises:

[0007] Acquire an original image and a depth image corresponding to the original image; determine a current position and posture of the mobile robot at a current moment based on the original image and the depth image;

[0008] Input the original image to the target detection model to obtain the initial position of the semantic target in the current posture; based on the conversion relationship between the current posture and the stored initial posture, convert the initial position into the target position of the semantic target in the initial posture; wherein the initial posture is the posture of the mobile robot at the initial moment, and the initial moment is the moment to start updating the target map;

[0009] Determine a target three-dimensional frame of the semantic target based on the target position;

[0010] The target map is updated based on the target three-dimensional box of the semantic target.

[0011] The present application provides a map updating device, which is applied to a mobile robot, and the device comprises:

[0012] An acquisition module, used to acquire an original image and a depth image corresponding to the original image; and determine a current posture of the mobile robot corresponding to a current moment based on the original image and the depth image;

[0013] A processing module, used for inputting the original image into a target detection model to obtain an initial position of the semantic target in a current posture; based on a conversion relationship between the current posture and the stored initial posture, converting the initial position into a target position of the semantic target in the initial posture; wherein the initial posture is the posture of the mobile robot at an initial moment, and the initial moment is the moment when the target map is started to be updated;

[0014] A determination module, configured to determine a target three-dimensional frame of the semantic target based on the target position;

[0015] A map management module is used to update the target map based on the target three-dimensional box of the semantic target.

[0016] The present application provides a mobile robot, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the map updating method of the above example.

[0017] The present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the map updating method of the above example is implemented.

[0018] The present application provides a machine-readable storage medium, which stores machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions, and when the machine-executable instructions are executed, the map updating method of the above example is implemented.

[0019] It can be seen from the above technical solutions that in the embodiment of the present application, the position of the semantic target is obtained based on the target detection model, the target three-dimensional frame of the semantic target is determined based on the position, and then the target three-dimensional frame is used to update the target map. When the target map is updated based on the target three-dimensional frame, a higher-precision target map update can be achieved, and the target map has higher precision and higher accuracy. In order to improve the estimation accuracy of the target three-dimensional frame in the three-dimensional space, the position of the semantic target is output in real time based on the target detection model (visual large model), and an accurate and reliable target three-dimensional frame is obtained, the association standard between the target three-dimensional frame and the target map is improved, and the accuracy of the target map is greatly improved. Data association and fusion with the target three-dimensional frame as the core can greatly reduce the "deformation" of the target frame caused by the increase in posture error, and achieve a higher-precision target map update. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of a map updating method in one embodiment of the present application;

[0021] Figure 2 is a schematic structural diagram of a mobile robot in one embodiment of the present application;

[0022] Figure 3 is a flowchart of a map updating method in one embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a coverage model in one embodiment of the present application;

[0024] Figure 5 is a flowchart of a map updating method in one embodiment of the present application;

[0025] Figure 6 is a structural schematic diagram of a map updating device in one embodiment of the present application;

[0026] Figure 7 It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0027] In the embodiment of the present application, a map updating method is proposed, which can be applied to a mobile robot. Figure 1 FIG. 1 is a flow chart of the map updating method, and the map updating method may include:

[0028] Step 101: Acquire an original image and a depth image corresponding to the original image; and determine a current position and posture of a mobile robot at a current moment based on the original image and the depth image.

[0029] Step 102: Input the original image to the target detection model to obtain the initial position of the semantic target in the current position; based on the conversion relationship between the current position and the stored initial position, convert the initial position into the target position of the semantic target in the initial position. Exemplarily, the initial position may be the position of the mobile robot at the initial moment, and the initial moment may be the moment when the target map is started to be updated.

[0030] Step 103: Determine a target 3D box of the semantic target based on the target position.

[0031] Step 104: Update the target map based on the target three-dimensional box of the semantic target.

[0032] Exemplarily, updating the target map based on the target 3D frame of the semantic target may include, but is not limited to: if the target map does not exist at the current moment, then the target map may be created, and the location may be added to the target map based on the target 3D frame. Figure 3 or, if the target map already exists at the current moment, and the target 3D frame is aligned with each location on the target map, Figure 3 If the distance between the target 3D frame and the target map is not less than the first threshold, a location may be added to the target map based on the target 3D frame. Figure 3 Then, for the ground Figure 3 The first-level attributes and second-level attributes of the semantic target are added to the dimension box. When the original image is input to the target detection model, the target detection model can also output the first-level attributes of the semantic target, and the first-level attributes can represent the category of the semantic target. In addition, the original image can also be input to the image retrieval model to obtain the second-level attributes of the semantic target, and the second-level attributes can represent the detailed information of the semantic target, and different semantic targets have different second-level attributes.

[0033] Exemplarily, updating the target map based on the target 3D frame of the semantic target may include but is not limited to: if a target map already exists at the current moment, and the target 3D frame is consistent with the first location on the target map; Figure 3 If the distance between the target three-dimensional frame and the first three-dimensional frame is less than the first threshold, Figure 3 Dimension frame to determine the second place Figure 3 Dimension frame; among them, the second Figure 3 The target 3D box or the second Figure 3 The first place Figure 3 Dimensional frame, or second place Figure 3 The target 3D box and the first Figure 3 Then, through the second Figure 3 The dimension box replaces the first place on the target map Figure 3 Dimension frame.

[0034] Exemplarily, based on the target three-dimensional frame and the first Figure 3Dimension frame to determine the second place Figure 3 The three-dimensional frame may include but is not limited to: based on the size of the target three-dimensional frame and the first Figure 3 If the size error is not greater than the second threshold, the point cloud data corresponding to the target three-dimensional frame and the first point cloud data can be compared. Figure 3 The point cloud data corresponding to the first point cloud frame are fused to obtain fused point cloud data; a fused 3D frame is generated based on the fused point cloud data. Figure 3 If the size of the 3D frame matches the size of the 3D frame, the fused 3D frame can be determined as the second Figure 3 dimensional frame; if the size of the fused 3D frame is Figure 3 If the size of the target 3D frame does not match, the target 2D frame can be obtained by back-projecting the target 3D frame onto the image coordinate system. Figure 3 The first coordinate system is obtained by back-projecting the dimensional frame onto the image coordinate system. Figure 2 If the distance between the target two-dimensional frame and the center of the image is less than the first Figure 2 The distance between the target 3D box and the center of the image can be determined as the second Figure 3 If the distance between the target two-dimensional frame and the center of the image is greater than the first Figure 2 The distance between the first dimension box and the center of the image can be Figure 3 The dimension frame is determined as the second Figure 3 Dimension frame.

[0035] Exemplarily, based on the target three-dimensional frame and the first Figure 3 Dimension frame to determine the second place Figure 3 The three-dimensional frame may include but is not limited to: based on the size of the target three-dimensional frame and the first Figure 3 If the size error is greater than the second threshold, the target three-dimensional frame and the first geomancy frame are obtained. Figure 3 The overlapping area between the two-dimensional frames corresponds to the overlapping area point cloud data. The target three-dimensional frame is back-projected into the image coordinate system to obtain the target two-dimensional frame. Figure 3 The first coordinate system is obtained by back-projecting the dimensional frame onto the image coordinate system. Figure 2 If the distance between the target two-dimensional frame and the center of the image is less than the first Figure 2 If the distance between the target 2D frame and the center of the image is greater than the first point cloud data, the point cloud data corresponding to the target 3D frame and the point cloud data of the overlapping area are fused to obtain fused point cloud data. Figure 2 The distance between the dimensional box and the center of the image is Figure 3 The point cloud data corresponding to the 3D frame is fused with the point cloud data of the overlapping area to obtain fused point cloud data. Then, the fused 3D frame corresponding to the fused point cloud data is determined as the second Figure 3 Dimension frame.

[0036] For example, if you traverse from the target map to the third place Figure 3 Dimensional frame and fourth place Figure 3 dimension frame, you can Figure 3 Dimensional frame and fourth place Figure 3 Dimension frame determines the fifth place Figure 3 Dimension frame; among them, the third place Figure 3 The first-level attributes and fourth-level attributes of the dimension box Figure 3 The first-level attributes of the dimension box are the same, the third Figure 3 Secondary attributes and fourth place of dimension box Figure 3 The secondary attributes of the dimension box are the same, the third Figure 3 Dimensional Frame and the Fourth Place Figure 3 The distance between the fifth dimension frame is not less than the first threshold and less than the third threshold. Figure 3 Dimension frame is the third place Figure 3 If you click on the dimension box, you can delete the fourth area from the target map. Figure 3 Dimension frame; or, if the fifth Figure 3 The fourth place Figure 3 If you select the dimension box, you can delete the third place from the target map. Figure 3 Dimension frame.

[0037] For example, based on a third Figure 3 Dimensional frame and fourth place Figure 3 Dimension frame determines the fifth place Figure 3 dimensions, including but not limited to: Figure 3 The third dimension frame is back-projected onto the image coordinate system to obtain Figure 2 Dimension frame, and the fourth Figure 3 The fourth dimension box is back-projected onto the image coordinate system to obtain Figure 2 Dimension frame; if the third Figure 2 The distance between the dimensional frame and the center of the image is less than the fourth Figure 2 The distance between the third dimension box and the center of the image is Figure 3 The fifth place is determined by the dimension frame Figure 3 Dimension frame; if the third Figure 2 The distance between the frame and the center of the image is greater than the fourth Figure 2 The distance between the fourth dimension box and the center of the image is Figure 3 The fifth place is determined by the dimension frame Figure 3 Dimension frame.

[0038] It can be seen from the above technical solutions that in the embodiment of the present application, the position of the semantic target is obtained based on the target detection model, the target three-dimensional frame of the semantic target is determined based on the position, and then the target three-dimensional frame is used to update the target map. When the target map is updated based on the target three-dimensional frame, a higher-precision target map update can be achieved, and the target map has higher precision and higher accuracy. In order to improve the estimation accuracy of the target three-dimensional frame in the three-dimensional space, the position of the semantic target is output in real time based on the target detection model (visual large model), and an accurate and reliable target three-dimensional frame is obtained, the association standard between the target three-dimensional frame and the target map is improved, and the accuracy of the target map is greatly improved. Data association and fusion with the target three-dimensional frame as the core can greatly reduce the "deformation" of the target frame caused by the increase in posture error, and achieve a higher-precision target map update.

[0039] The above technical solutions of the embodiments of the present application are described below in combination with specific application scenarios.

[0040] In an embodiment of the present application, a map updating method is proposed, which can be applied to a mobile robot. The mobile robot may include a camera, and the camera is used to collect raw images and depth images. The raw image may be any type of image, such as an RGB image, a RAW image, a YUV image, etc. There is no restriction on the type of the raw image. The raw image is an RGB image as an example. The depth image may include the depth value of each pixel in the raw image (used to indicate the distance between the physical position corresponding to the pixel and the camera).

[0041] For example, the mobile robot may include an RGB-D camera, and the RGB-D camera may be used to capture an RGB image and a depth image, that is, the RGB image is used as the original image. Of course, in addition to the RGB-D camera, the mobile robot may also include other types of cameras, as long as they can capture the original image and the depth image.

[0042] See also Figure 2 The figure shows a schematic diagram of the structure of the mobile robot. The mobile robot can be equipped with a mechanical arm and an RGB-D camera. The RGB-D camera is mounted at the end of the mechanical arm. The field of view of the RGB-D camera faces downward and forms a certain angle with the horizontal plane. The RGB-D camera can obtain more information by tilting. The mechanical arm can be a 6-degree-of-freedom mechanical arm. The mobile robot is equipped with a movable platform, which can control the movement of the mobile robot. The chassis of the mobile robot is equipped with a logic operation unit, and the map update method is executed by the logic operation unit.

[0043] When the mobile robot is stationary, the robotic arm can be controlled to move, thereby controlling the RGB-D camera to move with the robotic arm, so as to control the field of view of the RGB-D camera and collect images within a suitable range.

[0044] During the movement of the mobile robot, the robotic arm remains stationary, the field of view of the RGB-D camera remains different, and the chassis center of the mobile robot and the external parameter matrix of the RGB-D camera remain fixed.

[0045] In the above application scenario, a map updating method is proposed in the embodiment of the present application, which realizes the updating of the target map based on the fusion of RGB-D camera and large model (target detection model). Figure 3 FIG. 1 is a flow chart of the map updating method, and the map updating method may include:

[0046] Step 301: Obtain an original image and a depth image corresponding to the original image.

[0047] Exemplarily, the mobile robot can periodically acquire the original image and the depth image, and the acquisition time of the original image is the same as the acquisition time of the depth image. For example, when the current moment is time t1, the RGB image and the depth image are acquired by the RGB-D camera, that is, the acquisition time of the RGB image and the depth image is both time t1. When the current moment is time t2, the RGB image and the depth image are acquired by the RGB-D camera, that is, the acquisition time of the RGB image and the depth image is both time t2, and so on.

[0048] In summary, for each moment, when the current moment is the moment, the original image and the depth image can be obtained, and then the subsequent map update process is performed based on the original image and the depth image. For the convenience of description, the processing process of an original image and the depth image corresponding to the original image is taken as an example.

[0049] Step 302: determine the current posture of the mobile robot at the current moment based on the original image and the depth image, that is, the posture of the mobile robot at the current moment is called the current posture.

[0050] For example, based on the original image and depth image captured by the RGB-D camera, the SLAM algorithm can be used to update the current posture of the mobile robot at the current moment, and there is no restriction on the posture update process.

[0051] For example, based on the original image and the original image frame before the original image, multiple feature points can be found. For each feature point, the feature point corresponds to a first pixel point coordinate in the original image, the feature point corresponds to a second pixel point coordinate in the original image frame before the original image, and the feature point corresponds to a depth value in the depth image. Based on the first pixel point coordinate, the second pixel point coordinate and the depth value corresponding to each feature point, the current posture of the mobile robot can be determined, and there is no restriction on this determination process.

[0052] Step 303: input the original image to the target detection model (i.e., the large visual model), and the target detection model outputs the initial position of the semantic target in the current position and the primary attributes of the semantic target.

[0053] Exemplarily, the target detection model can be pre-trained, the input of the target detection model can be an image (such as an RGB image), and the output of the target detection model can be the location of the semantic target and the primary attributes of the semantic target. For example, the target detection model can be trained by a server and sent to each mobile robot, or each mobile robot can train the target detection model by itself, and there is no restriction on the training process of the target detection model. The target detection model can be a neural network model or a deep learning network model, and there is no restriction on the network structure of the target detection model.

[0054] Exemplarily, based on the trained object detection model, each time an original image is obtained, the original image can be input to the object detection model, that is, the object detection model is an online model for processing the original image in real time. After receiving the original image, the object detection model can process the original image to obtain the initial position of the semantic target and the primary attributes of the semantic target.

[0055] For example, a semantic target may be an object within the field of view of a camera, that is, an object within the field of view of a camera is called a semantic target, and a semantic target may be an object with attributes such as category, color, and text. During the movement of the mobile robot, each time the original image is captured by the camera, the original image may not include a semantic target, that is, the target detection model will not output the initial position and primary attributes of the semantic target. Alternatively, the original image may include a semantic target, that is, the target detection model outputs the initial position and primary attributes of the semantic target. The original image may include at least one semantic target, and the target detection model outputs the initial position and primary attributes of each semantic target. The following will be explained using a semantic target as an example.

[0056] For example, the original image includes multiple pixels, and the target detection model outputs which pixels belong to the semantic target and which pixels do not belong to the semantic target, that is, outputs the Mask information of the semantic target. The Mask information of the semantic target can determine the position of the semantic target, and the position of the semantic target includes the pixels belonging to the semantic target in the original image. Since the mobile robot is in the current posture at the current moment, the position of the semantic target can be called the initial position of the semantic target in the current posture.

[0057] For example, considering that each semantic target has complex semantic information, the semantic information of the semantic target can be divided into primary attributes and secondary attributes. The detection difficulty of the secondary attributes is higher than that of the primary attributes. The primary attributes can be detected and output by the target detection model, and the secondary attributes can be detected and output by the image retrieval model. The primary attributes can represent the category of the semantic target (i.e., category attributes), such as cups, tables, etc. The secondary attributes can represent the detailed information of the semantic target (i.e., detail attributes). For example, the secondary attributes of a cup can be the color of the cup, the material of the cup, the text on the cup, etc.

[0058] It should be noted that different semantic targets have different secondary attributes. For example, the secondary attributes of different categories are different, such as the secondary attributes of a cup are different from those of a table. The secondary attributes of the same category are different, such as the secondary attributes of cup 1 are different from those of cup 2. For example, different cups may be made of different materials, and the text on different cups may also be different.

[0059] When the original image is input to the object detection model, the object detection model can analyze the semantic target, obtain the primary attributes of the semantic target, and output the primary attributes of the semantic target.

[0060] Step 304: based on the conversion relationship between the current posture and the stored initial posture, convert the initial position into the target position of the semantic target in the initial posture. Exemplarily, the initial posture may be the posture of the mobile robot at an initial moment, and the initial moment may be the moment when the target map is started to be updated.

[0061] Exemplarily, if there is no target map at the current moment, the current moment can be taken as the initial moment, that is, the current moment is the moment to start updating the target map. In this case, the current posture can be the initial posture, and therefore, the initial position can be the target position of the semantic target in the initial posture.

[0062] Alternatively, if the target map does not exist at time t1, and time t1 is taken as the initial time, that is, the target map is updated starting at time t1, then the position of the mobile robot at time t1 can be taken as the initial position and stored. On this basis, after obtaining the current position of the mobile robot at the current moment, the position difference between the current position and the initial position can be calculated. This position difference represents the conversion relationship between the current position and the initial position. Then, the initial position of the semantic target at the current position is added to this position difference, and the target position of the semantic target at the initial position can be obtained. The target position represents the position of the semantic target at the initial position, not the position of the semantic target at the current position.

[0063] Step 305: Determine a target three-dimensional frame of the semantic target based on the target position of the semantic target.

[0064] Exemplarily, the target two-dimensional box of the semantic target (i.e., the 2D box of the semantic target) can be determined based on the target position of the semantic target. The target two-dimensional box is the smallest 2D box that envelops the semantic target with direction. That is, the target two-dimensional box can be the smallest 2D box that can envelop the target position of the semantic target.

[0065] For example, the coverage model can be used to determine the target two-dimensional box of the semantic target. The coverage model can include component convex hull, directional bounding box and axis-aligned bounding box. The component convex hull is the most fine-grained detection primitive, the axis-aligned bounding box is the coarsest-grained detection primitive, and the directional bounding box is the intermediate-grained detection primitive. Figure 4 The figure shows a schematic diagram of the coverage model. The deepest layer is the component convex hull. As a detection primitive, the component convex hull can be referred to as the convex hull. The component convex hull fits and surrounds the semantic target through multiple adjacent triangular facets. For example, the component convex hull is a method of representing convex polyhedrons. It is used in 3D reconstruction and detection. It simplifies the model by fitting and surrounding complex 3D models with many adjacent triangular facets. In this example, the component convex hull can be obtained by fitting and surrounding the semantic target through multiple adjacent triangular facets. There is no restriction on the acquisition of the component convex hull, and it can be configured.

[0066] The detection primitive of the second layer is the oriented bounding box, which can be called OBB (Oriented Bounding Box). The oriented bounding box is the smallest circumscribed cube used to envelop the convex hull of the component. The detection primitive of the outermost layer is the axis-aligned bounding box, which can be called AABB (Axis Aligned Bounding Box). The axis-aligned bounding box is the smallest axis-aligned circumscribed cube used to envelop the oriented bounding box.

[0067] Based on the above-mentioned coverage range model, the target two-dimensional box is enclosed by the convex hull of the component. Of course, the above is only an example of determining the target two-dimensional box of the semantic target, and there is no limitation on this determination method.

[0068] Exemplarily, after obtaining the target two-dimensional frame of the semantic target, the target two-dimensional frame of the semantic target can also be converted into a target three-dimensional frame of the semantic target, that is, the 2D frame of the semantic target is converted into a 3D frame. For example, the pixel points corresponding to the semantic target in the original image can be determined, and the depth image includes the depth value corresponding to each pixel point in the original image. Therefore, the depth value corresponding to the semantic target can be determined, and based on the depth value corresponding to the semantic target, the 2D frame of the semantic target can be converted into a 3D frame, that is, the target three-dimensional frame of the semantic target is obtained, and there is no restriction on the method of obtaining the target three-dimensional frame.

[0069] Exemplarily, after obtaining the target position of the semantic target, the target position of the semantic target can also be converted into point cloud data of the semantic target, and the point cloud data can include multiple three-dimensional points. For example, the pixel point corresponding to the target position of the semantic target in the original image can be determined, and the depth image includes the depth value corresponding to each pixel point in the original image. Therefore, the depth value corresponding to the target position can be determined, and based on the depth value corresponding to the target position, the target position can be converted into point cloud data of the semantic target, thereby obtaining the point cloud data of the semantic target, and there is no restriction on the method of obtaining this point cloud data.

[0070] In summary, the target 3D box of the semantic target, the primary attributes of the semantic target and the point cloud data of the semantic target can be obtained, and the target 3D box, the primary attributes and the point cloud data can be associated.

[0071] Step 306: Determine whether the target map already exists at the current moment.

[0072] If not, that is, the target map does not exist at the current moment, step 307 can be executed.

[0073] If yes, that is, the target map already exists at the current moment, step 308 can be executed.

[0074] Step 307: Create a target map (i.e., a three-dimensional target map), and add a map to the target map based on the target three-dimensional frame of the semantic target. Figure 3 Dimension frame, for the ground Figure 3 The dimension box adds the first-level attributes of the semantic target.

[0075] For example, if the target map does not exist at the current moment, it means that there is no established target map at the current moment, and the target map needs to be established first. This target map is an initial map, and in the subsequent process, this target map needs to be updated. In addition, when the target map does not exist at the current moment, the current moment can also be used as the initial moment, the current posture can be used as the initial posture, and the initial posture can be stored.

[0076] After the target map is established, the target 3D box of the semantic target can be added to the target map. Figure 3 Dimension frame, this place Figure 3 The dimension box corresponds to the semantic target, and this location Figure 3 The position of the 3D box is the same as the position of the target 3D box, that is, Figure 3 The size of the 3D box can be the same as the size of the target 3D box, and the Figure 3 The center position of the 3D frame can be the same as the center position of the target 3D frame, that is, the target 3D frame can be added to the target map to obtain the center position of the target 3D frame. Figure 3 Dimension frame.

[0077] Add the location to the target map Figure 3 After the frame is set, you can also Figure 3 The dimension box adds the first-level attributes of the semantic target, that is, the location is added to the target map. Figure 3 A dimension box is associated with the primary attribute.

[0078] Step 308: Calculate the target three-dimensional frame and each location on the target map. Figure 3 The distance of the dimension box.

[0079] For example, for each location on the target map Figure 3 3D box, the center position of the target 3D box and the location of the Figure 3 The distance between the center positions of the target 3D box (such as Euclidean distance, etc.). Obviously, the center position of the target 3D box can be a 3D position point. Figure 3 The center position of the dimension box can be a three-dimensional position point, and the distance between two three-dimensional position points can be calculated, and there is no restriction on the calculation method.

[0080] Step 309: Based on the target three-dimensional frame and each location on the target map Figure 3 The distance between the two boxes is determined to determine whether there is a distance less than a first threshold (such as TH1, the first threshold can be configured based on experience).

[0081] If not, the target 3D frame and each location on the target map Figure 3 If the distance between the target 3D frame and a certain location on the target map is not less than the first threshold, step 310 can be performed. Figure 3 Dimension frame (recorded as the first Figure 3 If the distance between the two dimensions (the dimension box) is less than the first threshold, step 311 can be executed.

[0082] Step 310: Add a target 3D frame based on the semantic target to the target map. Figure 3 Dimension frame, this place Figure 3 The dimension box corresponds to the semantic target and is the Figure 3 The dimension box adds the first-level attributes of the semantic target.

[0083] For example, if the target three-dimensional frame is aligned with each location on the target map Figure 3 If the distances of the target 3D frame and the semantic target are not less than the first threshold, it means that the target 3D frame is a new target 3D frame and the semantic target is a new semantic target. Therefore, the target 3D frame based on the semantic target can be added to the target map. Figure 3 The target map is updated by opening the dimension frame. Figure 3 The position of the 3D box is the same as that of the target 3D box.

[0084] Add the location to the target map Figure 3 After the frame is set, you can also Figure 3 The dimension box adds the first-level attributes of the semantic target, that is, the location is added to the target map. Figure 3 A dimension box is associated with the primary attribute.

[0085] Step 311: Based on the target 3D frame of the semantic target and the first Figure 3 Dimension frame to determine the second place Figure 3 dimension frame, and through the second Figure 3 The dimension box replaces the first place on the target map Figure 3 Dimension frame.

[0086] For example, if the target three-dimensional frame is aligned with the first location on the target map, Figure 3 The distance between the target 3D box and the first 3D box is less than the first threshold. Figure 3 The 3D box is the 3D box of the same semantic target, that is, the map of the semantic target that has been added to the target map before the current moment. Figure 3 When the target 3D frame of the semantic target is obtained again at the current moment, the second 3D frame of the semantic target needs to be re-determined. Figure 3 Dimension frame, through the second place Figure 3 The dimension box replaces the first place on the target map Figure 3 Dimension frame.

[0087] For example, add a second location to the target map Figure 3 Dimension box, delete the first Figure 3 The target map is updated by opening the dimension box. Figure 3 After the frame is restored, you can also Figure 3 Add the first-level attributes of the semantic target to the dimension box and delete the first-level attributes. Figure 3 The first-level attribute associated with the dimension frame, or you can directly Figure 3 Dimensional frame and first place Figure 3 The first-level attributes of the dimension box are associated.

[0088] Second place Figure 3 The 3D box can be a target 3D box, or a second Figure 3 The dimension frame can be the first Figure 3 Dimensional frame, or, second place Figure 3 The 3D frame can be the target 3D frame and the first Figure 3 On this basis, for step 311, the target map can be updated by the following steps:

[0089] Step S11: based on the size of the target three-dimensional frame and the first Figure 3 The size of the dimension box determines the size error.

[0090] For example, based on the length of the target 3D box and the first Figure 3The length of the target 3D box can be determined by calculating the length difference, that is, the difference between the two lengths. Figure 3 The width of the target 3D box can be determined by calculating the width difference, that is, the difference between the two widths. Figure 3 By measuring the height of the dimension box, you can determine the height difference, that is, the difference between the two heights.

[0091] The size error can be determined based on the length difference, width difference and height difference. For example, the average of the length difference, width difference and height difference is used as the size error, or the length difference, width difference and height difference are weighted to obtain the size error. There is no limitation on the method of determining the size error.

[0092] Step S12: determine whether the size error is greater than a second threshold TH2 (which can be configured based on experience).

[0093] If not, that is, the size error is not greater than the second threshold, step S13 may be executed.

[0094] If yes, that is, the size error is greater than the second threshold, step S18 may be executed.

[0095] Step S13: compare the point cloud data corresponding to the target three-dimensional frame and the first Figure 3 The point cloud data corresponding to the three-dimensional frame are fused to obtain fused point cloud data; and a fused three-dimensional frame is generated based on the fused point cloud data.

[0096] Exemplarily, if the size error is not greater than the second threshold, it means that the target three-dimensional frame and the first Figure 3 The size of the target 3D frame is close to that of the first 3D frame. Figure 3 Dimensional frames are merged.

[0097] Referring to step 305, the target 3D frame and the point cloud data can be associated, so the point cloud data corresponding to the target 3D frame can be found (referred to as point cloud data A). Figure 3 When maintaining the frame, the first Figure 3 The 3D box is also used as the target 3D box, and the first Figure 3 The dimension box is associated with the point cloud data, so the first location can be found Figure 3 The point cloud data corresponding to the dimension box (referred to as point cloud data B).

[0098] On this basis, the point cloud data A and the point cloud data B can be fused to obtain fused point cloud data, that is, the fused point cloud data can include the union of the point cloud data A and the point cloud data B.

[0099] After obtaining the fused point cloud data, a fused 3D box is generated based on the fused point cloud data. The fused point cloud data includes multiple 3D points, and the fused 3D box is the smallest 3D box that can envelop these 3D points. For example, the coverage range model of the component convex hull, the directional bounding box, and the axis-aligned bounding box can be used to determine the smallest 3D box that can envelop these 3D points, and there is no restriction on the method of obtaining the fused 3D box.

[0100] Step S14: Determine whether the size of the fused three-dimensional frame is consistent with the first Figure 3 Check whether the dimensions of the frame match.

[0101] If so, the size of the fused 3D frame is Figure 3 If the size of the fused 3D frame matches the size of the first 3D frame, then step S15 is executed. Figure 3 If the size of the dimension box does not match, step S16 is executed.

[0102] For example, if the size of the fused 3D box is equal to the first Figure 3 If the size of the fused 3D frame is the same as the size of the first 3D frame, Figure 3 If the size of the fused 3D frame matches the size of the first Figure 3 If the sizes of the 3D frames are different, the size of the fused 3D frame is the same as the size of the first Figure 3 The dimensions of the dimension boxes do not match.

[0103] For example, based on the size of the fused 3D frame and the first Figure 3 The size of the 3D frame is used to determine the size error. If the size error is less than the threshold, the size of the fused 3D frame is Figure 3 If the size error is not less than the threshold, the size of the fused 3D frame is equal to the size of the first Figure 3 The dimensions of the dimension boxes do not match.

[0104] Step S15: determine the fused 3D frame as the second Figure 3 dimension frame, and through the second Figure 3 The dimension box replaces the first place on the target map Figure 3 Dimension box, that is, you need to delete the first location on the target map Figure 3 Dimension frame, and the second Figure 3 Dimensional frame and first place Figure 3 The first-level attributes of the dimension box are associated.

[0105] Step S16: Back-project the target three-dimensional frame onto the image coordinate system to obtain the target two-dimensional frame, and Figure 3 The first coordinate system is obtained by back-projecting the dimensional frame onto the image coordinate system. Figure 2 Dimension frame.

[0106] Exemplarily, the target 3D frame can be back-projected into the image coordinate system to obtain the target 2D frame in the image coordinate system. That is, each 3D point of the target 3D frame can be back-projected into the image coordinate system to obtain the 2D point corresponding to each 3D point (how to map the 3D point to the 2D point in the image coordinate system is not limited in this embodiment). The area formed by these 2D points is the target 2D frame. Figure 3 The dimensional frame is back-projected into the image coordinate system to obtain the first Figure 2 dimension frame, that is, the first Figure 3 Each 3D point of the dimensional box is back-projected into the image coordinate system to obtain the 2D point corresponding to each 3D point. The area composed of these 2D points is the first Figure 2 Dimension frame.

[0107] Step S17: If the distance between the target two-dimensional frame and the image center is less than the first Figure 2 The distance between the target 3D box and the center of the image is determined as the second Figure 3 If the distance between the target two-dimensional frame and the center of the image is greater than the first Figure 2 The distance between the first dimension box and the center of the image is Figure 3 The dimension frame is determined as the second Figure 3 If the distance between the target two-dimensional frame and the center of the image is equal to the first Figure 2 The distance between the target 3D box and the center of the image is Figure 3 The dimension frame is determined as the second Figure 3 Dimension frame.

[0108] For example, the original image is located in the image coordinate system, and the central pixel point of the original image can be used as the image center, and the distance between the center point of the target two-dimensional box and the image center (such as the Euclidean distance, etc.) can be determined, and the first location can be determined. Figure 2 On this basis, if the distance between the target two-dimensional box and the image center is less than the first Figure 2 The distance between the target 3D box and the center of the image can indicate that the confidence of the target 3D box is higher, and the target 3D box is used as the second Figure 3 If the distance between the target two-dimensional frame and the center of the image is greater than the first Figure 2 The distance between the dimensional box and the center of the image can represent the first Figure 3 The confidence of the dimension box is higher, and the first Figure 3 Dimension frame as a second place Figure 3 Dimension frame.

[0109] If the target 3D box is used as the second Figure 3 dimension frame, you can use the second Figure 3 The dimension box replaces the first place on the target map Figure 3In this case, you need to delete the first location on the target map. Figure 3 Dimension frame, and the second Figure 3 Dimensional frame and first place Figure 3 The first-level attribute of the dimension box is associated. Figure 3 Dimension frame as a second place Figure 3 dimension frame, you can use the second Figure 3 The dimension box replaces the first place on the target map Figure 3 dimension box, in which case the first location on the target map is retained Figure 3 Just add the frame.

[0110] Step S18: Obtain the target three-dimensional frame and the first location Figure 3 For example, we can first determine the overlapped area between the target 3D frame and the first point cloud data. Figure 3 The overlapping area between the two dimensional frames can then be obtained, and the overlapping area point cloud data corresponding to the overlapping area can be obtained.

[0111] For example, if the size error is greater than the second threshold, it means that the target 3D frame and the first Figure 3 The size of the target 3D frame is quite different from that of the first 3D frame. Figure 3 The target 3D frame will intersect or overlap, so it can be determined that the target 3D frame is Figure 3 Based on the point cloud data A corresponding to the target 3D frame and the first Figure 3 The point cloud data B corresponding to the dimension box can find the point cloud data corresponding to the overlapping area from the point cloud data A and the point cloud data B, which is recorded as the overlapping area point cloud data.

[0112] Step S19: Back-project the target three-dimensional frame onto the image coordinate system to obtain the target two-dimensional frame. Figure 3 The first coordinate system is obtained by back-projecting the dimensional frame onto the image coordinate system. Figure 2 Dimension box, see step S16.

[0113] Step S20: If the distance between the target two-dimensional frame and the image center is less than the first Figure 2 If the distance between the target 2D frame and the image center is greater than the first ground frame, the point cloud data corresponding to the target 3D frame and the point cloud data in the overlapping area are fused to obtain fused point cloud data. Figure 2 The distance between the dimensional box and the center of the image is Figure 3 The point cloud data corresponding to the target two-dimensional frame and the point cloud data of the overlapping area are fused to obtain the fused point cloud data. Figure 2 The distance between the target 3D box and the center of the image is the distance between the point cloud data (or the first Figure 3The point cloud data corresponding to the dimensional box) is fused with the point cloud data of the overlapping area to obtain the fused point cloud data.

[0114] For example, when the point cloud data A corresponding to the target 3D frame and the point cloud data of the overlapping area are fused to obtain the fused point cloud data, the fused point cloud data includes the union of the point cloud data A and the point cloud data of the overlapping area. Figure 3 When the point cloud data B corresponding to the dimension box and the point cloud data of the overlapping area are fused to obtain the fused point cloud data, the fused point cloud data includes the union of the point cloud data B and the point cloud data of the overlapping area.

[0115] Step S21: determine the fused 3D frame corresponding to the fused point cloud data as the second Figure 3 Dimension frame, through the second place Figure 3 The dimension box replaces the first place on the target map Figure 3 Dimension box, that is, you need to delete the first location on the target map Figure 3 Dimension frame, and the second Figure 3 Dimensional frame and first place Figure 3 The first-level attributes of the dimension box are associated.

[0116] At this point, step 311 is completed, and the location of the semantic target can be updated on the target map. Figure 3 Dimension frame.

[0117] In one possible implementation, the image retrieval model may be pre-trained, the input of the image retrieval model may be an image (such as an RGB image), and the output of the image retrieval model may be a secondary attribute of a semantic target. For example, the image retrieval model may be trained by a server and sent to each mobile robot, or each mobile robot may train the image retrieval model itself, and there is no restriction on the training process of the image retrieval model. The image retrieval model may be a neural network model or a deep learning network model, and there is no restriction on the network structure of the image retrieval model.

[0118] For example, considering that the image retrieval model is used to analyze the secondary attributes of the semantic target, and the detection difficulty of the secondary attributes is higher than that of the primary attributes, it is impossible to realize the real-time detection of the secondary attributes. Therefore, the image retrieval model can be an offline model. When the original image is obtained, the original image is not directly input into the image retrieval model. Update the location of the semantic target on the target map. Figure 3 After the dimension box, for each location on the target map Figure 3 Dimension frame, if the place Figure 3 If the dimension box is already associated with a secondary attribute, the location will no longer be obtained. Figure 3 The secondary attribute associated with the dimension box. Figure 3 If the dimension box is not associated with a secondary attribute, then obtain the location Figure 3The original image corresponding to the dimensional box is input into the image retrieval model to obtain the secondary attributes of the semantic target, and the secondary attributes are compared with the location. Figure 3 On this basis, the Figure 3 The dimension box associates primary attributes and secondary attributes. The primary attributes can represent the category of the semantic target, and the secondary attributes can represent the detailed information of the semantic target. It should be noted that the secondary attributes of different semantic targets are different. For example, the secondary attributes for different categories are different, and the secondary attributes for the same category are different.

[0119] In some application scenarios, when a semantic target moves from location A to location B, multiple locations of the semantic target may be added to the target map. Figure 3 Therefore, based on the fact that the first-level and second-level attributes of the same semantic target are the same, and the second-level attributes of different semantic targets are different, it is possible to distinguish multiple locations belonging to the same semantic target. Figure 3 Dimension frame, and then delete some Figure 3 The dimension box only includes one location of the semantic target. Figure 3 dimensional box, thereby removing the ghosting problem of semantic targets in the target map.

[0120] Considering that the first-level attributes of the semantic targets in the target map will repeat and jump, Figure 3 The distance between the two boxes can be used to filter multiple locations of the same semantic target. Figure 3 Considering that different semantic targets have different secondary attributes, we can Figure 3 The secondary attributes associated with the dimension box filter multiple locations of the same semantic target Figure 3 Based on this, multiple locations with completely consistent secondary attributes and distances not less than the first threshold and less than the third threshold can be Figure 3 dimensional frame, determine the location of the same semantic target Figure 3 Dimension frame.

[0121] Based on the above principle, a map updating method is proposed in the embodiment of the present application. Figure 5 FIG. 1 is a flow chart of the map updating method, and the map updating method may include:

[0122] Step 501: traverse from the target map to the third location Figure 3 Dimensional frame and fourth place Figure 3 Dimension frame.

[0123] Exemplary, third place Figure 3 The first-level attributes and fourth-level attributes of the dimension box Figure 3 The first-level attributes of the dimension frame need to be the same, and the third-level Figure 3 Secondary attributes and fourth place of dimension box Figure 3 The secondary attributes of the dimension frame need to be the same, and the third Figure 3Dimensional Frame and the Fourth Place Figure 3 The distance between the two dimensions is not less than the first threshold and less than the third threshold.

[0124] For example, if there is a third place on the target map that meets the above conditions Figure 3 Dimensional frame and fourth place Figure 3 If the target map does not contain a third location that meets the above conditions, the target map can be updated using the subsequent steps. Figure 3 Dimensional frame and fourth place Figure 3 If the target map is not updated, the target map update process ends and the next cycle is waited for to continue traversing the third area from the target map. Figure 3 Dimensional frame and fourth place Figure 3 Dimension frame.

[0125] For example, if Figure 3 The first-level attributes and location of the dimension frame A Figure 3 The first-level attributes of dimension frame B are the same. Figure 3 The secondary attributes and location of the dimension box A Figure 3 The secondary attributes of dimension box B are the same. Figure 3 Dimension frame A and ground Figure 3 Dimension box B may be the location of the same semantic target Figure 3 Dimension frame. Figure 3 The first-level attributes and location of the dimension frame A Figure 3 The primary attributes of dimension box B are different, and / or Figure 3 The secondary attributes and location of the dimension frame A Figure 3 The secondary attributes of dimension box B are different. Figure 3 Dimension box A and ground Figure 3 Dimension box B is not the same semantic target Figure 3 Dimension frame.

[0126] For example, if Figure 3 Dimension frame A and ground Figure 3 The distance between the ground plane and the ground plane is less than the first threshold. Figure 3 Dimension frame A and ground Figure 3 Dimension box B is considered to be the location of the same semantic target. Figure 3 The implementation process of the dimension frame is shown in Figure 3 If the ground Figure 3 Dimension frame A and ground Figure 3 If the distance between the two semantic objects is not less than the third threshold, and the third threshold is determined based on the interval between the multiple objects, it means that the distance between the two semantic objects is far, exceeding the interval between the multiple objects, and it is unlikely that they are the same semantic object. Figure 3 Obviously, if the ground Figure 3 Dimension frame A and ground Figure 3 The distance between the dimension frame B is not less than the first threshold and less than the third threshold, indicating that the ground Figure 3 Dimension box A and ground Figure 3Dimension box B may be the location of the same semantic target Figure 3 Dimension frame.

[0127] Ruodi Figure 3 The first-level attributes and location of the dimension frame A Figure 3 The first-level attributes of dimension frame B are the same. Figure 3 The secondary attributes and location of dimension box A Figure 3 The secondary attributes of dimension box B are the same. Figure 3 Dimension frame A and ground Figure 3 If the distance between the dimension frame B is not less than the first threshold and less than the third threshold, then the ground is confirmed. Figure 3 Dimension frame A and ground Figure 3 Dimension box B is the location of the same semantic target Figure 3 Dimension frame, as a third place Figure 3 Dimensional frame and fourth place Figure 3 Dimension frame.

[0128] Exemplarily, there are multiple sets of matching pairs on the target map (each set of matching pairs includes a third place Figure 3 Dimensional frame and fourth place Figure 3 When multiple matching pairs are added to the target set to be associated, multiple matching pairs can be added to the target set to be associated. In the subsequent process, each matching pair in the target set to be associated needs to be traversed in sequence, and the target map is updated for the matching pairs currently traversed. The processing process of a set of matching pairs is used as an example to illustrate the following.

[0129] Step 502: The third Figure 3 The third dimension frame is back-projected onto the image coordinate system to obtain Figure 2 Dimension frame, the fourth Figure 3 The fourth dimension box is back-projected onto the image coordinate system to obtain Figure 2 Dimension box, see step S16.

[0130] Step 503: If the third party Figure 2 The distance between the dimensional frame and the center of the image is less than the fourth Figure 2 The distance between the third dimension frame and the center of the image (third Figure 3 The confidence of the third dimension box is higher). Figure 3 The fifth place is determined by the dimension frame Figure 3 If the third Figure 2 The distance between the frame and the center of the image is greater than the fourth Figure 2 The distance between the frame and the center of the image (the fourth Figure 3 The confidence of the fourth dimension box is higher). Figure 3 The fifth place is determined by the dimension frame Figure 3 If the third Figure 2 The distance between the dimensional box and the center of the image is equal to the fourth Figure 2 The distance between the third dimension frame and the center of the image is Figure 3 Dimensional frame or fourth place Figure 3The fifth place is determined by the dimension frame Figure 3 Dimension frame.

[0131] In summary, the third Figure 3 When the confidence of the third dimension box is higher, Figure 3 The fifth place is determined by the dimension frame Figure 3 Dimensional frame, fourth place Figure 3 When the confidence of the dimension box is higher, the fourth Figure 3 The fifth place is determined by the dimension frame Figure 3 In this way, the 3D frame closest to the camera center can be used as the core (when the camera is facing the target, the 3D frame is more accurate and the depth fluctuation is smaller, so the 3D frame estimation is more accurate), and the point cloud sets of the 3D frames in other postures can be filtered, registered and fused into the core 3D frame, thereby updating the target map.

[0132] So far, we can base on the third Figure 3 Dimensional frame and fourth place Figure 3 Dimension frame determines the fifth place Figure 3 Dimensional frame, fifth place Figure 3 Dimension frame is the third place Figure 3 Dimensional frame, or fifth place Figure 3 The fourth place Figure 3 Dimension frame.

[0133] Step 504: If the fifth Figure 3 Dimension frame is the third place Figure 3 If the dimension box is set, the fourth area will be deleted from the target map. Figure 3 Frame, keep third place Figure 3 dimensional box, thus retaining only one location of the same semantic target Figure 3 If the fifth Figure 3 The fourth place Figure 3 If the third area is not selected, the third area will be deleted from the target map. Figure 3 Frame, keep the fourth place Figure 3 dimensional box, thus retaining only one location of the same semantic target Figure 3 Dimension frame.

[0134] As can be seen from the above technical solutions, in the embodiment of the present application, the position of the semantic target is obtained based on the target detection model, the target three-dimensional frame of the semantic target is determined based on the position, and then the target three-dimensional frame is used to update the target map. When the target map is updated based on the target three-dimensional frame, a higher-precision target map update can be achieved, and the target map has higher precision and higher accuracy. In order to improve the estimation accuracy of the target three-dimensional frame in the three-dimensional space, the position of the semantic target is output in real time based on the target detection model (visual large model), and an accurate and reliable target three-dimensional frame is obtained. The size of the semantic target can be optimized, and the association standard between the target three-dimensional frame and the target map is improved. The accuracy of the target map is greatly improved, and it can be quickly applied to various target exploration and positioning products. Data association and fusion with the target three-dimensional frame as the core can greatly reduce the "deformation" of the target frame caused by the increase in posture error, and achieve a higher-precision target map update.

[0135] In order to improve the estimation accuracy of the three-dimensional target frame in the three-dimensional space and remove the target ghosting problem in the target map, in the embodiment of the present application, a high-precision map updating method combining an RGBD camera with a large visual model (real-time detection of the target detection model) + an image retrieval model (offline retrieval) is proposed. The target three-dimensional frame is obtained according to the real-time updated posture of the RGBD-SLAM, so as to record the position, size, point cloud data and primary attributes of the semantic target, and establish an initial semantic map (i.e., the target map). Since the primary attributes of the semantic target in the target map will repeat and jump, the Euclidean distance between the semantic targets is first used to be less than a certain threshold for initial screening and updating. Then, all semantic targets in the target map are input into the graphic retrieval model, and secondary semantic attribute detection is performed to obtain secondary attributes. The secondary attributes of all semantic targets are superimposed on the target map, and the targets with completely consistent secondary attributes but a Euclidean distance threshold greater than the first threshold and less than the third threshold (the interval between multiple targets) are listed as the target set to be fused; finally, the target set to be fused is traversed, and the 3D frame closest to the center of the camera is taken as the core (when the camera is facing the target, the 3D frame is more accurate and the depth fluctuation is smaller, and the 3D frame estimation is more accurate). The 3D frame point clouds in other postures are filtered, registered and fused into the core 3D frame to update the target map. The above method can be quickly applied to various robotic arm products, with the characteristics of strong adaptability, low cost and fast implementation, and has a wide range of application scenarios.

[0136] Based on the same application concept as the above method, a map updating device is proposed in the embodiment of the present application and applied to a mobile robot, see Figure 6 FIG. 1 is a schematic diagram of the structure of the device, wherein the device comprises:

[0137] An acquisition module 61 is used to acquire an original image and a depth image corresponding to the original image; based on the original image and the depth image, a current posture of the mobile robot at the current moment is determined; a processing module 62 is used to input the original image into a target detection model to obtain an initial position of the semantic target in the current posture; based on a conversion relationship between the current posture and the stored initial posture, the initial position is converted into a target position of the semantic target in the initial posture; wherein the initial posture is the posture of the mobile robot at the initial moment, and the initial moment is the moment to start updating the target map; a determination module 63 is used to determine a target three-dimensional box of the semantic target based on the target position; a map management module 64 is used to update the target map based on the target three-dimensional box of the semantic target.

[0138] Exemplarily, when the map management module 64 updates the target map based on the target three-dimensional frame of the semantic target, it is specifically used to: if the target map does not exist at the current moment, then create the target map, add the map to the target map based on the target three-dimensional frame, Figure 3 or, if the target map already exists at the current moment, and the target 3D frame is consistent with each location on the target map Figure 3 If the distance between the target three-dimensional frame and the target map is not less than a first threshold, a location is added to the target map based on the target three-dimensional frame. Figure 3 Dimension frame; for the place Figure 3 The first-level attributes and second-level attributes of the semantic target are added to the dimension box; wherein, when the original image is input into the target detection model, the target detection model outputs the first-level attributes of the semantic target, and the first-level attributes represent the category of the semantic target; the original image is input into the image retrieval model to obtain the second-level attributes of the semantic target, and the second-level attributes represent the detailed information of the semantic target, and different semantic targets have different second-level attributes.

[0139] Exemplarily, when the map management module 64 updates the target map based on the target three-dimensional frame of the semantic target, it is specifically used to: if a target map already exists at the current moment, the target three-dimensional frame is updated with the first location on the target map; Figure 3 If the distance between the target three-dimensional frame and the first ground frame is less than the first threshold, Figure 3 Dimension frame to determine the second place Figure 3 Dimensional frame; the second place Figure 3 The target three-dimensional frame is the target three-dimensional frame, or the first Figure 3 dimensional frame, or the target 3D frame and the first Figure 3 The fused three-dimensional frame of the second Figure 3 The first location frame on the target map is replaced by Figure 3 Dimension frame.

[0140] Exemplarily, the map management module 64 is based on the target three-dimensional frame and the first map Figure 3 Dimension frame to determine the second place Figure 3 The three-dimensional frame is specifically used to: based on the size of the target three-dimensional frame and the first Figure 3 If the size error is not greater than a second threshold, the point cloud data corresponding to the target three-dimensional frame and the first location Figure 3 The point cloud data corresponding to the first point cloud frame are fused to obtain fused point cloud data; a fused three-dimensional frame is generated based on the fused point cloud data; if the size of the fused three-dimensional frame is the same as that of the first point cloud frame, Figure 3 If the size of the fused three-dimensional frame matches that of the second three-dimensional frame, the fused three-dimensional frame is determined as the second Figure 3 If the size of the fused three-dimensional frame is the same as the first Figure 3 If the size of the target three-dimensional frame does not match, the target three-dimensional frame is back-projected onto the image coordinate system to obtain a target two-dimensional frame. Figure 3 The first coordinate system is obtained by back-projecting the dimensional frame onto the image coordinate system. Figure 2 If the distance between the target two-dimensional frame and the center of the image is less than the first Figure 2 The target three-dimensional frame is determined as the second three-dimensional frame. Figure 3 If the distance between the target two-dimensional frame and the center of the image is greater than the first Figure 2 The distance between the first dimension box and the center of the image is Figure 3 The dimension box is determined as the second place Figure 3 Dimension frame.

[0141] Exemplarily, the map management module 64 is based on the target three-dimensional frame and the first map Figure 3 Dimension frame to determine the second place Figure 3 The three-dimensional frame is specifically used to: based on the size of the target three-dimensional frame and the first Figure 3 If the size error is greater than a second threshold, obtaining the size error between the target three-dimensional frame and the first geo Figure 3 The overlapping area point cloud data corresponding to the overlapping area between the target three-dimensional frames; back-projecting the target three-dimensional frame to the image coordinate system to obtain the target two-dimensional frame, Figure 3 The first coordinate system is obtained by back-projecting the dimensional frame onto the image coordinate system. Figure 2 If the distance between the target two-dimensional frame and the center of the image is less than the first Figure 2 If the distance between the target 2D frame and the center of the image is greater than the first 3D frame, the point cloud data corresponding to the target 3D frame and the point cloud data of the overlapping area are fused to obtain fused point cloud data. Figure 2 The distance between the first dimension box and the center of the image is Figure 3The point cloud data corresponding to the 3D frame is fused with the point cloud data of the overlapping area to obtain fused point cloud data; and the fused 3D frame corresponding to the fused point cloud data is determined as the second Figure 3 Dimension frame.

[0142] Exemplarily, the map management module 64 is also used to: if the target map is traversed to a third map Figure 3 Dimensional frame and fourth place Figure 3 dimension frame, then based on the third place Figure 3 The fourth dimension frame and the fourth dimension frame Figure 3 Dimension frame determines the fifth place Figure 3 dimension frame; wherein the third Figure 3 The first-level attributes of the dimension box and the fourth Figure 3 The first-level attributes of the dimension box are the same, and the third Figure 3 The secondary attributes of the dimension box and the fourth Figure 3 The second-level attributes of the dimension box are the same, and the third Figure 3 The fourth area Figure 3 The distance between the first and third dimensions is not less than the first threshold and less than the third threshold; if the fifth dimension is Figure 3 The dimension frame is the third place Figure 3 dimension box, the fourth area is deleted from the target map Figure 3 dimension frame; or, if the fifth Figure 3 The fourth area Figure 3 If the third location is deleted from the target map Figure 3 Dimension frame.

[0143] Exemplarily, the map management module 64 is based on the third map Figure 3 The fourth dimension frame and the fourth dimension frame Figure 3 Dimension frame determines the fifth place Figure 3 The third place is specifically used to: Figure 3 The third dimension frame is back-projected onto the image coordinate system to obtain Figure 2 dimension frame, the fourth Figure 3 The fourth dimension box is back-projected onto the image coordinate system to obtain Figure 2 dimension frame; if the third place Figure 2 The distance between the dimensional frame and the center of the image is less than the fourth Figure 2 The distance between the third dimension frame and the center of the image is Figure 3 The fifth dimension frame is determined as the fifth Figure 3 dimension frame; if the third place Figure 2 The distance between the dimensional frame and the center of the image is greater than the fourth Figure 2 The distance between the fourth dimension frame and the center of the image is Figure 3 The fifth dimension frame is determined as the fifth Figure 3 Dimension frame.

[0144] Based on the same application concept as the above method, an electronic device (such as a mobile robot) is proposed in the embodiment of the present application, see Figure 7 As shown, it includes: a processor 71 and a machine-readable storage medium 72, the machine-readable storage medium 72 stores machine-executable instructions that can be executed by the processor 71; the processor 71 is used to execute the machine-executable instructions to implement the map updating method disclosed in the above example of this application.

[0145] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the map updating method disclosed in the above example of the present application can be implemented.

[0146] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0147] Based on the same application concept as the above method, an embodiment of the present application also provides a computer program product, which may include a computer program. When the computer program is executed by a processor, it can implement the map updating method disclosed in the above example of the present application.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0149] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A map updating method, characterized in that: Applied to a mobile robot, the method comprises: Acquire an original image and a depth image corresponding to the original image; determine a current position and posture of the mobile robot at a current moment based on the original image and the depth image; Input the original image to the target detection model to obtain the initial position of the semantic target in the current posture; based on the conversion relationship between the current posture and the stored initial posture, convert the initial position into the target position of the semantic target in the initial posture; wherein the initial posture is the posture of the mobile robot at the initial moment, and the initial moment is the moment to start updating the target map; Determine a target three-dimensional frame of the semantic target based on the target position; The target map is updated based on the target three-dimensional box of the semantic target.

2. The method according to claim 1, characterized in that The updating of the target map based on the target three-dimensional frame of the semantic target includes: If the target map does not exist at the current moment, then the target map is established, and a map 3D frame is added to the target map based on the target 3D frame; or, if the target map already exists at the current moment, and the distance between the target 3D frame and each map 3D frame on the target map is not less than a first threshold, then a map 3D frame is added to the target map based on the target 3D frame; Adding the primary attributes and the secondary attributes of the semantic target to the three-dimensional map frame; Wherein, when the original image is input to the target detection model, the target detection model outputs the primary attribute of the semantic target, and the primary attribute represents the category of the semantic target; The original image is input into an image retrieval model to obtain secondary attributes of the semantic target, wherein the secondary attributes represent detailed information of the semantic target, and different semantic targets have different secondary attributes.

3. The method according to claim 1, characterized in that The updating of the target map based on the target three-dimensional frame of the semantic target includes: If a target map already exists at the current moment, and a distance between the target 3D frame and a first map 3D frame on the target map is less than a first threshold, a second map 3D frame is determined based on the target 3D frame and the first map 3D frame; wherein the second map 3D frame is the target 3D frame, or the first map 3D frame, or a fused 3D frame of the target 3D frame and the first map 3D frame; The first map 3D frame on the target map is replaced by the second map 3D frame.

4. The method according to claim 3, characterized in that The determining of the second map three-dimensional frame based on the target three-dimensional frame and the first map three-dimensional frame includes: Determining a size error based on a size of the target three-dimensional box and a size of the first map three-dimensional box; If the size error is not greater than a second threshold, the point cloud data corresponding to the target 3D frame and the point cloud data corresponding to the first map 3D frame are fused to obtain fused point cloud data; a fused 3D frame is generated based on the fused point cloud data; if the size of the fused 3D frame matches the size of the first map 3D frame, the fused 3D frame is determined as the second map 3D frame; If the size of the fused 3D frame does not match the size of the first map 3D frame, back-projecting the target 3D frame onto the image coordinate system to obtain a target 2D frame, and back-projecting the first map 3D frame onto the image coordinate system to obtain a first map 2D frame; If the distance between the target two-dimensional frame and the image center is smaller than the distance between the first map two-dimensional frame and the image center, determining the target three-dimensional frame as the second map three-dimensional frame; If the distance between the target two-dimensional frame and the image center is greater than the distance between the first map two-dimensional frame and the image center, the first map three-dimensional frame is determined as the second map three-dimensional frame.

5. The method according to claim 3, characterized in that: The determining of the second map three-dimensional frame based on the target three-dimensional frame and the first map three-dimensional frame includes: Determining a size error based on a size of the target three-dimensional box and a size of the first map three-dimensional box; If the size error is greater than a second threshold, obtaining overlapping area point cloud data corresponding to the overlapping area between the target three-dimensional frame and the first map three-dimensional frame; Back-projecting the target three-dimensional frame onto the image coordinate system to obtain a target two-dimensional frame, and back-projecting the first map three-dimensional frame onto the image coordinate system to obtain a first map two-dimensional frame; If the distance between the target two-dimensional frame and the image center is less than the distance between the first map two-dimensional frame and the image center, the point cloud data corresponding to the target three-dimensional frame and the point cloud data of the overlapping area are fused to obtain fused point cloud data; if the distance between the target two-dimensional frame and the image center is greater than the distance between the first map two-dimensional frame and the image center, the point cloud data corresponding to the first map three-dimensional frame and the point cloud data of the overlapping area are fused to obtain fused point cloud data; The fused three-dimensional frame corresponding to the fused point cloud data is determined as the second map three-dimensional frame.

6. The method according to claim 1, characterized in that After the target three-dimensional frame based on the semantic target is updated to the target map, the method further includes: If a third map 3D frame and a fourth map 3D frame are traversed from the target map, a fifth map 3D frame is determined based on the third map 3D frame and the fourth map 3D frame; wherein the primary attribute of the third map 3D frame is the same as the primary attribute of the fourth map 3D frame, the secondary attribute of the third map 3D frame is the same as the secondary attribute of the fourth map 3D frame, and the distance between the third map 3D frame and the fourth map 3D frame is not less than the first threshold and less than the third threshold; If the fifth map 3D frame is the third map 3D frame, the fourth map 3D frame is deleted from the target map; or, if the fifth map 3D frame is the fourth map 3D frame, the third map 3D frame is deleted from the target map.

7. The method according to claim 6, characterized in that The determining of a fifth map 3D frame based on the third map 3D frame and the fourth map 3D frame comprises: Back-projecting the third map three-dimensional frame onto the image coordinate system to obtain a third map two-dimensional frame, and back-projecting the fourth map three-dimensional frame onto the image coordinate system to obtain a fourth map two-dimensional frame; If the distance between the third map two-dimensional frame and the image center is smaller than the distance between the fourth map two-dimensional frame and the image center, determining the third map three-dimensional frame as the fifth map three-dimensional frame; If the distance between the third map two-dimensional frame and the image center is greater than the distance between the fourth map two-dimensional frame and the image center, the fourth map three-dimensional frame is determined as the fifth map three-dimensional frame.

8. A map updating device, characterized in that: Applied to a mobile robot, the device comprises: An acquisition module, used to acquire an original image and a depth image corresponding to the original image; and determine a current posture of the mobile robot corresponding to a current moment based on the original image and the depth image; A processing module, used for inputting the original image into a target detection model to obtain an initial position of the semantic target in a current posture; based on a conversion relationship between the current posture and the stored initial posture, converting the initial position into a target position of the semantic target in the initial posture; wherein the initial posture is the posture of the mobile robot at an initial moment, and the initial moment is the moment when the target map is started to be updated; A determination module, configured to determine a target three-dimensional frame of the semantic target based on the target position; A map management module is used to update the target map based on the target three-dimensional box of the semantic target.

9. The device according to claim 8, characterized in that in, When the map management module updates the target map based on the target three-dimensional frame of the semantic target, it is specifically used to: if the target map does not exist at the current moment, then establish the target map, and add a map three-dimensional frame on the target map based on the target three-dimensional frame; or, if the target map already exists at the current moment, and the distance between the target three-dimensional frame and each map three-dimensional frame on the target map is not less than a first threshold, then add a map three-dimensional frame on the target map based on the target three-dimensional frame; Adding the primary attributes and secondary attributes of the semantic target to the three-dimensional map frame; wherein, when the original image is input to the target detection model, the target detection model outputs the primary attributes of the semantic target, and the primary attributes represent the category of the semantic target; the original image is input to the image retrieval model to obtain the secondary attributes of the semantic target, and the secondary attributes represent the detailed information of the semantic target, and different semantic targets have different secondary attributes; Wherein, when the map management module updates the target map based on the target 3D frame of the semantic target, it is specifically used to: if a target map already exists at the current moment, and the distance between the target 3D frame and the first map 3D frame on the target map is less than a first threshold, then determine a second map 3D frame based on the target 3D frame and the first map 3D frame; the second map 3D frame is the target 3D frame, or the first map 3D frame, or a fused 3D frame of the target 3D frame and the first map 3D frame; replace the first map 3D frame on the target map with the second map 3D frame; Wherein, when the map management module determines the second map 3D frame based on the target 3D frame and the first map 3D frame, it is specifically used to: determine the size error based on the size of the target 3D frame and the size of the first map 3D frame; if the size error is not greater than the second threshold, fuse the point cloud data corresponding to the target 3D frame and the point cloud data corresponding to the first map 3D frame to obtain fused point cloud data; generate a fused 3D frame based on the fused point cloud data; if the size of the fused 3D frame matches the size of the first map 3D frame, determine the fused 3D frame as the second map 3D frame; if the size of the fused 3D frame does not match the size of the first map 3D frame, back-project the target 3D frame onto the image coordinate system to obtain a target 2D frame, and back-project the first map 3D frame onto the image coordinate system to obtain a first map 2D frame; if the distance between the target 2D frame and the image center is less than the distance between the first map 2D frame and the image center, determine the target 3D frame as the second map 3D frame; if the distance between the target 2D frame and the image center is greater than the distance between the first map 2D frame and the image center, determine the first map 3D frame as the second map 3D frame; Wherein, when the map management module determines the second map three-dimensional frame based on the target three-dimensional frame and the first map three-dimensional frame, it is specifically used to: determine the size error based on the size of the target three-dimensional frame and the size of the first map three-dimensional frame; if the size error is greater than a second threshold, obtain the overlapping area point cloud data corresponding to the overlapping area between the target three-dimensional frame and the first map three-dimensional frame; back-project the target three-dimensional frame onto the image coordinate system to obtain the target two-dimensional frame, and back-project the first map three-dimensional frame onto the image coordinate system to obtain the first map two-dimensional frame; if the distance between the target two-dimensional frame and the image center is less than the distance between the first map two-dimensional frame and the image center, fuse the point cloud data corresponding to the target three-dimensional frame and the point cloud data of the overlapping area to obtain fused point cloud data; if the distance between the target two-dimensional frame and the image center is greater than the distance between the first map two-dimensional frame and the image center, fuse the point cloud data corresponding to the first map three-dimensional frame and the point cloud data of the overlapping area to obtain fused point cloud data; determine the fused three-dimensional frame corresponding to the fused point cloud data as the second map three-dimensional frame; The map management module is further used for determining a fifth map 3D frame based on the third map 3D frame and the fourth map 3D frame if the third map 3D frame and the fourth map 3D frame are traversed from the target map; wherein the primary attribute of the third map 3D frame is the same as the primary attribute of the fourth map 3D frame, the secondary attribute of the third map 3D frame is the same as the secondary attribute of the fourth map 3D frame, and the distance between the third map 3D frame and the fourth map 3D frame is not less than the first threshold and less than the third threshold; if the fifth map 3D frame is the third map 3D frame, deleting the fourth map 3D frame from the target map; or, if the fifth map 3D frame is the fourth map 3D frame, deleting the third map 3D frame from the target map; Among them, when the map management module determines the fifth map three-dimensional frame based on the third map three-dimensional frame and the fourth map three-dimensional frame, it is specifically used to: back-project the third map three-dimensional frame onto the image coordinate system to obtain a third map two-dimensional frame, and back-project the fourth map three-dimensional frame onto the image coordinate system to obtain a fourth map two-dimensional frame; if the distance between the third map two-dimensional frame and the center of the image is less than the distance between the fourth map two-dimensional frame and the center of the image, then determine the third map three-dimensional frame as the fifth map three-dimensional frame; if the distance between the third map two-dimensional frame and the center of the image is greater than the distance between the fourth map two-dimensional frame and the center of the image, then determine the fourth map three-dimensional frame as the fifth map three-dimensional frame.

10. A mobile robot, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method according to any one of claims 1 to 7.

Images