Underwater cleaning robot mapping method and device and underwater cleaning robot

Abstract

The invention discloses an underwater cleaning robot mapping method and device and an underwater cleaning robot. The method comprises the following steps: performing feature extraction on an environment image acquired by the underwater cleaning robot to obtain target structure information of the underwater cleaning robot in a target space; performing feature matching based on the target structure information of the two adjacent frames of environment images to obtain a target feature matching result; determining target depth information of matched target structure boundary points in the target structure information of the two adjacent frames of environment images in the target space based on a target feature matching result and the target acquisition position information of the underwater cleaning robot; the target acquisition position information is comprehensively determined based on positioning information acquired by various sensors of the underwater cleaning robot; based on the target structure information and the target depth information corresponding to the target structure boundary points, the target environment map of the target space is constructed, and the positioning precision and the mapping capability of the underwater cleaning robot in the underwater environment with a large number of weak textures can be ensured.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to an underwater cleaning robot mapping method, device, and underwater cleaning robot. Background Art

[0002] With the development of computer technology, robot technology has also developed rapidly. For example, users use sweeping robots to clean the floors of houses, use window cleaning robots to clean the windows of houses, and use underwater cleaning robots to clean pools.

[0003] In the related art, the positioning and mapping capabilities of underwater cleaning robots are easily invalidated in the case of weak textures, that is, they cannot meet the cleaning needs in weak texture environments and it is difficult to meet the needs of various user scenarios. Summary of the invention

[0004] The embodiments of the present application provide an underwater cleaning robot mapping method, device and underwater cleaning robot, which can ensure the positioning accuracy and mapping capability of the underwater cleaning robot in a large number of weak texture environments underwater. The above technical solutions are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for mapping an underwater cleaning robot, the method comprising:

[0006] Extracting features from the environment image collected by the underwater cleaning robot to obtain target structure information in the target space where the underwater cleaning robot is located;

[0007] Based on two adjacent frames of the environment image, feature matching is performed on the target structure information in the target space corresponding to the target, and a corresponding target feature matching result is obtained;

[0008] The target depth information of the target structure boundary point in the target space is determined based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot; the target acquisition position information is comprehensively determined based on the positioning information obtained by the various sensors of the underwater cleaning robot; the target structure boundary point is the matching structure boundary point in the target structure information corresponding to each of the two adjacent frames of the environment image;

[0009] Based on the target structure information and the target depth information corresponding to the boundary points of the target structure, a target environment map of the target space is constructed.

[0010] In a second aspect, an embodiment of the present application provides a mapping device for an underwater cleaning robot, the mapping device for an underwater cleaning robot comprising:

[0011] A feature extraction module is used to extract features from the environment image collected by the underwater cleaning robot to obtain target structure information in the target space where the underwater cleaning robot is located;

[0012] A feature matching module is used to perform feature matching based on the target structure information in the target space corresponding to two adjacent frames of the environment image to obtain a corresponding target feature matching result;

[0013] The first determination module is used to determine the target depth information of the target structure boundary point in the target structure information in the target space based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot; the target acquisition position information is comprehensively determined based on the positioning information obtained by the multiple sensors of the underwater cleaning robot; the target structure boundary point is the matching structure boundary point in the target structure information corresponding to each of the two adjacent frames of the environment image;

[0014] A mapping module is used to construct a target environment map of the target space based on the target structure information and the target depth information corresponding to the boundary points of the target structure.

[0015] In a possible implementation, the feature extraction module includes:

[0016] An extraction unit is used to extract structural boundary information from the environment image collected by the underwater cleaning robot; the structural boundary information includes first position information corresponding to the structural boundary existing in the environment image and second position information of the area included in the contour corresponding to the structural boundary;

[0017] The first determining unit is used to determine the target structure information in the target space based on the structure boundary information.

[0018] In a possible implementation, the feature extraction module is specifically used to:

[0019] By using the multi-task model, the target space component element information and the target space auxiliary element information in the environment image collected by the underwater cleaning robot are segmented and identified respectively, so as to obtain the target structure information in the target space where the underwater cleaning robot is located.

[0020] In a possible implementation, the feature matching module includes:

[0021] A collection position prediction unit, used to predict the predicted collection position information of the next frame of the environmental image between two adjacent frames of the environmental image corresponding to the underwater cleaning robot based on the previous frame of the environmental image between two adjacent frames of the environmental image corresponding to the previous frame of the underwater cleaning robot;

[0022] A structure information prediction unit, used to predict the predicted structure information in the next frame of the environment image based on the predicted acquisition position information and the target structure information in the previous frame of the environment image;

[0023] A structure information matching unit, used to match the target structure information in the next frame of the environment image with the predicted structure information to obtain a predicted structure matching range corresponding to the target structure information in the next frame of the environment image;

[0024] The second determination unit is used to determine the matching target structure boundary points in the target structure information corresponding to each of the two adjacent frames of the environmental image based on the geometric features of each structure boundary point in the previous frame of the environmental image and the geometric features of each structure boundary point within the predicted structure matching range.

[0025] In a possible implementation, the multiple sensors include a first sensor, a second sensor and an ultrasonic sensor, the first sensor is used to collect the environmental image of the underwater cleaning robot, the second sensor is used to collect the motion data of the underwater cleaning robot; the second sensor includes at least one of the following: an inertial measurement unit, an encoder, a tachometer; the ultrasonic sensor is used to collect the ultrasonic positioning data of the underwater cleaning robot;

[0026] The above-mentioned underwater cleaning robot mapping device also includes:

[0027] An acquisition module, used to acquire two adjacent frames of environmental images based on the first sensor, acquire actual motion data of the underwater cleaning robot based on one or more of the second sensors, and acquire ultrasonic positioning data of the underwater cleaning robot based on the ultrasonic sensor;

[0028] A second determination module is used to determine the first acquisition position information corresponding to the latter frame of the above two adjacent frames of environmental images based on the above two adjacent frames of environmental images and the previous frame acquisition position information corresponding to the previous frame of the above two adjacent frames of environmental images, determine the second acquisition position information corresponding to the above latter frame of environmental images based on the above actual motion data, and determine the third acquisition position information corresponding to the above latter frame of environmental images based on the above ultrasonic positioning data;

[0029] The third determining module is used to determine the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information, the second acquisition position information and the third acquisition position information.

[0030] In a possible implementation, the underwater cleaning robot mapping device further includes:

[0031] A fourth determination module is used to determine the position loss corresponding to the above-mentioned next frame of environmental image based on the predicted acquisition position information corresponding to the above-mentioned next frame of environmental image, the above-mentioned first acquisition position information, the above-mentioned second acquisition position information and the above-mentioned third acquisition position information;

[0032] A fifth determination module, configured to determine the image loss corresponding to the next frame of the environment image based on the predicted structure information and the target structure information corresponding to the next frame of the environment image;

[0033] The correction module is used to correct the position drift of the above-mentioned multiple sensors based on the position loss and image loss corresponding to the above-mentioned next frame of environmental image.

[0034] In a possible implementation, the underwater cleaning robot mapping device further includes:

[0035] A judgment module, used to judge whether the ultrasonic positioning data meets the non-line-of-sight condition based on the next frame of the environmental image; the non-line-of-sight condition is used to indicate that there is occlusion in the next frame of the environmental image and the ultrasonic positioning data is abnormally jittery;

[0036] a sixth determination module, configured to determine the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information and the second acquisition position information if the ultrasonic positioning data satisfies the non-line-of-sight condition;

[0037] The third determination module is specifically used to execute the step of determining the target acquisition position information corresponding to the next frame of the environmental image based on the first acquisition position information, the second acquisition position information and the third acquisition position information if the ultrasonic positioning data does not meet the non-line-of-sight condition.

[0038] In a possible implementation, the underwater cleaning robot mapping device further includes:

[0039] A statistical module, used to count the number of target structure boundary points that meet the preset constraint conditions in the above-mentioned next frame of environmental image, and the total number of structure boundary points in the target structure information corresponding to the above-mentioned next frame of environmental image; the above-mentioned preset constraint condition is that there are matching target structure boundary points in at least two consecutive frames of environmental images;

[0040] A seventh determination module, configured to determine a visual constraint weight corresponding to the first sensor based on a ratio between the number of target structure boundary points satisfying the preset constraint condition and the total number of structure boundary points;

[0041] An eighth determination module, configured to determine the weights corresponding to the second sensor and the ultrasonic sensor, respectively, based on the covariance matrices corresponding to the second sensor and the ultrasonic sensor, respectively;

[0042] a ninth determination module, configured to determine a target weight ratio corresponding to the first sensor, the second sensor, and the ultrasonic sensor based on the visual constraint weight corresponding to the first sensor, and the weights corresponding to the second sensor and the ultrasonic sensor;

[0043] The third determination module is specifically used for:

[0044] The first acquisition position information, the second acquisition position information and the third acquisition position information are fused according to the target weight ratio to obtain the target acquisition position information corresponding to the next frame of the environmental image.

[0045] In a possible implementation, the target feature matching result includes target structure boundary points that match in two adjacent frames of the environment image;

[0046] The first determination module is specifically used for:

[0047] The target depth information of the target structure boundary point in the target space is estimated based on the position of the target structure boundary point in two adjacent frames of the environment image and the target acquisition position information corresponding to the underwater cleaning robot.

[0048] In a possible implementation, the underwater cleaning robot mapping device further includes:

[0049] The mobile module is used to move according to the target mapping path;

[0050] An image acquisition module is used to acquire environmental images within a preset visual range at a preset frequency during movement;

[0051] The above-mentioned mapping module is specifically used for:

[0052] After the target mapping path is completed, a target environment map of the target space is constructed based on target structure information corresponding to each of the multiple environment images on the target mapping path and target depth information corresponding to the target structure boundary points in the multiple environment images.

[0053] In a possible implementation, the target structure information includes target space component element information and target space auxiliary element information in the target space; the target space component element information includes structural boundary information of the target space corresponding component elements, and the component elements include at least one of the following: bottom surface, wall surface, step plane, step facade; the target space auxiliary element information includes structural boundary information of the auxiliary elements corresponding to the target space, and the auxiliary elements include at least one of the following auxiliary devices: wall lamp, floor lamp, water inlet, and water outlet.

[0054] In a third aspect, an embodiment of the present application provides an underwater cleaning robot, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps provided by the first aspect of the embodiment of the present application or any possible implementation method of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the method steps provided by the first aspect of the embodiment of the present application or any possible implementation of the first aspect.

[0056] The technical solution provided in the embodiment of the present application, on the one hand, obtains the target acquisition position information corresponding to the underwater cleaning robot by comprehensively utilizing multiple sensors, thereby ensuring the positioning accuracy of the underwater cleaning robot in an environment with a large number of weak textures underwater; on the other hand, performs feature matching based on the target structure information in the target space corresponding to two adjacent frames of environmental images, and estimates the target depth information of the matching structure boundary points in the target space in two adjacent frames of environmental images based on the target acquisition position information obtained by multiple sensors, so as to construct a target environment map of the target space based on the extracted target structure information and the corresponding target depth information, thereby comprehensively utilizing multiple sensors, feature extraction and feature matching, and depth estimation to realize the perception and analysis of the internal structure of the swimming pool (target space), thereby improving the mapping capability and stability of the underwater cleaning robot.

[0057] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A schematic diagram of an application scenario of a control method for an underwater cleaning robot provided by an exemplary embodiment of the present application;

[0060] Figure 2 A schematic diagram of a flow chart of a method for mapping an underwater cleaning robot provided as an exemplary embodiment of the present application;

[0061] Figure 3 A schematic diagram of a process for implementing feature matching in a mapping process of an underwater cleaning robot provided by an exemplary embodiment of the present application;

[0062] Figure 4 A schematic diagram of a process for determining target acquisition location information provided by an exemplary embodiment of the present application;

[0063] Figure 5 A schematic diagram of an implementation process of position drift correction for an underwater cleaning robot provided by an exemplary embodiment of the present application;

[0064] Figure 6 A schematic diagram of a process for determining a target weight ratio provided for an exemplary embodiment of the present application;

[0065] Figure 7 A schematic structural diagram of a mapping device for an underwater cleaning robot provided by an exemplary embodiment of the present application;

[0066] Figure 8 A schematic structural diagram of another underwater cleaning robot provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0068] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0069] First, the terms involved in the embodiments of the present application are introduced.

[0070] Underwater cleaning robot: A robot that can dive into the water and automatically clean underwater areas. For example, the underwater cleaning robot is placed in a pool and can clean the bottom and / or walls of the pool or the water surface and / or water.

[0071] Visual inspection: Through the machine vision system, the machine is used to replace the human eye to make measurements and judgments. It uses image processing technology and computer vision technology to perform various processing on the images taken by the robot to achieve the function of automatically detecting and identifying the target object.

[0072] Target environment map: It is a visual data structure that can be used to describe and represent various features and information in the target space environment. These features may include terrain, landforms, buildings, etc., while information may include location, shape, size, color, etc.

[0073] Inertial measurement unit: a device that measures the motion state of an object through the principle of inertia, usually composed of a gyroscope and an accelerometer. It can sense the acceleration and angular velocity of the robot in real time, thereby calculating the robot's posture and position changes. The inertial measurement unit has a high update rate and dynamic performance, and is suitable for measurement of high-speed motion and complex environments.

[0074] Encoder: A device that calculates the displacement of an object by measuring the number of pulse signals, usually used with rotating parts such as motors. It can provide accurate rotation angle and speed information for measuring mechanical movement.

[0075] Flow meter: a device used to measure water flow velocity, commonly used in underwater robots, waterway monitoring and other fields. It can calculate the flow velocity by measuring the time or speed of water flow passing through the sensor, with high accuracy and real-time performance.

[0076] Monocular visual odometry: Based on monocular vision technology, it estimates the movement of an object by continuously acquiring images and analyzing the changes between images. During the robot's movement, the monocular camera will continuously acquire the current image at a very high frequency. What the visual odometry needs to do is to estimate the robot's movement (movement and rotation) between two adjacent frames of images.

[0077] After introducing the nouns involved in the embodiments of the present application, the application scenarios of the embodiments of the present application are described below.

[0078] The technical solution provided in the embodiment of the present application can be applied to the scenario of controlling an underwater cleaning robot to map a pool, see Figure 1, the underwater cleaning robot 110 can move along the target mapping path on the bottom of the pool 120 (for example, but not limited to first moving one circle clockwise along the edge of the pool 120, then moving one circle counterclockwise along the edge of the pool 120, and finally moving one circle across the middle of the pool 120, etc.), and extract features from the environmental images collected during the movement of the underwater cleaning robot to obtain target structure information in the target space where the underwater cleaning robot is located, and perform feature matching based on the target structure information in the target space corresponding to the above two adjacent frames of the environmental images to obtain a target feature matching result; then, based on the above target feature matching results and the underwater The target acquisition position information corresponding to the cleaning robot determines the target depth information of the target structure boundary points in the target structure information in the target space. The target acquisition position information is comprehensively determined based on the positioning information obtained by various sensors of the underwater cleaning robot. The target structure boundary points are the matching structure boundary points in the target structure information corresponding to two adjacent frames of environmental images. Finally, based on the above target structure information and the target depth information corresponding to the above target structure boundary points, a target environment map of the above target space is constructed, thereby ensuring the positioning accuracy, mapping capability and stability of the underwater cleaning robot in an environment with a large number of weak textures underwater.

[0079] Optionally, the underwater cleaning robot 110 can also, but is not limited to, move in any area to be cleaned, such as the bottom, wall, water surface, or water of the pool 120, so as to clean the area to be cleaned. When the underwater cleaning robot 110 inspects the area to be cleaned in the pool 120, the water pump of the underwater cleaning robot 110 is started, and the liquid in the pool 120 is sucked into the filter box of the underwater cleaning robot 110 through the water inlet at the bottom of the underwater cleaning robot 110, and the liquid is filtered through the filter box, and the dirt in the liquid is left in the filter box. The filtered liquid is discharged through the drain of the underwater cleaning robot 110, so as to clean the area to be cleaned in the pool 120. Of course, in addition to cleaning the area to be cleaned in the pool 120 by the cooperation of the water pump and the filter box, the underwater cleaning robot 110 can also clean the area to be cleaned in the pool 120 by other types of cleaning units, for example, by using a roller brush to clean the area to be cleaned in the pool 120.

[0080] It is understandable that the underwater cleaning robot mapping method provided in the embodiment of the present application can be executed by the underwater cleaning robot alone, or by a terminal or server connected to the underwater cleaning robot through a network, and the embodiment of the present application is not limited to this. The network can be a medium that provides a communication link between the underwater cleaning robot and the terminal or server, or it can be the Internet including network equipment and transmission media, but is not limited to this. The transmission medium can be a wired link, such as but not limited to coaxial cable, optical fiber and digital subscriber line (DSL), etc., or a wireless link, such as but not limited to wireless fidelity (WIFI), Hypertext Transfer Protocol (HTTP), Bluetooth and mobile device network, etc.

[0081] The above-mentioned terminal can interact with the underwater cleaning robot through the network to receive messages from the underwater cleaning robot or send messages to the underwater cleaning robot. The terminal can be hardware or software. When the terminal is hardware, it can be various electronic devices, including but not limited to smart watches, smart phones, tablet computers, laptop portable computers and desktop computers. When the terminal is software, it can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules (for example: used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0082] The above-mentioned server can be a server that provides control services for various underwater cleaning robots. It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here. The server can be, but is not limited to, a hardware server, a virtual server, a cloud server, etc.

[0083] Understandably, Figure 1 The number of underwater cleaning robots in the application scenario shown is for example only. In a specific implementation, any number of underwater cleaning robots can be used to map the pool 120 for full coverage. For example, the bottom of the pool 120 can be divided into multiple areas and mapped using underwater cleaning robots based on the technical solution provided in the embodiments of the present application.

[0084] The following takes the underwater cleaning robot executing underwater cleaning robot mapping as an example to introduce the underwater cleaning robot mapping method provided by an exemplary embodiment of the present application. Figure 2, which is a flow chart of a method for mapping an underwater cleaning robot provided by an exemplary embodiment of the present application. Figure 2 As shown, the underwater cleaning robot mapping method includes the following steps:

[0085] S201, extracting features from an environment image collected by the underwater cleaning robot to obtain target structure information in a target space where the underwater cleaning robot is located.

[0086] Specifically, when the underwater cleaning robot moves in the target space, it can collect the environment image within its visual range through the first sensor, and extract the features of the collected environment image to obtain the target structure information in the environment image. The above-mentioned first sensor may include, but is not limited to, image acquisition devices such as a monocular camera and a binocular camera. The above-mentioned target structure information may include, but is not limited to, at least one of the following: target space component element information and target space auxiliary element information in the target space; the above-mentioned target space component element information may include, but is not limited to, structural boundary information of the corresponding component element of the target space, and the above-mentioned component element may include, but is not limited to, at least one of the following: bottom surface, wall surface, step plane, step facade; the above-mentioned target space auxiliary element information may include, but is not limited to, structural boundary information of the corresponding auxiliary element of the target space, and the above-mentioned auxiliary element may include, but is not limited to, at least one of the following auxiliary devices: wall lamp, floor lamp, water inlet, and water outlet. The above-mentioned structural boundary information may include, but is not limited to, the first position information corresponding to the structural boundary of the component element or the auxiliary element in the environment image and the second position information of the contour containing area corresponding to the above-mentioned structural boundary in the environment image.

[0087] Optionally, the above-mentioned feature extraction of the environmental image collected by the underwater cleaning robot to obtain the target structure information in the target space where the underwater cleaning robot is located may include, but is not limited to: first extracting the structural boundary information in the environmental image collected by the underwater cleaning robot, the above-mentioned structural boundary information may include, but is not limited to, the first position information corresponding to the structural boundary existing in the environmental image and the second position information of the area containing the contour corresponding to the structural boundary, for example, but not limited to, using a segmentation model or a boundary extraction model or an edge recognition model to segment and extract the constituent elements of the target space area and the common devices (auxiliary elements) in the target space, and identify the pixel-level structural boundary coordinates (first position information) and the specific position information of the containing area (second position information) in the image coordinate system corresponding to the environmental image. Then, based on the above-mentioned structural boundary information, the target structure information in the target space is determined, for example, but not limited to, the structural boundary information corresponding to each of the multiple environmental images collected during the movement of the underwater cleaning robot can be fused to obtain the target structure information in the target space.

[0088] Optionally, the above-mentioned feature extraction of the environmental image collected by the underwater cleaning robot to obtain the target structure information in the target space where the underwater cleaning robot is located may also include, but is not limited to: directly using the multi-task model to segment and identify the target space component element information and the target space auxiliary element information in the environmental image collected by the underwater cleaning robot, respectively, to obtain the target structure information in the target space where the underwater cleaning robot is located. The above-mentioned target space component element information may include, but is not limited to, the structural boundary information of the component elements such as the pool bottom, pool wall, step plane, step facade, etc. that constitute the target space, and the above-mentioned target space auxiliary element information may include, but is not limited to, the structural boundary information of auxiliary devices (auxiliary elements) such as wall lamps, floor lamps, water inlets, and water outlets installed in the target space. The above-mentioned multi-task model may be, but is not limited to, trained based on multiple sample environmental images with known spatial component element information and spatial auxiliary element information.

[0089] S202, performing feature matching based on target structure information in the corresponding target space of two adjacent frames of environmental images to obtain corresponding target feature matching results.

[0090] Optionally, after obtaining the target structure information corresponding to the environment image, the target structure information in the target space corresponding to the two adjacent frames of the environment image can be feature matched, for example, but not limited to similarity calculation, to obtain target structure boundary points that match the two adjacent frames of the environment image, that is, structure boundary points at the same position on the same structure (component element or auxiliary element) in the two adjacent frames of the environment image or structure boundary points in the two adjacent frames of the environment image whose corresponding similarity is greater than a threshold. The above similarity calculation can be based on, but not limited to, features such as pixels, colors, positions, and slopes or curvatures of the corresponding positions of the boundary lines of the structure boundary points in the environment image.

[0091] Alternatively, if Figure 3 As shown, the implementation process of performing feature matching based on the target structure information in the target space corresponding to two adjacent frames of environmental images in S202 to obtain the corresponding target feature matching result may include but is not limited to the following steps:

[0092] S301, predicting predicted acquisition position information of an underwater cleaning robot corresponding to a subsequent frame of an environmental image between two adjacent frames of environmental images based on previous frame acquisition position information of an underwater cleaning robot corresponding to a previous frame of an environmental image between two adjacent frames of environmental images.

[0093] Specifically, the underwater cleaning robot can, but is not limited to, utilize its built-in kinematic model to estimate its moving direction and distance during the interval between two adjacent frames of environmental images based on the acquisition time interval corresponding to the two adjacent frames of environmental images, as well as the moving speed, steering angle, etc., and then combine the estimated moving direction and distance with the previous frame acquisition position information of the underwater cleaning robot corresponding to the previous frame of environmental image to determine the predicted acquisition position information of the underwater cleaning robot corresponding to the next frame of environmental image.

[0094] S302: predicting predicted structure information in a next frame of environment image based on the predicted acquisition position information and the target structure information in a previous frame of environment image.

[0095] Specifically, after obtaining the predicted acquisition position information of the underwater cleaning robot corresponding to the next frame of the environmental image, the target structure information in the previous frame of the environmental image can also be translated and / or rotated based on but not limited to the difference between the previous frame of the environmental image corresponding to the previous frame of the underwater cleaning robot and the predicted acquisition position information of the next frame of the environmental image, so as to obtain the corresponding predicted structure information in the next frame of the environmental image, that is, the structural boundary information of the structural boundaries of the constituent elements and / or auxiliary elements in the previous frame of the environmental image in the next frame of the environmental image.

[0096] S303, matching the target structure information in the next frame of the environment image with the predicted structure information to obtain a predicted structure matching range corresponding to the target structure information in the next frame of the environment image.

[0097] Specifically, after obtaining the target structure information actually extracted from the next frame of the environmental image and the predicted structure information predicted, the target structure information corresponding to the next frame of the environmental image can be directly matched with the predicted structure information, for example, but not limited to matching the target structure information with the corresponding structure boundary points in the predicted structure information (for example, but not limited to similarity calculation, etc.), and screening out a set of structure boundary points that match the target structure information and the predicted structure information (for example, but not limited to a similarity greater than a preset value), thereby obtaining the predicted structure matching range corresponding to the target structure information in the next frame of the environmental image.

[0098] S304, based on the geometric features of each structure boundary point in the previous frame of the environment image and the geometric features of each structure boundary point in the predicted structure matching range, determine the target structure boundary points that match in the target structure information corresponding to each of the two adjacent frames of the environment image.

[0099] Specifically, the above-mentioned geometric features may include, but are not limited to, slopes and curvatures corresponding to the structural boundary points. After obtaining the predicted structural matching range corresponding to the next frame of the environmental image, in order to ensure the accuracy of feature matching, it is also possible but not limited to directly calculating the difference or similarity between the geometric features of each structural boundary point in the previous frame of the environmental image and the geometric features of each structural boundary point in the predicted structural matching range corresponding to the next frame of the environmental image, and then determining the matching target structural boundary points in the target structural information corresponding to each of the two adjacent frames of the environmental image based on the above-mentioned difference or similarity. The above-mentioned matching target structural boundary points may be, but are not limited to, structural boundary points whose difference is less than the target value or whose similarity is greater than the threshold.

[0100] In an embodiment of the present application, by performing feature matching based on the target structure information in the corresponding target space of two adjacent frames of environmental images in the above manner, the matching target structure boundary points in the two adjacent frames of environmental images can be accurately found, thereby ensuring the continuity and accuracy of subsequent mapping.

[0101] Please continue to refer to Figure 2 ,like Figure 2 As shown, in the above S202, after performing feature matching based on target structure information in the target space corresponding to two adjacent frames of environmental images and obtaining the corresponding target feature matching result, the underwater cleaning robot mapping method further includes:

[0102] S203, based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot, the target depth information of the target structure boundary point in the target space is determined, the target acquisition position information is comprehensively determined based on the positioning information obtained by various sensors of the underwater cleaning robot, and the target structure boundary point is the matching structure boundary point in the target structure information corresponding to each of two adjacent frames of environmental images.

[0103] Specifically, the above-mentioned multiple sensors may include, but are not limited to, a first sensor, a second sensor, and an ultrasonic sensor. The above-mentioned first sensor is used to collect environmental images of the underwater cleaning robot, and the second sensor is used to collect motion data of the underwater cleaning robot. The above-mentioned second sensor may include, but is not limited to, at least one of the following: an inertial measurement unit, an encoder, and a tachometer; the above-mentioned ultrasonic sensor is used to collect ultrasonic positioning data of the underwater cleaning robot. The above-mentioned target acquisition position information is the actual acquisition position information of the underwater cleaning robot corresponding to the latter frame of the environmental image in the two adjacent frames of the target feature matching result. The above-mentioned target depth information may include, but is not limited to, the position coordinates of the boundary points of the target structure in the three-dimensional target space.

[0104] Further, the target feature matching result includes target structure boundary points that match in two adjacent frames of environmental images. The above S203, the process of implementing the target depth information of the target structure boundary points in the target structure information in the target space based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot may include, but is not limited to: estimating the target depth information of the target structure boundary points in the target space based on the positions of the target structure boundary points in two adjacent frames of environmental images and the target acquisition position information corresponding to the underwater cleaning robot, for example, but not limited to first determining the relative position information of the target structure boundary points and the underwater cleaning robot based on the difference between the positions of the target structure boundary points in two adjacent frames of environmental images, and then calculating the target depth information of the target structure boundary points in the target space based on the relative position information and the target acquisition position information corresponding to the underwater cleaning robot, i.e., the position coordinates of the underwater cleaning robot in the target space when the next frame of environmental image is acquired.

[0105] S204: construct a target environment map of the target space based on the target structure information and the target depth information corresponding to the target structure boundary points.

[0106] Specifically, the target environment map corresponding to the target space can be constructed directly using, but not limited to, target structure information corresponding to the collected environment image and target depth information corresponding to the target structure boundary points during the mapping process of the underwater cleaning robot.

[0107] Optionally, when it is necessary to use an underwater cleaning robot to map the target space, after receiving the corresponding mapping instructions, the underwater cleaning robot can move according to the target mapping path corresponding to the target space, but is not limited to, and collect environmental images within a preset visual range at a preset frequency during the movement. After the target mapping path is completed, the underwater cleaning robot can construct a target environment map of the target space based on the target structure information corresponding to each of the multiple environmental images on the above-mentioned target mapping path and the target depth information corresponding to the target structure boundary points that match two adjacent frames of environmental images in the multiple environmental images.

[0108] It can be understood that after traversing the target space according to the target mapping path, the underwater cleaning robot can construct a target environment map of the target space according to all target mapping paths obtained during the traversal process and the target structure information and the target depth information corresponding to the target structure boundary points. It can also construct the target environment map of the target space in real time according to the real-time changing target mapping path and the target structure information and the target depth information corresponding to the target structure boundary points during the traversal of the target space, that is, construct the target environment map for the corresponding traversed positions in the target space while traversing the target space, etc. The embodiments of the present application are not limited to this.

[0109] Optionally, after constructing the target environment map of the target space, the underwater cleaning robot can also use various optimization algorithms to optimize the target environment map, such as removing mismatched features, adjusting the scale of the map, etc., to improve the accuracy and quality of the target environment map.

[0110] In the embodiments of the present application, on the one hand, the target acquisition position information corresponding to the underwater cleaning robot is obtained by comprehensively utilizing multiple sensors to ensure the positioning accuracy of the underwater cleaning robot in an environment with a large number of weak textures underwater; on the other hand, feature matching is performed based on the target structure information in the target space corresponding to two adjacent frames of environmental images, and the target depth information of the matching structural boundary points in the target space in two adjacent frames of environmental images is estimated based on the target acquisition position information obtained by multiple sensors, so as to construct a target environment map of the target space based on the extracted target structure information and the corresponding target depth information, thereby comprehensively utilizing multiple sensors, feature extraction and feature matching, and depth estimation to realize the perception and analysis of the internal structure of the swimming pool (target space), thereby improving the mapping capability and stability of the underwater cleaning robot.

[0111] In some possible embodiments, Figure 4As shown, in the above S203, before determining the target depth information of the target structure boundary point in the target structure information in the target space based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot, the underwater cleaning robot mapping method may also include but is not limited to: first acquiring two adjacent frames of environmental images based on the first sensor, acquiring actual motion data of the underwater cleaning robot (such as but not limited to the actual moving distance, direction, speed, etc.) based on one or more second sensors, and acquiring ultrasonic positioning data of the underwater cleaning robot based on the ultrasonic sensor; then, determining the first acquisition position information corresponding to the next frame of the environmental image in the two adjacent frames of environmental images based on the two adjacent frames of environmental images and the previous frame acquisition position information corresponding to the previous frame of the environmental image in the two adjacent frames of environmental images (for example, but not limited to, the next frame of environmental image can be calculated based on the position change information of the same structural element in the two adjacent frames of environmental images and the previous frame acquisition position information corresponding to the previous frame of the environmental image); The method further comprises: determining the first acquisition position information corresponding to the next frame of the environment image based on the actual motion data (for example, but not limited to, the second acquisition position information corresponding to the next frame of the environment image can be calculated based on the corresponding moving distance and direction between two adjacent frames of the environment image and the previous frame of the environment image acquisition position information corresponding to the previous frame of the environment image), and determining the third acquisition position information corresponding to the next frame of the environment image based on the ultrasonic positioning data (for example, but not limited to, the position information indicated in the ultrasonic positioning data can be directly determined as the third acquisition position information corresponding to the next frame of the environment image); finally, determining the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information, the second acquisition position information and the third acquisition position information, for example, but not limited to, the first acquisition position information, the second acquisition position information and the third acquisition position information can be weightedly fused according to a preset weight ratio to obtain the target acquisition position information corresponding to the next frame of the environment image.

[0112] In an embodiment of the present application, the underwater cleaning robot can be positioned comprehensively through multiple sensors such as visual sensors (first sensors), motion sensors (second sensors) and ultrasonic sensors, thereby achieving perception of the current environment in a weak texture environment while ensuring real-time and accurate positioning of the underwater cleaning robot.

[0113] Optionally, in order to further ensure the accuracy of the real-time positioning of the underwater cleaning robot and improve the positioning accuracy and positioning capability of the underwater cleaning robot, Figure 5As shown, after the target acquisition position information corresponding to the next frame of the environment image is determined based on the first acquisition position information, the second acquisition position information and the third acquisition position information, the underwater cleaning robot mapping method further includes: first determining the position loss corresponding to the next frame of the environment image based on the predicted acquisition position information corresponding to the next frame of the environment image, the first acquisition position information, the second acquisition position information and the third acquisition position information, for example but not limited to first calculating the position differences corresponding to the predicted acquisition position information and the first acquisition position information, the second acquisition position information and the third acquisition position information, and then weighted fusion of the corresponding position differences to obtain the position loss corresponding to the next frame of the environment image; then, based on the predicted structure information and the target structure information corresponding to the next frame of the environment image The image loss corresponding to the next frame of the environment image is determined based on the information, for example, but not limited to comprehensively calculating the position loss corresponding to the next frame of the environment image according to the position deviation value and the geometric feature deviation value between the corresponding structure boundaries in the above-mentioned predicted structure information and the target structure information; finally, based on the position loss and image loss corresponding to the next frame of the environment image, the position drift of multiple sensors is corrected, for example, but not limited to adjusting the internal parameters of multiple sensors such as temperature drift, zero votes, etc., or the positioning data collected by the sensor (such as an ultrasonic sensor) itself, so as to minimize the position loss and image loss corresponding to the above-mentioned next frame of the environment image, thereby continuously correcting the positioning error of the underwater cleaning robot during the mapping process, and improving the positioning accuracy of the next frame of the environment image corresponding to the next frame of the environment image.

[0114] Optionally, in order to eliminate the interference of abnormal ultrasonic positioning data on the precise positioning of the underwater cleaning robot, before determining the target acquisition position information corresponding to the next frame of the environmental image based on the first acquisition position information, the second acquisition position information and the third acquisition position information, it is also possible but not limited to first judging whether the ultrasonic positioning data collected by the ultrasonic sensor meets the non-line-of-sight condition based on the next frame of the environmental image. The non-line-of-sight condition is used to characterize the situation where there is occlusion in the next frame of the environmental image and the ultrasonic positioning data is abnormally jumping. The abnormal jumping situation is used to characterize that the difference between the ultrasonic positioning data corresponding to the next frame of the environmental image and the ultrasonic positioning data corresponding to the previous frame of the environmental image is greater than the preset ultrasonic positioning threshold, that is, the ultrasonic positioning jump amplitude is large; if the next frame of the environmental image is blocked, the ultrasonic positioning data collected by the ultrasonic sensor meets the non-line-of-sight condition. If the ultrasonic positioning data corresponding to a frame of environmental image meets the non-line-of-sight condition, it means that the ultrasonic positioning data is abnormal positioning data and there may be a large positioning error. In this case, the ultrasonic positioning data will be discarded, and only the target acquisition position information corresponding to the next frame of environmental image will be determined based on the first acquisition position information and the second acquisition position information. If the ultrasonic positioning data does not meet the non-line-of-sight condition, it means that the ultrasonic positioning data is normal positioning data and the positioning is relatively accurate. In this case, the above-mentioned steps of determining the target acquisition position information corresponding to the next frame of environmental image based on the first acquisition position information, the second acquisition position information and the third acquisition position information can be directly executed, thereby only using relatively accurate ultrasonic positioning data to assist the underwater cleaning robot in meeting the more accurate positioning requirements in a weak texture environment.

[0115] Optionally, in order to avoid the influence of abnormal visual tracking of the underwater cleaning robot on its positioning accuracy, such as Figure 6As shown, before determining the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information, the second acquisition position information and the third acquisition position information, it is also possible but not limited to first counting the number of target structure boundary points in the next frame of the environment image that meet the preset constraint conditions, and the total number of structure boundary points in the target structure information corresponding to the next frame of the environment image, the preset constraint condition is that there are matching target structure boundary points in at least two consecutive frames of the environment image, and the preset constraint condition may include but is not limited to good constraint conditions (i.e., there are matching target structure boundary points in more than two consecutive frames of the environment image, that is, the same target structure boundary point is tracked in more than two consecutive frames of the environment image) and general constraints (i.e., there are matching target structure boundary points only in two consecutive frames of the environment image, that is, the same target structure boundary point is tracked only in two consecutive frames of the environment image). Since the higher the ratio between the number of target structure boundary points that meet the preset constraint conditions and the total number of structure boundary points, the better the tracking effect of the underwater cleaning robot is, and the higher the accuracy of visual positioning based on the environmental image collected by the underwater robot is, the visual constraint weight corresponding to the first sensor can be determined directly based on the ratio between the number of target structure boundary points that meet the preset constraint conditions and the total number of structure boundary points. For example, but not limited to, different ratios correspond to different visual constraint weights, or, when the ratio is greater than or equal to a first threshold (for example, but not limited to 0.9, 0.8, etc.), the visual constraint weight is set to a first weight (for example, but not limited to 1, 0.9, etc.), and when the ratio is less than the first threshold and greater than the second threshold (for example, but not limited to 0.6, 0.5, etc.), the visual constraint weight is set to a second weight (for example, but not limited to 0.6, 0.5, etc.). When the ratio is less than or equal to the second threshold, the visual constraint weight is set to a third weight (for example, but not limited to 0, 0.2, etc.); at the same time, the weights corresponding to the second sensor and the ultrasonic sensor can be determined based on their corresponding built-in covariance matrices; finally, the target weight ratios between the first sensor, the second sensor and the ultrasonic sensor can be determined based on the visual constraint weight corresponding to the first sensor and the weights corresponding to the second sensor and the ultrasonic sensor, and the first acquisition position information obtained by the first sensor, the second acquisition position information obtained by the second sensor, and the third acquisition position information obtained by the ultrasonic sensor are weightedly fused according to the above target weight ratio to obtain more accurate target acquisition position information corresponding to the next frame of the environmental image.

[0116] Please refer to the following Figure 7 , which is a schematic diagram of the structure of an underwater cleaning robot mapping device provided by an exemplary embodiment of the present application. Figure 7 As shown, the underwater cleaning robot mapping device 700 includes:

[0117] The feature extraction module 710 is used to extract features from the environment image collected by the underwater cleaning robot to obtain target structure information in the target space where the underwater cleaning robot is located;

[0118] A feature matching module 720 is used to perform feature matching based on the target structure information in the target space corresponding to two adjacent frames of the environment image to obtain a corresponding target feature matching result;

[0119] The first determination module 730 is used to determine the target depth information of the target structure boundary point in the target structure information in the target space based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot; the target acquisition position information is comprehensively determined based on the positioning information obtained by the various sensors of the underwater cleaning robot; the target structure boundary point is the structure boundary point matched in the target structure information corresponding to each of the two adjacent frames of the environment image;

[0120] The mapping module 740 is used to construct a target environment map of the target space based on the target structure information and the target depth information corresponding to the boundary points of the target structure.

[0121] In a possible implementation, the feature extraction module 710 includes:

[0122] An extraction unit is used to extract structural boundary information from the environment image collected by the underwater cleaning robot; the structural boundary information includes first position information corresponding to the structural boundary existing in the environment image and second position information of the area included in the contour corresponding to the structural boundary;

[0123] The first determining unit is used to determine the target structure information in the target space based on the structure boundary information.

[0124] In a possible implementation, the feature extraction module 710 is specifically used for:

[0125] By using the multi-task model, the target space component element information and the target space auxiliary element information in the environment image collected by the underwater cleaning robot are segmented and identified respectively, so as to obtain the target structure information in the target space where the underwater cleaning robot is located.

[0126] In a possible implementation, the feature matching module 720 includes:

[0127] A collection position prediction unit, used to predict the predicted collection position information of the next frame of the environmental image between two adjacent frames of the environmental image corresponding to the underwater cleaning robot based on the previous frame of the environmental image between two adjacent frames of the environmental image corresponding to the previous frame of the underwater cleaning robot;

[0128] A structure information prediction unit, used to predict the predicted structure information in the next frame of the environment image based on the predicted acquisition position information and the target structure information in the previous frame of the environment image;

[0129] A structure information matching unit, used to match the target structure information in the next frame of the environment image with the predicted structure information to obtain a predicted structure matching range corresponding to the target structure information in the next frame of the environment image;

[0130] The second determination unit is used to determine the matching target structure boundary points in the target structure information corresponding to each of the two adjacent frames of the environmental image based on the geometric features of each structure boundary point in the previous frame of the environmental image and the geometric features of each structure boundary point within the predicted structure matching range.

[0131] In a possible implementation, the multiple sensors include a first sensor, a second sensor and an ultrasonic sensor, the first sensor is used to collect the environmental image of the underwater cleaning robot, the second sensor is used to collect the motion data of the underwater cleaning robot; the second sensor includes at least one of the following: an inertial measurement unit, an encoder, a tachometer; the ultrasonic sensor is used to collect the ultrasonic positioning data of the underwater cleaning robot;

[0132] The above-mentioned underwater cleaning robot mapping device 700 also includes:

[0133] An acquisition module, used to acquire two adjacent frames of environmental images based on the first sensor, acquire actual motion data of the underwater cleaning robot based on one or more of the second sensors, and acquire ultrasonic positioning data of the underwater cleaning robot based on the ultrasonic sensor;

[0134] A second determination module is used to determine the first acquisition position information corresponding to the latter frame of the above two adjacent frames of environmental images based on the above two adjacent frames of environmental images and the previous frame acquisition position information corresponding to the previous frame of the above two adjacent frames of environmental images, determine the second acquisition position information corresponding to the above latter frame of environmental images based on the above actual motion data, and determine the third acquisition position information corresponding to the above latter frame of environmental images based on the above ultrasonic positioning data;

[0135] The third determining module is used to determine the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information, the second acquisition position information and the third acquisition position information.

[0136] In a possible implementation, the underwater cleaning robot mapping device 700 further includes:

[0137] A fourth determination module is used to determine the position loss corresponding to the above-mentioned next frame of environmental image based on the predicted acquisition position information corresponding to the above-mentioned next frame of environmental image, the above-mentioned first acquisition position information, the above-mentioned second acquisition position information and the above-mentioned third acquisition position information;

[0138] A fifth determination module, configured to determine the image loss corresponding to the next frame of the environment image based on the predicted structure information and the target structure information corresponding to the next frame of the environment image;

[0139] The correction module is used to correct the position drift of the above-mentioned multiple sensors based on the position loss and image loss corresponding to the above-mentioned next frame of environmental image.

[0140] In a possible implementation, the underwater cleaning robot mapping device 700 further includes:

[0141] A judgment module, used to judge whether the ultrasonic positioning data meets the non-line-of-sight condition based on the next frame of the environmental image; the non-line-of-sight condition is used to indicate that there is occlusion in the next frame of the environmental image and the ultrasonic positioning data is abnormally jittery;

[0142] a sixth determination module, configured to determine the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information and the second acquisition position information if the ultrasonic positioning data satisfies the non-line-of-sight condition;

[0143] The third determination module is specifically used to execute the step of determining the target acquisition position information corresponding to the next frame of the environmental image based on the first acquisition position information, the second acquisition position information and the third acquisition position information if the ultrasonic positioning data does not meet the non-line-of-sight condition.

[0144] In a possible implementation, the underwater cleaning robot mapping device 700 further includes:

[0145] A statistical module, used to count the number of target structure boundary points that meet the preset constraint conditions in the above-mentioned next frame of environmental image, and the total number of structure boundary points in the target structure information corresponding to the above-mentioned next frame of environmental image; the above-mentioned preset constraint condition is that there are matching target structure boundary points in at least two consecutive frames of environmental images;

[0146] A seventh determination module, configured to determine a visual constraint weight corresponding to the first sensor based on a ratio between the number of target structure boundary points satisfying the preset constraint condition and the total number of structure boundary points;

[0147] An eighth determination module, configured to determine the weights corresponding to the second sensor and the ultrasonic sensor, respectively, based on the covariance matrices corresponding to the second sensor and the ultrasonic sensor, respectively;

[0148] a ninth determination module, configured to determine a target weight ratio corresponding to the first sensor, the second sensor, and the ultrasonic sensor based on the visual constraint weight corresponding to the first sensor, and the weights corresponding to the second sensor and the ultrasonic sensor;

[0149] The third determination module is specifically used for:

[0150] The first acquisition position information, the second acquisition position information and the third acquisition position information are fused according to the target weight ratio to obtain the target acquisition position information corresponding to the next frame of the environmental image.

[0151] In a possible implementation, the target feature matching result includes target structure boundary points that match in two adjacent frames of the environment image;

[0152] The first determining module 730 is specifically used for:

[0153] The target depth information of the target structure boundary point in the target space is estimated based on the position of the target structure boundary point in two adjacent frames of the environment image and the target acquisition position information corresponding to the underwater cleaning robot.

[0154] In a possible implementation, the underwater cleaning robot mapping device 700 further includes:

[0155] The mobile module is used to move according to the target mapping path;

[0156] An image acquisition module is used to acquire environmental images within a preset visual range at a preset frequency during movement;

[0157] The above-mentioned mapping module 740 is specifically used for:

[0158] After the target mapping path is completed, a target environment map of the target space is constructed based on target structure information corresponding to each of the multiple environment images on the target mapping path and target depth information corresponding to the target structure boundary points in the multiple environment images.

[0159] In a possible implementation, the target structure information includes target space component element information and target space auxiliary element information in the target space; the target space component element information includes structural boundary information of the target space corresponding component elements, and the component elements include at least one of the following: bottom surface, wall surface, step plane, step facade; the target space auxiliary element information includes structural boundary information of the auxiliary elements corresponding to the target space, and the auxiliary elements include at least one of the following auxiliary devices: wall lamp, floor lamp, water inlet, and water outlet.

[0160] The division of each module in the above-mentioned underwater cleaning robot mapping device is only for illustration. In other embodiments, the underwater cleaning robot mapping device can be divided into different modules as needed to complete all or part of the functions of the above-mentioned underwater cleaning robot mapping device. The implementation of each module in the underwater cleaning robot mapping device provided in the embodiment of the present application may be in the form of a computer program. The computer program can be run on an underwater cleaning robot or a terminal or a server. The program modules constituted by the computer program can be stored in the memory of the underwater cleaning robot or a terminal or a server. When the computer program is executed by the processor, all or part of the steps of the underwater cleaning robot mapping method described in the embodiment of the present application are implemented.

[0161] See next Figure 8 , which is a schematic diagram of the structure of an underwater cleaning robot provided in an embodiment of the present application. Figure 8 As shown, the underwater cleaning robot 800 includes: at least one processor 810 , at least one communication bus 820 , a user interface 830 , at least one network interface 840 , and a memory 850 .

[0162] The communication bus 820 may be used to realize the connection and communication among the above-mentioned components.

[0163] The user interface 830 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 830 may also include a standard wired interface and a wireless interface.

[0164] The network interface 840 may include a Bluetooth module, a Near Field Communication (NFC) module, a Wireless Fidelity (Wi-Fi) module, and the like.

[0165] Among them, the processor 810 may include one or more processing cores. The processor 810 uses various interfaces and lines to connect the various parts of the entire underwater cleaning robot 800, and executes various functions and processes data of the underwater cleaning robot 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 850, and calling data stored in the memory 850. Optionally, the processor 810 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 810 can integrate one or more combinations of a processor (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 810, and it can be implemented by a single chip.

[0166] Among them, the memory 850 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 850 includes a non-transitory computer-readable medium. The memory 850 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 850 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a feature extraction function, a mapping function, a feature matching function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 850 may also be at least one storage device located away from the aforementioned processor 810. As Figure 8 As shown, the memory 850 as a computer storage medium may include an operating system, a network communication module, a user interface module, and program instructions.

[0167] Specifically, the processor 810 can be used to call the program instructions stored in the memory 850, and specifically execute any operation in the underwater cleaning robot mapping method provided in the above method embodiment of the present application.

[0168] The embodiment of the present application also provides a computer storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods. If the various component modules of the above underwater cleaning robot mapping device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the storage medium.

[0169] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes. In the absence of conflict, the technical features in this embodiment and the implementation scheme can be combined arbitrarily.

[0171] The embodiments described above are merely preferred embodiments of the present application and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present application.

Claims

1. A method for mapping an underwater cleaning robot, characterized in that: The method comprises: Performing feature extraction on the environment image collected by the underwater cleaning robot to obtain target structure information in the target space where the underwater cleaning robot is located; Performing feature matching on target structure information in the target space corresponding to two adjacent frames of the environment image to obtain a corresponding target feature matching result; The target depth information of the target structure boundary point in the target space in the target structure information is determined based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot; the target acquisition position information is comprehensively determined based on the positioning information obtained by the various sensors of the underwater cleaning robot; the target structure boundary point is the structure boundary point matched in the target structure information corresponding to each of two adjacent frames of the environment image; Based on the target structure information and the target depth information corresponding to the boundary points of the target structure, a target environment map of the target space is constructed.

2. The method according to claim 1, characterized in that The feature extraction of the environment image collected by the underwater cleaning robot to obtain the target structure information in the target space where the underwater cleaning robot is located includes: Extracting structural boundary information from an environment image collected by an underwater cleaning robot; the structural boundary information includes first position information corresponding to a structural boundary existing in the environment image and second position information of an area including a contour corresponding to the structural boundary; Target structure information in the target space is determined based on the structure boundary information.

3. The method according to claim 1, characterized in that The feature extraction of the environment image collected by the underwater cleaning robot to obtain the target structure information in the target space where the underwater cleaning robot is located includes: By using the multi-task model, the target space component element information and the target space auxiliary element information in the environment image collected by the underwater cleaning robot are segmented and identified respectively, so as to obtain the target structure information in the target space where the underwater cleaning robot is located.

4. The method according to claim 1, characterized in that The performing feature matching based on the target structure information in the target space corresponding to two adjacent frames of the environment image to obtain the corresponding target feature matching result includes: Predicting predicted acquisition position information of a subsequent frame of the environmental image between two adjacent frames of the environmental image corresponding to the underwater cleaning robot based on the previous frame acquisition position information of the underwater cleaning robot corresponding to the previous frame of the environmental image between two adjacent frames of the environmental image; Predicting predicted structure information in the next frame of environment image based on the predicted acquisition position information and the target structure information in the previous frame of environment image; Matching the target structure information in the next frame of the environment image with the predicted structure information to obtain a predicted structure matching range corresponding to the target structure information in the next frame of the environment image; Based on the geometric features of each structure boundary point in the previous frame of the environmental image and the geometric features of each structure boundary point within the predicted structure matching range, the target structure boundary points that match in the target structure information corresponding to each of two adjacent frames of the environmental image are determined.

5. The method according to claim 1, characterized in that The multiple sensors include a first sensor, a second sensor and an ultrasonic sensor, the first sensor is used to collect the environmental image of the underwater cleaning robot, the second sensor is used to collect the motion data of the underwater cleaning robot; the second sensor includes at least one of the following: an inertial measurement unit, an encoder, a tachometer; the ultrasonic sensor is used to collect the ultrasonic positioning data of the underwater cleaning robot; The method further comprises: determining the target structure boundary point in the target structure information based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot before the target depth information in the target space. Acquire two adjacent frames of environment images based on the first sensor, acquire actual motion data of the underwater cleaning robot based on one or more of the second sensors, and acquire ultrasonic positioning data of the underwater cleaning robot based on the ultrasonic sensor; Determine first acquisition position information corresponding to a later frame of the two adjacent frames of environmental images based on the two adjacent frames of environmental images and the previous frame acquisition position information corresponding to the previous frame of the two adjacent frames of environmental images, determine second acquisition position information corresponding to the later frame of environmental images based on the actual motion data, and determine third acquisition position information corresponding to the later frame of environmental images based on the ultrasonic positioning data; The target acquisition position information corresponding to the next frame of the environment image is determined based on the first acquisition position information, the second acquisition position information, and the third acquisition position information.

6. The method according to claim 5, characterized in that After determining the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information, the second acquisition position information, and the third acquisition position information, the method further includes: Determine the position loss corresponding to the next frame of the environment image based on the predicted acquisition position information corresponding to the next frame of the environment image, the first acquisition position information, the second acquisition position information, and the third acquisition position information; Determining an image loss corresponding to the next frame of the environment image based on the predicted structure information and the target structure information corresponding to the next frame of the environment image; The position drift of the multiple sensors is corrected based on the position loss and image loss corresponding to the next frame of the environment image.

7. The method according to claim 5, characterized in that Before determining the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information, the second acquisition position information, and the third acquisition position information, the method further includes: Based on the next frame of the environment image, determining whether the ultrasonic positioning data meets the non-line-of-sight condition; the non-line-of-sight condition is used to indicate that there is occlusion in the next frame of the environment image and the ultrasonic positioning data is abnormally jittery; If the ultrasonic positioning data meets the non-line-of-sight condition, determining the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information and the second acquisition position information; If the ultrasonic positioning data does not meet the non-line-of-sight condition, the step of determining the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information, the second acquisition position information and the third acquisition position information is performed.

8. The method according to claim 5, characterized in that Before determining the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information, the second acquisition position information, and the third acquisition position information, the method further includes: Counting the number of target structure boundary points that meet preset constraint conditions in the next frame of the environment image, and the total number of structure boundary points in the target structure information corresponding to the next frame of the environment image; the preset constraint condition is that there are matching target structure boundary points in at least two consecutive frames of the environment image; Determining a visual constraint weight corresponding to the first sensor based on a ratio between the number of target structure boundary points that meet the preset constraint condition and the total number of structure boundary points; Determine the weights corresponding to the second sensor and the ultrasonic sensor respectively based on the covariance matrices corresponding to the second sensor and the ultrasonic sensor respectively; Determine a target weight ratio corresponding to the first sensor, the second sensor, and the ultrasonic sensor based on the visual constraint weight corresponding to the first sensor and the weights corresponding to the second sensor and the ultrasonic sensor; The determining the target acquisition position information corresponding to the next frame of the environment image based on the first acquisition position information, the second acquisition position information, and the third acquisition position information includes: The first acquisition position information, the second acquisition position information and the third acquisition position information are fused according to a target weight ratio to obtain the target acquisition position information corresponding to the next frame of the environment image.

9. The method according to claim 1, characterized in that The target feature matching result includes target structure boundary points that match in two adjacent frames of the environment image; The determining the target depth information of the target structure boundary point in the target space in the target structure information based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot includes: The target depth information of the target structure boundary point in the target space is estimated based on the position of the target structure boundary point in two adjacent frames of the environment image and the target acquisition position information corresponding to the underwater cleaning robot.

10. The method according to claim 1, characterized in that Before constructing the target environment map of the target space based on the target structure information and the target depth information corresponding to the target structure boundary points, the method further includes: Move along the target mapping path and collect environmental images within a preset visual range at a preset frequency during the movement; The step of constructing a target environment map of the target space based on the target structure information and the target depth information corresponding to the boundary points of the target structure includes: After the target mapping path is completed, a target environment map of the target space is constructed based on target structure information corresponding to each of a plurality of environment images on the target mapping path and target depth information corresponding to the target structure boundary points in the plurality of environment images.

11. The method according to any one of claims 1 to 10, characterized in that: The target structure information includes target space component element information and target space auxiliary element information in the target space; the target space component element information includes structural boundary information of the corresponding component element of the target space, and the component element includes at least one of the following: bottom surface, wall surface, step plane, and step elevation; The target space auxiliary element information includes structural boundary information of the auxiliary element corresponding to the target space, and the auxiliary element includes at least one of the following auxiliary devices: a wall lamp, a floor lamp, a water inlet, and a water outlet.

12. A mapping device for an underwater cleaning robot, characterized in that: The underwater cleaning robot mapping device comprises: A feature extraction module is used to extract features from the environment image collected by the underwater cleaning robot to obtain target structure information in the target space where the underwater cleaning robot is located; A feature matching module, used for performing feature matching on target structure information in the target space corresponding to two adjacent frames of the environment image to obtain a corresponding target feature matching result; A first determination module is used to determine the target depth information of the target structure boundary point in the target structure information in the target space based on the target feature matching result and the target acquisition position information corresponding to the underwater cleaning robot; the target acquisition position information is comprehensively determined based on the positioning information obtained by the various sensors of the underwater cleaning robot; the target structure boundary point is the structure boundary point matched in the target structure information corresponding to each of two adjacent frames of the environment image; A mapping module is used to construct a target environment map of the target space based on the target structure information and the target depth information corresponding to the boundary points of the target structure.

13. An underwater cleaning robot, characterized in that: The underwater cleaning robot comprises a robot controller, which comprises a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 11.

14. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, which are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 11.

Images