Control method and device of underwater cleaning robot and underwater cleaning robot

Abstract

The application discloses a control method and device of an underwater cleaning robot and the underwater cleaning robot, and belongs to the field of robot control. Through the technical scheme provided by the embodiment of the application, in the case that the underwater cleaning robot executes a cleaning task, a first environment image around the underwater cleaning robot is acquired. Target recognition is performed on the first environment image to obtain a target recognition frame on the first environment image, and the target recognition frame is used to indicate the position of a target to be cleaned on the first environment image. Based on the first environment image and the target recognition frame, the underwater cleaning robot is controlled to clean the target to be cleaned, intelligent recognition and cleaning of the target to be cleaned by the underwater cleaning robot are realized, and the cleaning effect of the underwater cleaning robot on a pool is improved.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a control method, apparatus, device, and underwater cleaning robot. Background Technology

[0002] With the development of computer technology, robotics technology has also developed rapidly. For example, users use robotic vacuum cleaners to clean house floors, robotic window cleaners to clean house windows, and underwater cleaning robots to clean pools.

[0003] In related technologies, when using underwater cleaning robots to clean pools, the pool is often traversed and cleaned through a pre-set path.

[0004] However, the movement of underwater cleaning robots can cause water flow disturbances, which can carry debris to already cleaned areas, resulting in poor cleaning effectiveness of underwater cleaning robots for pools. Summary of the Invention

[0005] This application provides a control method, apparatus, and underwater cleaning robot for an underwater cleaning robot, which can improve the cleaning effect of using an underwater cleaning robot to clean pools. The technical solution is as follows:

[0006] On the one hand, a control method for an underwater cleaning robot is provided, the method comprising:

[0007] While the underwater cleaning robot is performing a cleaning task, a first environmental image of the area surrounding the underwater cleaning robot is acquired.

[0008] Target recognition is performed on the first environmental image to obtain a target recognition box on the first environmental image. The target recognition box is used to indicate the position of the target to be cleaned on the first environmental image.

[0009] Based on the first environmental image and the target recognition box, the underwater cleaning robot is controlled to clean the target to be cleaned.

[0010] In one possible implementation, the step of performing target recognition on the first environmental image to obtain a target recognition box on the first environmental image includes:

[0011] The first environmental image is input into the target recognition model, which is trained based on multiple sample environmental images, the labeled recognition boxes on each sample environmental image, and the target classification size. The labeled recognition boxes are used to indicate the actual location of the target to be cleaned on the sample environmental image, and the target classification size is determined based on the size of the labeled recognition boxes on the multiple sample environmental images.

[0012] Using the target recognition model, at least one candidate bounding box is slid and classified on the first environmental image to obtain the target bounding box on the first environmental image.

[0013] In one possible implementation, the step of sliding and classifying at least one candidate bounding box on the first environmental image to obtain a target bounding box on the first environmental image includes:

[0014] Control the at least one candidate bounding box to slide on the first environmental image;

[0015] Feature extraction is performed on multiple image regions in the first environmental image that are covered by the at least one candidate recognition box to obtain the regional image features of each image region;

[0016] Based on the regional image features of each of the image regions, a target recognition box is obtained on the first environmental image.

[0017] In one possible implementation, obtaining the target recognition box on the first environment image based on the regional image features of each of the image regions includes:

[0018] For any image region among the plurality of image regions, the regional image features of the image region are fully connected and normalized to obtain the regional classification value of the image region.

[0019] If the region classification value is greater than or equal to the first classification value threshold, the bounding box of the image region is determined as the target recognition box.

[0020] In one possible implementation, the training method for the target recognition model includes:

[0021] The multiple sample environment images are input into the initial model, and the initial model is used to determine the predicted recognition box on each of the sample environment images.

[0022] Based on the predicted recognition boxes and the labeled recognition boxes on each of the sample environment images, the initial predicted classification value of the predicted recognition boxes on each of the sample environment images is determined.

[0023] Based on the target classification size and the size of the predicted recognition box on each of the sample environment images, the classification value correction coefficient of the predicted recognition box on each of the sample environment images is determined;

[0024] Based on the target prediction classification value, first classification value threshold, and second classification value threshold of the predicted recognition box on each of the sample environment images, the initial model is trained to obtain the target recognition model. The target prediction classification value is determined based on the initial prediction classification value and classification value correction coefficient of the corresponding prediction recognition box. The first classification value threshold is used to identify the prediction recognition box containing the target to be cleared, and the second classification value threshold is used to identify the prediction recognition box that does not contain the target to be cleared.

[0025] In one possible implementation, determining the classification value correction coefficient for each predicted bounding box in each sample environment image based on the target classification size and the size of the predicted bounding box in each of the sample environment images includes:

[0026] For any predicted bounding box on any sample environment image, the square root of the division between the target classification size and the predicted bounding box size is used to obtain the classification value correction coefficient of the predicted bounding box.

[0027] In one possible implementation, training the initial model based on the target predicted classification value, a first classification value threshold, and a second classification value threshold of the predicted recognition box on each of the sample environment images includes:

[0028] For any predicted bounding box on any sample environment image, if the target predicted classification value of the predicted bounding box is greater than or equal to the first classification value threshold, the predicted bounding box is determined as a positive sample; the initial model is trained based on the positive sample;

[0029] If the target predicted classification value of the predicted recognition box is less than or equal to the second classification value threshold, the predicted recognition box is determined as a negative sample; the initial model is trained based on the negative sample.

[0030] In one possible implementation, the method for determining the target classification size includes:

[0031] The plurality of sample environment images are divided into a plurality of sample image groups, and each sample image group includes at least one sample environment image;

[0032] Based on the size of the bounding boxes in the sample environment images of each sample image group, the average size of the bounding boxes in each sample image group is determined.

[0033] The target classification size is obtained by iterating the reference classification size multiple times based on the average size of the bounding boxes of each of the sample image groups. Each iteration uses one of the sample image groups, and different sample image groups are used in different iterations.

[0034] In one possible implementation, the step of iterating the reference classification size based on the average size of the bounding boxes of each of the sample image groups to obtain the target classification size includes:

[0035] For any iteration in a multi-round iteration, the reference classification size determined in the previous iteration is weighted and fused with the average size of the bounding boxes of the sample image group used in the current iteration to obtain the reference classification size for the current iteration;

[0036] If this is the last iteration, the reference classification size for this iteration will be determined as the target classification size.

[0037] In one possible implementation, controlling the underwater cleaning robot to clean the target based on the first environmental image and the target recognition box includes:

[0038] Based on the relative position between the center point of the target recognition box and the central axis of the first environmental image, the underwater cleaning robot is controlled to turn so that the center point of the target recognition box in the subsequently acquired first environmental image is on the central axis of the first environmental image, and the direction indicated by the central axis of the first environmental image is the forward direction of the underwater cleaning robot.

[0039] The underwater cleaning robot is controlled to move forward to the location of the target to be cleaned, so as to clean the target.

[0040] In one possible implementation, controlling the underwater cleaning robot to turn based on the relative position between the center point of the target recognition box and the central axis of the first environmental image includes:

[0041] Based on the relative position between the center point of the target recognition box and the central axis of the first environmental image, the turning direction and turning angular velocity of the underwater cleaning robot are determined.

[0042] The underwater cleaning robot is controlled to turn in accordance with the turning direction and the turning angular velocity.

[0043] In one possible implementation, determining the turning direction and turning angular velocity of the underwater cleaning robot based on the relative position between the center point of the target recognition box and the central axis of the first environmental image includes:

[0044] When the center point of the target recognition box is located to the right of the central axis of the first environmental image, the turning direction of the underwater cleaning robot is determined to be turning to the right; when the center point of the target recognition box is located to the left of the central axis of the first environmental image, the turning direction of the underwater cleaning robot is determined to be turning to the left.

[0045] The turning angular velocity is determined based on the distance between the center point of the target recognition box and the central axis of the first environmental image, and the turning angular velocity is positively correlated with the distance.

[0046] In one possible implementation, the method further includes:

[0047] When the battery level of the underwater cleaning robot is less than or equal to a preset battery threshold, a second environmental image of the underwater cleaning robot is acquired.

[0048] The second environmental image is used to identify charging piles to obtain charging pile identification boxes on the second environmental image. The charging pile identification boxes are used to indicate the position of the charging piles on the second environmental image.

[0049] Based on the second environmental image and the charging pile identification frame, the underwater cleaning robot is controlled to move to the location of the charging pile so that the underwater cleaning robot can be charged through the charging pile.

[0050] In one possible implementation, the step of identifying charging piles in the second environmental image to obtain a charging pile identification box on the second environmental image includes:

[0051] Determine the target color region in the second environmental image. The target color region is a concentrated area of ​​pixels of a preset color, and the preset color is the color of the indicator light of the charging pile.

[0052] The bounding box of the target color area is defined as the charging pile identification box.

[0053] In one possible implementation, controlling the underwater cleaning robot to move to the location of the charging pile based on the second environmental image and the charging pile identification frame includes:

[0054] Based on the relative position between the center point of the charging pile identification frame and the central axis of the second environmental image, the underwater cleaning robot is controlled to turn so that the charging pile identification frame in the subsequently acquired second environmental image is on the central axis of the second environmental image, and the direction indicated by the central axis of the second environmental image is the forward direction of the underwater cleaning robot.

[0055] Control the underwater cleaning robot to move forward;

[0056] If the size of the charging pile identification frame in the subsequently acquired second environmental image is greater than or equal to a preset size, the underwater cleaning robot is determined to move to the location of the charging pile.

[0057] On one hand, a control device for an underwater cleaning robot is provided, the device comprising:

[0058] An environmental image acquisition module is used to acquire a first environmental image around the underwater cleaning robot when the underwater cleaning robot is performing a cleaning task.

[0059] The target recognition module is used to perform target recognition on the first environmental image to obtain a target recognition box on the first environmental image. The target recognition box is used to indicate the position of the target to be cleaned on the first environmental image.

[0060] The control module is used to control the underwater cleaning robot to clean the target to be cleaned based on the first environmental image and the target recognition box.

[0061] In one possible implementation, the target recognition module is used to input the first environmental image into a target recognition model, which is trained based on multiple sample environmental images, labeled recognition boxes on each of the sample environmental images, and target classification size. The labeled recognition boxes are used to indicate the actual location of the target to be cleaned on the sample environmental image, and the target classification size is determined based on the size of the labeled recognition boxes on the multiple sample environmental images. Through the target recognition model, at least one candidate recognition box is slid and classified on the first environmental image to obtain the target recognition box on the first environmental image.

[0062] In one possible implementation, the target recognition module is configured to control the at least one candidate recognition box to slide on the first environmental image; extract features from multiple image regions on the first environmental image covered by the at least one candidate recognition box to obtain regional image features of each image region; and obtain a target recognition box on the first environmental image based on the regional image features of each image region.

[0063] In one possible implementation, the target recognition module is configured to perform fully connected and normalized regional image features on any one of the plurality of image regions to obtain a regional classification value for the image region; and if the regional classification value is greater than or equal to a first classification value threshold, determine the bounding box of the image region as the target recognition box.

[0064] In one possible implementation, the training device for the target recognition model includes:

[0065] The prediction module is used to input the multiple sample environment images into an initial model and determine the predicted recognition box on each of the sample environment images through the initial model;

[0066] The prediction classification value determination module is used to determine the initial prediction classification value of the prediction recognition box on each of the sample environment images based on the predicted recognition box on each of the sample environment images and the labeled recognition box on each of the sample environment images.

[0067] The classification value correction coefficient determination module is used to determine the classification value correction coefficient of the predicted recognition box on each of the sample environment images based on the target classification size and the size of the predicted recognition box on each of the sample environment images;

[0068] The training module is used to train the initial model based on the target prediction classification value, the first classification value threshold, and the second classification value threshold of the predicted recognition box on each of the sample environment images to obtain the target recognition model. The target prediction classification value is determined based on the initial prediction classification value and the classification value correction coefficient of the corresponding prediction recognition box. The first classification value threshold is used to identify the prediction recognition box containing the target to be cleared, and the second classification value threshold is used to identify the prediction recognition box that does not contain the target to be cleared.

[0069] In one possible implementation, the classification value correction coefficient determination module is used to, for any predicted recognition box on any sample environment image, divide the target classification size by the size of the predicted recognition box and then take the square root to obtain the classification value correction coefficient of the predicted recognition box.

[0070] In one possible implementation, the training module is configured to, for any predicted bounding box on any sample environment image, determine the predicted bounding box as a positive sample if the target predicted classification value of the predicted bounding box is greater than or equal to the first classification value threshold; train the initial model based on the positive samples; and determine the predicted bounding box as a negative sample if the target predicted classification value of the predicted bounding box is less than or equal to the second classification value threshold; train the initial model based on the negative samples.

[0071] In one possible implementation, the target classification size determination device includes:

[0072] A grouping module is used to divide the plurality of sample environment images into a plurality of sample image groups, wherein a sample image group includes at least one sample environment image;

[0073] The average size determination module for annotation boxes is used to determine the average size of the annotation boxes in each of the sample image groups based on the size of the annotation recognition boxes in the sample environment images of each sample image group.

[0074] The target classification size determination module is used to perform multiple iterations on the reference classification size based on the average size of the bounding boxes of each of the sample image groups to obtain the target classification size. Each iteration uses one of the sample image groups, and different sample image groups are used in different iterations.

[0075] In one possible implementation, the target classification size determination module is used to, for any iteration in multiple iterations, perform a weighted fusion of the reference classification size determined in the previous iteration and the average size of the bounding boxes of the sample image group used in the current iteration to obtain the reference classification size for the current iteration; if the current iteration is the last iteration, the reference classification size for the current iteration is determined as the target classification size.

[0076] In one possible implementation, the control module is configured to control the underwater cleaning robot to turn based on the relative position between the center point of the target recognition box and the central axis of the first environmental image, so that the center point of the target recognition box in the subsequently acquired first environmental image is on the central axis of the first environmental image, the direction indicated by the central axis of the first environmental image being the forward direction of the underwater cleaning robot; and to control the underwater cleaning robot to move forward to the location of the target to be cleaned, so as to clean the target to be cleaned.

[0077] In one possible implementation, the control module is configured to determine the turning direction and turning angular velocity of the underwater cleaning robot based on the relative position between the center point of the target recognition box and the central axis of the first environmental image; and control the underwater cleaning robot to turn in accordance with the turning direction and the turning angular velocity.

[0078] In one possible implementation, the control module is configured to determine that the underwater cleaning robot turns right when the center point of the target recognition box is located to the right of the central axis of the first environmental image; and to determine that the underwater cleaning robot turns left when the center point of the target recognition box is located to the left of the central axis of the first environmental image; and to determine a turning angular velocity based on the distance between the center point of the target recognition box and the central axis of the first environmental image, wherein the turning angular velocity is positively correlated with the distance.

[0079] In one possible implementation, the device further includes:

[0080] The charging module is used to acquire a second environmental image around the underwater cleaning robot when the battery level of the underwater cleaning robot is less than or equal to a preset battery threshold; identify charging piles in the second environmental image to obtain a charging pile identification frame on the second environmental image, the charging pile identification frame indicating the position of the charging pile in the second environmental image; and control the underwater cleaning robot to move to the location of the charging pile based on the second environmental image and the charging pile identification frame, so as to charge the underwater cleaning robot through the charging pile.

[0081] In one possible implementation, the charging module is configured to determine a target color region in the second environmental image, the target color region being a concentrated area of ​​pixels of a preset color, the preset color being the color of the indicator light of the charging pile; and to define the bounding box of the target color region as the charging pile identification box.

[0082] In one possible implementation, the charging module is configured to control the underwater cleaning robot to turn based on the relative position between the center point of the charging pile identification frame and the central axis of the second environmental image, so that the charging pile identification frame in the subsequently acquired second environmental image is located on the central axis of the second environmental image, the direction indicated by the central axis of the second environmental image being the forward direction of the underwater cleaning robot; control the underwater cleaning robot to move forward; and determine that the underwater cleaning robot has moved to the location of the charging pile if the size of the charging pile identification frame in the subsequently acquired second environmental image is greater than or equal to a preset size.

[0083] On one hand, an underwater cleaning robot is provided, the underwater cleaning robot including a robot controller, the robot controller including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the control method of the underwater cleaning robot.

[0084] On one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the control method of the underwater cleaning robot.

[0085] On one hand, a computer program product or computer program is provided, which includes program code stored in a computer-readable storage medium. The processor of the robot controller reads the program code from the computer-readable storage medium and executes the program code, causing the robot controller to perform the control method of the underwater cleaning robot described above.

[0086] The technical solution provided in this application acquires a first environmental image of the area surrounding the underwater cleaning robot while it is performing a cleaning task. Target recognition is then performed on this first environmental image to obtain a target recognition box, which indicates the location of the target to be cleaned within the first environmental image. Based on the first environmental image and the target recognition box, the underwater cleaning robot is controlled to clean the target, achieving intelligent identification and cleaning of the target by the underwater cleaning robot and improving the cleaning effect of the underwater cleaning robot on pools. Attached Figure Description

[0087] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 This is a schematic diagram of an underwater cleaning robot for the bottom of a pool, provided in an embodiment of this application;

[0089] Figure 2 This is a flowchart of a control method for an underwater cleaning robot provided in an embodiment of this application;

[0090] Figure 3 This is a flowchart of another control method for an underwater cleaning robot provided in an embodiment of this application;

[0091] Figure 4 This is a schematic diagram of target recognition provided in an embodiment of this application;

[0092] Figure 5 This is a schematic diagram of a first environmental image provided in an embodiment of this application;

[0093] Figure 6 This is a flowchart of a training method for a target recognition model provided in an embodiment of this application;

[0094] Figure 7 This is a schematic diagram of a sample environment image provided in an embodiment of this application;

[0095] Figure 8 This is a schematic diagram of the structure of a control device for an underwater cleaning robot provided in an embodiment of this application;

[0096] Figure 9 This is a structural schematic diagram of an underwater cleaning robot provided in an embodiment of this application. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0098] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0099] First, the terms used in the embodiments of this application will be introduced.

[0100] Underwater cleaning robot: A robot used to perform underwater cleaning tasks, such as placing an underwater cleaning robot in a pool to clean the bottom of the pool. In some embodiments, the underwater cleaning robot also has a wall-climbing function, enabling it to clean the pool walls.

[0101] Computer vision: Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses image acquisition devices and computers to replace human eyes in tasks such as target recognition, tracking, and measurement. It further processes images to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting 'information' from images or multidimensional data.

[0102] Target recognition: Target recognition refers to the process of distinguishing one or more specific targets (or one or more types of targets) from other targets (or other types of targets). It includes both the identification of two very similar targets and the identification of one type of target from other types of targets. In this application embodiment, target recognition refers to identifying the target to be cleaned. Additionally, the charging pile identification involved in this application embodiment also falls under the category of target recognition.

[0103] After introducing the terms used in the embodiments of this application, the application scenarios of the embodiments of this application will be described below.

[0104] The technical solution provided in this application can be applied to scenarios where underwater cleaning robots are used to clean the bottom of pools. See also... Figure 1The underwater cleaning robot 100 can move on the bottom 101 of a pool to clean it. During its movement, the robot's water pump is activated, drawing liquid from the pool into its filter box through an inlet at the bottom. The filter removes impurities and discharges the filtered liquid through an outlet, thus cleaning the pool bottom 101. In addition to using the water pump and filter box, the robot can also clean the pool bottom 101 using other cleaning units, such as a roller brush.

[0105] Using the technical solution provided in this application, when an underwater cleaning robot is performing a cleaning task, a first environmental image of the environment surrounding the underwater cleaning robot is acquired. This first environmental image reflects the environmental conditions around the underwater cleaning robot. Target recognition is performed on the first environmental image to obtain a target recognition box on the first environmental image. This target recognition box is used to indicate the position of the target to be cleaned on the first environmental image. Based on the first environmental image and the target recognition box, the underwater cleaning robot is controlled to clean the target to be cleaned. Thus, the underwater cleaning robot can intelligently identify and clean the target to be cleaned during the cleaning of the pool, improving the cleaning effect of the pool.

[0106] The technical solutions provided in the embodiments of this application are described below. Figure 2 This is a flowchart of a control method for an underwater cleaning robot provided in an embodiment of this application. See also... Figure 2 Taking a robot controller that is an underwater cleaning robot as an example, the method includes the following steps.

[0107] 201. When the underwater cleaning robot is performing a cleaning task, the robot controller acquires a first environmental image of the area surrounding the underwater cleaning robot.

[0108] The cleaning task refers to the task of cleaning the pool. The cleaning task is initiated periodically by the robot controller or manually by the user; this embodiment does not limit this. The robot controller is built into the underwater cleaning robot and is used to control it. The underwater cleaning robot is used to clean the pool. When the underwater cleaning robot is located at the bottom of the pool, it can move on the pool bottom and perform cleaning actions, which refer to cleaning the pool bottom. The underwater cleaning robot includes an image acquisition device used to acquire environmental images around the robot. The first environmental image is part of the overall environmental image and reflects the environmental conditions around the underwater cleaning robot. The first environmental image refers to the environmental image acquired when the underwater cleaning robot performs the cleaning task.

[0109] 202. The robot controller performs target recognition on the first environmental image to obtain a target recognition box on the first environmental image. The target recognition box is used to indicate the position of the target to be cleaned on the first environmental image.

[0110] The target recognition process identifies targets to be cleaned in the first environmental image, such as trash and dirt. The target bounding box is the result of the target recognition and indicates the location of the target in the first environmental image. Ideally, the target bounding box completely surrounds the target.

[0111] 203. Based on the first environmental image and the target recognition box, the robot controller controls the underwater cleaning robot to clean the target to be cleaned.

[0112] The cleaning process includes moving the target to its location and performing cleaning actions.

[0113] The technical solution provided in this application acquires a first environmental image of the area surrounding the underwater cleaning robot while it is performing a cleaning task. Target recognition is then performed on this first environmental image to obtain a target recognition box, which indicates the location of the target to be cleaned within the first environmental image. Based on the first environmental image and the target recognition box, the underwater cleaning robot is controlled to clean the target, achieving intelligent identification and cleaning of the target by the underwater cleaning robot and improving the cleaning effect of the underwater cleaning robot on pools.

[0114] It should be noted that steps 201-203 above are a brief introduction to the technical solutions provided in the embodiments of this application. The technical solutions provided in the embodiments of this application will be described in more detail below with reference to some examples. See [link to relevant documentation]. Figure 3 The method includes the following steps.

[0115] 301. When the underwater cleaning robot is performing a cleaning task, the robot controller acquires a first environmental image of the area surrounding the underwater cleaning robot.

[0116] The cleaning task refers to the task of cleaning the pool. The cleaning task is initiated periodically by the robot controller or manually by the user. For example, the user can set the execution cycle of the cleaning task for the underwater cleaning robot, and the robot controller can initiate the cleaning task periodically according to the set execution cycle; or, the user can use the underwater cleaning robot's control terminal to send cleaning task execution commands to the underwater cleaning robot to control the underwater cleaning robot to perform the cleaning task. This application embodiment does not limit the method of initiating the cleaning task. The robot controller is built into the underwater cleaning robot and is used to control the underwater cleaning robot. The underwater cleaning robot is used to clean the pool. When the underwater cleaning robot is located at the bottom of the pool, it can move on the bottom of the pool and perform cleaning actions, which refer to the actions of cleaning the bottom of the pool. The underwater cleaning robot includes an image acquisition device used to acquire environmental images around the underwater cleaning robot. In some embodiments, the image acquisition device is an RGB (Red Green Blue) camera. The first environmental image is part of the environmental image and is used to reflect the environmental conditions around the underwater cleaning robot. The first environmental image refers to the environmental image collected by the underwater cleaning robot when it performs a cleaning task. The direction indicated by the central axis of the first environmental image is the forward direction of the underwater cleaning robot. That is, when a target is located on the central axis of the first environmental image, the underwater cleaning robot can move forward to reach the location of the target.

[0117] In one possible implementation, when the underwater cleaning robot is performing a cleaning task, the robot controller acquires a first environmental image of the area surrounding the underwater cleaning robot via an image acquisition device.

[0118] The number of image acquisition devices can be multiple or single, and this application embodiment does not limit this; when there are multiple image acquisition devices, the first environmental image is obtained by stitching together environmental images acquired by multiple image acquisition devices. It should be noted that when the underwater cleaning robot is performing the cleaning task, the robot controller will continuously acquire the first environmental image through the image acquisition devices.

[0119] Taking one image acquisition device as an example, this device is installed directly in front of the underwater cleaning robot and can capture environmental images in front of the robot. When the underwater cleaning robot is performing a cleaning task, this environmental image is the first environmental image. The direction indicated by the central axis of this first environmental image is the front of the underwater cleaning robot. If any target is to the left of the central axis of the first environmental image, it means that the target is to the left front of the underwater cleaning robot.

[0120] Taking a scenario with multiple image acquisition devices as an example, one of these devices is a target image acquisition device installed directly in front of the underwater cleaning robot. The other image acquisition devices are installed at equal intervals on either side of the target image acquisition device. The robot controller acquires environmental images from these multiple image acquisition devices. Using the environmental image acquired by the target image acquisition device as the center, and according to the relative positional relationship between the other image acquisition devices and the target image acquisition device, the environmental images acquired by the other image acquisition devices are stitched onto the environmental image acquired by the target image acquisition device to obtain the first environmental image.

[0121] 302. The robot controller performs target recognition on the first environmental image to obtain a target recognition box on the first environmental image. The target recognition box is used to indicate the position of the target to be cleaned on the first environmental image.

[0122] The target recognition process identifies targets to be cleaned in the first environmental image. These targets include trash and dirt, and encompass various types such as stains, cigarette butts, debris, and broken branches. The types of targets are set and adjusted by technicians according to actual conditions, and this embodiment does not limit this. The target recognition bounding box is the result of target recognition and indicates the position of the target to be cleaned in the first environmental image. Ideally, the target recognition bounding box completely surrounds the target. There can be one or more target recognition bounding boxes; for ease of understanding, step 302 will be explained below using one target recognition bounding box as an example.

[0123] In one possible implementation, the robot controller inputs the first environmental image into a target recognition model. This model is trained based on multiple sample environmental images, labeled bounding boxes on each sample image, and target classification sizes. The labeled bounding boxes indicate the actual location of the target to be cleaned in the sample environmental image, and the target classification sizes are determined based on the sizes of the labeled bounding boxes on the multiple sample environmental images. The robot controller then uses this target recognition model to slide and classify at least one candidate bounding box on the first environmental image to obtain the target bounding box on the first environmental image.

[0124] The target recognition model is used to identify objects to be cleaned, and multiple sample environment images serve as training samples for this model. The bounding boxes on each sample environment image are annotations used during training, indicating the actual location of the target to be cleaned within the image. The target classification size is an intermediate parameter used during training to improve the model's training performance.

[0125] In this implementation, the first environmental image is used to identify targets through a target recognition model, resulting in high accuracy and efficiency in target recognition.

[0126] To provide a clearer explanation of the above embodiments, the following will describe a method for obtaining target recognition boxes on the first environment image by sliding and classifying at least one candidate recognition box on the first environment image using a target recognition model.

[0127] In one possible implementation, the robot controller uses the target recognition model to control the sliding of at least one candidate bounding box on the first environmental image. The robot controller then uses the target recognition model to extract features from multiple image regions covered by the at least one candidate bounding box on the first environmental image, obtaining the regional image features of each image region. Based on the regional image features of each image region, the robot controller uses the target recognition model to obtain the target bounding box on the first environmental image.

[0128] The number and size of the candidate recognition boxes are set by technicians according to the actual situation, and this application embodiment does not limit this. When there are multiple candidate recognition boxes, the different candidate recognition boxes have different sizes to cover image areas of different sizes, thereby realizing the recognition of targets of different sizes. The step size of the candidate recognition box sliding on the first environmental image is set by technicians according to the actual situation, and this application embodiment does not limit this. The regional image features of the image area are an abstract representation of the image area. Using regional image features can more accurately reflect the characteristics of the image area, so as to facilitate processing by the robot controller.

[0129] In this implementation, at least one candidate bounding box is controlled to slide on the first environmental image by a target recognition model. Feature extraction is performed on the image area covered by the candidate bounding box to obtain the regional image features of each image area in the first environmental image. The target bounding box, that is, the position of the target to be cleaned in the first environmental image, is determined by the regional image features of each image area, which has high accuracy.

[0130] To provide a clearer explanation of the above embodiments, the following description will be divided into several parts.

[0131] The first part describes how the robot controller, through the target recognition model, controls at least one candidate bounding box to slide on the first environmental image.

[0132] In one possible implementation, when there is only one candidate recognition box, the robot controller controls the candidate recognition box to slide on the first environmental image with a preset step size through the target recognition model, thereby covering different image regions of the first environmental image, and performing target recognition on the first environmental image with image regions as the granularity.

[0133] The preset step size is set by technicians according to the actual situation, and this application embodiment does not limit it.

[0134] In one possible implementation, when there are multiple candidate recognition boxes, the robot controller uses the target recognition model to control the corresponding candidate recognition box to slide on the first environment image with a preset step size. The different candidate recognition boxes have different sizes, thereby covering different image areas of the first environment image, and target recognition is performed on the first environment image with image areas of different sizes as the granularity.

[0135] The second part involves the robot controller using the target recognition model to extract features from multiple image regions covered by the at least one candidate recognition box on the first environmental image, thereby obtaining the regional image features of each image region.

[0136] In one possible implementation, for any one of the plurality of image regions, the robot controller performs at least one convolution on the image region using the target recognition model to obtain the regional image features of the image region.

[0137] The number of convolutions is set by technicians according to the actual situation, and this application embodiment does not limit this.

[0138] In this implementation, by performing at least one convolution on the image region, the regional image features of the image region can be obtained, and the extraction efficiency of the regional image features is relatively high.

[0139] Taking one convolution as an example, the robot controller, using this target recognition model, slides a convolution kernel across multiple color channels of the image region. During this sliding process, the convolution kernel convolves with the covered areas of the channel images to obtain the image features of each color channel. The robot controller then fuses these image features from multiple color channels using this target recognition model to obtain the regional image features of the image region.

[0140] In one possible implementation, for any one of the plurality of image regions, the robot controller performs at least one full connection on the image region using the target recognition model to obtain the regional image features of the image region.

[0141] The number of full connections is set by technicians according to the actual situation, and this application embodiment does not limit this.

[0142] In this implementation, by performing at least one full connection on the image region, the regional image features of the image region can be obtained, and the extraction efficiency of the regional image features is relatively high.

[0143] Taking one full connection as an example, the robot controller uses this target recognition model to multiply the image region with the fully connected matrix and then add it to the bias matrix to obtain the regional image features of the image region.

[0144] In one possible implementation, for any one of the multiple image regions, the robot controller encodes the image region based on the attention mechanism using the target recognition model to obtain the regional image features of that image region.

[0145] In this implementation, an attention mechanism is used to focus on the important parts of the image region, and the encoded regional image features can more accurately represent the image region.

[0146] For example, the robot controller uses this target recognition model to divide the image region into multiple image blocks. The robot controller then uses the target recognition model to perform embedding encoding on each image block, obtaining the embedding features of each block. Next, the robot controller uses the target recognition model to perform a linear transformation on the embedding features of each image block, obtaining the query matrix, key matrix, and value matrix for each image block. Based on the query matrix and key matrix of every two image blocks, the robot controller uses the target recognition model to obtain the attention weights of each image block to the other image blocks. Finally, the robot controller uses the target recognition model and these attention weights to perform a weighted fusion of the value matrices of each image block to obtain the regional image features of the image region.

[0147] It should be noted that the robot controller can extract features from an image region using any of the above methods to obtain the region's image features, and this application embodiment does not limit this. Furthermore, the above embodiments are illustrated using feature extraction on a single image region as an example; for other image regions, the feature extraction methods all belong to the same inventive concept, and the implementation process will not be elaborated further.

[0148] Part Three: The robot controller obtains the target recognition bounding box on the first environmental image based on the regional image features of each image region through the target recognition model.

[0149] In one possible implementation, for any one of the plurality of image regions, the robot controller uses the target recognition model to perform fully connected and normalized regional image features of that region to obtain a region classification value. If the region classification value is greater than or equal to a first classification value threshold, the robot controller determines the bounding box of that image region as the target recognition box.

[0150] The first classification threshold is set by a technician according to the actual situation, and this application embodiment does not limit it. The first classification threshold is used to identify target recognition boxes containing the target to be cleaned. The bounding box of an image region is actually a candidate bounding box that covers the image region, and the size and shape of the candidate bounding box are the same.

[0151] In this implementation, a region classification value is obtained by performing fully connected and normalized processing on the region image features. If the region classification value is greater than or equal to a first classification value threshold, the bounding box of the image region is determined as the target recognition box, thereby achieving target recognition on the first environmental image.

[0152] For example, see Figure 4 After inputting the first environment image 401 into the target recognition model 402, a target recognition box 403 on the first environment image 401 can be obtained.

[0153] 303. The robot controller controls the underwater cleaning robot to clean the target based on the first environmental image and the target recognition box.

[0154] The cleaning process includes moving the target to its location and performing cleaning actions.

[0155] In one possible implementation, the robot controller controls the underwater cleaning robot to turn based on the relative position between the center point of the target identification box and the central axis of the first environmental image, so that the center point of the target identification box in a subsequently acquired first environmental image is aligned with the central axis of the first environmental image, the direction indicated by the central axis of the first environmental image being the forward direction of the underwater cleaning robot. The robot controller then controls the underwater cleaning robot to move forward to the location of the target to be cleaned, in order to clean the target.

[0156] The center point of the target recognition box refers to its geometric center, and the central axis of the first environmental image bisects the first environmental image. The direction indicated by the central axis of the first environmental image is the forward direction of the underwater cleaning robot. When the center point of the target recognition box is located on the central axis of the first environmental image, it means that the target to be cleaned in the target recognition box is in the forward direction of the underwater cleaning robot.

[0157] In this implementation, by utilizing the relative positional relationship between the center point of the target recognition box and the central axis of the first environmental image, the underwater cleaning robot can be controlled to turn so that the forward direction of the underwater cleaning robot is toward the target to be cleaned indicated by the target recognition box, so that the underwater cleaning robot can clean the target to be cleaned.

[0158] To provide a clearer explanation of the above embodiments, the following description will be divided into several parts.

[0159] The first part describes how the robot controller controls the underwater cleaning robot to turn based on the relative position between the center point of the target recognition box and the central axis of the first environmental image.

[0160] In one possible implementation, the robot controller determines the turning direction and turning angular velocity of the underwater cleaning robot based on the relative position between the center point of the target recognition box and the central axis of the first environmental image. The robot controller controls the underwater cleaning robot to turn in accordance with the turning direction and the turning angular velocity, so that the center point of the target recognition box in the subsequently acquired first environmental image is located on the central axis of the first environmental image.

[0161] In this implementation, the turning direction and turning angular velocity of the underwater cleaning robot can be determined by using the relative position between the center point of the target recognition box and the central axis of the first environmental image. The underwater cleaning robot can be controlled by using the turning direction and turning angular velocity, which is highly efficient.

[0162] For example, if the center point of the target recognition box is located to the right of the central axis of the first environmental image, the robot controller determines that the underwater cleaning robot will turn to the right. If the center point of the target recognition box is located to the left of the central axis of the first environmental image, the robot controller determines that the underwater cleaning robot will turn to the left. The robot controller determines the turning angular velocity based on the distance between the center point of the target recognition box and the central axis of the first environmental image, and this turning angular velocity is positively correlated with the distance.

[0163] The positive correlation between the steering angular velocity and the distance means that the larger the distance, the larger the steering angular velocity; and the smaller the distance, the smaller the steering angular velocity. The correspondence between the steering angular velocity and the distance is set by a technician according to the actual situation, and this application embodiment does not limit this.

[0164] In one possible implementation, the robot controller determines the turning direction and turning angular velocity of the underwater cleaning robot based on the relative position between the center point of the target recognition box and the central axis region of the first environmental image, where the central axis bisects the central axis region. The robot controller controls the underwater cleaning robot to turn according to the turning direction and the turning angular velocity, so that the center point of the target recognition box in a subsequently acquired first environmental image is located on the central axis region of the first environmental image.

[0165] The central axis region can be considered as the region obtained by shifting the central axis to the left and right by the same distance. The length of the central axis region is the width of the first environmental image. The width of the central axis region is set by the technician according to the actual situation, and this application embodiment does not limit it.

[0166] In this implementation, the turning direction and turning angular velocity of the underwater cleaning robot can be determined by using the relative position between the center point of the target recognition box and the central axis region of the first environmental image. The underwater cleaning robot can be controlled by using the turning direction and turning angular velocity, which is highly efficient.

[0167] For example, if the center point of the target recognition box is located to the right of the central axis region of the first environmental image, the robot controller determines that the underwater cleaning robot will turn to the right. If the center point of the target recognition box is located to the left of the central axis region of the first environmental image, the robot controller determines that the underwater cleaning robot will turn to the left. The robot controller determines the turning angular velocity based on the distance between the center point of the target recognition box and the central axis region of the first environmental image.

[0168] For example, see Figure 5The first environmental image 500 includes a central axis region 501. When the center point 502 of the target recognition box is to the left of the central axis region 501, the robot controller determines that the underwater cleaning robot turns to the left.

[0169] Part Two: The robot controller controls the underwater cleaning robot to move forward to the location of the target to be cleaned.

[0170] In one possible implementation, the robot controller controls the underwater cleaning robot to move forward. If the size of the target recognition box in a subsequently acquired first environmental image is greater than or equal to a preset size, the robot controller determines that the underwater cleaning robot has moved to the location of the target to be cleaned.

[0171] The preset size is set by technicians according to actual conditions, and this application embodiment does not limit it. Moving forward means moving directly forward. The method of identifying the target recognition box in the subsequently acquired first environmental image belongs to the same inventive concept as the method described in step 302 above, and the implementation process will not be repeated.

[0172] In this implementation, the underwater cleaning robot is controlled to move forward, and during the movement, the robot determines whether it has moved to the location of the target to be cleaned based on the change in the size of the target recognition box, which is highly efficient.

[0173] In one possible implementation, the robot controller controls the underwater cleaning robot to move forward. If the target recognition box disappears in a subsequently acquired first environmental image, the robot controller determines that the underwater cleaning robot has moved to the location of the target to be cleaned.

[0174] The disappearance of the target recognition box indicates that the underwater cleaning robot has covered the target to be cleaned.

[0175] In this implementation, the underwater cleaning robot is controlled to move forward, and during the movement, the robot determines whether it has moved to the location of the target based on whether the target recognition box disappears, which is highly efficient.

[0176] Optionally, after step 303, the robot controller can also execute steps 304-306 below. Alternatively, steps 304-306 below can also be executed independently of steps 301-303 above, and this application embodiment does not limit this.

[0177] 304. When the battery level of the underwater cleaning robot is less than or equal to a preset battery threshold, the robot controller acquires a second environmental image of the area surrounding the underwater cleaning robot.

[0178] The preset power threshold is set by technicians according to actual conditions, and this application embodiment does not limit this setting. The second environmental image refers to the image captured when the underwater cleaning robot's power is less than or equal to the preset power threshold. The second environmental image and the first environmental image are simply environmental images captured under different conditions. The acquisition methods of the second environmental image and the first environmental image belong to the same inventive concept, and the implementation process is described in the relevant description of step 301 above, and will not be repeated here.

[0179] 305. The robot controller performs charging pile identification on the second environmental image to obtain a charging pile identification box on the second environmental image. The charging pile identification box is used to indicate the position of the charging pile on the second environmental image.

[0180] Among them, the charging pile identification is used to identify the location of the charging pile in the second environmental image.

[0181] In one possible implementation, the robot controller determines a target color region in the second environmental image, which is a concentrated area of ​​pixels of a preset color, the preset color being the color of the indicator light on the charging station. The robot controller then defines the bounding box of this target color region as the charging station identification frame.

[0182] The target color area is defined as the region in the second environmental image where the number of pixels of a preset color is greater than or equal to a first preset number. This first preset number is set by technicians based on actual conditions, and this embodiment does not limit it. The color of the charging pile's indicator light is obtained experimentally, and the preset color is one that produces good underwater imaging.

[0183] In this implementation, the charging station is identified by recognizing its indicator light, which reduces the amount of computation and helps the underwater cleaning robot quickly find the charging station when the battery is low, resulting in higher efficiency.

[0184] 306. Based on the second environmental image and the charging pile identification frame, the robot controller controls the underwater cleaning robot to move to the location of the charging pile so as to charge the underwater cleaning robot through the charging pile.

[0185] In one possible implementation, the robot controller controls the underwater cleaning robot to turn based on the relative position between the center point of the charging pile identification frame and the central axis of the second environmental image, so that the charging pile identification frame in a subsequently acquired second environmental image is positioned on the central axis of the second environmental image, the direction indicated by the central axis of the second environmental image being the forward direction of the underwater cleaning robot. The robot controller controls the underwater cleaning robot to move forward. If the size of the charging pile identification frame in a subsequently acquired second environmental image is greater than or equal to a preset size, the robot controller determines that the underwater cleaning robot has moved to the location of the charging pile. Subsequently, once the underwater cleaning robot has moved to the location of the charging pile, the robot controller controls the underwater cleaning robot to dock with the charging pile to charge the underwater cleaning robot.

[0186] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0187] The technical solution provided in this application acquires a first environmental image of the area surrounding the underwater cleaning robot while it is performing a cleaning task. Target recognition is then performed on this first environmental image to obtain a target recognition box, which indicates the location of the target to be cleaned within the first environmental image. Based on the first environmental image and the target recognition box, the underwater cleaning robot is controlled to clean the target, achieving intelligent identification and cleaning of the target by the underwater cleaning robot and improving the cleaning effect of the underwater cleaning robot on pools.

[0188] The training method for the target recognition model used in step 302 above is explained below. See [link to documentation]. Figure 6 Taking electronic devices as the executing entity as an example, the training method of this target recognition model includes the following steps.

[0189] 601. The electronic device inputs the multiple sample environment images into the initial model, and uses the initial model to determine the predicted recognition box on each sample environment image.

[0190] The electronic device used to train the target recognition model can be a server, computer, or other type of electronic device. The initial model is an initialized target recognition model with the same model structure as the target recognition model. This initial model can be a pre-trained model or a completely new model; this embodiment does not limit the specific model. In some embodiments, the multiple sample environment images are all environmental images collected underwater by the underwater cleaning robot. These sample environment images are relatively close to the images collected by the underwater cleaning robot during actual operation, thereby improving the model training effect.

[0191] In one possible implementation, for any one of the plurality of sample environment images, the robot controller, using the initial model, controls at least one candidate bounding box to slide across the sample environment image. The robot controller, using the initial model, extracts features from multiple image regions on the sample environment image covered by the at least one candidate bounding box, obtaining the region image features of each image region on the sample environment image. Based on the region image features of each image region on the sample environment image, the robot controller, using the initial model, obtains the predicted bounding box on the sample environment image.

[0192] The above-described implementation method and step 302 belong to the same inventive concept. The implementation process is described in the relevant description of step 302 above, and will not be repeated here.

[0193] 602. The electronic device determines the initial predicted classification value of the predicted recognition box on each sample environment image based on the predicted recognition box on each sample environment image and the labeled recognition box on each sample environment image.

[0194] The labeled recognition box is used to indicate the actual location of the target to be cleaned on the sample environment image.

[0195] In one possible implementation, for any one of the plurality of sample environment images, the electronic device divides the area of ​​the intersection of the predicted recognition box and the labeled recognition box of the sample environment image by the area of ​​the union of the predicted recognition box and the labeled recognition box of the sample environment image to obtain the initial predicted classification value of the predicted recognition box on the sample environment image.

[0196] The intersection of the predicted and labeled bounding boxes in the sample environment image refers to the region where the predicted and labeled bounding boxes intersect in the sample environment image, and the union of the predicted and labeled bounding boxes refers to the region occupied by the predicted and labeled bounding boxes in the sample environment image. For example, see... Figure 7The sample environment image 700 includes a predicted bounding box 701 and a labeled bounding box 702. The intersection of the predicted bounding box 701 and the labeled bounding box 702 is A, and the union of the predicted bounding box 701 and the labeled bounding box 702 is A+B+C. The larger the area of ​​the intersection between the predicted bounding box and the labeled bounding box of the sample environment image, the higher the degree of overlap between the predicted bounding box and the labeled bounding box, and the higher the accuracy of the predicted bounding box. The accuracy is highest when they overlap completely. Conversely, the smaller the area of ​​the intersection between the predicted bounding box and the labeled bounding box of the sample environment image, the lower the degree of overlap between the predicted bounding box and the labeled bounding box, and the lower the accuracy of the predicted bounding box. The accuracy is lowest when they do not overlap at all.

[0197] In this implementation, the initial predicted classification value of the predicted bounding box can be determined by using the intersection and union of the predicted bounding box and the labeled bounding box, which is highly efficient.

[0198] 603. The electronic device determines the classification value correction coefficient of the predicted recognition box on each sample environment image based on the target classification size and the size of the predicted recognition box on each sample environment image.

[0199] In one possible implementation, for any predicted bounding box on any sample environment image, the electronic device divides the target classification size by the size of the predicted bounding box and then takes the square root to obtain the classification value correction coefficient of the predicted bounding box.

[0200] For example, electronic devices determine the classification value correction factor using the following formula (1).

[0201]

[0202] Where k is the classification value correction coefficient, m t For the target classification size, s i Let i be the size of the predicted bounding box numbered i.

[0203] To provide a clearer explanation of the above implementation methods, the method for determining the target classification size will be described below.

[0204] In one possible implementation, the electronic device divides the multiple sample environment images into multiple sample image groups, each sample image group including at least one sample environment image. The electronic device determines the average size of the bounding boxes in each sample image group based on the dimensions of the bounding boxes in the sample environment images within each sample image group. The electronic device iterates through a reference classification size multiple times based on the average size of the bounding boxes in each sample image group to obtain the target classification size. Each iteration uses one sample image group, and different sample image groups are used in different iterations.

[0205] A group of sample images is also called a batch.

[0206] In this implementation, multiple sample environment images are divided into multiple sample image groups, and the target classification size is determined by the image group, which is highly efficient.

[0207] To provide a clearer explanation of the above embodiments, the following description will be divided into several parts.

[0208] Part 1: The electronic device divides the multiple sample environment images into multiple sample image groups.

[0209] In one possible implementation, the electronic device groups the multiple sample environment images at intervals of a second preset number to obtain multiple sample image groups.

[0210] The second preset quantity is set by technicians according to the actual situation, and this application embodiment does not limit it.

[0211] Part Two: The electronic device determines the average size of the bounding boxes in each sample image group based on the size of the bounding boxes in the sample environment images of each sample image group.

[0212] In one possible implementation, for any one of a plurality of sample image groups, the electronic device determines the average size of the labeled recognition boxes in the plurality of sample environment images in the sample image group, thereby obtaining the average size of the labeled boxes in the sample image group.

[0213] Part Three: The electronic device iterates through multiple rounds of tests on the reference classification size based on the average size of the bounding boxes of each sample image group to obtain the target classification size.

[0214] In one possible implementation, for any iteration in a multi-round iteration, the electronic device performs a weighted fusion of the reference classification size determined in the previous iteration and the average size of the bounding boxes of the sample image group used in the current iteration to obtain the reference classification size for the current iteration. If the current iteration is the last iteration, the electronic device determines the reference classification size for the current iteration as the target classification size.

[0215] In the first iteration, the reference classification size determined in the previous iteration is a preset classification size, such as 0. One set of sample images is used per iteration, and the used set of sample images is not used again in subsequent iterations.

[0216] For example, an electronic device determines the target classification size using the following formula (2).

[0217] m t =0.9×mj +0.1×m t-1 (2)

[0218] Where, m t The reference classification size for this iteration is t, where t is the iteration number and m is m. t-1 m is the reference classification size for the previous iteration. j The average size of the bounding box for the sample image group numbered j.

[0219] 604. The electronic device trains the initial model based on the target prediction classification value, the first classification value threshold, and the second classification value threshold of the predicted recognition box on each sample environmental image to obtain the target recognition model. The target prediction classification value is determined based on the initial prediction classification value and the classification value correction coefficient of the corresponding prediction recognition box. The first classification value threshold is used to identify the prediction recognition box containing the target to be cleaned, and the second classification value threshold is used to identify the prediction recognition box that does not contain the target to be cleaned.

[0220] In one possible implementation, for any predicted bounding box on any sample environment image, if the target predicted classification value of the predicted bounding box is greater than or equal to the first classification value threshold, the electronic device determines the predicted bounding box as a positive sample. The electronic device trains the initial model based on the positive sample. If the target predicted classification value of the predicted bounding box is less than or equal to the second classification value threshold, the electronic device determines the predicted bounding box as a negative sample. The electronic device trains the initial model based on the negative sample.

[0221] In this model, positive samples refer to predicted bounding boxes containing the target to be cleared, and negative samples refer to predicted bounding boxes that do not contain the target to be cleared. Training the initial model using both positive and negative samples aims to make the model's predictions closer to positive samples and further away from negative samples. The first and second classification thresholds are set by those skilled in the art based on actual conditions, and this application embodiment does not limit this. In some embodiments, predicted bounding boxes with target prediction classification values ​​greater than the second classification threshold and less than the first classification threshold are discarded during model training.

[0222] To provide a clearer explanation of the above implementation methods, the method for determining the target predicted classification value is described below.

[0223] In one possible implementation, for any predicted bounding box, the electronic device multiplies the initial predicted classification value of the predicted bounding box by a classification value correction coefficient to obtain the target predicted classification value of the predicted bounding box.

[0224] For example, an electronic device determines the target predicted classification value of the predicted recognition box using the following formula (3).

[0225] t=k×IoU(b i s i (3)

[0226] Where t is the predicted recognition box b i The target predicted classification value, k is the classification value correction coefficient in the above formula (1), s i This indicates the relationship between the predicted recognition box b and the actual box. i Matched bounding box, IoU(b i s i ) refers to the predicted recognition box b i The initial predicted classification value.

[0227] By setting target prediction classification values, the predicted classification values ​​for most small targets are greater than 1, while the predicted classification values ​​for large targets are less than 1, but not excessively low. This balances the probability of generating positive samples for targets of different scales. Here, "large targets" refers to larger targets to be cleaned, and "small targets" refers to smaller targets to be cleaned. Introducing these target prediction classification values ​​during the training of the target recognition model gives it a stronger ability to recognize small targets. Therefore, when the underwater cleaning robot uses this model for target recognition, it can more accurately identify smaller targets to be cleaned, thereby improving the cleaning effect of the pool.

[0228] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0229] Figure 8 This is a schematic diagram of the control device for an underwater cleaning robot provided in an embodiment of this application. See also... Figure 8 The device includes: an environmental image acquisition module 801, a target recognition module 802, and a control module 803.

[0230] The environmental image acquisition module 801 is used to acquire a first environmental image around the underwater cleaning robot when the underwater cleaning robot is performing a cleaning task.

[0231] The target recognition module 802 is used to perform target recognition on the first environmental image to obtain a target recognition box on the first environmental image. The target recognition box is used to indicate the position of the target to be cleaned on the first environmental image.

[0232] The control module 803 is used to control the underwater cleaning robot to clean the target to be cleaned based on the first environmental image and the target recognition box.

[0233] In one possible implementation, the target recognition module 802 is used to input the first environmental image into a target recognition model. This model is trained based on multiple sample environmental images, bounding boxes labeled on each sample environmental image, and a target classification size. The bounding boxes indicate the actual location of the target to be cleaned on the sample environmental image, and the target classification size is determined based on the size of the bounding boxes on the multiple sample environmental images. Using the target recognition model, at least one candidate bounding box is slid across and classified on the first environmental image to obtain the target bounding box on the first environmental image.

[0234] In one possible implementation, the target recognition module 802 is used to control the at least one candidate bounding box to slide on the first environmental image. Feature extraction is performed on multiple image regions on the first environmental image covered by the at least one candidate bounding box to obtain region image features for each image region. Based on the region image features of each image region, a target bounding box on the first environmental image is obtained.

[0235] In one possible implementation, the target recognition module 802 is used to perform fully connected and normalized regional image features on any one of the plurality of image regions to obtain a region classification value for that image region. If the region classification value is greater than or equal to a first classification value threshold, the bounding box of that image region is determined as the target recognition box.

[0236] In one possible implementation, the training apparatus for the target recognition model includes:

[0237] The prediction module is used to input the multiple sample environment images into the initial model, and to determine the predicted recognition box on each of the sample environment images through the initial model.

[0238] The prediction classification value determination module is used to determine the initial prediction classification value of the prediction recognition box on each sample environment image based on the predicted recognition box on each sample environment image and the labeled recognition box on each sample environment image.

[0239] The classification value correction coefficient determination module is used to determine the classification value correction coefficient of the predicted recognition box on each sample environment image based on the target classification size and the size of the predicted recognition box on each sample environment image.

[0240] The training module is used to train the initial model based on the target prediction classification value, the first classification value threshold, and the second classification value threshold of the predicted recognition box on each sample environment image to obtain the target recognition model. The target prediction classification value is determined based on the initial prediction classification value and the classification value correction coefficient of the corresponding prediction recognition box. The first classification value threshold is used to identify the prediction recognition box containing the target to be cleaned, and the second classification value threshold is used to identify the prediction recognition box that does not contain the target to be cleaned.

[0241] In one possible implementation, the classification value correction coefficient determination module is used to obtain the classification value correction coefficient of any predicted recognition box on any sample environment image by dividing the target classification size by the size of the predicted recognition box and then taking the square root.

[0242] In one possible implementation, the training module is configured to, for any predicted bounding box on any sample environment image, determine the predicted bounding box as a positive sample if the target predicted classification value of the predicted bounding box is greater than or equal to a first classification value threshold. The initial model is then trained based on these positive samples. If the target predicted classification value of the predicted bounding box is less than or equal to a second classification value threshold, the predicted bounding box is determined as a negative sample. The initial model is then trained based on these negative samples.

[0243] In one possible implementation, the device for determining the target classification size includes:

[0244] The grouping module is used to divide the multiple sample environment images into multiple sample image groups, and each sample image group includes at least one sample environment image.

[0245] The module for determining the average size of the bounding boxes is used to determine the average size of the bounding boxes in each sample image group based on the size of the bounding boxes in the sample environment images of each sample image group.

[0246] The target classification size determination module is used to perform multiple iterations on the reference classification size based on the average size of the bounding boxes of each sample image group to obtain the target classification size. Each iteration uses one sample image group, and different sample image groups are used in different iterations.

[0247] In one possible implementation, the target classification size determination module is used to, for any iteration in a multi-round iteration, perform a weighted fusion of the reference classification size determined in the previous iteration and the average size of the bounding boxes of the sample image group used in the current iteration to obtain the reference classification size for the current iteration. If the current iteration is the last iteration, the reference classification size for the current iteration is determined as the target classification size.

[0248] In one possible implementation, the control module 803 is used to control the underwater cleaning robot to turn based on the relative position between the center point of the target recognition box and the central axis of the first environmental image, so that the center point of the target recognition box in a subsequently acquired first environmental image is located on the central axis of the first environmental image, the direction indicated by the central axis of the first environmental image being the forward direction of the underwater cleaning robot. The underwater cleaning robot is then controlled to move forward to the location of the target to be cleaned, in order to clean the target.

[0249] In one possible implementation, the control module 803 is used to determine the turning direction and turning angular velocity of the underwater cleaning robot based on the relative position between the center point of the target recognition box and the central axis of the first environmental image. The control module then controls the underwater cleaning robot to turn according to the turning direction and the turning angular velocity.

[0250] In one possible implementation, the control module 803 is configured to determine that the underwater cleaning robot's turning direction is to the right when the center point of the target recognition frame is located to the right of the central axis of the first environmental image; and to determine that the underwater cleaning robot's turning direction is to the left when the center point of the target recognition frame is located to the left of the central axis of the first environmental image. A turning angular velocity is determined based on the distance between the center point of the target recognition frame and the central axis of the first environmental image, and this turning angular velocity is positively correlated with the distance.

[0251] In one possible implementation, the device further includes:

[0252] The charging module is used to acquire a second environmental image of the underwater cleaning robot when its battery level is less than or equal to a preset battery threshold. It then identifies charging stations within the second environmental image, obtaining a charging station identification box that indicates the location of the charging station. Based on the second environmental image and the charging station identification box, the module controls the underwater cleaning robot to move to the location of the charging station to recharge it.

[0253] In one possible implementation, the charging module is used to determine a target color region in the second environmental image, the target color region being a concentrated area of ​​pixels of a preset color, the preset color being the color of the indicator light of the charging pile. The bounding box of the target color region is defined as the charging pile identification box.

[0254] In one possible implementation, the charging module controls the underwater cleaning robot to steer based on the relative position between the center point of the charging pile identification frame and the central axis of the second environmental image, so that the charging pile identification frame in a subsequently acquired second environmental image is positioned on the central axis of the second environmental image, the direction indicated by the central axis of the second environmental image being the forward direction of the underwater cleaning robot. The underwater cleaning robot is controlled to move forward. If the size of the charging pile identification frame in a subsequently acquired second environmental image is greater than or equal to a preset size, the underwater cleaning robot is determined to have moved to the location of the charging pile.

[0255] It should be noted that the control device for the underwater cleaning robot provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the robot controller can be divided into different functional modules to complete all or part of the functions described above. In addition, the control device for the underwater cleaning robot provided in the above embodiments and the control method embodiments for the underwater cleaning robot belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0256] The technical solution provided in this application acquires a first environmental image of the area surrounding the underwater cleaning robot while it is performing a cleaning task. Target recognition is then performed on this first environmental image to obtain a target recognition box, which indicates the location of the target to be cleaned within the first environmental image. Based on the first environmental image and the target recognition box, the underwater cleaning robot is controlled to clean the target, achieving intelligent identification and cleaning of the target by the underwater cleaning robot and improving the cleaning effect of the underwater cleaning robot on pools.

[0257] This application also provides an underwater cleaning robot. Figure 9 This is a schematic diagram of the structure of a robot controller provided in an embodiment of this application. Typically, an underwater cleaning robot includes a robot controller 900, which includes one or more processors 901 and one or more memories 902.

[0258] Processor 901 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0259] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one computer program, which is executed by the processor 901 to implement the control method for the underwater cleaning robot provided in the method embodiments of this application.

[0260] In some embodiments, the underwater cleaning robot 900 may also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board.

[0261] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the underwater cleaning robot 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0262] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the control method for the underwater cleaning robot described in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, or an optical data storage device, etc.

[0263] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. The processor of the robot controller reads the program code from the computer-readable storage medium and executes the program code, causing the robot controller to perform the control method of the underwater cleaning robot described above.

[0264] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a robot controller, or on multiple robot controllers located in one location, or on multiple robot controllers distributed in multiple locations and interconnected through a communication network. Multiple robot controllers distributed in multiple locations and interconnected through a communication network may form a blockchain system.

[0265] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0266] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A control method for an underwater cleaning robot, characterized in that, The method includes: While the underwater cleaning robot is performing a cleaning task, a first environmental image of the area surrounding the underwater cleaning robot is acquired. Target recognition is performed on the first environmental image to obtain a target recognition box on the first environmental image. The target recognition box is used to indicate the position of the target to be cleaned on the first environmental image. The target recognition box is obtained by a target recognition model. Based on the first environmental image and the target recognition box, the underwater cleaning robot is controlled to clean the target to be cleaned; The training steps of the target recognition model include: inputting multiple sample environment images into an initial model, and determining the predicted recognition box on each of the sample environment images through the initial model; Based on the predicted recognition boxes and the labeled recognition boxes on each of the sample environment images, the initial predicted classification value of the predicted recognition boxes on each of the sample environment images is determined. The labeled recognition boxes are used to indicate the actual location of the target to be cleaned on the sample environment image. Based on the target classification size and the size of the predicted recognition box on each of the sample environment images, the classification value correction coefficient of the predicted recognition box on each of the sample environment images is determined, wherein the target classification size is determined based on the size of the labeled recognition box on the multiple sample environment images; Based on the target prediction classification value, first classification value threshold, and second classification value threshold of the predicted recognition box on each of the sample environment images, the initial model is trained to obtain the target recognition model. The target prediction classification value is determined based on the initial prediction classification value and classification value correction coefficient of the corresponding prediction recognition box. The first classification value threshold is used to identify the prediction recognition box containing the target to be cleared, and the second classification value threshold is used to identify the prediction recognition box that does not contain the target to be cleared.

2. The method according to claim 1, characterized in that, The step of performing target recognition on the first environmental image to obtain a target recognition box on the first environmental image includes: The first environmental image is input into the target recognition model, which is trained based on multiple sample environmental images, the labeled recognition boxes on each sample environmental image, and the target classification size. The labeled recognition boxes are used to indicate the actual location of the target to be cleaned on the sample environmental image, and the target classification size is determined based on the size of the labeled recognition boxes on the multiple sample environmental images. Using the target recognition model, at least one candidate bounding box is slid and classified on the first environmental image to obtain the target bounding box on the first environmental image.

3. The method according to claim 2, characterized in that, The step of sliding and classifying at least one candidate bounding box on the first environmental image to obtain a target bounding box on the first environmental image includes: Control the at least one candidate bounding box to slide on the first environmental image; Feature extraction is performed on multiple image regions in the first environmental image that are covered by the at least one candidate recognition box to obtain the regional image features of each image region; Based on the regional image features of each of the image regions, a target recognition box is obtained on the first environmental image.

4. The method according to claim 3, characterized in that, The step of obtaining the target recognition box on the first environment image based on the regional image features of each of the image regions includes: For any image region among the plurality of image regions, the regional image features of the image region are fully connected and normalized to obtain the regional classification value of the image region. If the region classification value is greater than or equal to the first classification value threshold, the bounding box of the image region is determined as the target recognition box.

5. The method according to claim 1, characterized in that, The step of determining the classification value correction coefficient of the predicted recognition box on each of the sample environment images based on the target classification size and the size of the predicted recognition box on each of the sample environment images includes: For any predicted bounding box on any sample environment image, the square root of the division between the target classification size and the predicted bounding box size is used to obtain the classification value correction coefficient of the predicted bounding box.

6. The method according to claim 1, characterized in that, The training of the initial model based on the target prediction classification value, the first classification value threshold, and the second classification value threshold of the predicted recognition box on each of the sample environment images includes: For any predicted bounding box on any sample environment image, if the target predicted classification value of the predicted bounding box is greater than or equal to the first classification value threshold, the predicted bounding box is determined as a positive sample; the initial model is trained based on the positive sample; If the target predicted classification value of the predicted recognition box is less than or equal to the second classification value threshold, the predicted recognition box is determined as a negative sample; the initial model is trained based on the negative sample.

7. The method according to claim 1, characterized in that, The method for determining the target classification size includes: The plurality of sample environment images are divided into a plurality of sample image groups, and each sample image group includes at least one sample environment image; Based on the size of the bounding boxes in the sample environment images of each sample image group, the average size of the bounding boxes in each sample image group is determined. The target classification size is obtained by iterating the reference classification size multiple times based on the average size of the bounding boxes of each of the sample image groups. Each iteration uses one of the sample image groups, and different sample image groups are used in different iterations.

8. The method according to claim 7, characterized in that, The process of iterating the reference classification size based on the average size of the bounding boxes in each of the sample image groups to obtain the target classification size includes: For any iteration in a multi-round iteration, the reference classification size determined in the previous iteration is weighted and fused with the average size of the bounding boxes of the sample image group used in the current iteration to obtain the reference classification size for the current iteration; If this is the last iteration, the reference classification size for this iteration will be determined as the target classification size.

9. The method according to claim 1, characterized in that, The step of controlling the underwater cleaning robot to clean the target based on the first environmental image and the target recognition box includes: Based on the relative position between the center point of the target recognition box and the central axis of the first environmental image, the underwater cleaning robot is controlled to turn so that the center point of the target recognition box in the subsequently acquired first environmental image is on the central axis of the first environmental image, and the direction indicated by the central axis of the first environmental image is the forward direction of the underwater cleaning robot. The underwater cleaning robot is controlled to move forward to the location of the target to be cleaned, so as to clean the target.

10. The method according to claim 9, characterized in that, The step of controlling the underwater cleaning robot to turn based on the relative position between the center point of the target recognition box and the central axis of the first environmental image includes: Based on the relative position between the center point of the target recognition box and the central axis of the first environmental image, the turning direction and turning angular velocity of the underwater cleaning robot are determined. The underwater cleaning robot is controlled to turn in accordance with the turning direction and the turning angular velocity.

11. The method according to claim 10, characterized in that, Determining the turning direction and turning angular velocity of the underwater cleaning robot based on the relative position between the center point of the target recognition box and the central axis of the first environmental image includes: When the center point of the target recognition box is located to the right of the central axis of the first environmental image, the turning direction of the underwater cleaning robot is determined to be turning to the right; when the center point of the target recognition box is located to the left of the central axis of the first environmental image, the turning direction of the underwater cleaning robot is determined to be turning to the left. The turning angular velocity is determined based on the distance between the center point of the target recognition box and the central axis of the first environmental image, and the turning angular velocity is positively correlated with the distance.

12. The method according to claim 1, characterized in that, The method further includes: When the battery level of the underwater cleaning robot is less than or equal to a preset battery threshold, a second environmental image of the underwater cleaning robot is acquired. The second environmental image is used to identify charging piles to obtain charging pile identification boxes on the second environmental image. The charging pile identification boxes are used to indicate the position of the charging piles on the second environmental image. Based on the second environmental image and the charging pile identification frame, the underwater cleaning robot is controlled to move to the location of the charging pile so that the underwater cleaning robot can be charged through the charging pile.

13. The method according to claim 12, characterized in that, The step of identifying charging piles in the second environmental image to obtain a charging pile identification box on the second environmental image includes: Determine the target color region in the second environmental image. The target color region is a concentrated area of ​​pixels of a preset color, and the preset color is the color of the indicator light of the charging pile. The bounding box of the target color area is defined as the charging pile identification box.

14. The method according to claim 12, characterized in that, The step of controlling the underwater cleaning robot to move to the location of the charging pile based on the second environmental image and the charging pile identification frame includes: Based on the relative position between the center point of the charging pile identification frame and the central axis of the second environmental image, the underwater cleaning robot is controlled to turn so that the charging pile identification frame in the subsequently acquired second environmental image is on the central axis of the second environmental image, and the direction indicated by the central axis of the second environmental image is the forward direction of the underwater cleaning robot. Control the underwater cleaning robot to move forward; If the size of the charging pile identification frame in the subsequently acquired second environmental image is greater than or equal to a preset size, the underwater cleaning robot is determined to move to the location of the charging pile.

15. A control device for an underwater cleaning robot, characterized in that, The device includes: An environmental image acquisition module is used to acquire a first environmental image around the underwater cleaning robot when the underwater cleaning robot is performing a cleaning task. The target recognition module is used to perform target recognition on the first environmental image to obtain a target recognition box on the first environmental image. The target recognition box is used to indicate the position of the target to be cleaned on the first environmental image. The target recognition box is obtained by a target recognition model. The control module is used to control the underwater cleaning robot to clean the target to be cleaned based on the first environmental image and the target recognition box; The training steps of the target recognition model include: inputting multiple sample environment images into an initial model, and determining the predicted recognition box on each of the sample environment images through the initial model; Based on the predicted recognition boxes and the labeled recognition boxes on each of the sample environment images, the initial predicted classification value of the predicted recognition boxes on each of the sample environment images is determined. The labeled recognition boxes are used to indicate the actual location of the target to be cleaned on the sample environment image. Based on the target classification size and the size of the predicted recognition box on each of the sample environment images, the classification value correction coefficient of the predicted recognition box on each of the sample environment images is determined, wherein the target classification size is determined based on the size of the labeled recognition box on the multiple sample environment images; Based on the target prediction classification value, first classification value threshold, and second classification value threshold of the predicted recognition box on each of the sample environment images, the initial model is trained to obtain the target recognition model. The target prediction classification value is determined based on the initial prediction classification value and classification value correction coefficient of the corresponding prediction recognition box. The first classification value threshold is used to identify the prediction recognition box containing the target to be cleared, and the second classification value threshold is used to identify the prediction recognition box that does not contain the target to be cleared.

16. An underwater cleaning robot, characterized in that, The underwater cleaning robot includes a robot controller, which includes one or more processors and one or more memories. The one or more memories store at least one computer program, which is loaded and executed by the one or more processors to implement the control method of the underwater cleaning robot as described in any one of claims 1 to 14.

Images