Underwater cleaning method based on multi-target recognition, terminal device and storage medium

Abstract

Embodiments of the present application provide an underwater cleaning method based on multi-target identification, a terminal device and a storage medium. The method obtains a plurality of three-dimensional partitions by three-dimensionally layering a second swimming pool three-dimensional map. When identifying and tracking the targets of the plurality of three-dimensional partitions for cleaning, an intelligent identification master device acquires target images and predicts the motion trajectory of the target, so that a tracking device tracks and cleans the target according to the motion trajectory. The intelligent identification master device predicts the motion trajectory of the target by using a multi-target identification and tracking model. The multi-target identification and tracking model includes a target identification sub-model based on a YOLOv7 algorithm and a target tracking sub-model based on a DeepSORT algorithm. The tracking device and the intelligent identification master device are connected by wireless communication. Through the implementation of the present application, the underwater cleaning path can be reasonably planned, and the swimming pool can be thoroughly cleaned, thereby improving the underwater cleaning efficiency and cleaning completion degree.

Description

Technical Field

[0001] This invention relates to the field of underwater cleaning technology, and in particular to an underwater cleaning method based on multi-target recognition. Background Technology

[0002] Current research on underwater robots for pool cleaning mainly focuses on multimodal perception and environmental modeling, underwater target detection algorithms, intelligent path planning and control, and cleaning actuators. Regarding intelligent path planning and control, underwater robots typically rely on random paths or inertial navigation devices for path planning. Inertial navigation has poor accuracy, which deteriorates over time, leading to high repetition rates and low cleaning efficiency. Furthermore, the complex environment of a pool can interfere with the robot's path, causing it to deviate from its original route, further exacerbating path confusion and repetition. In terms of underwater target detection, underwater targets are typically small, such as leaves and hair. Current deep learning-based target detectors often have poor performance in detecting small objects, easily resulting in missed detections.

[0003] Therefore, there are problems such as unreasonable underwater path planning and low target recognition accuracy, resulting in low underwater cleaning efficiency, low cleaning task completion rate, and incomplete cleaning. Summary of the Invention

[0004] This invention provides an underwater cleaning method based on multi-target identification and tracking, which realizes the rational planning of underwater cleaning paths and thorough cleaning of swimming pools, thereby improving underwater cleaning efficiency and cleaning completion.

[0005] An embodiment of the present invention provides an underwater cleaning method based on multi-target identification and tracking, comprising the following steps: Load the first 3D map of the pool to be cleaned, and update the first 3D map of the pool according to the real-time measured pool depth to obtain the second 3D map of the pool. The second swimming pool 3D map is layered into multiple 3D partitions. A first path is planned for the multiple 3D partitions. The targets in the multiple 3D partitions are identified, tracked and cleaned in sequence according to the first path. When identifying and tracking targets in the multiple three-dimensional partitions, the intelligent identification main device acquires target images and predicts the target's motion trajectory, so that the tracking device tracks and cleans the target based on the motion trajectory. The intelligent identification main device uses a multi-target identification and tracking model to predict the target's motion trajectory. The multi-target identification and tracking model includes a target identification sub-model based on the YOLOv7 algorithm and a target tracking sub-model based on the DeepSORT algorithm. The tracking device and the intelligent identification main device are connected wirelessly. Furthermore, when tracking and cleaning the target in any of the three-dimensional partitions, the following is included: The first camera sequentially captures a first image of the target in each of the three-dimensional partitions, and the first image is an image captured from a single perspective; the first camera is installed on the intelligent recognition main device. The first image is subjected to fuzziness analysis and target motion recognition. Multiple targets identified in the first image are divided into a first target and a second target, and the first target and the second target have different motion states. A first tracking path is planned for the first target, so that the first tracking device tracks and cleans the first target according to the first tracking path; A multi-target recognition and tracking model is used to predict the motion trajectory of the second target, so that the second tracking device can track and clean the second target according to the motion trajectory; the first tracking device and the second tracking device are slave devices of the intelligent recognition master device, and the slave devices and the intelligent recognition master device are connected through wireless communication.

[0006] Furthermore, the step of performing three-dimensional layering on the second swimming pool 3D map to obtain multiple 3D partitions, planning a first path for the multiple 3D partitions, and sequentially identifying, tracking, and cleaning targets in the multiple 3D partitions according to the first path specifically includes: The second swimming pool 3D map is divided into multiple 3D partitions at preset distances along the X, Y and Z axes. Obtain the center point coordinates of each of the three-dimensional partitions. Using the center point coordinates of the three-dimensional partition where the starting position is located as the starting coordinates, plan the shortest path that passes through all the center point coordinates as the first path. The first path is a set of center point coordinates sorted according to the shortest path. The targets in the multiple three-dimensional partitions are identified, tracked, and cleaned sequentially according to the first path.

[0007] Furthermore, the tracking and cleaning of the target in the three-dimensional partition includes: Step S1: Traverse the set of center point coordinates to obtain the current 3D partition; Step S2: Divide the current 3D partition into N view partitions; Step S3: The intelligent recognition main device captures a first image of the target in the current field of view partition, wherein the first image is a single-view image; Step S4: Perform blur analysis on the first image to obtain blur data of the first image, and perform target motion recognition on the blur data to obtain target motion recognition data of the first image; Step S5: Based on the motion recognition data, the target identified in the first image is divided into a first target and a second target, wherein the first target and the second target have different motion states; Step S6: Simultaneously activate the first tracking device and the second tracking device, so that the first tracking device cleans all the first targets in the current field of view partition, and so that the second tracking device cleans all the second targets in the current field of view partition; The step of enabling the first tracking device to clean all the first targets in the current field of view includes: The first coordinates of all the first targets in the current field of view are obtained to form a first coordinate set. The shortest cleaning path is planned based on the first coordinate set as the first tracking path, so that the first tracking device can track and clean all the first targets according to the first tracking path. The step of enabling the second tracking device to clean all the second targets in the current field of view includes: The multi-target recognition and tracking model is used to predict the motion trajectory of the second target, so that the second tracking device can track and clean the second target according to the motion trajectory. Step S7: Determine whether the N field of view partitions have been cleaned. If yes, proceed to step S8; otherwise, control the intelligent recognition main device to rotate a preset angle on a preset plane and enter the next field of view partition, and set the next field of view partition as the current field of view partition, and jump to step S3. Step S8: Determine whether the set of center point coordinates has been traversed. If yes, complete the cleaning task. If no, obtain the next center point coordinate from the set of center point coordinates, take the three-dimensional partition of the next center point coordinate as the current three-dimensional partition, and jump to step S2.

[0008] Furthermore, the step of using the multi-target recognition and tracking model to predict the motion trajectory of the second target, so that the second tracking device tracks and cleans the second target according to the motion trajectory, includes: The second image of the second target is obtained by capturing the second target with the second camera of the intelligent recognition main device. The second image is then input into the multi-target recognition and tracking model to obtain the predicted trajectory of the second target. The tracking trajectory of the second tracking device is then set according to the predicted trajectory.

[0009] Furthermore, the target recognition sub-model based on the YOLOv7 algorithm is trained according to the following steps: Training set partitioning steps: The training set is labeled according to a preset ratio, and the training set is divided into a first training set and a second training set. The first training set is the labeled training set, and the second training set is the unlabeled training set. Model training steps: The first target recognition sub-model is trained using the first training set to obtain the second target recognition sub-model; the first target recognition sub-model is the initial model of the target recognition sub-model, and the second target recognition sub-model is the intermediate model of the target recognition sub-model; Determine whether the second target recognition sub-model meets the training termination condition. If yes, end the training. If no, update the first and second training sets and repeat the model training steps until the second target recognition sub-model converges.

[0010] Furthermore, updating the first training set and the second training set includes the following steps: The second training set is input into the second target recognition sub-model to obtain the first recognition result. The first recognition result is filtered according to a preset result filtering strategy to obtain the second recognition result. The second training set is labeled according to the second recognition result to obtain the third training set. The third training set is added to the first training set to update the first training set, and the third training set is subtracted from the second training set to update the second training set.

[0011] Furthermore, the intelligent recognition main device includes a first form and a second form; when the intelligent recognition main device moves to the current center point coordinates, it switches from the first form to the second form, and when the intelligent recognition main device reaches the current center point coordinates, it switches from the second form to the first form; the first form is a stationary form, and the second form is a moving form.

[0012] Based on the above-described method embodiments, the present invention provides another embodiment; Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the underwater cleaning method based on multi-target recognition provided in any of the above-described method embodiments of the present application.

[0013] Based on the embodiments of the method of the present invention, another embodiment is provided: Another embodiment of the present invention provides a storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to perform the underwater cleaning method based on multi-target recognition provided in any of the above-described method embodiments of the present invention.

[0014] The embodiments of the present invention have the following beneficial effects: This invention provides an underwater cleaning method, terminal device, and storage medium based on multi-target recognition. The method involves dividing the swimming pool into three-dimensional partitions based on a three-dimensional map, and then planning the shortest cleaning path based on the center point coordinates of each partition. This allows an underwater robot to clean each three-dimensional partition of the pool sequentially along the shortest path, achieving rational planning of the cleaning path, improving underwater cleaning efficiency, and ensuring thorough cleaning of the pool, thus increasing the cleaning completion rate. Simultaneously, this method sets up a tracking device and an intelligent recognition main device during underwater cleaning. The intelligent recognition main device identifies underwater targets and predicts their movement trajectories using a multi-target recognition and tracking model. The tracking device performs underwater tracking and cleaning based on the target movement trajectories predicted by the intelligent recognition main device. Therefore, by placing the recognition and prediction functions and the tracking functions on different underwater devices, this invention improves the flexibility and efficiency of the underwater cleaning equipment and ensures the effectiveness of underwater cleaning. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating an underwater cleaning method based on multi-target recognition provided in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the process of tracking and cleaning the target in the three-dimensional partition according to an embodiment of the underwater cleaning method provided by the present invention.

[0017] Figure 3 This is a schematic diagram illustrating the process of training the target recognition sub-model based on the YOLOv7 algorithm in an underwater cleaning method according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] See Figure 1 This is a flowchart illustrating an underwater cleaning method based on multi-target recognition, provided by an embodiment of the present invention. The underwater cleaning method is applied to an underwater robot and includes the following steps: Step S01: Load the first three-dimensional map of the pool to be cleaned, and update the first three-dimensional map of the pool according to the real-time measured pool depth to obtain the second three-dimensional map of the pool.

[0020] In a preferred embodiment, step S01 includes: Step S011: Establish a swimming pool map database, which includes a swimming pool ID and a first swimming pool 3D map. The swimming pool ID and the first swimming pool 3D map have a one-to-one association. The first swimming pool 3D map is an empty pool map, which refers to a 3D map that lacks swimming pool water level information.

[0021] Step S012: Obtain the first three-dimensional map of the swimming pool from the swimming pool map database according to the swimming pool ID, and update the first three-dimensional map of the swimming pool according to the real-time measured swimming pool depth to obtain the second three-dimensional map of the swimming pool.

[0022] Step S02: The second swimming pool 3D map is layered into multiple 3D partitions. A first path is planned for each of the multiple 3D partitions. Targets in each of the multiple 3D partitions are identified and tracked for cleaning sequentially according to the first path. This embodiment of the invention divides the swimming pool into 3D partitions based on the 3D map of the pool to be cleaned, and then plans the shortest cleaning path based on the center point coordinates of each 3D partition. This allows the underwater robot to clean each 3D partition of the pool sequentially according to the shortest cleaning path, achieving reasonable planning of the pool cleaning path, improving the efficiency of underwater cleaning, achieving thorough cleaning of the pool, and improving the cleaning completion rate.

[0023] In a preferred embodiment, step S02 includes: The second swimming pool 3D map is divided into multiple 3D partitions at preset distances along the X, Y, and Z axes; preferably, the preset distance is the optimal recognition distance of the intelligent recognition main device. Obtain the center point coordinates of each of the three-dimensional partitions. Using the center point coordinates of the three-dimensional partition where the starting position is located as the starting coordinates, plan the shortest path that passes through all the center point coordinates as the first path. The first path is a set of center point coordinates sorted according to the shortest path. The targets in the multiple three-dimensional partitions are identified, tracked, and cleaned sequentially according to the first path.

[0024] In another preferred embodiment, step S02 includes: The second swimming pool 3D map is divided into a first layer, a second layer, and a third layer according to the optimal recognition distance to obtain multiple 3D partitions. The first layer, the second layer, and the third layer are layers in the Z-axis, Y-axis, and X-axis directions, respectively. The second swimming pool 3D map takes a corner of the bottom of the pool as the origin, the vertical upward direction as the positive Z-axis, and the two sides along the corner on the horizontal plane as the X-axis and Y-axis, respectively. The first layer is formed by dividing a plane at every predetermined distance from the first horizontal plane of the pool towards the positive Z-axis until the first layer is completed. The first horizontal plane is the horizontal plane containing the X-axis and Y-axis. The second layer is formed by dividing a plane at every predetermined distance from the first vertical plane containing the X-axis of the pool towards the positive Y-axis until the second layer is completed. The first vertical plane includes the X-axis and is perpendicular to the first horizontal plane. The third layer is formed by dividing a plane at every predetermined distance from the second vertical plane containing the Y-axis of the pool towards the positive X-axis until the third layer is completed. The second vertical plane includes the Y-axis and is perpendicular to the first horizontal plane.

[0025] Step S03: When identifying and tracking the targets in the multiple three-dimensional partitions, the intelligent identification main device acquires the target image and predicts the target's motion trajectory, so that the tracking device tracks and cleans the target according to the motion trajectory; the intelligent identification main device uses a multi-target identification and tracking model to predict the target's motion trajectory, the multi-target identification and tracking model includes a target identification sub-model based on the YOLOv7 algorithm and a target tracking sub-model based on the DeepSORT algorithm; the tracking device and the intelligent identification main device are connected wirelessly.

[0026] In a preferred embodiment, the intelligent identification main device is equipped with a depth measuring device. If the first 3D map of the pool cannot be obtained from the pool map database based on the pool ID, the pool is determined to be cleaning for the first time. For the pool being cleaned for the first time, the depth measuring device measures the first, second, and third depths of the current pool, and a first 3D map of the current pool is drawn based on the first, second, and third depths. The first depth is the sum of the fourth and fifth depths, where the fourth depth is the depth from the intelligent identification main device to the bottom of the pool, the fifth depth is the depth from the intelligent identification main device to the water surface, the second depth is the width of the current pool, and the third depth is the length of the current pool.

[0027] When the current swimming pool has inconsistent depths, during the identification and tracking cleaning of targets in each of the three-dimensional zones, a first depth of each three-dimensional zone is measured using the depth measuring device to obtain a first depth set. A depth coordinate association relationship is established between the first depth set and the center point coordinate set. Based on the first depth set and the center point coordinate set, the depth of the swimming pool at each location can be determined. Based on the second depth, third depth, first depth set, center point coordinate set, and depth coordinate association relationship, a first three-dimensional map of the current swimming pool is drawn and saved to the swimming pool map database. This embodiment of the invention can, for swimming pools without a map, record a first three-dimensional map of the current swimming pool and save it to the swimming pool map database during the initial cleaning process, thereby expanding the swimming pool map database and preparing for the three-dimensional partitioning.

[0028] In a preferred embodiment, the tracking and cleaning of the target in the three-dimensional partition includes: Step S031: Sequentially capture a first image of the target in each of the three-dimensional partitions using the first camera. The first image is a single-view image. The first camera is mounted on the intelligent recognition main device. Step S032: Perform fuzziness analysis and target motion recognition on the first image, and divide the multiple targets identified in the first image into a first target and a second target, wherein the first target and the second target have different motion states; Step S033: Plan a first tracking path for the first target, so that the first tracking device can track and clean the first target according to the first tracking path; Step S034: For the second target, a multi-target recognition and tracking model is used to predict the motion trajectory of the second target, so that the second tracking device can track and clean the second target according to the motion trajectory; the first tracking device and the second tracking device are slave devices of the intelligent recognition master device, and the slave devices and the intelligent recognition master device are connected through wireless communication.

[0029] In another preferred embodiment, the underwater robot includes one intelligent identification master device and two tracking devices. The first and second tracking devices are slave devices of the intelligent identification master device. The intelligent identification master device is used to identify underwater targets, predict tracking trajectories, and plan cleaning paths. The tracking devices are used to track and clean underwater objects. Preferably, both the first and second tracking devices are streamlined in shape, preferably fish-shaped, and each has a waste storage chamber inside.

[0030] In another preferred embodiment, see Figure 2The step of tracking and cleaning the target in the three-dimensional partition includes: Step S035: Traverse the set of center point coordinates to obtain the current three-dimensional partition; Step S036: Divide the current 3D partition into N view areas. Preferably, N is a positive integer, and the N view areas are 4 view areas.

[0031] Step S037: The first image of the target in the current field of view is captured by the first camera of the intelligent recognition main device. The first image is a single-view image. Step S038: Perform blur analysis on the first image to obtain blur data of the first image, and perform target motion recognition on the blur data to obtain target motion recognition data of the first image; Step S039: Based on the motion recognition data, the multiple targets identified in the first image are divided into a first target and a second target, where the first target and the second target have different motion states. Preferably, the first target is a static target, and the second target is a moving target. By classifying underwater targets and using different tracking devices to track and clean different targets, different tracking devices can focus on a specific type of target, improving the efficiency and effectiveness of underwater cleaning work.

[0032] Step S040: Simultaneously activate the first tracking device and the second tracking device, so that the first tracking device cleans all the first targets in the current field of view partition, and so that the second tracking device cleans all the second targets in the current field of view partition; The step of enabling the first tracking device to clean all the first targets in the current field of view includes: The first coordinates of all the first targets in the current field of view are obtained to form a first coordinate set. The shortest cleaning path is planned based on the first coordinate set as a first tracking path, so that the first tracking device can track and clean all the first targets according to the first tracking path. Specifically, the intelligent recognition main device plans to obtain the first coordinates of all the first targets in the current field of view to form a first coordinate set. The shortest cleaning path is planned based on the first coordinate set as a first tracking path, and the first tracking path is sent to the first tracking device so that the first tracking device can track and clean all the first targets according to the first tracking path.

[0033] The step of enabling the second tracking device to clean all the second targets in the current field of view includes: The multi-target recognition and tracking model is used to predict the motion trajectory of the second target, so that the second tracking device can track and clean the second target according to the motion trajectory. Specifically, the intelligent recognition main device obtains the second coordinates of the second target in the first image and assigns a target ID to the second target; the second camera is activated so that the second camera continuously tracks and captures the second target to obtain a second image according to the second coordinates and the target ID; the second camera is set on the intelligent recognition main device; the second image is input to the multi-target recognition and tracking model to predict the first motion trajectory of the second target, and the first motion trajectory is sent to the second tracking device. The second tracking device sets a second motion trajectory according to the first motion trajectory and tracks and cleans the second target according to the second motion trajectory. Preferably, the Hungarian algorithm is used to perform frame matching of the second target in the second image to maintain the continuity of tracking the second target.

[0034] Step S041: Determine whether the N visual field zones have been cleaned. If yes, proceed to step S042; otherwise, proceed to step S0411. Step S0411: Control the intelligent recognition main device to rotate a preset angle on a preset plane and enter the next visual field zone, and set the next visual field zone as the current visual field zone, then jump to step S037. Preferably, the preset angle is 90°.

[0035] Step S042: Determine whether the set of center point coordinates has been traversed. If yes, proceed to step S043; otherwise, proceed to step S0421. Step S0421: Obtain the next center point coordinate from the set of center point coordinates, take the three-dimensional partition of the next center point coordinate as the current three-dimensional partition, and jump to step S036.

[0036] Step S043: Complete this cleaning task.

[0037] In a preferred embodiment, after dividing the multiple targets identified in the first image into a first target and a second target based on the motion recognition data in step S039, the first coordinates of the first target and the second coordinates of the second target are obtained. The first coordinates of all the three-dimensional partitions are collected into a first coordinate set, and the second coordinates of all the three-dimensional partitions are collected into a second coordinate set. The second pool three-dimensional map is updated based on the first coordinate collection and the second coordinate set, and a target analysis report of the current pool is generated. This allows the underwater robot to perform secondary cleaning of key cleaning areas based on the key cleaning marks in the target analysis report when cleaning the same pool next time. The target analysis report includes the number of targets in each of the three-dimensional partitions and key cleaning marks. The key cleaning marks refer to marking the three-dimensional partitions with more than a preset threshold of targets as key cleaning areas. By generating target analysis reports and marking key cleaning areas, this embodiment of the invention can help the underwater robot to repeatedly clean certain areas with a focus, improving the cleaning effect and reducing the occurrence of incomplete cleaning.

[0038] In a preferred embodiment, the intelligent identification main device includes a first form and a second form; when the intelligent identification main device moves to the current center point coordinates, it switches from the first form to the second form; when the intelligent identification main device reaches the current center point coordinates, it switches from the second form to the first form; the first form is a stationary form, and the second form is a moving form. Preferably, the first form is spherical, and the second form is streamlined; preferably, the streamlined form is fish-shaped. In this embodiment of the invention, during underwater cleaning, a tracking device and an intelligent identification main device are respectively set up. The intelligent identification main device is used to identify underwater targets and predict the target's trajectory through a multi-target identification and tracking model. The tracking device is used to perform underwater tracking and cleaning based on the target's predicted trajectory. Therefore, by setting the identification prediction function and the tracking function on different underwater devices, this embodiment of the invention improves the flexibility and efficiency of the underwater cleaning equipment and ensures the effectiveness of underwater cleaning.

[0039] This invention provides different configurations for different devices based on their functional requirements in different cleaning stages. Specifically, the intelligent identification master device is located at the center point of the three-dimensional partition during identification and prediction, maintaining a stationary state. In this state, it needs to maintain a spherical shape to facilitate smooth rotation and attitude adjustment, as well as maintain a stable balance to achieve identification and prediction functions. When moving to the next three-dimensional partition for cleaning, it needs to switch to a moving state for faster and more flexible underwater movement. Simultaneously, the tracking device is configured as a mobile type, enabling it to track and clean underwater targets more quickly and flexibly.

[0040] In a preferred embodiment, the training set, test set, and validation set of the target recognition sub-model based on the YOLOv7 algorithm are constructed according to the following steps: Step S051: First video data of the debris to be cleaned in a real swimming pool environment is collected in advance using an underwater video acquisition device, and second video data of the debris to be cleaned in a real swimming pool environment is collected using a surface video acquisition device. Preferably, the surface monitoring device is 0.2 meters above the pool surface, and preferably, the surface video acquisition device is a surface drone.

[0041] Step S052: Extract image data from the first video data and the second video data at preset time intervals to obtain a first image dataset and a second image dataset. Preferably, the time interval is 0.5s.

[0042] Step S053: Select the first image data and the second image data from the first image dataset and the second image dataset in a ratio of 6:4, merge them into a third image dataset, and divide the third image dataset into a training set, a test set and a validation set in a ratio of 7:2:1.

[0043] Step S054: Label the training set according to a preset ratio, and divide the training set into a first training set and a second training set, where the first training set is the labeled training set and the second training set is the unlabeled training set. Preferably, the preset ratio is 6%. Label all image data in the test set and validation set. During data labeling, Labelimg is used to identify each second target in the image, and a corresponding recognition box is labeled for the second target. The location information and category information of each recognition box are stored. Preferably, the location information and category information are stored in the form of an XML file.

[0044] In a preferred embodiment, see Figure 3 The target recognition sub-model based on the YOLOv7 algorithm is trained according to the following steps: Step S055: Pre-train the YOLOv7-based target recognition sub-model using an open-source target detection dataset. This step improves the robustness and generalization ability of the YOLOv7-based target recognition sub-model. Preferably, the open-source target detection dataset is the PASCAL VOC series target detection dataset or the MS COCO target detection dataset.

[0045] Step S056: Train the first target recognition sub-model using the first training set to obtain the second target recognition sub-model; the first target recognition sub-model is the initial model of the target recognition sub-model based on the YOLOv7 algorithm, and the second target recognition sub-model is the intermediate model of the target recognition sub-model based on the YOLOv7 algorithm; Step S057: Determine whether the second target recognition sub-model meets the training termination condition. If yes, end the training. If no, update the first training set and the second training set and repeat step S056 until the second target recognition sub-model converges.

[0046] Updating the first training set and the second training set includes the following steps: Step S058: Input the second training set into the second target recognition sub-model to obtain the first recognition result, filter the first recognition result according to the preset result filtering strategy to obtain the second recognition result, and label the second training set according to the second recognition result to obtain the third training set; Step S059: Add the third training set to the first training set to update the first training set, and subtract the third training set from the second training set to update the second training set.

[0047] In a preferred embodiment, the result filtering strategy is specifically as follows: calculate the uncertainty of the first identification result according to formula (1), that is, calculate the uncertainty of the first identification result according to the mean confidence of the identified target, the number of targets after filtering the first identification result and the confidence of the first identification result; select the first identification result with the uncertainty greater than a preset threshold as the second identification result, and label the second training set according to the second identification result to obtain the third training set.

[0048] (1) in, denoted by the mean confidence level of the identified target, and m represents the number of targets after filtering from the first identification result. Let Q represent the confidence level of the j-th recognition result in the first recognition results, and let Q represent the uncertainty of the recognition result. In existing technologies, high-confidence recognition results are often directly selected from the first recognition results to label the second training set, resulting in a fourth training set. This fourth training set is then added to the first training set to update it, and the updated first training set is used to train the recognition sub-model. This causes the model to ignore targets with low confidence levels, lacking training on such targets. Targets with low confidence levels are often blurry or have indistinct features, making them difficult to recognize. Therefore, models trained in existing technologies have high requirements for the input images and struggle to recognize targets in blurry or feature-indistinct images. In this embodiment of the invention, by calculating the uncertainty, the recognition results with larger uncertainties are selected to annotate the second training set to obtain a third training set. The third training set is then added to the first training set to update the first training set. This can achieve the annotation of relatively blurry or feature-indistinct images and increase a certain proportion of relatively blurry or feature-indistinct images in the first training set. This improves the recognition accuracy and detection range of the target recognition sub-model based on the YOLOv7 algorithm, so that the target recognition sub-model based on the YOLOv7 algorithm is no longer limited to detecting and recognizing clear and easily identifiable targets, but can also accurately recognize relatively blurry or feature-indistinct images.

[0049] In a preferred embodiment, the target recognition sub-model based on the YOLOv7 algorithm incorporates a first attention module, which processes the input first feature map according to the following steps to enhance the representational ability of the first feature map: The input first feature map is divided into a first part and a second part. The first result map of the first part is subjected to average pooling to obtain a first pooled map, and the first feature map of the second part is subjected to global max pooling to obtain a second pooled map. The first pooled map and the second pooled map are added to obtain a first feature vector. The first feature vector is then subjected to a local one-dimensional convolution operation to obtain a second feature vector.

[0050] The convolution kernel h of the one-dimensional convolution is calculated according to formula (2), that is, the convolution kernel of the one-dimensional convolution is calculated based on the number of channels, the scaling factor of the convolution kernel size, and the offset of the convolution kernel size. h : (2) in, E Indicates the number of channels. The scaling factor representing the kernel size. The offset representing the kernel size, preferably, is... The value is 2, the The value is 1.

[0051] The second eigenvector is calculated according to formula (3). : (3) in, This represents a one-dimensional convolution operation. This represents the feature vector obtained through global average pooling. This represents the feature vector obtained by global max pooling. H This represents a one-dimensional convolution kernel.

[0052] The second feature vector is activated according to formula (4) to obtain the first attention weight vector of each channel. P The first attention weight vector and the first feature map have the same size.

[0053] (4) According to formula (5), the first feature map is weighted in the channel direction to obtain the third feature map. This embodiment of the invention enhances the model's ability to represent the target by multiplying the first feature map and the first attention weight vector, i.e., by weighting in the channel direction. L This is the first feature map. This is the third feature map.

[0054] (5) Existing target recognition models using the YOLOv7 algorithm suffer from a significantly reduced performance and increased recognition time due to the use of fully connected layers and their extremely large number of parameters. Furthermore, they are highly dependent on labeled data; insufficient or incomplete labeled data hinders proper training and drastically reduces recognition accuracy. Specifically, the attention mechanism in existing technologies is located deep within the network, potentially having a high number of channels. The attention mechanism will result in hundreds of thousands of network parameters. This embodiment of the invention improves the YOLOv7 algorithm by introducing a first attention module, which calculates the convolution kernel size according to formula (2). h The convolution kernel calculated by this method h The value can drop to single digits, at which point the network parameters are: h × This significantly reduces the number of network parameters, making the model more concise and focused on target features. It also improves the utilization and representation capabilities of labeled data, reduces the requirements for labeled data, and enhances the model's recognition accuracy and performance.

[0055] The target recognition sub-model based on the YOLOv7 algorithm incorporates a first attention module, which improves upon existing technologies by increasing the utilization rate of labeled data, thereby enhancing the performance of the target recognition sub-model based on the YOLOv7 algorithm.

[0056] The target tracking sub-model based on the DeepSORT algorithm uses the DeepSORT algorithm for multi-target tracking, including the Karman filter, Hungarian algorithm, appearance feature extraction network, cascaded matching mechanism and trajectory management module.

[0057] Based on the above-described method embodiments, another embodiment is provided; Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the underwater cleaning method based on multi-target recognition provided in any of the above-described method embodiments of the present invention.

[0058] For example, a computer program can be divided into one or more modules / units, one or more of which are stored in memory and executed by a processor to complete the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a terminal device.

[0059] Terminal devices can be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. Terminal devices may include, but are not limited to, processors and memory. Those skilled in the art will understand that, for example, terminal devices may also include input / output devices, network access devices, buses, etc.

[0060] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device through various interfaces and lines.

[0061] Memory can be used to store computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as sound display function, image display function, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). In addition, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, Flash Card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0062] Based on the above embodiments of the invention, corresponding embodiments of storage media are provided; Another embodiment of the present invention provides a storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the storage medium is located to execute the underwater cleaning method based on multi-target recognition provided in any of the above-described method embodiments of the present invention.

[0063] Wherein, the storage medium is a computer storage medium. Modules / units integrated into the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0064] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An underwater cleaning method based on multi-target recognition, applied to underwater robots, characterized in that, Includes the following steps: Load the first 3D map of the pool to be cleaned, and update the first 3D map of the pool according to the real-time measured pool depth to obtain the second 3D map of the pool. The second swimming pool 3D map is layered into multiple 3D partitions. A first path is planned for the multiple 3D partitions. The targets in the multiple 3D partitions are identified, tracked and cleaned in sequence according to the first path. When identifying and tracking targets in the multiple three-dimensional partitions, the main intelligent identification device acquires target images and predicts the target's motion trajectory, so that the tracking device can track and clean the target according to the motion trajectory. The intelligent identification main device uses a multi-target identification and tracking model to predict the trajectory of the target. The multi-target identification and tracking model includes a target identification sub-model based on the YOLOv7 algorithm and a target tracking sub-model based on the DeepSORT algorithm. The tracking device and the intelligent identification main device are connected wirelessly. The process of performing three-dimensional layering on the second swimming pool 3D map to obtain multiple 3D partitions, planning a first path for the multiple 3D partitions, and sequentially identifying, tracking, and cleaning targets in the multiple 3D partitions according to the first path specifically includes: The second swimming pool 3D map is divided into multiple 3D partitions at preset distances along the X, Y and Z axes. Obtain the center point coordinates of each of the three-dimensional partitions. Using the center point coordinates of the three-dimensional partition where the starting position is located as the starting coordinates, plan the shortest path that passes through all the center point coordinates as the first path. The first path is a set of center point coordinates sorted according to the shortest path. The targets in the multiple three-dimensional partitions are identified, tracked, and cleaned sequentially according to the first path; The tracking and cleaning of the target in the three-dimensional partition includes: Step S1: Traverse the set of center point coordinates to obtain the current 3D partition; Step S2: Divide the current 3D partition into N view partitions; Step S3: The intelligent recognition main device captures a first image of the target in the current field of view partition, wherein the first image is a single-view image; Step S4: Perform blur analysis on the first image to obtain blur data of the first image, and perform target motion recognition on the blur data to obtain target motion recognition data of the first image; Step S5: Based on the motion recognition data, the target identified in the first image is divided into a first target and a second target, wherein the first target and the second target have different motion states; Step S6: Simultaneously activate the first tracking device and the second tracking device, so that the first tracking device cleans all the first targets in the current field of view partition, and so that the second tracking device cleans all the second targets in the current field of view partition; The step of enabling the first tracking device to clean all the first targets in the current field of view includes: The first coordinates of all the first targets in the current field of view are obtained to form a first coordinate set. The shortest cleaning path is planned based on the first coordinate set as the first tracking path, so that the first tracking device can track and clean all the first targets according to the first tracking path. The step of enabling the second tracking device to clean all the second targets in the current field of view includes: The multi-target recognition and tracking model is used to predict the motion trajectory of the second target, so that the second tracking device can track and clean the second target according to the motion trajectory. Step S7: Determine whether the N field of view partitions have been cleaned. If yes, proceed to step S8; otherwise, control the intelligent recognition main device to rotate a preset angle on a preset plane and enter the next field of view partition, and set the next field of view partition as the current field of view partition, and jump to step S3. Step S8: Determine whether the set of center point coordinates has been traversed. If yes, complete the cleaning task. If no, obtain the next center point coordinate from the set of center point coordinates, take the three-dimensional partition of the next center point coordinate as the current three-dimensional partition, and jump to step S2.

2. The underwater cleaning method based on multi-target recognition as described in claim 1, characterized in that, When tracking and cleaning a target in any of the three-dimensional partitions, the following is included: The first camera sequentially captures a first image of the target in each of the three-dimensional partitions, and the first image is an image captured from a single perspective; the first camera is installed on the intelligent recognition main device. The first image is subjected to fuzziness analysis and target motion recognition. Multiple targets identified in the first image are divided into a first target and a second target, and the first target and the second target have different motion states. A first tracking path is planned for the first target, so that the first tracking device can track and clean the first target according to the first tracking path; A multi-target recognition and tracking model is used to predict the motion trajectory of the second target, so that the second tracking device can track and clean the second target according to the motion trajectory; the first tracking device and the second tracking device are slave devices of the intelligent recognition master device, and the slave devices and the intelligent recognition master device are connected through wireless communication.

3. The underwater cleaning method based on multi-target recognition as described in claim 1, characterized in that, The step of using the multi-target recognition and tracking model to predict the motion trajectory of the second target, so that the second tracking device can track and clean the second target according to the motion trajectory, includes: The second image of the second target is obtained by capturing the second target with the second camera of the intelligent recognition main device. The second image is then input into the multi-target recognition and tracking model to obtain the predicted trajectory of the second target. The tracking trajectory of the second tracking device is then set according to the predicted trajectory.

4. The underwater cleaning method based on multi-target recognition as described in claim 3, characterized in that, The target recognition sub-model based on the YOLOv7 algorithm is trained according to the following steps: Training set partitioning steps: The training set is labeled according to a preset ratio, and the training set is divided into a first training set and a second training set. The first training set is the labeled training set, and the second training set is the unlabeled training set. Model training steps: The first target recognition sub-model is trained using the first training set to obtain the second target recognition sub-model; the first target recognition sub-model is the initial model of the target recognition sub-model, and the second target recognition sub-model is the intermediate model of the target recognition sub-model; Determine whether the second target recognition sub-model meets the training termination condition. If yes, end the training. If no, update the first and second training sets and repeat the model training steps until the second target recognition sub-model converges.

5. The underwater cleaning method based on multi-target recognition as described in claim 4, characterized in that, Updating the first training set and the second training set includes the following steps: The second training set is input into the second target recognition sub-model to obtain the first recognition result. The first recognition result is filtered according to the preset result filtering strategy to obtain the second recognition result. The second training set is labeled according to the second recognition result to obtain the third training set. The third training set is added to the first training set to update the first training set, and the third training set is subtracted from the second training set to update the second training set.

6. The underwater cleaning method based on multi-target recognition as described in any one of claims 1 to 5, characterized in that, The intelligent recognition main device includes a first form and a second form; when the intelligent recognition main device moves to the current center point coordinates, it switches from the first form to the second form, and when the intelligent recognition main device reaches the current center point coordinates, it switches from the second form to the first form; the first form is a stationary form, and the second form is a moving form.

7. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the underwater cleaning method based on multi-target recognition as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the storage medium to perform the underwater cleaning method based on multi-target recognition as described in any one of claims 1 to 6.

Images