Swimming pool cleaning robot path optimization method, system, device and medium

Abstract

The application relates to a swimming pool cleaning robot path optimization method, system, device and storage medium. The method comprises the following steps: acquiring a continuous video stream captured by a camera arranged in a swimming pool area; performing camera distortion correction on the continuous video stream to obtain a plurality of frames of corrected swimming pool images; using a pre-trained deep learning model to identify the position of a cleaning robot from each frame of the corrected swimming pool images; performing coordinate conversion on the identified cleaning robot position based on a projection correction matrix to obtain an actual position sequence of the cleaning robot in a world coordinate system; and performing real-time optimization on the cleaning planning path of the cleaning robot according to the actual position sequence to obtain an optimized path, and performing subsequent cleaning based on the optimized path.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, system, device and storage medium for optimizing the path of a pool cleaning robot. Background Technology

[0002] Pool cleaning robots are a type of cleaning robot. They are characterized by saving time and effort in cleaning pools, having diverse functions, and being lightweight and compact. They can greatly reduce the labor burden on humans and are becoming a hot topic in robot research. The cleaning coverage rate of a given pool area cleaned by a cleaning robot has become an important indicator for measuring the performance of a cleaning robot. The planning of the cleaning path of a pool cleaning robot directly affects the cleaning coverage rate of the cleaning robot. Accurate and efficient path planning plays an important role in improving cleaning efficiency and optimizing cleaning algorithms. At present, pool cleaning robots usually use pre-programmed random paths or fixed patterns (such as bow-shaped) for cleaning. The drawbacks of this method are: (1) lack of environmental adaptability: it cannot perceive its own precise position in the pool, resulting in blind cleaning paths, easy repetition of work in already cleaned areas, and insufficient cleaning in missed areas. (2) low cleaning efficiency: due to the fixed path and inability to adjust according to the real-time cleaning effect, there are a lot of invalid movements and repeated coverage, resulting in a long time and high energy consumption to complete the cleaning of the entire pool. (3) Poor ability to cope with interference: When the robot deviates from the predetermined path due to obstacles, water flow or other factors, it cannot return to the efficient cleaning path on its own and may get stuck in a local area loop or stagnate for a long time.

[0003] How to effectively optimize the cleaning path planning of pool cleaning robots has become an urgent problem to be solved. Summary of the Invention

[0004] In view of the above, this application provides a method, system, device and storage medium for optimizing the path of a swimming pool cleaning robot, the purpose of which is to solve the above-mentioned technical problems.

[0005] In a first aspect, this application provides a method for optimizing the path of a swimming pool cleaning robot, the method comprising:

[0006] Acquire a continuous video stream captured by cameras positioned in the pool area;

[0007] Camera distortion correction is performed on the continuous video stream to obtain multi-frame corrected pool images;

[0008] Using a pre-trained deep learning model, the location of the cleaning robot is identified from each frame of the corrected pool image;

[0009] Based on the projection correction matrix, coordinate transformation is performed on the identified cleaning robot position to obtain the actual position sequence of the cleaning robot in the world coordinate system;

[0010] The cleaning robot's cleaning planning path is optimized in real time based on the actual location sequence to obtain an optimized path, and subsequent cleaning is performed based on the optimized path.

[0011] In some embodiments, performing camera distortion correction on the continuous video stream includes:

[0012] Obtain the camera intrinsic parameter matrix and distortion coefficients obtained in advance through camera calibration;

[0013] The camera intrinsic parameter matrix, the distortion coefficients, and the continuous video stream input distortion correction model are processed to obtain the multi-frame corrected pool image.

[0014] In some embodiments, processing the camera intrinsic parameter matrix, the distortion coefficients, and the continuous video stream input distortion correction model to obtain the multi-frame corrected pool image includes:

[0015] Obtain the four corner pixel coordinates of the pool surface area in the image coordinate system and its actual physical coordinates in the world coordinate system;

[0016] Based on the four corner pixel coordinates and the actual physical coordinates, calculate the geometric features of the pool boundary in the bird's-eye view;

[0017] Calculate the deviation between the geometric features and the pool boundary in the corrected pool image;

[0018] If the deviation exceeds a preset threshold, the camera intrinsic parameter matrix or distortion coefficient is adjusted, and the camera intrinsic parameter matrix, the distortion coefficient, and the continuous video stream are reprocessed into the distortion correction model. The deviation between the geometric features and the pool boundary in the corrected pool image is repeatedly calculated until the deviation is less than or equal to the preset threshold.

[0019] In some embodiments, optimizing the cleaning planning path of the cleaning robot based on the actual location sequence to obtain an optimized path includes:

[0020] Based on the actual location sequence, the cleaning trajectory of the cleaning robot is generated;

[0021] Based on the cleaning trajectory and the width of the cleaning robot's brush, the area that has been cleaned and covered is determined;

[0022] The optimized path is determined based on the cleaning trajectory, the already cleaned and covered area, the global cleaning map, and the cleaning planning path.

[0023] In some embodiments, determining the optimized path based on the cleaning trajectory, the cleaned area, the global cleaning map, and the planned cleaning path includes:

[0024] Based on the cleaning trajectory, cleaning trajectory features are determined; wherein, the cleaning trajectory features reflect the cleaning efficiency of the cleaning robot;

[0025] Based on the cleaned and covered areas and the global cleaning map, the uncleaned areas are determined;

[0026] The optimized path is determined based on the cleaning trajectory characteristics, the uncleaned areas, and the cleaning planning path.

[0027] In some embodiments, determining the optimized path based on the cleaning trajectory features, the uncleaned areas, and the cleaning planning path includes:

[0028] Determine the current position of the cleaning robot;

[0029] Select a target area from the uncleaned area that meets preset conditions; wherein, the preset conditions include one or more of the following conditions: closest to the current position of the cleaning robot, largest uncleaned area, and the smallest angle change of the cleaning robot's current direction of travel upon reaching the target area;

[0030] Based on the current location, the target area, and the cleaning trajectory features, a path is planned to generate the current planned path.

[0031] The current planned path and the cleaning planned path are matched, and the inconsistent trajectory points in the cleaning planned path are replaced with trajectory points in the current planned path to obtain the optimized path.

[0032] In some embodiments, the deep learning model is trained through the following steps:

[0033] The camera captures training videos of the cleaning robot moving in the pool.

[0034] The training image set is extracted from the training video;

[0035] While the camera captures training videos including the cleaning robot moving in the pool, a drone acquires tagged videos from directly above the pool area.

[0036] A set of labeled images is obtained by extracting and labeling the labeled videos; wherein, there is a one-to-one correspondence between the image frames in the set of labeled images and the image frames in the set of training images in terms of acquisition time.

[0037] The model is trained using the training image set and the labeled image set to obtain the pre-trained deep learning model.

[0038] Secondly, this application provides a swimming pool cleaning robot path optimization system, which includes:

[0039] The correction module is used to perform camera distortion correction on the continuous video stream to obtain multi-frame corrected pool images;

[0040] The recognition module is used to identify the location of the cleaning robot from each frame of the corrected pool image using a pre-trained deep learning model.

[0041] The transformation module is used to perform coordinate transformation on the identified cleaning robot position based on the projection correction matrix to obtain the actual position sequence of the cleaning robot in the world coordinate system;

[0042] The optimization module is used to optimize the cleaning planning path of the cleaning robot in real time according to the actual position sequence, obtain the optimized path, and perform subsequent cleaning based on the optimized path.

[0043] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0044] Memory, used to store computer programs;

[0045] When the processor executes a program stored in memory, it implements the steps of the swimming pool cleaning robot path optimization method according to any embodiment of the first aspect.

[0046] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the swimming pool cleaning robot path optimization method as described in any embodiment of the first aspect.

[0047] The technical solutions provided in this application have the following advantages compared with the prior art:

[0048] (1) Through precise camera calibration and deep model training, accurate identification of machine position and continuous tracking of trajectory are achieved. (2) Significantly improve cleaning efficiency and coverage: Through global visual perception and real-time path optimization, repeated cleaning and missed areas can be effectively reduced, making the robot's movement more purposeful, thereby achieving a higher cleaning coverage in a shorter time. (3) Achieve adaptive intelligent cleaning: The robot no longer mechanically executes fixed patterns, but can "think" about the best action for the next step based on the real-time cleaning progress, possessing environmental adaptability and a more intelligent cleaning strategy. (4) Strong ability to cope with complex scenarios: When the robot is blocked by obstacles or washed away by water, the base station can reposition and plan a new path to the uncleaned area, helping the robot quickly return to an efficient working state and becoming more robust. (5) Save energy and time: By optimizing the path, ineffective movement and spinning in place can be reduced, reducing the robot's energy consumption and shortening the time required for a single complete cleaning. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating a preferred embodiment of the swimming pool cleaning robot path optimization method of this application;

[0052] Figure 2 A flowchart illustrating the process for determining the cleaning coverage of the cleaning robot for this application;

[0053] Figure 3 This is a schematic diagram of a preferred embodiment of the electronic device of this application;

[0054] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0056] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0057] Reference Figure 1 The diagram shown is a flowchart illustrating an embodiment of the swimming pool cleaning robot path optimization method of this application. The method is executed by an electronic device, which can be implemented by a software system and / or a hardware system. The swimming pool cleaning robot path optimization method includes:

[0058] Step 101: Acquire a continuous video stream captured by a camera set in the pool area.

[0059] The pool area refers to the pool and its surrounding area, which is the space in which cleaning robots perform cleaning operations.

[0060] A continuous video stream is a series of image frames captured by a camera in succession and arranged in chronological order to form a dynamic video.

[0061] In some embodiments, continuous video data can be read from or received from a camera positioned in the pool area.

[0062] Step 102: Perform camera distortion correction on the continuous video stream to obtain multi-frame corrected pool images.

[0063] Camera distortion correction is used to correct image distortion caused by the camera lens. Distortions include radial distortion and tangential distortion, and the corrected image is closer to the real scene.

[0064] Multi-frame corrected pool images refer to the image sequence obtained by correcting the distortion of multiple frames in a continuous video stream. Each frame represents a corrected image of the pool area.

[0065] In some embodiments, performing camera distortion correction on the continuous video stream includes: obtaining a camera intrinsic parameter matrix and distortion coefficients obtained in advance through camera calibration; and processing the camera intrinsic parameter matrix, the distortion coefficients, and the continuous video stream into a distortion correction model to obtain the multi-frame corrected pool image.

[0066] Camera calibration is the process of determining camera intrinsic parameters and lens distortion parameters by analyzing images of known patterns. For example, using a checkerboard pattern, images are taken from multiple angles, and algorithms are used to calculate the camera intrinsic parameter matrix and distortion coefficients.

[0067] The camera intrinsic parameter matrix is ​​a matrix that includes the camera's internal parameters and is used to describe how the camera projects points in three-dimensional space onto a two-dimensional image plane.

[0068] Distortion coefficients are used to quantify image distortion caused by camera lenses. Distortion coefficients include radial distortion coefficients and tangential distortion coefficients, used to correct bending or deformation in images.

[0069] In some embodiments, the camera intrinsic matrix and distortion coefficients can be loaded by accessing a database, file system, or memory storage location. For example, after camera calibration, the camera intrinsic matrix and distortion coefficients are saved in a configuration file, such as an XML or JSON file. When needed, the configuration file is loaded through a file reading interface to extract the values ​​of the camera intrinsic matrix and distortion coefficients.

[0070] In some embodiments, the camera intrinsic matrix, distortion coefficients, and each frame of the continuous video stream can be passed as input to the distortion correction model. The distortion correction model uses the camera intrinsic matrix and distortion coefficients to perform distortion correction on each frame.

[0071] In some embodiments, processing the camera intrinsic parameter matrix, the distortion coefficients, and the continuous video stream input distortion correction model to obtain the multi-frame corrected pool image may include the following operations:

[0072] S10, obtain the four corner pixel coordinates of the pool water surface area in the image coordinate system and its actual physical coordinates in the world coordinate system.

[0073] The pool surface area refers to the visible portion of the pool surface in an image captured by a camera. The pool surface area can be represented as a polygonal region.

[0074] An image coordinate system is a two-dimensional coordinate system used to describe the position of pixels in a digital image. The origin of the image coordinate system can be located at the top left corner of the image, the horizontal axis is to the right, and the vertical axis is downward, with the unit being pixels.

[0075] The four-corner pixel coordinates refer to the pixel coordinates of the four corners of the pool surface area in the image coordinate system. These four points correspond to the four vertices of the pool surface.

[0076] A world coordinate system is a three-dimensional coordinate system used to describe real physical space. In this context, a world coordinate system represents the actual geographical location of a swimming pool and its surrounding environment. For example, it could be a coordinate system established with a fixed corner point of the pool as the origin, the length of the pool as the X-axis, the width as the Y-axis, and the vertical direction as the Z-axis.

[0077] Actual physical coordinates refer to the three-dimensional spatial coordinates of an object in the world coordinate system, representing its physical location in the real world. For the four corner points of the pool's water surface area, the actual physical coordinates are the coordinate values ​​of these corner points in the world coordinate system.

[0078] The pixel coordinates of the four corners in the image coordinate system can be obtained from an uncorrected or corrected pool image through manual annotation or automatic image segmentation and corner detection algorithms.

[0079] S11, Based on the four corner pixel coordinates and the actual physical coordinates, calculate the geometric features of the pool boundary in the bird's-eye view.

[0080] Geometric features refer to attributes such as shape, size, angle, parallelism, and perpendicularity of a graphic or object.

[0081] In some embodiments, a transformation matrix (e.g., homography matrix) from the image coordinate system to the world coordinate system (bird's-eye view plane) can be calculated using the four corner pixel coordinates and the actual physical coordinates. Then, using this transformation matrix, the theoretically regular pool boundary (defined by the actual physical coordinates) can be mapped to the bird's-eye view coordinate system, thereby obtaining its geometric features in the bird's-eye view.

[0082] S12, calculate the deviation between the geometric features and the pool boundary in the corrected pool image.

[0083] In some embodiments, edge detection algorithms (such as Canny edge detection) or image segmentation algorithms can be used to identify the outline of the pool area, thereby obtaining a set of pixels of the pool boundary. These boundary pixels are mapped to a bird's-eye view coordinate system. In the bird's-eye view coordinate system, the geometric properties of the lines formed by the mapped actual boundary pixels are analyzed (e.g., fitting a straight line and then calculating the distance from the point to the fitted line; or calculating the angle between the fitted line segments). These actually measured geometric properties and features are compared to calculate the deviation value.

[0084] S13, if the deviation exceeds a preset threshold, the camera intrinsic parameter matrix or distortion coefficient is adjusted, the camera intrinsic parameter matrix, the distortion coefficient and the continuous video stream are reprocessed into the distortion correction model, and the deviation between the geometric features and the pool boundary in the corrected pool image is repeatedly calculated until the deviation is less than or equal to the preset threshold.

[0085] The preset threshold is a pre-defined numerical limit used to determine whether the deviation is within an acceptable range.

[0086] Adjusting the camera intrinsic parameter matrix or distortion coefficients refers to modifying the values ​​of parameters (such as focal length and principal point coordinates) or distortion coefficients in the camera intrinsic parameter matrix.

[0087] Step 103: Using a pre-trained deep learning model, identify the location of the cleaning robot from each frame of the corrected pool image.

[0088] Pre-trained deep learning models are machine learning models that have been trained on a large amount of data and are used to perform specific tasks, such as image recognition.

[0089] The position of the cleaning robot refers to its coordinates within the image. This can be represented by a bounding box or center point coordinates in the image coordinate system. For example, the bounding box is defined by the coordinates of the top-left and bottom-right corners, or by the center point coordinates and width and height.

[0090] When using a pre-trained deep learning model to identify the location of the cleaning robot from each frame of the corrected pool image, each frame of the corrected pool image is input into the pre-trained deep learning model.

[0091] Step 104: Based on the projection correction matrix, perform coordinate transformation on the identified cleaning robot position to obtain the actual position sequence of the cleaning robot in the world coordinate system.

[0092] The projection correction matrix is ​​a mathematical transformation matrix used to map points in the image coordinate system to points in the world coordinate system.

[0093] The actual position sequence refers to a series of position coordinates of the cleaning robot in the world coordinate system, arranged in chronological order. Each position corresponds to the actual coordinates at a specific point in time.

[0094] In some embodiments, a pre-calculated or known projection correction matrix can be used to convert each identified cleaning robot position into coordinates in the world coordinate system.

[0095] Step 105: Optimize the cleaning planning path of the cleaning robot in real time according to the actual position sequence to obtain an optimized path, and perform subsequent cleaning based on the optimized path.

[0096] A cleaning planning path is the planned cleaning route or trajectory of a cleaning robot within a swimming pool area. In some embodiments, the cleaning planning path can be a bow-shaped path, or a path of other shapes, such as a U-shaped path, an irregular path, etc. The bow-shaped path can be planned as follows: Based on a preset time allocation strategy, the pool cleaning process is divided into a pool wall cleaning stage and a pool bottom cleaning stage; based on the division of the pool wall cleaning stage and the pool bottom cleaning stage, a hierarchical state machine framework containing pool wall state sets and pool bottom state sets is constructed, where each state set is associated with a different bow-shaped path motion control strategy; according to the motion control strategy in the pool wall state set, the robot is controlled to clean the pool walls; after the pool wall cleaning stage ends and the robot enters the pool bottom, it switches from the pool wall state to the pool bottom state, and according to the motion control strategy in the pool bottom state set, the robot is controlled to clean the pool bottom; based on a preset path optimization algorithm, the coverage rate, overlap rate, and turning energy consumption of parallel paths on the pool bottom are evaluated in real time, and the robot's movement path is updated according to the evaluation results, finally obtaining the planned cleaning path.

[0097] In some embodiments, according to the motion control strategy of the pool wall state set, the robot is controlled to clean the pool wall, including: based on a preset vertical movement step size, the robot is controlled to perform periodic bow-shaped up-and-down movement along the pool wall; when the robot moves vertically to the water surface, the horizontal translation direction and distance are determined according to a pre-configured target value; after moving according to the horizontal translation direction and distance, the robot is controlled to continue to perform periodic bow-shaped up-and-down movement along the pool wall.

[0098] In some embodiments, when the robot performs lateral translation, it is determined whether a wall-foot collision event has occurred based on the trigger frequency of the collision sensor and the pressure threshold. In response to determining that a wall-foot collision event has occurred, a safe path to retreat to the bottom of the pool is calculated based on the robot's current pose and the 3D map of the pool bottom, and the robot is controlled to move along the safe path. After retreating to the bottom of the pool, the target detection direction is determined according to the direction switching rules in the motion control strategy in the state machine framework, and the robot is controlled to move forward a preset detection distance. Based on the trigger state of the collision sensor during the movement, it is determined whether there is a new wall in the target detection direction. If there is, the state machine is switched to the new wall climbing state; otherwise, the robot is controlled to turn back to the original wall direction and re-execute the motion control strategy in the pool wall cleaning stage.

[0099] In some embodiments, the cleaning duration of the pool wall cleaning stage can be obtained; it can be determined whether the cleaning duration of the pool wall cleaning stage has reached a preset threshold for cleaning time allocation; if so, based on the robot's sinking depth data, it can be determined whether the robot has entered the bottom of the pool; in response to the robot entering the bottom of the pool, a state switch to the pool bottom cleaning stage can be triggered.

[0100] In some embodiments, a motion control strategy based on the pool bottom state set is used to control a robot to clean the bottom of the pool, including: acquiring the robot's current position and the pool's boundary data; determining the robot's initial bow-shaped round-trip path based on the current position and boundary data; cleaning the pool bottom based on the initial bow-shaped round-trip path; and dynamically adjusting the spacing between adjacent parallel paths in the initial bow-shaped round-trip path based on the boundary signals fed back by the collision sensor.

[0101] In some embodiments, dynamically adjusting the spacing between adjacent parallel paths in the initial bow-shaped round-trip path based on the boundary signal fed back by the collision sensor includes: in response to detecting a collision with the bottom boundary of the pool, determining the trial movement direction after turning based on the coordinates of the collision point and the robot's movement direction; calculating the trial movement distance based on the robot's current speed, preset detection time, and safety redundancy distance, and controlling the robot to move the trial movement distance along the trial movement direction; determining whether a collision with the bottom boundary of the pool is triggered again during the movement, and if so, determining that the current area is a corner, generating a turning path away from the corner based on the corner coordinates, and if not, using the trial movement distance as the new spacing between adjacent parallel paths.

[0102] In some embodiments, the safety redundancy distance is determined as follows: the braking slip distance of the robot is calculated using dynamic equations based on the robot's mass, hub motor torque, and pool bottom friction coefficient; a delay compensation distance is calculated based on the response delay time of the ranging sensor; and the safety redundancy distance is obtained by weighted summing of the braking slip distance and the delay compensation distance. Cleaning coverage is the ratio of the actual cleaned area by the cleaning robot to the total area of ​​the pool area, representing the completeness of the cleaning operation. It can be expressed as a percentage.

[0103] Real-time optimization refers to the process of dynamically adjusting or improving the cleaning plan path based on real-time input data in order to improve cleaning efficiency or adaptability.

[0104] Optimized paths refer to new paths generated after real-time optimization. Optimized paths are better suited to actual cleaning conditions, such as reducing cleaning time or increasing coverage.

[0105] In some embodiments, the actual position sequence can be analyzed to determine the robot's real-time position, trajectory, and covered area. The actual position is compared with the planned cleaning path, and an optimized path is generated using optimization algorithms (such as path replanning or dynamic adjustment algorithms). Optimization objectives may include reducing repetitive cleaning, increasing coverage, or adapting to obstacles. The optimized path is then sent to the cleaning robot's control system, and the robot performs subsequent cleaning tasks based on the optimized path.

[0106] In some embodiments, optimizing the cleaning planning path of the cleaning robot based on the actual location sequence to obtain an optimized path includes: generating a cleaning trajectory of the cleaning robot based on the actual location sequence; determining the cleaned area based on the cleaning trajectory and the width of the cleaning robot's brush; and determining the optimized path based on the cleaning trajectory, the cleaned area, the global cleaning map, and the cleaning planning path.

[0107] A cleaning trajectory is a continuous trajectory generated from the actual position sequence that represents the actual movement path of a cleaning robot.

[0108] In some embodiments, the cleaning trajectory of the cleaning robot can be generated based on the actual location sequence through data processing and trajectory generation algorithms.

[0109] The cleaned area refers to the two-dimensional area that the cleaning robot has actually cleaned, based on the cleaning trajectory and the width of the brush.

[0110] In some embodiments, the cleaned area can be determined by geometric calculations based on the cleaning trajectory and the width of the cleaning robot's brush.

[0111] A global cleaning map refers to a digital map that describes the boundaries, obstacles, and passable areas of the entire pool area to be cleaned.

[0112] In some embodiments, determining the optimized path based on the cleaning trajectory, the cleaned coverage area, the global cleaning map, and the planned cleaning path includes: determining cleaning trajectory features based on the cleaning trajectory; wherein the cleaning trajectory features reflect the cleaning efficiency of the cleaning robot; determining uncleaned areas based on the cleaned coverage area and the global cleaning map; and determining the optimized path based on the cleaning trajectory features, the uncleaned areas, and the planned cleaning path.

[0113] Cleaning trajectory features are data extracted from a cleaning trajectory to quantify or describe its attributes. These features can be used to evaluate the efficiency of a cleaning robot in generating that trajectory. Examples include the total length of the cleaning trajectory, its curvature, the area covered per unit time, the number of times the trajectory repeatedly passes through a certain area, whether small loops occur, and whether long-distance backtracking occurs.

[0114] In some embodiments, the features of the cleaning trajectory can be calculated based on the cleaning trajectory through geometric analysis, statistical analysis, or kinematic analysis.

[0115] In some embodiments, uncleaned areas can be determined by spatial geometric set difference operations based on the cleaned coverage area and the global cleaning map.

[0116] In some embodiments, an optimized path can be determined by integrating evaluation and path planning algorithms based on the cleaning trajectory characteristics, the uncleaned areas, and the cleaning planned path.

[0117] In some embodiments, the optimized path can also be determined based on the cleaning trajectory features, the uncleaned area, and the cleaning planning path, according to the operations described in the embodiments below, including:

[0118] S20, determine the current position of the cleaning robot.

[0119] In some embodiments, the last location point can be extracted from the actual location sequence as the current location of the cleaning robot.

[0120] In some embodiments, the current position of the cleaning robot can also be determined by a positioning device installed on the cleaning robot.

[0121] S21, Select a target area that meets the preset conditions from the uncleaned areas.

[0122] The target area refers to the sub-area selected from the never-cleaned areas based on preset conditions.

[0123] The preset conditions are one or more criteria or rules used to evaluate and select target areas. In some embodiments, the preset conditions include one or more of the following: closest to the current position of the cleaning robot, largest uncleaned area, and minimal angular change to the current direction of travel of the cleaning robot upon reaching the target area.

[0124] "Closest to the cleaning robot's current position" means the shortest spatial distance between the target area and the cleaning robot's current position. This distance can be the actual route the cleaning robot needs to travel, or it can be the straight-line distance.

[0125] Reaching the target area requires minimal angular change to the cleaning robot's current direction of travel. This means that the cleaning robot needs to adjust its heading angle as little as possible when moving from its current position to the target area along its current direction of travel. For example, it can maximize the extension of the current direction of travel to avoid sharp turns at one or more target points / areas.

[0126] S22, perform path planning based on the current location, the target area, and the cleaning trajectory features to generate the current planned path.

[0127] The current planned path refers to a newly planned movement path that starts from the current position of the cleaning robot and moves to and covers the target area.

[0128] When generating the current planned path, an optimized path should be planned from the current location to the target point (area). This path should pass through uncovered areas as much as possible and avoid unnecessary intersections with covered areas. At the same time, the cleaning robot should avoid inefficient cleaning paths with cleaning trajectory characteristics such as small-scale loops and long-distance backtracking under the optimized path.

[0129] The current planned path can be a straight line, a curve, or a series of waypoints.

[0130] S23, match the current planned path and the cleaning planned path, and replace the inconsistent trajectory points in the cleaning planned path with trajectory points in the current planned path to obtain the optimized path.

[0131] Matching refers to comparing the current planned path with the clean planning path, with the aim of finding the correspondence and differences between the two paths in spatial and logical sequence.

[0132] Inconsistent trajectory points refer to trajectory points that, during the matching process, are identified as having a significant spatial deviation from the corresponding point on the currently planned path, or that do not logically match the path sequence.

[0133] Replacement refers to using information (such as coordinates and order) of trajectory points in the current planned path to cover or replace the information of corresponding inconsistent trajectory points in the clean planned path.

[0134] When performing subsequent cleaning based on the optimized path, the optimized path can first be converted into control commands that the robot can understand. For example, for a robot with heading control capabilities, the command is "turning angle θ, straight-line distance d". For a robot with only basic forward movement, turning, and turning in place, the command is a series of basic action sequences, such as "turn left 30 degrees, move forward 5 meters, turn right 15 degrees...". Then, the base station sends the generated control commands to the pool cleaning robot via wired or wireless communication (such as Wi-Fi, Bluetooth, or radio frequency). Pool cleaning robots can be divided into wired and wireless types. Wired pool cleaning robots are connected to the base station via cables for power supply and communication control; wired robots are more suitable for large pools.

[0135] The cleaning robot receives instructions to interrupt its current random or fixed-pattern movement (i.e., to interrupt cleaning based on the cleaning plan path) and execute optimized instructions from the base station.

[0136] After execution, or after continuous movement in a new position for a period of time, the cleaning robot can resume autonomous local obstacle avoidance mode. At the same time, the system will repeatedly execute the path optimization method disclosed in this application, start a new round of perception, analysis and optimization decision-making, and form closed-loop control.

[0137] In some embodiments, the deep learning model in the foregoing embodiments is trained in the following manner:

[0138] S30, the camera captures training videos including the cleaning robot moving in the pool.

[0139] The training video consists of continuous video data captured by a camera, including scenes of a cleaning robot moving in a swimming pool. This training video is used to subsequently extract training images to train a deep learning model.

[0140] S31, extract the training image set from the training video.

[0141] Extraction is the process of separating individual still image frames from a continuous video stream. For example, it involves decoding the video stream and acquiring image frames sequentially or at fixed intervals.

[0142] The training image set is a collection of image frames extracted from the training video. This training image set will be used as input data for training the deep learning model.

[0143] S32, while acquiring training videos including the cleaning robot moving in the pool through the camera, a tagged video is acquired from directly above the pool area by a drone.

[0144] "Directly above the pool area" refers to the spatial position perpendicular to the pool's surface and above its geographical location. From this position, the drone's shooting perspective is a vertical overhead view.

[0145] Tag videos are continuous video data captured by drones directly above the pool area, including footage of the cleaning robots in motion. Tag videos are used to provide accurate reference information about the cleaning robots' location.

[0146] S33, extract and label the labeled video to obtain a set of labeled images; wherein, the image frames in the set of labeled images have a one-to-one correspondence with the image frames in the training image set in terms of acquisition time.

[0147] Annotation is the process of marking the location information of a target object (in this case, a cleaning robot) on an image. Annotations can be represented in the form of bounding boxes, center points, or masks. For example, a rectangle is drawn on the image to enclose the cleaning robot, and the coordinates of the rectangle are recorded.

[0148] A labeled image set is a collection of images and their corresponding labels obtained by extracting image frames from labeled videos and annotating them. Labeled image sets provide real labels for supervised learning. For example, a set of image files, each corresponding to a frame extracted from a labeled video, is accompanied by an annotation file that records the bounding box coordinates of the cleaning robot in that frame image.

[0149] The capture time is the specific moment a video frame is captured by a camera or drone. The capture time can be represented by a timestamp.

[0150] A one-to-one correspondence means that there is a strict pairing relationship between elements in two sets. For example, each image frame in the labeled image set and each image frame in the training image set are extracted from video frames captured at the same time. For example, for an image frame in the training image set acquired at time t, there is one and only one corresponding image frame in the labeled image set acquired at time t.

[0151] S34, use the training image set and the label image set to train the model and obtain the pre-trained deep learning model.

[0152] Model training is the process of using training data and corresponding labels to adjust the internal parameters of a deep learning model through optimization algorithms. The goal is to enable the model to learn the ability to predict the location of a target from an input image. For example, the stochastic gradient descent algorithm is used to iteratively update the weight parameters of a convolutional neural network.

[0153] In some embodiments, the deep learning model used may be a Faster R-CNN model, etc.

[0154] Reference Figure 2 The diagram shown is a functional module schematic of the swimming pool cleaning robot path optimization system 100 of this application.

[0155] The swimming pool cleaning robot path optimization system 100 described in this application is installed in an electronic device. Depending on the functions it performs, the swimming pool cleaning robot path optimization system 100 includes an acquisition module 110, a correction module 120, an identification module 130, a conversion module 140, and an optimization module 150. These modules can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, and are stored in the memory of the electronic device.

[0156] In this embodiment, the functions of each module / unit are as follows:

[0157] Acquisition module 110 acquires a continuous video stream captured by a camera positioned in the pool area;

[0158] The correction module 120 is used to perform camera distortion correction on the continuous video stream to obtain multi-frame corrected pool images;

[0159] The recognition module 130 is used to identify the position of the cleaning robot from each frame of the corrected pool image using a pre-trained deep learning model.

[0160] The conversion module 140 is used to perform coordinate transformation on the identified cleaning robot position based on the projection correction matrix to obtain the actual position sequence of the cleaning robot in the world coordinate system;

[0161] The optimization module 150 is used to optimize the cleaning planning path of the cleaning robot in real time according to the actual position sequence, obtain the optimized path, and perform subsequent cleaning based on the optimized path.

[0162] The specific implementation of the swimming pool cleaning robot path optimization system in this application is largely the same as the specific implementation of the swimming pool cleaning robot path optimization method described above, and will not be repeated here.

[0163] Reference Figure 3 The diagram shown is a schematic representation of a preferred embodiment of the electronic device of this application.

[0164] The electronic device includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0165] Memory 113 is used to store computer programs, such as a path optimization program for a pool cleaning robot;

[0166] In some embodiments, the processor 111 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 111 can be used to control the overall operation of the electronic device, such as performing data interaction or communication-related control and processing. In this embodiment, the processor 111 is used to run program code stored in the memory 113 or process data.

[0167] The communication interface 112 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The communication interface 112 may also be used to establish a communication connection between the electronic device and other electronic devices.

[0168] The memory 113 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 113 may be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. In other embodiments, the memory 113 may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. of the electronic device. Of course, the memory 113 may include both internal storage units and external storage devices of the electronic device. In this embodiment, the memory 113 may be used to store the operating system and various computer programs installed on the electronic device, such as the program code of a swimming pool cleaning robot path optimization program. In addition, the memory 113 may also be used to temporarily store various types of data that have been output or will be output.

[0169] Figure 3 Only an electronic device with components 111-114 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0170] In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the swimming pool cleaning robot path optimization method provided in any of the foregoing method embodiments, including:

[0171] Acquire a continuous video stream captured by cameras positioned in the pool area;

[0172] Camera distortion correction is performed on the continuous video stream to obtain multi-frame corrected pool images;

[0173] Using a pre-trained deep learning model, the location of the cleaning robot is identified from each frame of the corrected pool image;

[0174] Based on the projection correction matrix, coordinate transformation is performed on the identified cleaning robot position to obtain the actual position sequence of the cleaning robot in the world coordinate system;

[0175] The cleaning robot's cleaning planning path is optimized in real time based on the actual location sequence to obtain an optimized path, and subsequent cleaning is performed based on the optimized path.

[0176] For a detailed explanation of the above steps, please refer to the above. Figure 1 A flowchart illustrating an embodiment of a path optimization method for pool cleaning robots.

[0177] Furthermore, this application also proposes a computer-readable storage medium that is both non-volatile and volatile. This computer-readable storage medium is any one or any combination of several of the following: hard disk, multimedia card, SD card, flash memory card, SMC, read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, etc. The computer-readable storage medium includes a data storage area and a program storage area. The program storage area stores a path optimization program for a pool cleaning robot. When executed by a processor, the pool cleaning robot path optimization program performs the following operations:

[0178] Acquire a continuous video stream captured by cameras positioned in the pool area;

[0179] Camera distortion correction is performed on the continuous video stream to obtain multi-frame corrected pool images;

[0180] Using a pre-trained deep learning model, the location of the cleaning robot is identified from each frame of the corrected pool image;

[0181] Based on the projection correction matrix, coordinate transformation is performed on the identified cleaning robot position to obtain the actual position sequence of the cleaning robot in the world coordinate system;

[0182] The cleaning robot's cleaning planning path is optimized in real time based on the actual location sequence to obtain an optimized path, and subsequent cleaning is performed based on the optimized path.

[0183] The specific implementation of the computer-readable storage medium in this application is largely the same as the specific implementation of the above-described swimming pool cleaning robot path optimization method, and will not be repeated here.

[0184] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, system, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, system, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, system, article, or method that includes that element.

[0185] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware simulation platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0186] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method of optimizing a path of a pool cleaning robot, the method comprising: The method comprises: acquiring a continuous video stream captured by a camera arranged at a pool area; performing camera distortion correction on the continuous video stream to obtain a plurality of frames of corrected pool images; using a pre-trained deep learning model to identify a position of a cleaning robot from each frame of the corrected pool images; based on a projection correction matrix, performing coordinate conversion on the identified position of the cleaning robot to obtain an actual position sequence of the cleaning robot in a world coordinate system; based on the actual position sequence, optimizing a cleaning planning path of the cleaning robot in real time to obtain an optimized path, and performing subsequent cleaning based on the optimized path; the camera distortion correction on the continuous video stream comprises: obtaining a camera intrinsic parameter matrix and distortion coefficients obtained in advance through camera calibration; inputting the camera intrinsic parameter matrix, the distortion coefficients and the continuous video stream into a distortion correction model for processing to obtain the plurality of frames of corrected pool images; the inputting the camera intrinsic parameter matrix, the distortion coefficients and the continuous video stream into the distortion correction model for processing to obtain the plurality of frames of corrected pool images comprises: obtaining four-corner pixel coordinates of a pool water surface area in an image coordinate system and actual physical coordinates thereof in the world coordinate system, wherein the four-corner pixel coordinates refer to pixel coordinates of four corner positions of the pool water surface area in the image coordinate system; based on the four-corner pixel coordinates and the actual physical coordinates, calculating geometric features of a pool boundary in a bird's-eye view; calculating a deviation of the geometric features from the pool boundary in the corrected pool image; if the deviation exceeds a preset threshold, adjusting the camera intrinsic parameter matrix or the distortion coefficients, re-inputting the camera intrinsic parameter matrix, the distortion coefficients and the continuous video stream into the distortion correction model for processing, and repeatedly calculating the deviation of the geometric features from the pool boundary in the corrected pool image until the deviation is less than or equal to the preset threshold.

2. The pool cleaning robot path optimization method of claim 1, wherein, the optimization of the cleaning planning path of the cleaning robot based on the actual position sequence to obtain the optimized path comprises: based on the actual position sequence, generating a cleaning trajectory of the cleaning robot; based on the cleaning trajectory and a width of a rolling brush of the cleaning robot, determining a cleaned coverage area; based on the cleaning trajectory, the cleaned coverage area, a cleaning global map and the cleaning planning path, determining the optimized path.

3. The pool cleaning robot path optimization method of claim 2, wherein, the determination of the optimized path based on the cleaning trajectory, the cleaned coverage area, the cleaning global map and the cleaning planning path comprises: based on the cleaning trajectory, determining a cleaning trajectory feature; wherein the cleaning trajectory feature reflects a cleaning efficiency of the cleaning robot; based on the cleaned coverage area and the cleaning global map, determining an uncleaned area; based on the cleaning trajectory feature, the uncleaned area and the cleaning planning path, determining the optimized path.

4. The pool cleaning robot path optimization method of claim 3, wherein, the determination of the optimized path based on the cleaning trajectory feature, the uncleaned area and the cleaning planning path comprises: determining a current position of the cleaning robot; select a target area from the unswept area that meets a preset condition; wherein the preset condition includes one or more of the following conditions: closest to the current position of the cleaning robot, largest unswept area, smallest angle change to the current travel direction of the cleaning robot when reaching the target area; path planning based on the current position, the target area, and the cleaning trajectory feature to generate a current planning path; matching the current planning path and the cleaning planning path, and replacing inconsistent trajectory points in the cleaning planning path with trajectory points in the current planning path to obtain the optimized path.

5. The pool cleaning robot path optimization method of claim 1, wherein, The deep learning model is trained by the following steps: acquiring a training video including the movement of the cleaning robot in the pool through the camera; extracting a training image set from the training video; acquiring a label video from directly above the pool area through a drone while acquiring the training video including the movement of the cleaning robot in the pool through the camera; extracting and labeling a label image set from the label video; wherein the image frames in the label image set and the image frames in the training image set have a one-to-one correspondence in acquisition time using the training image set and the label image set for model training to obtain the pre-trained deep learning model.

6. A swimming pool cleaning robot path optimization system characterized by, The system comprises: an acquisition module that acquires a continuous video stream captured by a camera arranged in a pool area; a correction module for camera distortion correction on the continuous video stream to obtain a plurality of frames of corrected pool images; an identification module for identifying the position of the cleaning robot from each frame of the corrected pool images using a pre-trained deep learning model; a conversion module for coordinate conversion of the identified position of the cleaning robot based on a projection correction matrix to obtain an actual position sequence of the cleaning robot in a world coordinate system; an optimization module for real-time optimization of a cleaning planning path of the cleaning robot based on the actual position sequence to obtain an optimized path, and subsequent cleaning based on the optimized path; The camera distortion correction on the continuous video stream comprises: obtaining a camera intrinsic parameter matrix and distortion coefficients obtained in advance through camera calibration; inputting the camera intrinsic parameter matrix, the distortion coefficients, and the continuous video stream into a distortion correction model for processing to obtain the plurality of frames of corrected pool images; The inputting the camera intrinsic parameter matrix, the distortion coefficients, and the continuous video stream into a distortion correction model for processing to obtain the plurality of frames of corrected pool images comprises: obtaining four-corner pixel coordinates of the pool water surface area in an image coordinate system and their actual physical coordinates in the world coordinate system; the four-corner pixel coordinates refer to the pixel coordinates of the four corner positions of the pool water surface area in the image coordinate system; calculating the geometric features of the pool boundary in the bird's eye view based on the four-corner pixel coordinates and the actual physical coordinates; calculating the deviation of the geometric features from the pool boundary in the corrected pool image; If the deviation exceeds a preset threshold, the camera intrinsic matrix or the distortion coefficient is adjusted, the camera intrinsic matrix, the distortion coefficient and the continuous video stream are re-input into a distortion correction model for processing, and the deviation of the geometric feature from the pool boundary in the corrected pool image is repeatedly calculated until the deviation is less than or equal to the preset threshold.

7. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used for storing a computer program; The processor is used for executing the program stored on the memory, and the pool cleaning robot path optimization method in any one of claims 1 to 5 is realized.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the pool cleaning robot path optimization method in any one of claims 1 to 5.

Images