Automatic pool cleaning device, stuck state detection method and computer storage medium

Abstract

The invention provides an automatic pool cleaning device, a detection method for the stuck state of the automatic pool cleaning device and a computer storage medium. The detection method comprises the following steps: respectively acquiring environment images at two adjacent moments during the cleaning operation of the automatic pool cleaning device; feature point sets are extracted from the two obtained environment images respectively, so that two feature point sets are obtained, and the two feature point sets respectively comprise M feature points; calculating an optical flow value of a corresponding feature point between the two feature point sets through a predetermined optical flow algorithm, thereby obtaining M optical flow values; according to the obtained M optical flow values, the driving state of the automatic pool cleaning device is judged, and the driving state comprises a pre-clamping state and a non-clamping state.

Description

Technical Field

[0001] The present application relates to the technical field of cleaning devices, and in particular to an automatic pool cleaning device, a method for detecting a stuck state of the automatic pool cleaning device, and a computer storage medium. Background Art

[0002] With the popularity of swimming pools and the development of robot technology, more and more consumers hope to complete the cleaning of swimming pools through automated pool cleaning robots. When the robot performs the task of cleaning the bottom of the swimming pool, obstacles such as the water inlet, stones, steps, etc. at the bottom of the swimming pool may get stuck on the machine chassis, causing the pool cleaning robot to be unable to continue cleaning operations and become stuck. If the pool cleaning robot is stuck for a long time, it may cause damage to its motor or other components, increasing the risk of failure. Therefore, timely judging whether the pool cleaning robot is stuck is of great significance in improving the cleaning efficiency of the swimming pool, protecting the safety of the equipment, and improving the level of intelligence. Summary of the invention

[0003] In view of the deficiencies of the above-mentioned prior art, the present application provides a method for detecting a stuck state of an automatic pool cleaning device, comprising: acquiring environmental images at two adjacent moments during a cleaning operation of the automatic pool cleaning device; extracting feature point sets from the two acquired environmental images, thereby obtaining two feature point sets, wherein the two feature point sets respectively include M feature points; calculating optical flow values ​​of corresponding feature points between the two feature point sets by a predetermined optical flow algorithm, thereby obtaining M optical flow values; and judging the driving state of the automatic pool cleaning device based on the obtained M optical flow values, wherein the driving state includes: a pre-stuck state and a non-stuck state.

[0004] Furthermore, if the driving state is the pre-stuck state, the detection method also includes: as time goes by, repeating in sequence the steps of acquiring environmental images at two adjacent moments, extracting a feature point set, obtaining M optical flow values, and judging the driving state of the automatic pool cleaning device; if the driving state judged each time is the pre-stuck state, it is determined that the automatic pool cleaning device is in a stuck state.

[0005] Furthermore, judging the driving state of the automatic pool cleaning device based on the obtained M optical flow values ​​includes: sorting the M optical flow values ​​in descending order to obtain an optical flow value sequence; extracting the first N optical flow values ​​in the optical flow value sequence; calculating the average value of the extracted N optical flow values; comparing the average value with a predetermined optical flow threshold value, and if the average value is less than the predetermined optical flow threshold value, judging that the driving state of the automatic pool cleaning device is a pre-stuck state.

[0006] Furthermore, judging the driving state of the automatic pool cleaning device based on the M optical flow values ​​obtained includes: calculating the average value of the M optical flow values ​​obtained; comparing the average value with a predetermined optical flow threshold, and if the average value is less than the predetermined optical flow threshold, judging that the driving state of the automatic pool cleaning device is a pre-stuck state.

[0007] Further, if the average value is greater than or equal to the predetermined optical flow threshold, it is determined that the driving state of the automatic pool cleaning device is a non-stuck state.

[0008] Furthermore, the extracting of feature point sets from the two acquired environmental images respectively, thereby obtaining two feature point sets, includes: performing grayscale conversion on the two acquired environmental images respectively, to obtain a plurality of pixel points corresponding to the two extracted environmental images respectively; calculating a first gradient amplitude in the x-axis direction and a second gradient amplitude in the y-axis direction for each pixel point; constructing an autocorrelation matrix corresponding to the pixel point based on the first gradient amplitude and the second gradient amplitude; and determining the feature point set based on the autocorrelation matrix corresponding to each pixel point.

[0009] Further, constructing the autocorrelation matrix corresponding to the pixel point based on the first gradient amplitude and the second gradient amplitude includes: calculating the first gradient amplitude to obtain a first calculation result; calculating the second gradient amplitude to obtain a second calculation result; calculating the first gradient amplitude and the second gradient amplitude to obtain a third calculation result; constructing the autocorrelation matrix corresponding to the pixel point based on the first calculation result, the second calculation result and the third calculation result.

[0010] Furthermore, determining the feature point set based on the autocorrelation matrix corresponding to each pixel point includes: calculating the autocorrelation matrix corresponding to each pixel point to obtain the eigenvalue corresponding to each pixel point; calculating the eigenvalue corresponding to each pixel point to obtain the corner point responsiveness corresponding to each pixel point; sorting the corner point responsiveness corresponding to each pixel point in descending order to obtain a plurality of sorted corner point responsivenesses; and screening the plurality of sorted corner point responsivenesses according to a first preset value to obtain the feature point set.

[0011] The present application also discloses an automatic pool cleaning device, comprising an image acquisition unit and a control unit, wherein the control unit is configured to: control the image acquisition unit to respectively acquire environmental images at two adjacent moments during the cleaning operation of the automatic pool cleaning device; extract feature point sets from the two acquired environmental images respectively, thereby obtaining two feature point sets, wherein the two feature point sets respectively include M feature points; calculate the optical flow values ​​of corresponding feature points between the two feature point sets by a predetermined optical flow algorithm, thereby obtaining M optical flow values; and judge the driving state of the automatic pool cleaning device based on the obtained M optical flow values, wherein the driving state includes: a pre-stuck state and a non-stuck state.

[0012] The present application also discloses a computer storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0013] The embodiments described in this application have the following beneficial effects:

[0014] The method for detecting the stuck state of the automatic pool cleaning device provided in the present application enables the automatic pool cleaning device to extract feature point sets based on two environmental images corresponding to the automatic pool cleaning device at consecutive moments during the process of cleaning the pool bottom, and judge whether the automatic pool cleaning device is in a stuck state through the optical flow values ​​of the corresponding feature points between the two feature point sets, thereby assisting the automatic pool cleaning device to get out of the jam in time, and further enabling the automatic pool cleaning device to perform the cleaning operation of the pool bottom normally and efficiently, thereby improving the cleaning efficiency of the pool. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. The drawings described below are only exemplary embodiments of the present application.

[0016] Figure 1 Detailed description of the invention The invention is a flow chart showing a method for detecting a stuck state of an automatic pool cleaning device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in this application will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of this application. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict.

[0018] The present application provides a method for detecting the stuck state of an automatic pool cleaning device, an automatic pool cleaning device using the detection method, and a computer storage medium. The automatic pool cleaning device of the present application can clean a pool. The pool is, for example, a pool-shaped building. The pool-shaped building can be a swimming pool, a reservoir, a spa pool, a water tank, a water tank, etc. The automatic pool cleaning device can be a device such as an automatic cleaning device, a pool cleaning robot, etc., which can clean the pool-shaped building. The present application does not limit the specific presentation of the automatic pool cleaning device and the pool-shaped building, as long as the principle of the present application can be realized. In the following, if not otherwise specified, the robot will be used as an example of the automatic pool cleaning device, and the swimming pool will be used as an example of a pool or a pool-shaped building. In the following, if not otherwise specified, the terms "pool bottom", "pool bottom", and "pool bottom" all refer to the bottom surface of the swimming pool.

[0019] The following is a detailed description of the method 100 for detecting a stuck state of the automatic pool cleaning device of the present application in conjunction with the accompanying drawings. Figure 1 The flowchart of the method for detecting the stuck state of the automatic pool cleaning device according to an embodiment of the present application is shown. The control method 100 includes steps S101 to S104. Steps S101 to S104 will be described below.

[0020] In step S101, environmental images are respectively acquired at two adjacent moments during the cleaning operation of the automatic pool cleaning device.

[0021] In one embodiment, the robot needs to first plan the path of the cleaning operation according to the contour of the pool in order to obtain the cleaning path required for the cleaning operation, and then the robot performs the cleaning operation on the bottom surface of the pool along the cleaning path. The robot can use a path planning algorithm to obtain the cleaning path; it can also obtain the cleaning path by updating and iterating historical cleaning paths; the cleaning path can also be provided or set by the user; the cleaning path can also be pre-stored in the robot's memory. The above description of the method for obtaining the cleaning path is only exemplary, and those skilled in the art can select the cleaning path according to actual conditions, as long as the technical principles of this application can be implemented.

[0022] When the robot performs cleaning operations on the bottom surface of the pool along the cleaning path, the robot can obtain environmental images at two adjacent moments (e.g., moment t-1 and moment t) through an image acquisition device (e.g., an image acquisition device disposed in front of the robot), and the objects included in the environmental image are one or more static reference objects, thereby avoiding image distortion caused when the objects included in the environmental image are one or more dynamic reference objects. The term "two adjacent moments" refers to two adjacent time points when the image acquisition device acquires the environmental image. In other words, moment t can represent the current moment, and moment t-1 represents the moment when the environmental image was last acquired. For example, if the image acquisition device acquires an environmental image once per second, the two adjacent moments (i.e., moment t-1 and moment t) can be two adjacent seconds, such as the t-1 second and the t second.

[0023] For example, the robot can collect images of static reference objects on the bottom of the pool and the surrounding environment of the bottom of the pool at adjacent moments t-1 and t by using a monocular camera and / or binocular camera and / or depth camera installed on the top of the machine. In order to improve the environmental adaptability and target recognition accuracy, other sensors can also be combined with the image acquisition device, including but not limited to: high-definition visible light cameras, infrared thermal imaging cameras, and laser radars. When the robot performs cleaning operations on the bottom of the pool, it collects images of static reference objects on the bottom of the pool and the surrounding environment of the bottom of the pool at adjacent moments t-1 and t. The information contained in the collected environmental image includes but is not limited to: information such as the color, texture, size, outline, and relative position of the static reference object.

[0024] It should be noted that after obtaining the environmental images at adjacent times t-1 and t, image preprocessing operations can be performed on them to improve image quality, making subsequent detection and analysis more accurate and reliable. Image preprocessing operations include but are not limited to: filtering the environmental image, sharpening the environmental image, removing noise from the environmental image, enhancing the contrast of the environmental image, and normalizing the data in the environmental image.

[0025] It should be noted that the above description of the method for obtaining environmental images, the information of environmental images collected, and the pre-processing operations on environmental images is only exemplary. The method for obtaining environmental images, the information of environmental images collected, and the pre-processing operations on environmental images protected by this application are not limited to the contents listed above. Those skilled in the art can adjust the method for obtaining environmental images and the information of environmental images collected according to actual conditions, and can also adopt one or several pre-processing operation methods to process the environmental images according to actual conditions, as long as the technical principles of this application can be implemented.

[0026] Next, the process proceeds to step S102. In step S102, feature point sets are respectively extracted from the two acquired environment images, thereby obtaining two feature point sets, wherein the two feature point sets respectively include M feature points.

[0027] In one embodiment, the extracting of feature point sets from the two acquired environmental images respectively to obtain two feature point sets includes: performing grayscale conversion on the two acquired environmental images respectively to obtain a plurality of pixel points corresponding to the two extracted environmental images respectively; calculating a first gradient amplitude in the x-axis direction and a second gradient amplitude in the y-axis direction for each pixel point; constructing an autocorrelation matrix corresponding to the pixel point based on the first gradient amplitude and the second gradient amplitude; and determining the feature point set based on the autocorrelation matrix corresponding to each pixel point.

[0028] For example, after grayscale conversion is performed on the environment image A at time t-1 and the environment image B at time t (hereinafter collectively referred to as "two environment images"), multiple pixel points corresponding to the two environment images are obtained, and the number of the multiple pixel points is M. all . Corner detection is performed on multiple pixel points corresponding to the two environmental images respectively (corner detection is a method used to obtain image features in computer vision systems, also known as feature point detection). Corner detection algorithms include but are not limited to: Harris corner detection algorithm, Shi-Tomasi corner detection algorithm (also known as Shi-Tomasi corner detection algorithm or GoodFeatures to Track algorithm), FAST corner detection algorithm, AGAST corner detection algorithm, Phase Congruency corner detection algorithm, and SuperPoint corner detection algorithm. Multiple corner points corresponding to environmental image A at time t-1 and multiple corner points corresponding to environmental image B at time t are extracted by one or more of the above corner detection algorithms. In other words, a feature point set corresponding to environmental image A at time t-1 and a feature point set corresponding to environmental image B at time t are extracted by one or more of the above corner detection algorithms, and each feature point set includes M feature points (M all ≥M), in other words, the corner detection algorithm can detect M corner points in each environment image. all From the pixels, M key feature points are extracted to analyze whether the robot is stuck.

[0029] It should be noted that a corner point refers to a point where two edges in an image intersect, which is usually manifested as a point where the intensity changes significantly in a local area and there are edge responses in two orthogonal directions. In other words, a corner point is like an "inflection point" in an image, which refers to the place where two edge lines meet in an image. For example, if two intersecting lines are drawn on a piece of paper, the point where the two intersect is a corner point. In a photo or image, a corner point is usually where the edges of an object meet, such as a corner of a wall, a corner of a table, or where any two edges intersect. It can be understood that a corner point is a point in an image from which edges can be seen in two different directions. The following will take the Shi-Tomasi corner point detection algorithm as an example, and combine it with a specific embodiment to illustrate the process and method of obtaining two feature point sets.

[0030] Since both environmental images are color images, they contain a variety of information from three color channels (red channel, green channel and blue channel), and corner detection is to extract key feature points in the image, which are usually related to the structure and shape of the image and have nothing to do with color. Therefore, in order to reduce the complexity and processing time of subsequent calculations, and to provide sufficient brightness information to detect corner points in the image, it is necessary to perform grayscale conversion on the two environmental images respectively, so as to obtain multiple pixel points corresponding to the two environmental images. Grayscale conversion can be achieved by a variety of methods, including but not limited to: linear weighted method, average method, brightness information method (if the image has a brightness channel) and grayscale conversion using programming language.

[0031] If the two environment images are converted to grayscale separately by the linear weighted method, different weights need to be assigned to different color channels in the two environment images. For example, the weight of the red channel is assigned to 0.299, the weight of the green channel is assigned to 0.587, and the weight of the blue channel is assigned to 0.114. The grayscale conversion of the two environment images is performed by the conversion formula. The conversion formula is as follows:

[0032] I gray =0.299×I red +0.587×I green +0.114×I blue

[0033] Among them, I gray is the gray value, I red ,I green and I blue They are the corresponding values ​​of the red channel, green channel, and blue channel respectively.

[0034] If the grayscale conversion of the two environment images is performed separately by the average method, it is necessary to simply add the values ​​of the three color channels in the image and divide by 3 to obtain the grayscale values ​​of the two environment images. The conversion formula is as follows:

[0035] I gray =(I red +I green +I blue ) / 3

[0036] Among them, I gray is the gray value, I red ,I green and I blue They are the corresponding values ​​of the red channel, green channel, and blue channel respectively.

[0037] If the brightness information method is used to perform grayscale conversion on the two environment images respectively, the brightness values ​​in the two environment images are directly used as their corresponding grayscale values.

[0038] It should be noted that the above description of the method of performing grayscale conversion on two environmental images is only exemplary. The method of performing grayscale conversion on two environmental images protected by the present application is not limited to the contents listed above. Those skilled in the art can adjust the method of performing grayscale conversion on two environmental images according to actual conditions, as long as the technical principles of the present application can be implemented.

[0039] After grayscale conversion is performed on the two environment images, for each pixel point in the two environment images, a Sobel operator, a Laplacian operator, a Prewitt operator, a Scharr operator, or other methods are used to calculate the first gradient amplitude of the pixel point in the horizontal direction (i.e., the x direction) and the second gradient amplitude in the vertical direction (i.e., the y direction), so as to indicate how fast the brightness changes in the two environment images. The following will take the Sobel operator as an example, and in combination with a specific embodiment, describe the process and method of calculating the first gradient amplitude of each pixel point in the x-axis direction and the second gradient amplitude in the y-axis direction.

[0040] The x-direction convolution kernel G using the Sobel operator x and the y-direction convolution kernel G y Convolution operations are performed with each pixel (i, j) in the two environment images respectively, and the gradient of each pixel in the x and y directions is calculated.

[0041] in,

[0042] For each pixel (i, j) in each environment image, its gradient I in the x direction x (i, j) (i.e., the first gradient magnitude) and the gradient I in the y direction y (i,j) (i.e., the second gradient magnitude) can be obtained by the following convolution operation:

[0043] Ix (i,j)=G x *I(i,j)

[0044] I y (i,j)=G y *I(i,j)

[0045] Among them, I(i,j) is the grayscale value of the environment image at pixel (i,j).

[0046] In another embodiment, after obtaining the first gradient amplitude and the second gradient amplitude, constructing the autocorrelation matrix corresponding to the pixel point based on the first gradient amplitude and the second gradient amplitude includes: calculating the first gradient amplitude to obtain a first calculation result; calculating the second gradient amplitude to obtain a second calculation result; calculating the first gradient amplitude and the second gradient amplitude to obtain a third calculation result; constructing the autocorrelation matrix corresponding to the pixel point based on the first calculation result, the second calculation result and the third calculation result.

[0047] For example, for each pixel (i, j) in each environment image, its gradient I in the x direction is x (i, j) (i.e., the first gradient amplitude) is squared to obtain the first calculation result I x 2 , which indicates the gradient change of the pixel in the horizontal direction; its gradient in the y direction I y (i, j) (i.e., the second gradient amplitude) is squared to obtain the second calculation result I y 2 , which indicates the gradient change of the pixel in the vertical direction; the gradient I in the x direction x (i, j) (i.e., the first gradient magnitude) and its gradient I in the y direction y (i, j) (i.e., the second gradient amplitude) is multiplied to obtain the third calculation result I x I y , which represents the cross-change of the pixel in two directions, and uses this to construct the autocorrelation matrix A (also called the structure tensor matrix or gradient covariance matrix), which is used to capture the direction and magnitude information of the gradient around each pixel in each environment image. The expression of the autocorrelation matrix A is as follows:

[0048]

[0049] It should be noted that in some cases, in order to improve the stability of the algorithm, it may be necessary to normalize the autocorrelation matrix A by dividing each element in the matrix by the sum of the squares of the gradient amplitudes:

[0050]

[0051] In another embodiment, after obtaining multiple autocorrelation matrices corresponding to multiple pixel points in two environmental images, determining the feature point set based on the autocorrelation matrix corresponding to each pixel point includes: calculating the autocorrelation matrix corresponding to each pixel point to obtain the eigenvalue corresponding to each pixel point; calculating the eigenvalue corresponding to each pixel point to obtain the corner point responsiveness corresponding to each pixel point; sorting the corner point responsiveness corresponding to each pixel point in descending order to obtain multiple sorted corner point responsiveness; and screening the sorted multiple corner point responsiveness according to a first preset value to obtain the feature point set.

[0052] For example, by solving the eigenvalue λ and the unit matrix I, the characteristic equation A′ corresponding to the autocorrelation matrix A corresponding to each pixel point is constructed. The expression of A′ is as follows:

[0053] A′=A-λ×I

[0054] The eigenvalue λ can be obtained by solving the characteristic equation A′:

[0055] det(A′)=0

[0056] Right now

[0057] det(A-λI)=0

[0058] Simplifying it, we get:

[0059] λ 2 -tr(A)λ+det(A)=0

[0060] Among them, tr(A) is the trace of matrix A (that is, the sum of the diagonal elements), and det(A) is the determinant of matrix A. The specific calculation is as follows:

[0061] tr(A)=I x 2 +I y 2

[0062] det(A)=I x 2 ×I y 2 -I x I y 2

[0063] Therefore, the eigenvalue λ corresponding to each pixel in each environment image 1 and eigenvalue λ 2 for:

[0064]

[0065] The size of the eigenvalue reflects the edge strength of the pixel in different directions on the environment image. In some cases, a pixel may have a larger gradient value in one direction and a smaller gradient value in another direction due to noise or image inhomogeneity. If both eigenvalues ​​corresponding to a pixel are large, it may mean that the pixel is an edge point rather than a corner point. However, if a pixel is a corner point, both eigenvalues ​​of its autocorrelation matrix will be large because the gradient at the corner point changes significantly in both directions.

[0066] After getting the eigenvalue λ corresponding to each pixel 1 and eigenvalue λ 2 After that, we can choose the eigenvalue λ 1 and eigenvalue λ 2 The smaller eigenvalue in is used as the corner point response corresponding to the pixel point, which is used to determine whether the pixel point in the environment image is a corner point; the eigenvalue λ can also be selected 1 and eigenvalue λ 2 The larger eigenvalue in is used as the corner point response corresponding to the pixel point, which is used to determine whether the pixel point in the environment image is a corner point; the eigenvalue λ 1 and eigenvalue λ 2 The average value of is used as the corner point responsivity corresponding to the pixel point, which is used to determine whether the pixel point in the environment image is a corner point. The above description of the calculation method of the corner point responsivity is only exemplary, and those skilled in the art can select the calculation method of the corner point responsivity according to actual conditions, as long as the technical principle of the present application can be implemented.

[0067] In one embodiment, the corner point responsiveness of all pixels in the environment image A at time t-1 may be sorted in descending order, and the first N pixels with the highest corner point responsiveness are selected as the final corner point results. In other words, the first N pixels with the highest corner point responsiveness are used as the feature point set corresponding to the environment image A at time t-1. The sorted corner point responsiveness may also be screened according to a first preset value, and the pixels corresponding to the corner point responsiveness greater than or equal to the first preset value are used as the corner point results of the environment image at time t-1. In other words, the pixels corresponding to the corner point responsiveness greater than or equal to the first preset value are used as the feature point set corresponding to the environment image at time t-1.

[0068] Accordingly, the corner point responsiveness of all pixels in the environment image B at time t can be sorted in descending order, and the first N pixels with the highest corner point responsiveness are selected as the final corner point results. In other words, the first N pixels with the highest corner point responsiveness are used as the feature point set corresponding to the environment image at time t. The sorted corner point responsiveness can also be screened according to a first preset value, and the pixels corresponding to the corner point responsiveness greater than or equal to the first preset value are used as the corner point results of the environment image at time t. In other words, the pixels corresponding to the corner point responsiveness greater than or equal to the first preset value are used as the feature point set corresponding to the environment image at time t.

[0069] Then, the process proceeds to step S103. In step S103, optical flow values ​​of corresponding feature points between the two feature point sets are calculated using a predetermined optical flow algorithm, thereby obtaining M optical flow values.

[0070] In one embodiment, in order to obtain the motion state of the robot in the two frames of images (i.e., the environment image A at time t-1 and the environment image B at time t), it is necessary to calculate the optical flow values ​​of the corresponding feature points between the two feature point sets in the two frames of images. In other words, it is necessary to calculate the optical flow values ​​between the M feature points in the environment image A at time t-1 and the corresponding M feature points in the environment image B at time t. For example, the M feature points in the environment image A at time t-1 are T 1 , T 2 , T 3 , ..., T M , the M feature points in the environment image B at time t are T′ 1 , T′ 2 , T′ 3 , ..., T′ M If T 1 and T′ 1 Correspondingly, T 2 and T′ 2 Correspondingly, T 3 and T′ 3 Correspondingly, by analogy, T M and T′ M Correspondingly, we need to calculate T 1 and T′ 1 The corresponding optical flow value, T 2 and T′ 2 The corresponding optical flow value, T 3 and T′ 3 The corresponding optical flow value, and so on, T M and T′ M The corresponding optical flow values ​​are obtained to obtain M optical flow values.

[0071] It should be noted that the optical flow value refers to the displacement of the corresponding pixel points (or feature points) in two environment images from one frame to another. It describes the motion information of the object or scene in the image, including the direction and speed of the motion. The optical flow value is usually expressed as a vector, which contains the displacement in the horizontal direction (x-axis) and the vertical direction (y-axis).

[0072] For each feature point in the environmental image A at time t-1 and the environmental image B at time t, the optical flow value is the displacement value of its position in the subsequent frame and the position in the previous frame. Specifically, for each pair of feature points in the two environmental images, a predetermined optical flow algorithm is used, including but not limited to: Lucas-Kanade optical flow algorithm, Horn-Schunck optical flow algorithm, FlowNet optical flow algorithm, RAFT optical flow algorithm, etc., to calculate its displacement in the x and y directions, that is, the optical flow vector (u, v), where u is the horizontal displacement and v is the vertical displacement, thereby obtaining M optical flow values. The above description of the predetermined optical flow algorithm is only exemplary, and those skilled in the art can select the predetermined optical flow algorithm according to actual conditions, as long as the technical principles of the present application can be implemented.

[0073] Finally, the process proceeds to step S104, in which the driving state of the automatic pool cleaning device is determined based on the M optical flow values ​​obtained, wherein the driving state includes: a pre-stuck state and a non-stuck state.

[0074] It is understandable that when the robot is performing cleaning operations normally, the rotation of its wheels will drive the robot to move at the bottom of the swimming pool, and then the M optical flow values ​​corresponding to the M pairs of feature points in the robot's environmental image at time t-1 and the environmental image at time t will also show relatively obvious motion information and displacement data. When the robot is trapped at the bottom of the pool (for example, trapped by cables, steps, obstacles, etc. at the bottom of the pool), its wheels may still be rotating, but due to obstructions, the robot cannot travel normally, that is, the robot does not move in space or moves a very limited distance. At this time, the optical flow values ​​corresponding to each pair of feature points in the robot's environmental image A at time t-1 and the environmental image B at time t will show a small displacement or almost no displacement, which is significantly different from the optical flow values ​​between the environmental images collected at two adjacent moments when the robot is driving normally to perform cleaning operations.

[0075] In one embodiment, judging the driving state of the automatic pool cleaning device based on the obtained M optical flow values ​​includes: sorting the M optical flow values ​​in descending order to obtain an optical flow value sequence; extracting the first N optical flow values ​​in the optical flow value sequence; calculating the average of the extracted N optical flow values; comparing the average with a predetermined optical flow threshold, if the average is less than the predetermined optical flow threshold, judging that the driving state of the automatic pool cleaning device is a pre-stuck state; if the average is greater than or equal to the predetermined optical flow threshold, judging that the driving state of the automatic pool cleaning device is a non-stuck state.

[0076] For example, the M optical flow values ​​corresponding to the M pairs of feature points in the robot's environment image A at time t-1 and the environment image B at time t are sorted in descending order to obtain an optical flow value sequence. In the two environment images, the largest optical flow value usually represents the most significant movement of the robot from time t-1 to time t. By selecting the largest first N optical flow values ​​in the optical flow value sequence, the dynamic changes corresponding to multiple pairs of pixel points of the robot from time t-1 to time t can be more accurately analyzed. At the same time, selecting the largest first N optical flow values ​​in the optical flow value sequence can also filter out the optical flow values ​​caused by noise to a certain extent, thereby improving the accuracy of judgment. The average value of the first N optical flow values ​​extracted is calculated to evaluate the overall movement of the robot from time t-1 to time t. When the average value is lower than the predetermined optical flow threshold, it means that the displacement of the robot at the bottom of the pool between two adjacent moments is too small, and it may be in a stuck state, also known as a "pre-stuck state". The reason for the pre-stuck state may be that the robot is performing a turning action, so the robot's movement speed is slow, so the displacement generated in a relatively short period of time is small; the reason for the pre-stuck state may also be that the robot touches the wall or other obstacles, causing its movement speed to drop significantly or stop; the reason for the pre-stuck state may also be that the robot is trapped by cables or other obstacles, causing the robot to be unable to move or only able to move in a small range. Therefore, after identifying that the robot is in the pre-stuck state, it is also necessary to judge the movement state of the robot at subsequent continuous moments to determine whether the robot is actually stuck (the specific judgment method of the stuck state will be described in detail below). In the present application, unless otherwise specified, the operations and technical principles described in the above steps S101 to S104 may be collectively referred to as “pre-stuck state determination”.

[0077] In another embodiment, judging the driving state of the automatic pool cleaning device based on the M optical flow values ​​obtained includes: calculating the average value of the M optical flow values ​​obtained; comparing the average value with a predetermined optical flow threshold, if the average value is less than the predetermined optical flow threshold, judging that the driving state of the automatic pool cleaning device is a pre-stuck state; if the average value is greater than or equal to the predetermined optical flow threshold, judging that the driving state of the automatic pool cleaning device is a non-stuck state.

[0078] In another embodiment, if the average value is greater than or equal to the predetermined optical flow threshold, the driving state of the automatic pool cleaning device is judged to be a non-stuck state. It should be noted that when the robot is judged to be in a non-stuck state through the environmental image of the robot at time t-1 and the environmental image of the robot at time t, it is necessary to continuously extract feature point sets from the environmental images of the robot at subsequent consecutive moments, obtain M optical flow values, and judge the driving state of the robot.

[0079] For example, the average value of the M optical flow values ​​corresponding to the M pairs of feature points in the robot's environment image A at time t-1 and the environment image B at time t can be directly calculated to evaluate the overall movement of the robot from time t-1 to time t. When the average value is lower than the predetermined optical flow threshold, it means that the robot's displacement at the bottom of the swimming pool is too small and may be in a pre-stuck state. The pre-stuck state has been described above and will not be repeated here.

[0080] In another embodiment, if the driving state is the pre-stuck state, the detection method further includes: as time goes by, repeating in sequence the steps of acquiring environmental images at two adjacent moments, extracting a feature point set, obtaining M optical flow values, and judging the driving state of the automatic pool cleaning device; if the driving state judged each time is the pre-stuck state, it is determined that the automatic pool cleaning device is in a stuck state.

[0081] For example, if the robot is judged to be in a pre-stuck state through the environment image of the robot at time t-1 and the environment image of the robot at time t, then continue to extract feature point sets from the environment images of the robot at subsequent times t and t+1, obtain M optical flow values ​​and judge whether the driving state of the robot is a pre-stuck state. If the driving state of the robot at times t and t+1 is still a pre-stuck state, it is determined that the robot is in a stuck state. In this case, it means that the movement of the robot is restricted from time t-1 to time t, and the movement of the robot is still restricted from time t to time t+1. It can be determined that the robot is stuck. The principle of collecting environmental images and extracting feature point sets from environmental images is the same or similar to the principle described in steps S101 and S102 above, and will not be repeated here. The principle of calculating optical flow values ​​from feature point sets is the same or similar to the principle described in step S103 above, and will not be repeated here.

[0082] In one embodiment, in order to improve the accuracy of the judgment of the stuck state, the above-mentioned "pre-stuck state judgment" can be repeated multiple times at multiple consecutive adjacent moments. If the robot's driving state is judged to be the pre-stuck state each time, it is determined that the automatic pool cleaning device is in a stuck state. For example, the above steps S101 to 103 can be performed once from time t-1 to time t to determine whether the robot's driving state is the pre-stuck state; the principles of steps S101 to 103 can be performed once from time t to time t+1 to determine whether the robot's driving state is the pre-stuck state; the principles of steps S101 to 103 can be performed once from time t+1 to time t+2 to determine whether the robot's driving state is the pre-stuck state. As time goes by, and so on, it is repeated multiple times. If the robot's driving state is judged to be the pre-stuck state each time, it is determined that the robot is in a stuck state.

[0083] In one embodiment, as described above, the above-mentioned "pre-stuck state judgment" can be repeated multiple times at multiple consecutive adjacent moments, and the number of times the robot's driving state is in the pre-stuck state is counted. If the number is greater than or equal to a threshold, it is determined that the robot is in the stuck state. For example, if the "pre-stuck state judgment" is performed once from time t-1 to time t, from time t to time t+1, from time t+1 to time t+2, and from time t+2 to t+3, respectively, for a total of 4 judgments, of which 3 judgments result in that the robot's driving state is in the pre-stuck state, it can be determined that the robot is in the stuck state.

[0084] The detection method 100 for the stuck state of the automatic pool cleaning device provided in the present application enables the automatic pool cleaning device to extract feature point sets based on two environmental images corresponding to the automatic pool cleaning device at consecutive moments during the cleaning operation at the pool bottom, and determine whether the automatic pool cleaning device is in a stuck state through the optical flow values ​​of the corresponding feature points between the two feature point sets, thereby assisting the automatic pool cleaning device to get out of the jam in time, and further enabling the automatic pool cleaning device to perform the cleaning operation of the pool bottom normally and efficiently, thereby improving the cleaning efficiency of the pool.

[0085] The present application also provides an automatic pool cleaning device. The automatic pool cleaning device includes an image acquisition unit and a control unit, wherein the control unit is configured to: control the image acquisition unit to acquire environmental images at two adjacent moments during the cleaning operation of the automatic pool cleaning device; extract feature point sets from the two acquired environmental images, thereby obtaining two feature point sets, wherein the two feature point sets each include M feature points; calculate the optical flow values ​​of the corresponding feature points between the two feature point sets by a predetermined optical flow algorithm, thereby obtaining M optical flow values; and determine the driving state of the automatic pool cleaning device based on the obtained M optical flow values, wherein the driving state includes: a pre-stuck state and a non-stuck state.

[0086] The technical contents such as the environmental image, the feature point set, the predetermined optical flow algorithm, the M optical flow values ​​and the driving state of the automatic pool cleaning device have been described in detail in the above embodiments and will not be repeated here.

[0087] This embodiment discloses a computer storage medium, in which a computer program is stored. When the computer program is executed by a processor, the control method described above is implemented.

[0088] It should be understood that, in this embodiment, the computer storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0089] It should be noted that the sequence of the above embodiments of the present application is only for description and does not represent the advantages or disadvantages of the embodiments.

[0090] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0091] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0092] In the present application, unless otherwise specified, directional words such as "up" and "down" are generally used with reference to the directions shown in the drawings, or with reference to the vertical, perpendicular or gravity direction; similarly, for ease of understanding and description, "left" and "right" are generally used with reference to the left and right shown in the drawings; "inside" and "outside" refer to the inside and outside relative to the outline of each component itself, but the above directional words are not used to limit the present application.

[0093] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for detecting a stuck state of an automatic pool cleaning device, comprising: Acquire environmental images at two adjacent moments during the cleaning operation of the automatic pool cleaning device; Extracting feature point sets from the two acquired environment images respectively, thereby obtaining two feature point sets, wherein the two feature point sets respectively include M feature points; Calculating the optical flow values ​​of the corresponding feature points between the two feature point sets by using a predetermined optical flow algorithm, thereby obtaining M optical flow values; as well as The driving state of the automatic pool cleaning device is determined based on the M optical flow values ​​obtained, wherein the driving state includes: a pre-stuck state and a non-stuck state.

2. The detection method according to claim 1, wherein: If the driving state is the pre-stuck state, the detection method further includes: as time goes by, repeating in sequence the steps of acquiring environmental images at two adjacent moments, extracting a feature point set, obtaining M optical flow values, and judging the driving state of the automatic pool cleaning device; if the driving state judged each time is the pre-stuck state, it is determined that the automatic pool cleaning device is in a stuck state.

3. The detection method according to claim 1, wherein: The step of judging the driving state of the automatic pool cleaning device based on the obtained M optical flow values ​​includes: Sorting the M optical flow values ​​in descending order to obtain an optical flow value sequence; Extracting the first N optical flow values ​​in the optical flow value sequence; Calculate the average value of the N extracted optical flow values; The average value is compared with a predetermined optical flow threshold value, and if the average value is less than the predetermined optical flow threshold value, it is determined that the driving state of the automatic pool cleaning device is a pre-stuck state.

4. The detection method according to claim 1, wherein: The step of judging the driving state of the automatic pool cleaning device based on the obtained M optical flow values ​​includes: Calculate the average of the M optical flow values ​​obtained; The average value is compared with a predetermined optical flow threshold value, and if the average value is less than the predetermined optical flow threshold value, it is determined that the driving state of the automatic pool cleaning device is a pre-stuck state.

5. The detection method according to claim 3 or 4, wherein: If the average value is greater than or equal to the predetermined optical flow threshold, it is determined that the driving state of the automatic pool cleaning device is a non-stuck state.

6. The detection method according to claim 1, wherein: The extracting feature point sets from the two acquired environment images respectively, thereby obtaining two feature point sets, comprises: Performing grayscale conversion on the two acquired environment images respectively to obtain a plurality of pixel points corresponding to the two extracted environment images respectively; Calculate the first gradient magnitude in the x-axis direction and the second gradient magnitude in the y-axis direction for each pixel point; constructing an autocorrelation matrix corresponding to the pixel point according to the first gradient amplitude and the second gradient amplitude; The feature point set is determined according to the autocorrelation matrix corresponding to each pixel point.

7. The detection method according to claim 6, wherein: The constructing the autocorrelation matrix corresponding to the pixel point according to the first gradient amplitude and the second gradient amplitude comprises: Calculating the first gradient amplitude to obtain a first calculation result; Calculating the second gradient amplitude to obtain a second calculation result; Calculating the first gradient amplitude and the second gradient amplitude to obtain a third calculation result; An autocorrelation matrix corresponding to the pixel is constructed according to the first calculation result, the second calculation result and the third calculation result.

8. The detection method according to claim 6, wherein: Determining the feature point set according to the autocorrelation matrix corresponding to each pixel point includes: Calculate the autocorrelation matrix corresponding to each pixel point to obtain the eigenvalue corresponding to each pixel point; Calculate the eigenvalue corresponding to each pixel to obtain the corner point response corresponding to each pixel; The corner point responsivity corresponding to each pixel point is sorted in descending order to obtain a plurality of sorted corner point responsivities; The sorted plurality of corner point responsiveness are screened according to a first preset value to obtain the feature point set.

9. An automatic pool cleaning device, comprising an image acquisition unit and a control unit, wherein: The control unit is configured to: Control the image acquisition unit to respectively acquire environmental images at two adjacent moments during the cleaning operation of the automatic pool cleaning device; Extracting feature point sets from the two acquired environment images respectively, thereby obtaining two feature point sets, wherein the two feature point sets respectively include M feature points; Calculating the optical flow values ​​of the corresponding feature points between the two feature point sets by using a predetermined optical flow algorithm, thereby obtaining M optical flow values; as well as The driving state of the automatic pool cleaning device is determined based on the M optical flow values ​​obtained, wherein the driving state includes: a pre-stuck state and a non-stuck state.

10. A computer storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

Images