Smudge detection method, robot, and computer-readable storage medium

Abstract

Provided in the present application are a smudge detection method, a robot, and a computer-readable storage medium. The method comprises: when a robot is in an operational state, collecting a first image sequence by means of a first camera of the robot; on the basis of an order in which images in the first image sequence are arranged, sequentially performing the following processing on each first image in the first image sequence: acquiring a difference value that represents a difference between the first image and an image previous thereto, i.e., a second image, if the difference value between the first image and the second image is less than a first difference threshold value, updating the number of first marks by incrementing the current number by one, and if the difference value between the first image and the second image is greater than or equal to the first difference threshold value, updating the number of first marks to zero; and when the number of the first marks exceeds a first smudge threshold value, determining that there is a smudge on the first camera. Therefore, whether a camera is obscured by a smudge is quickly and accurately determined, thereby improving the operational efficiency and safety.

Description

Dirty detection method, robot and computer readable storage medium

[0001] The present application claims priority to the Chinese patent application No. 2024110489639, filed on August 01, 2024, and entitled "Dirty detection method, robot and computer readable storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of robots, and in particular to a dirty detection method, a robot and a computer readable storage medium. BACKGROUND

[0003] Intelligent robots, such as lawn mowing robots, sweeping robots, etc., collect image information of the surrounding environment through a camera, thereby realizing corresponding functions.

[0004] Taking a lawn mowing robot as an example, when the lawn mowing robot is in a working state, splashed mud and weeds may block the camera module, which will hinder the normal work of the camera. After the camera is blocked by dirt, the user needs to discover and handle it in time.

[0005] However, when the lawn mowing robot is in a working state, the user usually does not check whether there is dirt or other blocking objects on the camera at all times. If there is dirt on the camera and it is not discovered and handled in time, it will affect the normal work of the lawn mowing robot and reduce the work efficiency and safety. SUMMARY

[0006] The present application provides a dirty detection method, a robot and a computer readable storage medium to solve the problem that the camera of the robot is blocked by dirt and affects the work efficiency and safety.

[0007] In a first aspect, the present application provides a dirty detection method, comprising:

[0008] collecting a first image sequence by a first camera of a robot when the robot is in a working state;

[0009] in the order of arrangement of images in the first image sequence, sequentially performing the following processing on each first image in the first image sequence: obtaining a difference degree value between the first image and a second image, the second image being a previous image of the first image; if the difference degree value between the first image and the second image is less than a first difference degree threshold, updating the number of first marks to be the current number plus one, the initial value of the number of first marks being zero; if the difference degree value between the first image and the second image is greater than or equal to the first difference degree threshold, updating the number of first marks to be zero;

[0010] determine that the first camera is dirty when a number of the first marks exceeds a first dirt threshold.

[0011] In one embodiment, the obtaining the difference degree value between the first image and the second image comprises:

[0012] obtaining an energy matrix of the first image;

[0013] obtaining an energy matrix of the second image;

[0014] obtaining a distance between the first image and the second image according to the energy matrix of the first image and the energy matrix of the second image;

[0015] determining that the distance is the difference degree value between the first image and the second image.

[0016] In one embodiment, the obtaining the energy matrix of the first image comprises:

[0017] dividing the first image into N first regions of the same size, N being a positive integer;

[0018] obtaining a composite gradient value of each pixel point in the first image according to a pixel value of the pixel point;

[0019] obtaining an energy value of each first region according to the composite gradient value of each pixel point in the first region;

[0020] obtaining the energy matrix of the first image according to the energy value of each first region;

[0021] the obtaining the energy matrix of the second image comprises:

[0022] dividing the second image into N second regions of the same size, the size of the second region being the same as the size of the first region;

[0023] obtaining a composite gradient value of each pixel point in the second image according to a pixel value of the pixel point;

[0024] obtaining an energy value of each second region according to the composite gradient value of each pixel point in the second region;

[0025] obtaining the energy matrix of the second image according to the energy value of each second region.

[0026] In one embodiment, the obtaining the composite gradient value of each pixel point in the first image according to the pixel value of the pixel point comprises:

[0027] determining gradient values of each pixel point in the first image in a horizontal direction according to pixel values of the pixel points in the first image and a horizontal convolution kernel;

[0028] determining gradient values of each pixel point in the first image in a vertical direction according to pixel values of the pixel points in the first image and a vertical convolution kernel;

[0029] for each pixel point in the first image, obtaining a composite gradient value of the pixel point according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction;

[0030] the obtaining of the composite gradient values of the pixel points in the second image according to the pixel values of the pixel points in the second image comprises:

[0031] determining gradient values of each pixel point in the second image in a horizontal direction according to pixel values of the pixel points in the second image and the horizontal convolution kernel;

[0032] determining gradient values of each pixel point in the second image in a vertical direction according to pixel values of the pixel points in the second image and the vertical convolution kernel;

[0033] for each pixel point in the second image, obtaining a composite gradient value of the pixel point according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction.

[0034] In one embodiment, the obtaining of the distance between the first image and the second image according to the energy matrix of the first image and the energy matrix of the second image comprises:

[0035] the distance between the first image and the second image is obtained according to the following formula:

[0036] distance=sqrt(sum((A[i][j]-B[i][j])^2))

[0037] wherein, distance is the distance between the first image and the second image; A[i][j] is an element value in the i-th row and the j-th column of the energy matrix of the first image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is an element value in the i-th row and the j-th column of the energy matrix of the second image; sum() represents a summation operation within the range of i from 0 to the number of rows of the energy matrix and j from 0 to the number of columns of the energy matrix; sqrt() represents a square root function.

[0038] In one embodiment, the method further comprises:

[0039] acquire a second image sequence through a second camera of the robot; the second camera and the first camera belong to a same binocular camera;

[0040] in a sequence of images in the second image sequence, sequentially for each third image in the second image sequence, the following processing is performed: obtaining a difference value between the third image and a fourth image, the fourth image being an image in the first image sequence and being the same as the time of acquiring the third image; if the difference value between the third image and the fourth image is greater than a second difference threshold, the number of second marks is incremented by one, and the initial value of the number of second marks is zero; if the difference value between the third image and the fourth image is less than the second difference threshold, the number of second marks is updated to zero;

[0041] when the number of second marks exceeds a second dirt threshold, it is determined that the binocular camera has dirt.

[0042] In one embodiment, the method further comprises:

[0043] when the robot is in a self-checking state, controlling the binocular camera of the robot to acquire an image pair at M angles, respectively, the binocular camera comprising the first camera and the second camera, M being a positive integer, the image pair comprising a fifth image acquired through the first camera and a sixth image acquired through the second camera at the same time;

[0044] for the image pair acquired at each of the M angles, the following processing is performed: obtaining a difference value between the fifth image in the image pair and the sixth image in the image pair; if the difference value between the fifth image in the image pair and the sixth image in the image pair is greater than a third difference threshold, the number of third marks is incremented by one, and the initial value of the number of third marks is zero;

[0045] when the total number of third marks exceeds a third dirt threshold, it is determined that the binocular camera has dirt.

[0046] In one embodiment, after the determination that the first camera has dirt, the method further comprises:

[0047] outputting dirt alarm information, the dirt alarm information being used to indicate that the first camera has dirt.

[0048] In a second aspect, the embodiments of the present application provide a robot, comprising a first camera and a processor, the processor being configured to execute the method of the first aspect.

[0049] In a third aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the method of the first aspect.

[0050] The present application provides a dirt detection method, a robot and a computer readable storage medium. When the robot is in a working state, a first camera of the robot acquires a first image sequence. According to the order of arrangement of images in the first image sequence, the following processing is sequentially performed on each first image in the first image sequence: obtaining a difference value representing the difference degree between the first image and a previous image, i.e., a second image; if the difference value between the first image and the second image is less than a first difference threshold, updating the number of first marks to be the current number plus one, and the initial value of the number of first marks is zero; if the difference value between the first image and the second image is greater than or equal to the first difference threshold, updating the number of first marks to be zero. When the number of first marks exceeds a first dirt threshold, it is determined that the first camera is dirty. During the working process of the robot, due to the change of the scene around the robot, there is usually a difference between every two adjacent images continuously acquired by the first camera of the robot. If the first camera is blocked by dirt, the difference between every two adjacent images is small. Therefore, by detecting that the difference value between every two adjacent images acquired by the camera in a period of time is less than the first difference threshold, it can be determined that the camera is blocked by dirt, so as to prompt the user to clean the dirt or automatically clean the dirt by the robot. The robot can quickly and accurately determine whether the camera is blocked by dirt during the working process, ensure the normal working of the robot, and improve the working efficiency and safety. In addition, the dirt detection is completed using the existing camera and processing resources, without increasing the hardware cost. BRIEF DESCRIPTION OF DRAWINGS

[0051] FIG. 1 is a flowchart of a dirt detection method according to an embodiment of the present application;

[0052] FIG. 2 is a flowchart of another dirt detection method according to an embodiment of the present application;

[0053] FIG. 3A is a schematic diagram of a convolution kernel according to an embodiment of the present application;

[0054] FIG. 3B is a schematic diagram of another convolution kernel according to an embodiment of the present application;

[0055] FIG. 4 is a flowchart of another dirt detection method according to an embodiment of the present application;

[0056] FIG. 5 is a flowchart of another dirt detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] In the present application, "at least one" means one or more, and "multiple" means two or more. The term "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c alone can represent: a alone, b alone, c alone, combination of a and b, combination of a and c, combination of b and c, or combination of a, b and c, where a, b, c can be single or multiple. In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0058] The terms "center", "longitudinal", "transverse", "upper", "lower", "left", "right", "front", "rear", and the like indicate the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0059] The terms "connected" and "connected" should be broadly understood, for example, the "connected" or "connected" of the circuit structure can mean not only physical connection, but also electrical connection or signal connection, for example, it can be directly connected, that is, physically connected, or indirectly connected through at least one element in the middle, as long as the circuit is connected, it can also be the internal connection of two elements; In addition to signal connection through the circuit, signal connection through media medium, such as radio waves, can also be referred to. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0060] The present application provides a robot, a camera is arranged on the robot, which can be a monocular camera or a binocular camera. It can be understood that the binocular camera includes two cameras. For example, a lawn mower is usually provided with a binocular camera in front of the robot.

[0061] When the robot is working, the camera can collect environmental information around the robot.

[0062] The application provides a dirt detection method, which can be applied to the robot described above. During the operation of the robot, each two adjacent images continuously collected by a camera of the robot usually have differences due to changes in the scene around the robot. If the camera is blocked by dirt, the differences between each two adjacent images are small. Therefore, if the difference values between each two adjacent images collected by the camera in a period of time are all less than a first difference threshold, it can be determined that the camera is blocked by dirt, so that the user can be prompted to clean the dirt, or the robot can automatically clean the dirt. The camera can be quickly and accurately determined whether it is blocked by dirt during the operation of the robot, so that the normal operation of the robot is ensured, and the operation efficiency and safety are improved. In addition, the dirt detection is completed by using the existing camera and processing resources, without increasing the hardware cost.

[0063] The technical solutions and beneficial effects provided by the application will be described in detail below with specific embodiments.

[0064] Please refer to FIG. 1, which is a flowchart of a dirt detection method provided by an embodiment of the application. As shown in FIG. 1, the method provided by the embodiment is executed by a robot provided with a camera, which can be the robot mentioned in the above embodiments. The method provided by the embodiment can include the following steps 101-103.

[0065] Step 101: When the robot is in an operation state, a first image sequence is collected by a first camera of the robot.

[0066] The operation state refers to a state in which the corresponding function of the robot is completed, for example, the lawn mower is in the operation state when it is mowing. The robot in the operation state refers to the robot executing the steps of the embodiment in the process of operation.

[0067] The first camera is provided on the robot. The first camera can be a monocular camera or one of the two cameras of a binocular camera. If the first camera is one of the two cameras of the binocular camera, the two cameras of the binocular camera can execute the method of the embodiment respectively, or one of the cameras can execute the method of the embodiment.

[0068] The first image sequence is an image sequence arranged in the order of collection time. Generally, when the robot is in the operation state, the first camera is also in the working state, so the first camera will collect the image sequence. The first image sequence can be composed of all the images collected by the first camera, or can be obtained by sampling all the images collected by the first camera, which is not limited by the application. It can be understood that the size and resolution of the images in the first image sequence are the same.

[0069] When the robot is in the working state, the corresponding first camera is also in the state of collecting images, and the first image sequence collected by the camera.

[0070] Step 102, for each first image in the first image sequence, the following steps 1021-1024 are performed according to the order of the images in the first image sequence.

[0071] Step 1021, obtain the difference value between the first image and the second image.

[0072] The second image is also an image in the first image sequence, and in the first image sequence, the second image is the previous image of the first image.

[0073] The difference value between the first image and the second image refers to a numerical representation of the degree of difference between the image contents of the first image and the second image, which can indicate the difference between the first image and the second image.

[0074] Step 1022, determine whether the difference value between the first image and the second image is less than the first difference threshold.

[0075] If yes, continue to step 1023; if no, continue to step 1024.

[0076] The first difference threshold is a preset value, and the first difference threshold can be an empirical value. For example, when the first camera has the smallest dirt that affects the normal operation of the robot, the first difference threshold is determined as the difference value between the two consecutive images collected by the first camera at this time.

[0077] Step 1023, update the number of first marks to the current number plus one.

[0078] The first mark is used to mark the first image and the second image with a difference value less than the first difference threshold, so the first mark is used to mark the smaller difference between the current first image and the second image. The initial value of the number of first marks is zero, and during the processing of the sampled first image sequence, the difference value between the current first image and the second image is compared, and when the difference value is less than the first difference threshold, a first mark is obtained. It can be understood that the number of first marks is a cumulative process.

[0079] Further, the first mark can be a first image, a second image, information of the first image, information of the second image, or a number, a character, etc.

[0080] For example, the first mark can be stored in the form of a queue, and the number of first marks can be determined according to the length of the queue.

[0081] Step 1024, update the number of first marks to zero.

[0082] When the robot is in a working state, a first image sequence is collected by a first camera of the robot. The images in the first image sequence collected by the first camera have a sequence, and the images in the first image sequence are arranged according to the sequence of collection. In the processing process, the images in the first image sequence are processed according to the sequence of collection time, that is, when the first camera currently collects a first image, the first image is processed, when a new first image is collected at the next collection time, the first image is continuously processed, and so on. In the process of processing the first image, the difference degree value between the first image and the previous image, that is, the second image, is obtained. If the difference degree value between the first image and the second image is less than the first difference degree threshold, it means that the difference between the first image and the second image is small, and the current first camera may be dirty. A first mark can be obtained, and the number of first marks is updated to the current number plus 1 accordingly. If the difference degree value between the first image and the second image is greater than or equal to the first difference degree threshold, it means that there is a certain degree of difference between the first image and the second image, that is, the current first camera does not exist dirty, and the number of first marks obtained can be cleared. Thus, the number of first marks is cumulative. When the difference degree value between the first image and the second image is less than the first difference degree threshold, the number of first marks is cumulatively added by one. Once the difference degree value between the first image and the second image is greater than or equal to the first difference degree threshold, it can be that the dirt previously existing automatically falls off during the robot working process, or the difference degree of the pictures taken at different positions in the environment where the robot is located is small. At this time, the number of first marks obtained can be cleared.

[0083] Step 103, when the number of first marks exceeds the first dirt threshold, it is determined that the first camera exists dirt.

[0084] The first dirt threshold is a preset positive integer, and the first dirt threshold is used to determine whether the first camera exists dirt through the number of first marks.

[0085] After each time the first image is processed in step 102, it can be determined whether the current number of first marks exceeds the first dirt threshold. If the current number of first marks exceeds the first dirt threshold, it means that the difference degree between the first image and the second image is small in a period of time, and it is determined that the first camera exists dirt. If the number of first marks does not reach the first dirt threshold, the next first image is continuously processed.

[0086] In the embodiment, when the robot is in a working state, a first image sequence is collected by a first camera of the robot, and for each first image in the first image sequence, the following processing is performed in the order of the arrangement of the images in the first image sequence: a difference value representing a difference degree between the first image and a previous image, i.e., a second image, is obtained, if the difference degree between the first image and the second image is less than a first difference degree threshold, the number of first marks is updated to be the current number plus one, and the initial value of the number of first marks is zero; if the difference degree between the first image and the second image is greater than or equal to the first difference degree threshold, the number of first marks is updated to be zero. When the number of first marks exceeds a first dirt threshold, it is determined that the first camera is dirty. In the working process of the robot, due to the change of the scene around the robot, each two adjacent images collected by the first camera of the robot usually have a difference, and if the first camera is blocked by dirt, the difference between each two adjacent images is small. Therefore, if the difference degree values between each two adjacent images collected by the camera in a period of time are all less than the first difference degree threshold, it can be determined that the camera is blocked by dirt, so as to prompt the user to clean the dirt or automatically clean the dirt by the robot. In the working process of the robot, whether the camera is blocked by dirt can be quickly and accurately determined, the normal work of the robot is ensured, and the working efficiency and safety are improved. In addition, the dirt detection is completed by using the existing camera and processing resources, without increasing the hardware cost.

[0087] Referring to FIG. 2, FIG. 2 is a flowchart of another dirt detection method provided by the embodiment. As shown in FIG. 2, the embodiment of FIG. 2 is based on the embodiment of FIG. 1, and further, step 1021 can be implemented by steps 10211-10214.

[0088] In step 10211, an energy matrix of the first image is obtained.

[0089] The energy matrix of the first image is a quantity used to represent the change rate or texture feature of the first image. The energy matrix of the first image usually has a smaller size than the first image.

[0090] In step 10212, an energy matrix of the second image is obtained.

[0091] The energy matrix of the second image is a quantity used to represent the change rate or texture feature of the second image. The energy matrix of the second image usually has a smaller size than the second image. The energy matrix of the first image and the energy matrix of the second image have the same form.

[0092] It should be noted that the execution of step 10211 and step 10212 has no sequence. Step 10211 can be executed first, and step 10212 can be executed; step 10212 can be executed first, and step 10211 can be executed; or step 10211 and step 10212 can be executed simultaneously, and the application does not limit the sequence of execution of step 10211 and step 10212.

[0093] Step 10213, obtaining the distance between the first image and the second image according to the energy matrix of the first image and the energy matrix of the second image.

[0094] According to the energy matrix of the first image and the energy matrix of the second image, the distance between the energy matrix of the first image and the energy matrix of the second image is obtained, that is, the distance between the first image and the second image. The distance can be Euclidean distance, Manhattan distance, etc., which is not limited by the application.

[0095] Further, the distance between the first image and the second image is obtained according to the following formula (1):

[0096] distance=sqrt(sum((A[i][j]-B[i][j])^2)) formula (1)

[0097] Wherein, distance is the distance between the first image and the second image; A[i][j] is the element value of the i-th row and j-th column in the energy matrix of the first image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is the element value of the i-th row and j-th column in the energy matrix of the second image; sum() represents the summation operation in the range of i from 0 to the number of rows of the energy matrix, j from 0 to the number of columns of the energy matrix; sqrt() represents the square root function.

[0098] Step 10214, determining the distance as the difference value between the first image and the second image.

[0099] In the embodiment, the energy matrixes of the first image and the second image are obtained respectively, the texture features of the first image and the second image are obtained respectively, the energy matrix of the first image is compared with the energy matrix of the second image, so that the change rate difference or the texture feature difference between the first image and the second image is compared, and the distance between the first image and the second image is obtained. The distance between the first image and the second image is determined as the difference degree value between the first image and the second image, so that the obtained difference degree value can accurately reflect the difference between the first image and the second image. Since the distance between the energy matrixes is used to obtain the difference between the first image and the second image, the processing speed is fast, and whether there is dirt can be quickly detected in the robot operation process.

[0100] In some embodiments, step 10211 can be implemented by steps 102111-102114.

[0101] Step 102111, dividing the first image into N first regions with the same size, N being a positive integer.

[0102] For the first image, the first image can be divided into N first regions with the same size, the N first regions do not overlap, which is equivalent to macroblock division of the first image according to the same size, and each macroblock is a first region.

[0103] Further, the size of the first region can be predetermined, for example, the size of the first region can be 8x8, that is, a square with 8 pixel points in length and width.

[0104] Step 102112, obtaining the composite gradient value of each pixel point according to the pixel value of each pixel point in the first image.

[0105] For all pixel points in the first image, the composite gradient value of the pixel point is obtained according to the pixel value of the pixel point and the pixel value of the pixel points around the pixel point. The composite gradient value of the pixel point is used to represent the comprehensive change rate of the pixel point in the x and y directions of the image.

[0106] Further, step 102112 can be implemented by the following way:

[0107] According to the pixel value of each pixel point in the first image and the horizontal convolution kernel, the gradient value of each pixel point in the first image in the horizontal direction is determined.

[0108] According to the pixel value of each pixel point in the first image and the vertical convolution kernel, the gradient value of each pixel point in the first image in the vertical direction is determined.

[0109] For each pixel point in the first image, the composite gradient value of the pixel point is obtained according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction.

[0110] In this embodiment, the horizontal convolution kernel can be used to perform convolution operation on the first image along the horizontal direction to obtain the gradient value of each pixel in the first image along the horizontal direction. The vertical convolution kernel can be used to perform convolution operation on the first image along the vertical direction to obtain the gradient value of each pixel in the first image along the vertical direction. For each pixel in the first image, the composite gradient value of the pixel can be obtained according to the gradient value along the horizontal direction and the gradient value along the vertical direction.

[0111] Optionally, the convolution kernel shown in FIG. 3A can be used as the horizontal convolution kernel. The horizontal convolution kernel is used to perform convolution operation on each pixel in the first image and the pixels around the pixel. For each pixel, the pixel values in the 3x3 region around the pixel are multiplied by the elements of the horizontal convolution kernel, and the results are added to obtain the gradient value of the pixel along the horizontal direction. The convolution kernel shown in FIG. 3B can be used as the vertical convolution kernel. The vertical convolution kernel is used to perform convolution operation on each pixel in the first image and the pixels around the pixel. For each pixel, the pixel values in the 3x3 region around the pixel are multiplied by the elements of the vertical convolution kernel, and the results are added to obtain the gradient value of the pixel along the vertical direction.

[0112] Optionally, the composite gradient value can be obtained by the sum of the square of the gradient value along the horizontal direction and the gradient value along the vertical direction. The composite gradient value can be obtained by the following formula (2):

[0113] In the formula, gradient_magnitude is the composite gradient value of the pixel, horizontal_gradient is the gradient value of the pixel along the horizontal direction, vertical_gradient is the gradient value of the pixel along the vertical direction, and sqrt represents the square root function.

[0114] In step 102113, the energy value of each first region can be obtained according to the composite gradient value of each pixel in the first region.

[0115] For each first region, the following processing is performed: the energy value of the first region is obtained according to the composite gradient value of all the pixels in the first region. The energy value of the first region can comprehensively reflect the image energy of the first region.

[0116] Optionally, if the size of the first region is 8x8, the energy value of the first region can be obtained by the following formula (3):

[0117] Where E is the energy value of the first region, Ix is the gradient value of pixel I in the i-th row and j-th column in the x-direction, and Iy is the gradient value of pixel I in the i-th row and j-th column in the y-direction. Let i represent the sum of the composite gradient values ​​of all pixels in the first region, where 1 ≤ i ≤ 8 and 1 ≤ j ≤ 8.

[0118] Step 102114: Obtain the energy matrix of the first image based on the energy values ​​of each first region.

[0119] The energy values ​​of all the first regions are arranged according to the positional relationship between the first regions to obtain the energy matrix of the first image.

[0120] In this embodiment, the first image is divided into N first regions of the same size. The energy value of the first region is obtained by combining the gradient values ​​of the pixels. The energy value can reflect the rate of change and texture features of the first image. The energy matrix of the first image is obtained based on the energy values ​​of all the first regions. The energy matrix of the first image can reflect the rate of change and texture features of the first image, making the subsequent determination of whether there is dirt based on the energy matrix more accurate.

[0121] Furthermore, step 10212 can be achieved through the following steps 102121-102124.

[0122] Step 102121: Divide the second image into N second regions of the same size.

[0123] The second region has the same size as the first region. The method by which the second image is divided into the second region is the same as the method by which the first image is divided into the first region, and will not be repeated here.

[0124] Step 102122: Obtain the composite gradient value of each pixel in the second image based on the pixel value of each pixel in the second image.

[0125] Furthermore, step 102122 can be implemented in the following way:

[0126] Based on the pixel values ​​of each pixel in the second image and the horizontal convolution kernel, determine the gradient value of each pixel in the second image in the horizontal direction.

[0127] Based on the pixel values ​​of each pixel in the second image and the vertical convolution kernel, determine the gradient value of each pixel in the vertical direction in the second image.

[0128] For each pixel in the second image, the composite gradient value of the pixel is obtained based on the gradient value of the pixel in the horizontal direction and the gradient value of the pixel in the vertical direction.

[0129] The step 102123 is similar to the implementation and principle of the step 102113, and details are not described herein.

[0130] The step 102123 obtains an energy value of each second region according to the composite gradient value of each pixel in each second region.

[0131] The step 102123 is similar to the implementation and principle of the step 102113, and details are not described herein.

[0132] The step 102124 obtains an energy matrix of the second image according to the energy value of each second region.

[0133] The step 102124 is similar to the implementation and principle of the step 102114, and details are not described herein.

[0134] In this embodiment, the second image is divided into N second regions of the same size, and the energy value of each second region is obtained by comprehensively using the gradient value of each pixel. The energy value can reflect the change rate and texture feature of the second image. The energy matrix of the second image is obtained according to the energy value of each second region, so that the energy matrix of the second image can reflect the change rate and texture feature of the second image, and the result of determining whether there is dirt according to the energy matrix is more accurate.

[0135] In some embodiments, the step 103 can further include the following step 104.

[0136] The step 104 outputs the dirt alarm information.

[0137] The dirt alarm information is used to indicate that the first camera has dirt. The dirt alarm information can be voice alarm information, for example, the robot can issue “please clean the camera” or “the camera is blocked, please check” and the like. The dirt alarm information can also be a dirt indicator light that is turned on or flashes. The dirt alarm information can also be voice or text form of dirt alarm information sent to a device connected to the robot. The form of the dirt alarm information is not limited in the present application.

[0138] In this embodiment, when it is detected that there is dirt, the dirt alarm information is issued, which can prompt the user to clean the dirt in time and improve the efficiency and safety of the robot.

[0139] In some scenarios, the first camera and the second camera belong to the same binocular camera, and the scenes viewed by the two cameras of the binocular camera at the same time are basically consistent, therefore, the difference between the images obtained by the two cameras is low, if the two cameras are dirty to different degrees, the scenes viewed by the two cameras will be different, so the difference between the images obtained by the two cameras can determine whether the two cameras are dirty. The following will be described in detail with specific embodiments.

[0140] Please refer to FIG. 4, which is a flowchart of another method for detecting dirt provided by an embodiment of the present application. The method steps provided by the present embodiment can be executed simultaneously with the method steps provided by the above-mentioned embodiments, or can be executed after the method steps of the above-mentioned embodiments, or can be executed before the method steps of the above-mentioned embodiments, or can be executed alone. The present application does not make any limitation thereon. For the convenience of understanding, FIG. 4 shows the case of simultaneous execution, which does not constitute a limitation on the present application. The method of the present embodiment comprises the following steps 401-403.

[0141] Step 401: acquiring a second image sequence by a second camera of a robot.

[0142] The second camera and the first camera belong to the same binocular camera. The binocular camera is arranged on the robot. The second image sequence is a sequence arranged in the order of acquisition time by the images acquired by the first camera. Generally, when the robot is in a working state, the binocular camera is also in a working state, the first camera and the second camera simultaneously acquire images, the first camera acquires a first image sequence, and the second camera acquires a second image sequence. The second image sequence can be composed of all images acquired by the second camera, or can be obtained by sampling all images acquired by the second camera, and the present application does not make any limitation thereon. It can be understood that the size and resolution of the images in the second image sequence are the same. It can be understood that each image in the first image sequence has a corresponding image in the second image sequence, that is, the acquisition time of the image in the first image sequence and the corresponding image in the second image sequence is the same.

[0143] Step 402: in the order of arrangement of the images in the second image sequence, sequentially performing the following steps 4021-4024 on each third image in the second image sequence.

[0144] Step 4021: obtaining a difference value between the third image and a fourth image.

[0145] The fourth image is an image in the first image sequence which has the same acquisition time as the third image. It can be understood that the third image and the fourth image are an image pair acquired simultaneously by the binocular camera.

[0146] The difference value between the third image and the fourth image refers to a numerical representation of the difference between the image contents of the third image and the fourth image, and can indicate the difference between the third image and the fourth image.

[0147] Step 4022, determining whether the difference value between the third image and the fourth image is greater than a second difference threshold value.

[0148] If yes, continue to step 4023, and if no, continue to step 4024.

[0149] The second difference threshold value is a preset numerical value, and the second difference threshold value can be an empirical value. For example, when the binocular camera has a small amount of dirt that does not affect the normal operation of the robot, the second difference threshold value is determined as the difference value of the image pair collected by the binocular camera at this time.

[0150] Step 4023, updating the number of second marks to be the current number plus one.

[0151] The initial value of the number of second marks is zero. The second mark is used to mark the third image and the fourth image with a difference value greater than the second difference threshold value, so the second mark is used to mark the current third image and the fourth image with a large difference. The initial value of the number of second marks is zero, and in the process of processing the sampled second image sequence, the difference value between the current third image and the fourth image is compared, and when the difference value is greater than the first difference threshold value, a second mark is obtained. It can be understood that the number of second marks is a cumulative process.

[0152] Further, the second mark can be a third image, a fourth image, information of the third image, information of the fourth image, or a number, a character, etc.

[0153] For example, the second mark can be stored in the form of a queue, and the number of second marks can be determined according to the length of the queue.

[0154] Step 4024, updating the number of second marks to be zero.

[0155] When the robot is in a working state, a first image sequence is collected by a first camera in the binocular camera, and a second image sequence is collected by a second camera in the binocular camera. The images in the second image sequence collected by the second camera have a sequence, and the images in the second image sequence are arranged according to the sequence of collection. In the processing, the images in the second image sequence are processed according to the sequence of collection time, that is, the fourth image is currently collected by the first camera, and the third image is currently collected by the second camera, then the third image and the fourth image are processed, when a new third image and a new fourth image are collected at the next collection time, the new third image and the new fourth image are continuously processed, and so on. In the process of processing the third image and the fourth image, the difference degree value between the third image and the fourth image is obtained. Since the images collected by the binocular camera at the same time are usually similar, if the difference degree value between the third image and the fourth image is greater than a second difference degree threshold, it indicates that the difference between the third image and the fourth image is large, then the first camera and the second camera may be contaminated to different degrees, that is, the binocular camera may be contaminated, and a second mark can be obtained, and the number of the second mark is updated to the current number plus 1. If the difference degree value between the third image and the fourth image is less than or equal to the second difference degree threshold, it indicates that the third image and the fourth image are similar, then the binocular camera is not contaminated, and the number of the second mark obtained can be cleared. Therefore, the number of the second mark is cumulative, and when the difference degree value between the third image and the fourth image is greater than the second difference degree threshold, the number of the second mark is cumulatively added by one. Once it is detected that the difference degree value between the third image and the fourth image is less than or equal to the second difference degree threshold, the existing contamination may automatically fall off during the robot working process, and at this time, the number of the second mark obtained can be cleared.

[0156] In step 403, when the number of the second mark exceeds a second contamination threshold, it is determined that the binocular camera is contaminated.

[0157] The second contamination threshold is a preset positive integer, and the second contamination threshold is used to determine whether the binocular camera is contaminated by the number of the second mark.

[0158] After the third image and the fourth image are processed in step 402 each time, it can be judged whether the current number of the second mark exceeds the second contamination threshold. If the current number of the second mark exceeds the second contamination threshold, it indicates that the difference degree between the third image and the fourth image is small in a period of time, then it is determined that the binocular camera is contaminated. If the number of the second mark does not reach the second contamination threshold, the next group of third image and fourth image is continuously processed.

[0159] In the embodiment, when the robot is in the working state, a first image sequence is collected by a first camera in the binocular camera of the robot, a second image sequence is collected by a second camera, and for each third image in the second image sequence, the following processing is performed in the order of arrangement of the images in the second image sequence: obtaining a difference value representing the difference between the simultaneously collected third image and fourth image, if the difference value between the third image and the fourth image is greater than a second difference threshold, updating the number of second marks to be the current number plus one, and the initial value of the number of second marks being zero; if the difference value between the third image and the fourth image is less than or equal to the second difference threshold, updating the number of second marks to be zero. When the number of second marks exceeds a second dirt threshold, it is determined that the binocular camera is dirty. During the working process of the robot, the scenes seen by the binocular camera of the robot are the same, and therefore the images in the image pair collected at the same time are similar. If the binocular camera is blocked by dirt to different degrees, the two images in the image pair are different. Therefore, if the difference value between the two images in the image pair collected by the binocular camera in a period of time is greater than the second difference threshold, it can be determined that the binocular camera is dirty to different degrees, so as to prompt the user to clean the dirt or automatically clean the dirt by the robot. During the working process of the robot, the binocular camera can be quickly and accurately determined whether it is blocked by dirt, the normal working of the robot is ensured, and the working efficiency and safety are improved. In addition, the dirt detection is completed using the existing binocular camera and processing resources, without increasing the hardware cost. If the method of the above embodiment and the method of the embodiment are executed at the same time, the binocular camera can be more accurately determined whether it is blocked by dirt during the working process of the robot, the normal working of the robot is ensured, and the working efficiency and safety are improved.

[0160] In some embodiments, on the basis of the embodiment shown in FIG. 4, further, step 4021 can be implemented by steps 40211-step 40214 as follows.

[0161] Step 40211, obtaining the energy matrix of the third image.

[0162] The energy matrix of the third image is a quantity used to represent the rate of change or texture feature of the third image. The energy matrix of the third image is usually smaller in size than the third image.

[0163] Step 40212, obtaining the energy matrix of the fourth image.

[0164] The energy matrix of the fourth image is a quantity used to represent the rate of change or texture feature of the fourth image. The energy matrix of the fourth image is usually smaller in size than the fourth image. The energy matrix of the third image and the energy matrix of the fourth image are of the same form.

[0165] It should be noted that the execution of step 40211 and step 40212 has no sequence. Step 40211 can be executed first, and step 40212 can be executed; step 40212 can be executed first, and step 40211 can be executed; or step 40211 and step 40212 can be executed simultaneously, and the application does not limit the sequence of execution of step 40211 and step 40212.

[0166] Step 40213, obtaining the distance between the third image and the fourth image according to the energy matrix of the third image and the energy matrix of the fourth image.

[0167] According to the energy matrix of the third image and the energy matrix of the fourth image, the distance between the energy matrix of the third image and the energy matrix of the fourth image is obtained, that is, the distance between the third image and the fourth image. The distance can be Euclidean distance, Manhattan distance, etc., and the application does not limit it.

[0168] Further, the distance between the third image and the fourth image is obtained according to the following formula (4):

[0169] distance=sqrt(sum((A[i][j]-B[i][j])^2)) formula (4)

[0170] Wherein, distance is the distance between the third image and the fourth image; A[i][j] is the element value of the i-th row and j-th column in the energy matrix of the third image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is the element value of the i-th row and j-th column in the energy matrix of the fourth image; sum() represents the summation operation within the range of i from 0 to the number of rows of the energy matrix, j from 0 to the number of columns of the energy matrix; sqrt() represents the square root function.

[0171] Step 40214, determining the distance as the difference value between the third image and the fourth image.

[0172] In the embodiment, the energy matrixes of the third image and the fourth image are obtained respectively, the texture features of the third image and the fourth image are obtained respectively, the energy matrix of the third image is compared with the energy matrix of the fourth image, and the change rate difference or the texture feature difference between the third image and the fourth image is compared, so that the distance between the third image and the fourth image is obtained. The distance between the third image and the fourth image is determined as the difference degree value between the third image and the fourth image, so that the obtained difference degree value can accurately reflect the difference between the third image and the fourth image. Since the distance between the energy matrixes is used to obtain the difference between the third image and the fourth image, the processing speed is fast, and whether there is dirt can be quickly detected in the robot operation process.

[0173] In some embodiments, step 40211 can be implemented by steps 402111-402114.

[0174] Step 402111, the third image is divided into K third regions with the same size, K is a positive integer.

[0175] For the third image, the third image can be divided into K third regions with the same size, the K third regions do not overlap, which is equivalent to macroblock division of the third image according to the same size, and each macroblock is a third region.

[0176] Further, the size of the third region can be determined in advance, for example, the size of the third region can be 8x8, that is, a square with 8 pixel points in length and width.

[0177] Step 402112, according to the pixel value of each pixel point in the third image, a composite gradient value of each pixel point is obtained.

[0178] For all pixel points in the third image, according to the pixel value of the pixel point and the pixel value of the pixel points around the pixel point, a composite gradient value of the pixel point is obtained. The composite gradient value of the pixel point is used to represent the comprehensive change rate of the pixel point in the x and y directions of the image.

[0179] Further, step 402112 can be implemented by the following way:

[0180] According to the pixel value of each pixel point in the third image and the horizontal convolution kernel, the gradient value of each pixel point in the third image in the horizontal direction is determined.

[0181] According to the pixel value of each pixel point in the third image and the vertical convolution kernel, the gradient value of each pixel point in the third image in the vertical direction is determined.

[0182] For each pixel in the third image, according to the gradient value of the pixel in the horizontal direction and the gradient value of the pixel in the vertical direction, a composite gradient value of the pixel is obtained.

[0183] In this embodiment, the third image can be convolved along the horizontal direction by a horizontal convolution kernel to obtain the gradient value of each pixel in the third image in the horizontal direction. The third image can be convolved along the vertical direction by a vertical convolution kernel to obtain the gradient value of each pixel in the third image in the vertical direction. For each pixel in the third image, according to the gradient value in the horizontal direction and the gradient value in the vertical direction, a composite gradient value of the pixel can be obtained.

[0184] Optionally, the convolution kernel shown in FIG. 3A can be used as the horizontal convolution kernel. The horizontal convolution kernel is convolved with each pixel in the third image and the pixels around the pixel. For each pixel, the pixel values in the 3x3 region around the pixel are multiplied by the horizontal convolution kernel element by element, and the results are added to obtain the gradient value of the pixel in the horizontal direction. The convolution kernel shown in FIG. 3B is used as the vertical convolution kernel. The vertical convolution kernel is convolved with each pixel in the third image and the pixels around the pixel. For each pixel, the pixel values in the 3x3 region around the pixel are multiplied by the vertical convolution kernel element by element, and the results are added to obtain the gradient value of the pixel in the vertical direction.

[0185] Optionally, the composite gradient value can be obtained by the sum of the square of the gradient value of the pixel in the horizontal direction and the gradient value in the vertical direction. The composite gradient value can be obtained by the following formula (5):

[0186] Wherein, gradient_magnitude is the composite gradient value of the pixel, horizontal_gradient is the gradient value of the pixel in the horizontal direction, vertical_gradient is the gradient value of the pixel in the vertical direction, and sqrt represents the square root function.

[0187] Step 402113, according to the composite gradient value of each pixel in each third region, the energy value of each third region is obtained.

[0188] For each third region, the following processing is performed respectively: according to the composite gradient value of all the pixels in the third region, the energy value of the third region is obtained. The energy value of the third region comprehensively reflects the image energy of the third region.

[0189] Optionally, if the size of the third region is 8x8, the energy value of the third region can be obtained by the following formula (6).

[0190] wherein E is the energy value of the third region, Ix is the gradient value of the pixel point I in the x direction of the i-th row and the j-th column, Iy is the gradient value of the pixel point I in the y direction of the i-th row and the j-th column, which represents the sum of the composite gradient values of all pixel points in the third region, 1≤i≤8, 1≤j≤8.

[0191] Step 402114, obtaining the energy matrix of the third image according to the energy values of the third regions.

[0192] The energy values of all third regions are arranged according to the positional relationship between the third regions to obtain the energy matrix of the third image.

[0193] In this embodiment, the third image is divided into K third regions of the same size, and the energy value of each third region is obtained by synthesizing the gradient values of the pixel points, so that the energy value can reflect the change rate and texture features of the third image. The energy matrix of the third image is obtained according to the energy values of all third regions, so that the energy matrix of the third image can reflect the change rate and texture features of the third image, and the result of determining whether there is dirt according to the energy matrix is more accurate.

[0194] Further, step 40212 can be implemented by the following steps 402121-402124.

[0195] Step 402121, dividing the fourth image into K fourth regions of the same size.

[0196] wherein the size of the fourth region is the same as that of the third region. The fourth image is divided into fourth regions in the same way as the third image is divided into third regions, which will not be described here.

[0197] Step 402122, obtaining the composite gradient value of each pixel point in the fourth image according to the pixel value of each pixel point in the fourth image.

[0198] Further, step 402122 can be implemented by the following ways:

[0199] According to the pixel value of each pixel point in the fourth image and the horizontal convolution kernel, the gradient value of each pixel point in the fourth image in the horizontal direction is determined.

[0200] According to the pixel value of each pixel point in the fourth image and the vertical convolution kernel, the gradient value of each pixel point in the fourth image in the vertical direction is determined.

[0201] For each pixel point in the fourth image, a composite gradient value of the pixel point is obtained according to a gradient value of the pixel point in a horizontal direction and a gradient value of the pixel point in a vertical direction.

[0202] The step 402122 has similar implementation manners and principles to those of the step 402112, details are not described herein.

[0203] The step 402123 obtains an energy value of each fourth region according to the composite gradient value of each pixel point in the fourth region.

[0204] The step 402123 has similar implementation manners and principles to those of the step 402113, details are not described herein.

[0205] The step 402124 obtains an energy matrix of the fourth image according to the energy value of each fourth region.

[0206] The step 402124 has similar implementation manners and principles to those of the step 402114, details are not described herein.

[0207] In this embodiment, the fourth image is divided into K fourth regions of the same size, and the energy value of each fourth region is obtained according to the gradient value of each pixel point, so that the energy value can reflect the change rate and texture feature of the fourth image. The energy matrix of the fourth image is obtained according to the energy value of each fourth region, so that the energy matrix of the fourth image can reflect the change rate and texture feature of the fourth image, and the result of determining whether there is dirt according to the energy matrix is more accurate.

[0208] In some embodiments, the step 403 can further include the following step 404.

[0209] The step 404 outputs a dirt alarm information.

[0210] The dirt alarm information is used to indicate that the binocular camera has dirt. The dirt alarm information can be an alarm information in the form of voice, for example, the robot can issue “please clean the camera” or “the camera is blocked, please check” and the like. The dirt alarm information can also be a dirt indicator light that is turned on or flashes, and the dirt alarm information can also be a voice or text form of dirt alarm information sent to a device connected to the robot. The form of the dirt alarm information is not limited in the present application.

[0211] In this embodiment, when it is detected that there is dirt, the dirt alarm information is issued, which can prompt the user to clean the dirt in time, and improve the efficiency and safety of the robot.

[0212] In some scenarios, the robot is in a non-working state, for example, the robot has just finished working, or the robot is about to start working, etc. It is necessary to detect whether the current robot camera is dirty. The following will be described in detail with specific embodiments.

[0213] Please refer to FIG. 5, which is a flowchart of another dirty detection method provided by the embodiments of the present application. The embodiments of the present application can be executed after the method steps of the above embodiments, can be executed before the method steps of the above embodiments, and can be executed alone. The present application does not make any limitation. For the convenience of understanding, FIG. 5 shows the case of execution after the method steps shown in FIG. 1, which does not constitute a limitation on the present application. The method of the present embodiment comprises the following steps 501-503.

[0214] Step 501, when the robot is in a self-checking state, control the binocular camera of the robot to collect an image pair at M angles respectively.

[0215] The self-checking state refers to the state of detecting whether the binocular camera is dirty in the non-working state of the robot. The robot can enter the self-checking state when receiving a self-checking trigger instruction or when the robot work is finished, and correspondingly, exit the self-checking state when the self-checking is finished or when receiving a self-checking end instruction, etc.

[0216] The binocular camera comprises a first camera and a second camera. The binocular camera is arranged on the robot. When the robot is in a self-checking state, the first camera and the second camera simultaneously collect images, the first camera collects a fifth image, and the second camera collects a sixth image. It can be understood that the fifth image and the sixth image have the same size and resolution.

[0217] The image pair comprises the fifth image collected by the first camera and the sixth image collected by the second camera.

[0218] The binocular camera of the robot collects image pairs at M angles respectively. The robot can rotate different angles with the center of the robot as the midpoint, and at each angle, the binocular camera collects the image pair at the current angle. The robot can also not move, and the binocular camera rotates different angles, and at each angle, the binocular camera collects the image pair at the current angle. Wherein, M is a positive integer, for example, the robot rotates 0 degrees, 90 degrees, 180 degrees and 270 degrees at the current position to obtain image pairs. In this scenario, M can be 4.

[0219] Step 502, for the image pair collected at each of the M angles, the following steps 5021-5024 are performed respectively.

[0220] Step 5021, obtaining a difference degree value between the fifth image in the image pair and the sixth image in the image pair.

[0221] The difference degree value between the fifth image and the sixth image refers to a numerical representation of the difference degree between the image contents of the fifth image and the sixth image, and can indicate the difference between the fifth image and the sixth image.

[0222] Step 5022, determining whether the difference degree value between the fifth image in the image pair and the sixth image in the image pair is greater than a third difference degree threshold value.

[0223] If yes, continue to perform step 5023, and if no, continue to perform step 5034.

[0224] The third difference degree threshold value is a preset numerical value, and the third difference degree threshold value can be an empirical value. For example, when the first camera has a small amount of dirt, the third difference degree threshold value is determined as the difference degree value between the images collected by the first camera and the second camera at this time.

[0225] Further, the third difference degree threshold value can be the same as the second difference degree threshold value. The third difference degree threshold value can also be different from the second difference degree threshold value.

[0226] Step 5023, increasing the number of third marks by one.

[0227] The third mark is used to mark the fifth image and the sixth image with a difference degree value greater than the third difference degree threshold value, so that the third mark is used to mark that the difference degree between the current fifth image and the sixth image is large. The initial value of the number of third marks is zero. During the processing of the image pair, the difference degree value between the current fifth image and the sixth image is compared at M angles. When the difference degree value is greater than the third difference degree threshold value, a third mark is obtained. It can be understood that the number of third marks is a cumulative process.

[0228] Further, the third mark can be the fifth image, the sixth image, information of the fifth image, information of the sixth image, or a number, a character, etc.

[0229] For example, the third mark can be stored in the form of a queue, and the number of third marks can be determined according to the length of the queue.

[0230] Step 5024, processing the image pair at the next angle.

[0231] When the robot is in the self-checking state, the binocular camera collects image pairs at M angles, processes each image pair at each of the M angles respectively: obtains a difference degree value between the fifth image and the sixth image in the current image pair. Since the image pairs collected by the binocular camera are usually similar, if the difference degree value between the fifth image and the sixth image is greater than a third difference degree threshold, it means that the difference between the fifth image and the sixth image is large, and the current first camera and the second camera may be contaminated to different degrees, that is, the current binocular camera may be contaminated, and a third mark can be obtained, and the number of the third mark is updated to the current number plus 1. If the difference degree value between the fifth image and the sixth image is less than or equal to the third difference degree threshold, it means that the fifth image and the sixth image are similar, and the current binocular camera is not contaminated, and the next image pair can be processed. Therefore, the number of the third mark is cumulative, and when the difference degree value between the fifth image and the sixth image is greater than the third difference degree threshold, the number of the third mark is cumulatively added by one.

[0232] It should be noted that the execution order of steps 501 and 502 can be to execute step 501 first and then execute step 502. For example, after the image pairs at M angles are collected by step 501, the processing process of the image pairs in step 502 is executed. The execution order of steps 501 and 502 can also be to execute steps 501 and 502 at the same time. For example, after the image pair at one angle is collected in step 501, the processing process in step 502 is executed on the image pair.

[0233] Step 503, when the total number of the third mark exceeds a third contamination threshold, it is determined that the binocular camera is contaminated.

[0234] The third contamination threshold is a preset positive integer, and the third contamination threshold is used to determine whether the binocular camera is contaminated by the number of the third mark.

[0235] After the image pair is processed by step 502 each time, it can be judged whether the current number of the third mark exceeds the third contamination threshold. If the current number of the third mark exceeds the third contamination threshold, it is determined that the binocular camera is contaminated. If the number of the third mark does not reach the third contamination threshold, the next image pair is processed.

[0236] In this embodiment, when the robot is in a self-checking state, the binocular camera is used to collect image pairs at M angles, and the image pairs at each of the M angles are processed respectively: the difference value between the fifth image and the sixth image in the current image pair is obtained, if the difference value between the fifth image and the sixth image is greater than a third difference threshold, the first camera and the second camera may be contaminated to different degrees, that is, the binocular camera may be contaminated, a third mark can be obtained, and the number of the third mark is updated to the current number plus one; if the difference value between the fifth image and the sixth image is less than or equal to the third difference threshold, it is indicated that the fifth image and the sixth image are similar, and the binocular camera is not contaminated, and the next image pair can be processed. When the total number of the third mark exceeds a third contamination threshold, it is determined that the binocular camera is contaminated. Therefore, the user can be prompted to clean the contamination, or the robot can automatically clean the contamination. In the non-working state of the robot, whether the binocular camera is contaminated can be automatically, quickly and accurately determined, so that the subsequent work of the robot can be normally carried out, and the work efficiency and safety are improved. In addition, the contamination detection is completed by using the existing binocular camera and processing resources, without increasing the hardware cost.

[0237] In some embodiments, on the basis of the embodiment shown in Figure 5, further, step 5021 can be implemented by steps 50211-step 50214 as follows.

[0238] Step 50211, obtaining the energy matrix of the fifth image.

[0239] The energy matrix of the fifth image is a quantity used to represent the change rate or texture feature of the fifth image. The energy matrix of the fifth image is usually smaller in size than the fifth image.

[0240] Step 50212, obtaining the energy matrix of the sixth image.

[0241] The energy matrix of the sixth image is a quantity used to represent the change rate or texture feature of the sixth image. The energy matrix of the sixth image is usually smaller in size than the sixth image. The energy matrix of the fifth image and the energy matrix of the sixth image are of the same form.

[0242] It should be noted that the execution of step 50211 and step 50212 has no sequence. Step 50211 can be executed first, and then step 50212 can be executed; or step 50212 can be executed first, and then step 50211 can be executed; or step 50211 and step 50212 can be executed simultaneously, and the application does not limit the sequence of the execution of step 50211 and step 50212.

[0243] Step 50213, obtaining the distance between the fifth image and the sixth image according to the energy matrix of the fifth image and the energy matrix of the sixth image.

[0244] According to the energy matrix of the fifth image and the energy matrix of the sixth image, the distance between the energy matrix of the fifth image and the energy matrix of the sixth image is obtained, and the distance between the energy matrix of the fifth image and the energy matrix of the sixth image is the distance between the fifth image and the sixth image. The distance can be Euclidean distance, Manhattan distance, etc., which is not limited by the present application.

[0245] Further, the distance between the fifth image and the sixth image is obtained according to the following formula (7):

[0246] distance = sqrt(sum((A[i][j]-B[i][j])^2)) formula (7)

[0247] Wherein, distance is the distance between the fifth image and the sixth image; A[i][j] is the element value of the i-th row and j-th column in the energy matrix of the fifth image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is the element value of the i-th row and j-th column in the energy matrix of the sixth image; sum() represents the summation operation within the range of i from 0 to the number of rows of the energy matrix and j from 0 to the number of columns of the energy matrix; sqrt() represents the square root function.

[0248] Step 50214, determining the distance as the difference degree value between the fifth image and the sixth image.

[0249] In this embodiment, the energy matrix of the fifth image and the energy matrix of the sixth image are obtained respectively, and the texture features of the fifth image and the sixth image are obtained respectively. The energy matrix of the fifth image and the energy matrix of the sixth image are compared, so as to compare the change rate difference or the texture feature difference between the fifth image and the sixth image, and obtain the distance between the fifth image and the sixth image. The distance between the fifth image and the sixth image is determined as the difference degree value between the fifth image and the sixth image, so that the obtained difference degree value can accurately reflect the difference between the fifth image and the sixth image. Since the distance between the fifth image and the sixth image is obtained by the distance between the energy matrix, the processing speed is fast, and whether there is dirt can be quickly detected in the robot operation process.

[0250] In some embodiments, step 50211 can be implemented by the following steps 502111-502114.

[0251] Step 502111, divide the fifth image into K third regions with the same size, K is a positive integer.

[0252] For the fifth image, the fifth image can be divided into K third regions with the same size, the K third regions do not overlap, which is equivalent to macroblock division of the fifth image according to the same size, and each macroblock is a third region.

[0253] Further, the size of the third region can be predetermined, for example, the size of the third region can be 8x8, that is, a square with 8 pixels in length and width. The size of the third region can be different from the size of the first region.

[0254] Step 502112, according to the pixel value of each pixel point in the fifth image, obtain the composite gradient value of each pixel point.

[0255] For all pixel points in the fifth image, according to the pixel value of the pixel point and the pixel value of the pixel points around it, the composite gradient value of the pixel point is obtained. The composite gradient value of the pixel point is used to represent the comprehensive change rate of the pixel point in the x and y directions of the image.

[0256] Further, step 502112 can be implemented by the following way:

[0257] According to the pixel value of each pixel point in the fifth image and the horizontal convolution kernel, the gradient value of each pixel point in the fifth image in the horizontal direction is determined.

[0258] According to the pixel value of each pixel point in the fifth image and the vertical convolution kernel, the gradient value of each pixel point in the fifth image in the vertical direction is determined.

[0259] For each pixel point in the fifth image, according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction, the composite gradient value of the pixel point is obtained.

[0260] In this embodiment, the horizontal convolution kernel can be used to perform convolution operation on the fifth image in the horizontal direction to obtain the gradient value of each pixel point in the fifth image in the horizontal direction. The vertical convolution kernel can be used to perform convolution operation on the fifth image in the vertical direction to obtain the gradient value of each pixel point in the fifth image in the vertical direction. For each pixel point in the fifth image, according to the gradient value in the horizontal direction and the gradient value in the vertical direction, the composite gradient value of the pixel point can be obtained.

[0261] Optionally, the convolution kernel shown in FIG. 3A can be used as a horizontal convolution kernel. The horizontal convolution kernel is convolved with each pixel in the fifth image and its surrounding pixels. For each pixel, the pixel values in the surrounding 3x3 area are multiplied by the horizontal convolution kernel element by element, and the results are added to obtain the horizontal gradient value of the pixel point. The convolution kernel shown in FIG. 3B can be used as a vertical convolution kernel. The vertical convolution kernel is convolved with each pixel in the fifth image and its surrounding pixels. For each pixel, the pixel values in the surrounding 3x3 area are multiplied by the vertical convolution kernel element by element, and the results are added to obtain the vertical gradient value of the pixel point.

[0262] Optionally, the composite gradient value can be obtained by the sum of the square of the horizontal gradient value and the vertical gradient value of the pixel point. The composite gradient value can be obtained by the following formula (8):

[0263] where gradient_magnitude is the composite gradient value of the pixel point, horizontal_gradient is the horizontal gradient value of the pixel point, vertical_gradient is the vertical gradient value of the pixel point, and sqrt represents the square root function.

[0264] Step 502113, obtaining the energy value of each fifth region according to the composite gradient value of each pixel point in the fifth region.

[0265] For each fifth region, the following processing is performed: obtaining the energy value of the fifth region according to the composite gradient value of all pixel points in the fifth region. The energy value of the fifth region comprehensively reflects the change rate of the pixel points in the fifth region, and can comprehensively reflect the image energy of the fifth region.

[0266] Optionally, if the size of the fifth region is 8x8, the energy value of the fifth region can be obtained by the following formula (9):

[0267] where E is the energy value of the fifth region, Ix is the gradient value of the pixel point I in the x direction, Iy is the gradient value of the pixel point I in the y direction, and represents the sum of the composite gradient values of all pixel points in the fifth region, 1≤i≤8, 1≤j≤8.

[0268] Step 502114, obtaining the energy matrix of the fifth image according to the energy value of each fifth region.

[0269] All the energy values of the fifth regions are arranged according to the positional relationship between the fifth regions, and an energy matrix of the fifth image is obtained.

[0270] In this embodiment, the fifth image is divided into K fifth regions of the same size, and the energy value of each fifth region is obtained by comprehensively considering the gradient values of the pixel points, so that the energy value can reflect the change rate and texture features of the fifth image. The energy matrix of the fifth image is obtained according to all the energy values of the fifth regions, so that the energy matrix of the fifth image can reflect the change rate and texture features of the fifth image, and the result of determining whether there is dirt according to the energy matrix is more accurate.

[0271] Further, the step 50212 can be implemented by the following steps 502121-502124.

[0272] The step 502121 divides the sixth image into K fourth regions of the same size.

[0273] The size of the sixth region is the same as that of the fifth region. The way of dividing the sixth image into the sixth regions is the same as the way of dividing the fifth image into the fifth regions, which will not be described herein again.

[0274] The step 502122 obtains the composite gradient value of each pixel point in the sixth image according to the pixel value of each pixel point in the sixth image.

[0275] Further, the step 502122 can be implemented in the following manner:

[0276] The gradient value of each pixel point in the sixth image in the horizontal direction is determined according to the pixel value of each pixel point in the sixth image and the horizontal convolution kernel.

[0277] The gradient value of each pixel point in the sixth image in the vertical direction is determined according to the pixel value of each pixel point in the sixth image and the vertical convolution kernel.

[0278] For each pixel point in the sixth image, the composite gradient value of the pixel point is obtained according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction.

[0279] The step 502122 and the implementation manner and principle of the step 502112 are similar, which will not be described herein again.

[0280] The step 502123 obtains the energy value of each fourth region according to the composite gradient value of each pixel point in each fourth region.

[0281] The step 502123 and the implementation manner and principle of the step 502113 are similar, which will not be described herein again.

[0282] In step 502124, an energy matrix of the sixth image is obtained according to the energy values of the fourth regions.

[0283] The implementation and principle of step 502124 are similar to those of step 502114, and thus are not described herein.

[0284] In this embodiment, the sixth image is divided into K fourth regions of the same size, and the energy value of each fourth region is obtained according to the gradient values of the pixels. The energy value can reflect the change rate and texture features of the sixth image. The energy matrix of the sixth image is obtained according to the energy values of all the fourth regions, so that the energy matrix of the sixth image can reflect the change rate and texture features of the sixth image, and the result of determining whether there is dirt according to the energy matrix is more accurate.

[0285] In some embodiments, step 503 can be followed by step 504.

[0286] In step 504, dirty alarm information is output.

[0287] The dirty alarm information is used to indicate that the binocular camera has dirt. The dirty alarm information can be voice alarm information, for example, the robot can issue "please clean the camera" or "the camera is blocked, please check". The dirty alarm information can also be a dirty indicator light that is turned on or flashes. The dirty alarm information can also be voice or text dirty alarm information sent to a device connected to the robot. The form of the dirty alarm information is not limited in this application.

[0288] In this embodiment, when it is detected that there is dirt, dirty alarm information is issued to prompt the user to clean the dirt in time, thereby improving the efficiency and safety of the robot.

[0289] An embodiment of the present application provides a robot, which comprises a first camera and a processor. The processor is configured to execute the method of any of the above embodiments.

[0290] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of any of the above embodiments is implemented.

Claims

1. A method for detecting dirt and grime, characterized in that, include: When the robot is in operation, a first image sequence is acquired through the robot's first camera; According to the order in which the images are arranged in the first image sequence, each first image in the first image sequence is processed as follows: obtain the difference value between the first image and the second image, where the second image is the image preceding the first image; if the difference value between the first image and the second image is less than a first difference threshold, then update the number of first markers to the current number plus one, and the initial value of the number of first markers is zero; if the difference value between the first image and the second image is greater than or equal to the first difference threshold, then update the number of first markers to zero. When the number of the first markers exceeds the first dirt threshold, it is determined that the first camera is dirty.

2. The method according to claim 1, characterized in that, The step of obtaining the difference value between the first image and the second image includes: Obtain the energy matrix of the first image; Obtain the energy matrix of the second image; The distance between the first image and the second image is obtained based on the energy matrix of the first image and the energy matrix of the second image; The distance is determined as the difference value between the first image and the second image.

3. The method according to claim 2, characterized in that, The step of obtaining the energy matrix of the first image includes: The first image is divided into N first regions of the same size, where N is a positive integer; Based on the pixel values ​​of each pixel in the first image, the composite gradient value of each pixel in the first image is obtained; The energy value of each first region is obtained based on the composite gradient value of each pixel in each first region; The energy matrix of the first image is obtained based on the energy values ​​of each first region; The step of obtaining the energy matrix of the second image includes: The second image is divided into N second regions of the same size, and the size of the second region is the same as the size of the first region; Based on the pixel values ​​of each pixel in the second image, the composite gradient value of each pixel in the second image is obtained; The energy value of each second region is obtained based on the composite gradient value of each pixel in each second region; The energy matrix of the second image is obtained based on the energy values ​​of each second region.

4. The method according to claim 3, characterized in that, The step of obtaining the composite gradient value of each pixel in the first image based on the pixel value of each pixel in the first image includes: Based on the pixel values ​​of each pixel in the first image and the horizontal convolution kernel, determine the gradient value of each pixel in the first image in the horizontal direction; Based on the pixel values ​​of each pixel in the first image and the vertical convolution kernel, determine the gradient value of each pixel in the first image in the vertical direction; For each pixel in the first image, the composite gradient value of the pixel is obtained based on the gradient value of the pixel in the horizontal direction and the gradient value of the pixel in the vertical direction. The step of obtaining the composite gradient value of each pixel in the second image based on the pixel value of each pixel in the second image includes: Based on the pixel values ​​of each pixel in the second image and the horizontal convolution kernel, the gradient value of each pixel in the second image in the horizontal direction is determined; Based on the pixel values ​​of each pixel in the second image and the vertical convolution kernel, the gradient value of each pixel in the second image in the vertical direction is determined; For each pixel in the second image, a composite gradient value is obtained based on the pixel's gradient value in the horizontal direction and the pixel's gradient value in the vertical direction.

5. The method according to claim 2, characterized in that, The step of obtaining the distance between the first image and the second image based on the energy matrix of the first image and the energy matrix of the second image includes: The distance between the first image and the second image can be obtained using the following formula: distance=sqrt(sum((A[i][j]-B[i][j])^2)) Where distance is the distance between the first image and the second image; A[i][j] is the element value in the i-th row and j-th column of the energy matrix of the first image, where i is greater than 0 and less than or equal to the number of rows in the energy matrix, and j is greater than 0 and less than or equal to the number of columns in the energy matrix; B[i][j] is the element value in the i-th row and j-th column of the energy matrix of the second image; sum() represents the summation operation performed on i from 0 to the number of rows in the energy matrix, and j from 0 to the number of columns in the energy matrix; sqrt() represents the square root function.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The robot acquires a second image sequence using its second camera; the second camera is the same type of binocular camera as the first camera. According to the order of the images in the second image sequence, each third image in the second image sequence is processed as follows: The difference value between the third image and the fourth image, where the fourth image is in the first image sequence and was acquired at the same time as the third image; if the difference value between the third image and the fourth image is greater than a second difference threshold, the number of second markers is incremented by one, with an initial value of zero; if the difference value between the third image and the fourth image is less than the second difference threshold, the number of second markers is updated to zero. When the number of the second markers exceeds the second dirt threshold, it is determined that the binocular camera is dirty.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: When the robot is in self-test mode, the binocular camera controlling the robot captures image pairs at M angles. The binocular camera includes a first camera and a second camera, where M is a positive integer. The image pair includes a fifth image captured simultaneously by the first camera and a sixth image captured simultaneously by the second camera. For each image pair acquired at each of the M angles, the following processing is performed: the difference value between the fifth image and the sixth image in the image pair is obtained; if the difference value between the fifth image and the sixth image in the image pair is greater than the third difference threshold, the number of the third marker is incremented by one, and the initial value of the number of the third marker is zero. When the total number of third markers exceeds the third dirt threshold, it is determined that the binocular camera is dirty.

8. The method according to any one of claims 1-5, characterized in that, After determining that the first camera is dirty, the process further includes: Output a dirt alarm message, which is used to indicate that the first camera is dirty.

9. A robot, characterized in that, It includes a first camera and a processor, the processor being used to perform the method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images