Contamination detection method, and robot and computer-readable storage medium

Abstract

Provided in the present application are a contamination detection method, and a robot and a computer-readable storage medium. The method comprises: collecting a plurality of image pairs by means of a binocular camera of a robot, wherein a first camera and a second camera in the binocular camera simultaneously collect images to obtain each image pair; acquiring a difference value between a first image and a second image in each image pair; on the basis of the difference value between the first image and the second image, recording a first mark; and when the number of recorded first marks exceeds a preset contamination threshold, determining that contamination is present on the binocular camera. Therefore, it can be quickly and accurately determined whether a binocular camera is obscured by contamination, thereby ensuring the normal operation of a robot, and improving the operational efficiency and safety. In addition, contamination detection is completed by using existing binocular cameras and processing resources, without increasing hardware costs.

Description

Dirty detection method, robot and computer readable storage medium

[0001] The present application claims priority to the Chinese patent application No. 2024110490674, filed on August 01, 2024, and entitled "Dirty detection method, robot and computer readable storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of robots, and in particular to a dirty detection method, a robot and a computer readable storage medium. BACKGROUND

[0003] Intelligent robots, such as lawn mowing robots, sweeping robots, etc., obtain image information of the surrounding environment through binocular cameras, thereby realizing corresponding functions.

[0004] Taking a lawn mowing robot as an example, when the lawn mowing robot is in a working state, splashed mud, weeds, etc. may block the binocular camera, which will hinder the normal work of the binocular camera. After the dirty blocks the binocular camera, the user needs to find and handle it in time.

[0005] However, when the lawn mowing robot is in a working state, the user usually does not check whether there is dirt or other blocking objects on the binocular camera at all times. If there is dirt on the binocular camera and it is not found and handled in time, it will affect the normal work of the lawn mowing robot and reduce the working efficiency and safety. SUMMARY

[0006] The present application provides a dirty detection method, a robot and a computer readable storage medium to solve the problem that the binocular camera of the robot is blocked by dirt and affects the working efficiency and safety.

[0007] In a first aspect, the present application provides a dirty detection method, comprising:

[0008] collecting a plurality of image pairs through a binocular camera of a robot; the binocular camera comprises a first camera and a second camera; the image pair comprises a first image collected through the first camera and a second image collected through the second camera;

[0009] determining the number of first marks according to the difference degree value between the first image and the second image in each image pair; the first mark is obtained when the difference degree value between the first image and the second image in the image pair is greater than a preset difference degree threshold;

[0010] when the number of first marks exceeds a preset dirty threshold, determining that the binocular camera has dirt.

[0011] In one embodiment, the image pair is captured by the binocular camera when the robot is in a working state; and the determining the number of first marks according to the difference value between the first image and the second image in each image pair comprises:

[0012] In order of time sequence of capturing the image pairs, the following processing is sequentially performed for each image pair: obtaining a difference value between the first image in the image pair and the second image in the image pair; if the difference value between the first image and the second image is greater than the preset difference threshold, updating the number of first marks to the current number plus one, and the initial value of the number of first marks is zero; and if the difference value between the first image and the second image is less than or equal to the preset difference threshold, updating the number of first marks to zero.

[0013] In one embodiment, the image pair is captured by the binocular camera at M angles when the robot is in a self-checking state, M being a positive integer; and the determining the number of first marks according to the difference value between the first image and the second image in each image pair comprises:

[0014] For each image pair captured at each of the M angles, the following processing is performed: obtaining a difference value between the first image in the image pair and the second image in the image pair; and if the difference value between the first image and the second image in the image pair is greater than a preset difference threshold, updating the number of first marks to the current number plus one.

[0015] In one embodiment, the obtaining the difference value between the first image in the image pair and the second image in the image pair comprises:

[0016] Obtaining an energy matrix of the first image in the image pair;

[0017] Obtaining an energy matrix of the second image in the image pair;

[0018] According to the energy matrix of the first image and the energy matrix of the second image, obtaining a distance between the first image and the second image;

[0019] Determining the distance as the difference value between the first image and the second image.

[0020] In one embodiment, the obtaining the energy matrix of the first image in the image pair comprises:

[0021] Dividing the first image in the image pair into N first regions of the same size, N being a positive integer;

[0022] According to the pixel value of each pixel point in the first image, a composite gradient value of each pixel point in the first image is obtained;

[0023] According to the composite gradient value of each pixel point in each first region, an energy value of each first region is obtained;

[0024] According to the energy value of each first region, an energy matrix of the first image is obtained;

[0025] The energy matrix of the second image in the image pair is obtained, comprising:

[0026] The second image in the image pair is divided into N second regions of the same size, and the size of the second region is the same as that of the first region;

[0027] According to the pixel value of each pixel point in the second image, a composite gradient value of each pixel point in the second image is obtained;

[0028] According to the composite gradient value of each pixel point in each second region, an energy value of each second region is obtained;

[0029] According to the energy value of each second region, an energy matrix of the second image is obtained.

[0030] In one embodiment, the composite gradient value of each pixel point in the first image is obtained according to the pixel value of each pixel point in the first image, comprising:

[0031] According to the pixel value of each pixel point in the first image and a horizontal convolution kernel, a gradient value of each pixel point in the first image in the horizontal direction is determined;

[0032] According to the pixel value of each pixel point in the first image and a vertical convolution kernel, a gradient value of each pixel point in the first image in the vertical direction is determined;

[0033] For each pixel point in the first image, according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction, a composite gradient value of the pixel point is obtained;

[0034] The composite gradient value of each pixel point in the second image is obtained according to the pixel value of each pixel point in the second image, comprising:

[0035] According to the pixel value of each pixel point in the second image and the horizontal convolution kernel, a gradient value of each pixel point in the second image in the horizontal direction is determined;

[0036] According to the pixel value of each pixel point in the second image and the vertical convolution kernel, a gradient value of each pixel point in the second image in the vertical direction is determined;

[0037] For each pixel point in the second image, a composite gradient value of the pixel point is obtained according to a gradient value of the pixel point in a horizontal direction and a gradient value of the pixel point in a vertical direction.

[0038] In one embodiment, the distance between the first image and the second image is obtained according to the energy matrix of the first image and the energy matrix of the second image, comprising:

[0039] The distance between the first image and the second image is obtained according to the following formula: distance=sqrt(sum((A[i][j]-B[i][j])^2)

[0040] Wherein, distance is the distance between the first image and the second image; A[i][j] is the element value of the i-th row and the j-th column in the energy matrix of the first image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is the element value of the i-th row and the j-th column in the energy matrix of the second image; sum() represents the summation operation in the range of i from 0 to the number of rows of the energy matrix and j from 0 to the number of columns of the energy matrix; sqrt() represents the square root function.

[0041] In one embodiment, after determining that the binocular camera is dirty, the method further comprises:

[0042] Outputting dirty alarm information, the dirty alarm information being used to indicate that the first camera is dirty.

[0043] In a second aspect, the application provides a robot, comprising a binocular camera and a processor, the processor being used to execute the method of the first aspect.

[0044] In a third aspect, the application provides a computer readable storage medium, which stores a computer program, the computer program being executed by a processor to implement the method of the first aspect.

[0045] This application provides a dirt detection method, a robot, and a computer-readable storage medium. The method involves a robot using binocular cameras to acquire multiple image pairs. The first and second cameras in the binocular camera simultaneously capture images, resulting in image pairs. The method obtains the difference value between the first and second images in each image pair and records a first marker based on this difference value. When the number of recorded first markers exceeds a preset dirt threshold, dirt is determined to be present in the binocular camera. Since the two cameras in the robot's binocular camera view the same scene, the two images in each image pair are similar. If the binocular camera is obstructed by dirt to varying degrees, the two images in the image pair will show significant differences. Therefore, by detecting that the difference value between two images in multiple image pairs acquired by the binocular camera exceeds the preset difference threshold, it can be determined that the binocular camera is dirty to varying degrees, thus prompting the user to clean the dirt or allowing the robot to clean it automatically. This method can quickly and accurately determine whether the binocular camera is obstructed by dirt, ensuring normal robot operation and improving operational efficiency and safety. Furthermore, the dirt detection is performed using existing binocular cameras and processing resources, without increasing hardware costs. Attached Figure Description

[0046] Figure 1 is a schematic flowchart of a dirt detection method provided in an embodiment of this application;

[0047] Figure 2 is a flowchart illustrating another dirt detection method provided in an embodiment of this application;

[0048] Figure 3 is a flowchart illustrating another dirt detection method provided in an embodiment of this application;

[0049] Figure 4A is a schematic diagram of a convolution kernel provided in an embodiment of this application;

[0050] Figure 4B is a schematic diagram of another convolution kernel provided in an embodiment of this application. Detailed Implementation

[0051] In the present application, “at least one” means one or more, and “multiple” means two or more. “And / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character “ / ” generally represents an “or” relationship between the associated objects before and after it. “At least one of the following” or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c alone can represent: a alone, b alone, c alone, combination of a and b, combination of a and c, combination of b and c, or combination of a, b and c, where a, b and c can be single or multiple. In addition, the terms “first”, “second” are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0052] The terms “center”, “longitudinal”, “transverse”, “upper”, “lower”, “left”, “right”, “front”, “back” and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and cannot be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0053] The terms “connected” and “connected” should be broadly understood, for example, the “connected” or “connected” of the circuit structure can mean physical connection, but also means electrical connection or signal connection, for example, it can be directly connected, that is, physically connected, or indirectly connected through at least one intermediate element, as long as the circuit is connected, it can also be the internal connection of two elements; Signal connection can be signal connection through circuit, but also means signal connection through media medium, for example, radio waves. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0054] The present application provides a robot, which comprises a binocular camera, and the binocular camera comprises a first camera and a second camera. Taking a lawn mower as an example, the binocular camera is usually arranged in front of the robot. When the robot is working, the binocular camera can obtain environmental information around the robot.

[0055] The application provides a dirt detection method, which can be applied to the robot described above. The method comprises the following steps: collecting a plurality of image pairs by a binocular camera of the robot, wherein the first camera and the second camera of the binocular camera collect images simultaneously to obtain an image pair, obtaining a difference value between the first image and the second image in the image pair, and recording a first mark according to the difference value between the first image and the second image. When the number of recorded first marks exceeds a preset dirt threshold, it is determined that the binocular camera is dirty. Since the two cameras of the binocular camera of the robot see the same scene, the two images in the image pair collected by the binocular camera are similar. If the binocular camera is blocked by dirt to different degrees, the two images in the image pair are quite different. Therefore, if the difference value between the two images in the plurality of image pairs collected by the binocular camera is greater than a preset difference threshold, it can be determined that the binocular camera is dirty to different degrees, so as to prompt the user to clean the dirt or automatically clean the dirt by the robot. The method can quickly and accurately determine whether the binocular camera is blocked by dirt, ensure the normal operation of the robot, and improve the operation efficiency and safety. In addition, the dirt detection is completed by using the existing binocular camera and processing resources, without increasing the hardware cost.

[0056] The technical solutions and beneficial effects provided by the application will be described in detail in specific embodiments.

[0057] Please refer to FIG. 1, which is a flowchart of a dirt detection method provided by an embodiment of the application. As shown in FIG. 1, the method provided by the embodiment is executed by a robot provided with a binocular camera, which can be the robot mentioned in the above embodiments. The method provided by the embodiment can comprise the following steps 101-103.

[0058] Step 101: collecting a plurality of image pairs by a binocular camera of the robot.

[0059] The binocular camera comprises a first camera and a second camera. The image pair comprises a first image collected by the first camera and a second image collected by the second camera. During the image collection, the first camera and the second camera collect images simultaneously, the first camera collects the first image, and the second camera collects the second image. The first image and the second image collected simultaneously form an image pair. It can be understood that the collection time, size and resolution of the first image and the second image in the image pair are the same.

[0060] Step 102: determining the number of first marks according to the difference value between the first image and the second image in each image pair.

[0061] The difference value between the first image and the second image refers to a numerical representation of the degree of difference between the image contents of the first image and the second image, and can indicate the difference between the first image and the second image.

[0062] The first mark is obtained when the difference value between the first image and the second image in the image pair is greater than a preset difference threshold. The preset difference threshold is a preset numerical value, and the preset difference threshold can be an empirical value. For example, when the binocular camera has a small amount of dirt that affects the normal operation of the robot, the preset difference threshold is determined as the difference value of the image pair captured by the binocular camera at this time.

[0063] Each image pair is processed individually to obtain the difference value between the first image in each image pair and the second image in the image pair, and the difference value of the image pair is obtained. The difference value is compared with the preset difference threshold, and if the difference value is greater than the preset difference threshold, it means that the two images in the current image pair have a large difference, and a first mark can be recorded.

[0064] Step 103, when the number of first marks exceeds a preset dirt threshold, it is determined that the binocular camera has dirt.

[0065] The preset dirt threshold is a preset positive integer, and the preset dirt threshold is used to determine whether the binocular camera has dirt by the number of first marks.

[0066] The number of first marks is counted, and if the number of first marks exceeds the preset dirt threshold, it is determined that the binocular camera has dirt. After processing each image pair in step 102, it can be determined whether the current number of first marks exceeds the preset dirt threshold. If the current number of first marks exceeds the preset dirt threshold, it means that the difference between the two images in multiple image pairs is small, and it is determined that the binocular camera has dirt. If the number of first marks does not reach the preset dirt threshold, the next image pair is processed.

[0067] In this embodiment, a plurality of image pairs are collected by the binocular camera of the robot, wherein the first camera and the second camera in the binocular camera collect images at the same time to obtain an image pair, a difference value between the first image and the second image in the image pair is obtained, and the first mark is recorded according to the difference value between the first image and the second image. When the number of recorded first marks exceeds a preset dirt threshold, it is determined that the binocular camera is dirty. Since the two cameras in the binocular camera of the robot see the same scene, the two images in the image pair collected by the binocular camera are similar. If the binocular camera is blocked by dirt to different degrees, there is a large difference between the two images in the image pair. Therefore, by detecting that the difference value between the two images in the plurality of image pairs collected by the binocular camera is greater than a preset difference threshold, it can be determined that the binocular camera is dirty to different degrees, so as to prompt the user to clean the dirt or automatically clean the dirt by the robot. The binocular camera can be quickly and accurately determined whether it is blocked by dirt, ensuring the normal operation of the robot and improving the efficiency and safety of the operation. In addition, the dirt detection is completed using the existing binocular camera and processing resources, without increasing the hardware cost.

[0068] In some scenarios, the robot is in a working state, and the corresponding binocular camera is in an image collecting state. The binocular camera can process the image pair collected by the binocular camera to determine whether the binocular camera is dirty. The following will be described in detail with specific embodiments.

[0069] Please refer to FIG. 2, which is a flowchart of another dirt detection method provided by an embodiment of the present application. This embodiment is based on the embodiment shown in FIG. 1, and further, the image pair is collected by the binocular camera when the robot is in a working state. The plurality of image pairs in this embodiment can be all the image pairs collected by the binocular camera, or can be obtained by sampling the image pairs collected by the binocular camera, which is not limited by the present application. Correspondingly, in step 102, each image pair can be processed in the following steps 1021-1024 in the order of the time when the image pair is collected.

[0070] In step 1021, a difference value between the first image in the image pair and the second image in the image pair is obtained.

[0071] In step 1022, it is determined whether the difference value between the first image and the second image is greater than a preset difference threshold.

[0072] If yes, step 1023 is continued, and if no, step 1024 is continued.

[0073] In step 1023, the number of first marks is updated to the current number plus one.

[0074] The first mark is used to mark the first image and the second image with the difference value greater than the preset difference threshold, so that the first mark is used to mark that the difference between the first image and the second image in the current image pair is small. The initial value of the number of first marks is zero. In the process of processing the collected image pair, the difference value between the first image and the second image in the current image pair is compared. When the difference value is greater than the preset difference threshold, a first mark is obtained. It can be understood that the number of first marks is a cumulative process.

[0075] Further, the first mark can be the first image, the second image, information of the first image, information of the second image, or a number, a character, etc.

[0076] Exemplarily, the first mark can be stored in the form of a queue, and the number of first marks is determined according to the length of the queue.

[0077] Step 1024, update the number of first marks to zero.

[0078] In the embodiment, when the robot is in the working state, the image pair is collected in real time by the binocular camera. The image pair collected by the binocular camera has the sequence of collection time. In the processing process, the image pair is processed according to the sequence of collection time, that is, when the binocular camera currently collects an image pair, the currently collected image pair is processed, when a new image pair is collected at the next collection time, the new image pair is continuously processed, and so on. In the process of processing each image pair, the difference value between the first image and the second image is obtained. Since the image pair collected by the binocular camera is usually similar, if the difference value between the first image and the second image is greater than the preset difference threshold, it indicates that the difference between the first image and the second image is large, and then the current first camera and the second camera may have different degrees of dirt, that is, the current binocular camera may have dirt, and a first mark can be obtained. Accordingly, the number of first marks is updated to the current number plus 1. If the difference value between the first image and the second image is less than or equal to the preset difference threshold, it indicates that the first image and the second image are similar, and then the current binocular camera does not have dirt, and the number of first marks obtained can be cleared. Thus, the number of first marks is cumulative. When the difference value between the first image and the second image is greater than the preset difference threshold, the number of first marks is cumulatively added by one. Once it is detected that the difference value between the first image and the second image is less than or equal to the preset difference threshold, the dirt existing before can be automatically dropped in the robot working process, etc. At this time, the number of first marks obtained can be cleared.

[0079] In the embodiment, when the robot is in the working state, a plurality of image pairs are collected by the binocular camera of the robot, and in sequence for each image pair, the two images in the image pair are processed as follows according to the order of the collection time of the image pair: obtaining a difference value representing the difference between the first image and the second image collected at the same time, if the difference value between the first image and the second image is greater than a preset difference threshold, updating the number of first marks to the current number plus one, and the initial value of the number of first marks is zero; if the difference value between the first image and the second image is less than or equal to the preset difference threshold, updating the number of first marks to zero. When the number of first marks exceeds a preset dirt threshold, it is determined that the binocular camera is dirty. In the process of robot working, because the scene seen by the binocular camera of the robot is the same, the image pairs collected at the same time should be similar, if the binocular camera is blocked by dirt to different degrees, there is a large difference between the two images in the image pair, therefore, by detecting that the difference value between the two images in the image pairs collected by the binocular camera in a period of time is greater than the preset difference threshold, it can be determined that the binocular camera is dirty to different degrees, so as to prompt the user to clean the dirt, or automatically clean the dirt by the robot. In the process of robot working, the binocular camera can be quickly and accurately determined whether it is blocked by dirt, to ensure the normal working of the robot, improve the working efficiency and safety. In addition, the dirt detection is completed using the existing binocular camera and processing resources, without increasing the hardware cost.

[0080] In some scenarios, the robot is in a non-working state, for example, the robot has just completed working, or the robot is about to start working, etc., it is necessary to detect whether the current binocular camera of the robot is dirty. The following will be described in detail with specific embodiments.

[0081] Please refer to FIG. 3, which is a flowchart of another dirt detection method provided by the embodiment of the application. The embodiment is based on the embodiment shown in FIG. 1, further, the image pairs are collected by the binocular camera at M angles when the robot is in a self-checking state, and M is a positive integer. The self-checking state refers to the state of detecting whether the binocular camera is dirty in the non-working state of the robot. The robot can enter the self-checking state when receiving a self-checking trigger instruction or the robot working is completed, etc., and correspondingly, exit the self-checking state when the self-checking is completed or a self-checking end instruction is received, etc.

[0082] Correspondingly, the image pairs collected at the M angles can be processed by the following steps 102a-102d in step 102.

[0083] Step 102a, obtaining a difference value between the first image in the image pair and the second image in the image pair.

[0084] Wherein, when the robot is in the self-checking state, the first camera and the second camera of the binocular camera simultaneously perform image acquisition, the first camera acquires a first image, and the second camera acquires a second image, and the first image and the second image simultaneously acquired are included in the image pair.

[0085] The binocular camera of the robot acquires image pairs at M angles, which can be that the robot rotates at different angles with the center of the robot as the midpoint, at each angle, the binocular camera acquires the image pair at the current angle, or the robot does not move, and the binocular camera rotates at different angles, at each angle, the binocular camera acquires the image pair at the current angle. Wherein, M is a positive integer, for example, the robot acquires image pairs at 0 degrees, 90 degrees, 180 degrees and 270 degrees in the current position, and in this case, M can be 4.

[0086] Step 102b, determining whether the difference value between the first image in the image pair and the second image in the image pair is greater than a preset difference threshold.

[0087] If yes, step 102c is continued to be executed, and if no, step 102d is continued to be executed.

[0088] Step 102c, updating the number of first marks to the current number plus one.

[0089] In the process of processing the image pair, at M angles, the difference value between the first image and the second image in the current image pair is compared, and when the difference value is greater than the preset difference threshold, a first mark is obtained.

[0090] Step 102d, processing the image pair at the next angle.

[0091] It should be noted that the execution order of step 101 and step 102 can be to execute step 101 first and then execute step 102. For example, after the image pairs at M angles are acquired through step 101, the processing process of step 102 on the image pairs is executed. The execution order of step 101 and step 102 can also be to execute step 101 and step 102 at the same time. For example, after the image pair at an angle is acquired in step 101, the processing process in step 102 is executed on the image pair.

[0092] In the embodiment, when the robot is in the self-checking state, the binocular camera collects image pairs at M angles, and processes each image pair at each of the M angles: obtains a difference degree value between the first image and the second image in the current image pair, if the difference degree value between the first image and the second image is greater than a preset difference degree threshold, the current first camera and the second camera may be contaminated to different degrees, that is, the current binocular camera may be contaminated, a first mark can be obtained, and the number of the first mark is updated to the current number plus one; if the difference degree value between the first image and the second image is less than or equal to the preset difference degree threshold, it is indicated that the first image and the second image are similar, and the current binocular camera is not contaminated, and the next image pair can be processed. When the total number of the first mark exceeds a preset contamination threshold, it is determined that the binocular camera is contaminated. Therefore, the user can be prompted to clean the contamination, or the robot can automatically clean the contamination. In the non-working state of the robot, whether the binocular camera is contaminated can be automatically, quickly and accurately determined, the subsequent work of the robot is ensured to be normally performed, and the working efficiency and safety are improved. In addition, the contamination detection is completed by using the existing binocular camera and processing resources, without increasing hardware cost.

[0093] In some embodiments, steps 1021 and 102a can be implemented by steps 10211-10214.

[0094] Step 10211, obtaining an energy matrix of the first image in the image pair.

[0095] The energy matrix of the first image is a quantity used to represent the change rate or texture feature of the first image. The energy matrix of the first image is usually smaller in size than the first image.

[0096] Step 10212, obtaining an energy matrix of the second image in the image pair.

[0097] The energy matrix of the second image is a quantity used to represent the change rate or texture feature of the second image. The energy matrix of the second image is usually smaller in size than the second image. The energy matrix of the first image and the energy matrix of the second image are of the same form.

[0098] It should be noted that steps 10211 and 10212 have no execution sequence. Step 10211 can be executed first, and then step 10212 can be executed; step 10212 can be executed first, and then step 10211 can be executed; or steps 10211 and 10212 can be executed simultaneously. The execution sequence of steps 10211 and 10212 is not limited in the application.

[0099] Step 10213, obtaining the distance between the first image and the second image according to the energy matrix of the first image and the energy matrix of the second image.

[0100] According to the energy matrix of the first image and the energy matrix of the second image, the distance between the energy matrix of the first image and the energy matrix of the second image is obtained, that is, the distance between the first image and the second image. The distance can be Euclidean distance, Manhattan distance, etc., which is not limited by the application.

[0101] Further, the distance between the first image and the second image is obtained according to the following formula (1):

[0102] Wherein, distance is the distance between the first image and the second image; A[i][j] is the element value of the i-th row and j-th column in the energy matrix of the first image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is the element value of the i-th row and j-th column in the energy matrix of the second image; sum() represents the summation operation within the range of i from 0 to the number of rows of the energy matrix and j from 0 to the number of columns of the energy matrix; sqrt() represents the square root function.

[0103] Step 10214, determining the distance as the difference value between the first image and the second image.

[0104] In this embodiment, the energy matrix of the first image and the energy matrix of the second image are obtained respectively, and the texture features of the first image and the second image are obtained respectively. The energy matrix of the first image and the energy matrix of the second image are compared, so as to compare the change rate difference or the texture feature difference between the first image and the second image, and obtain the distance between the first image and the second image. The distance between the first image and the second image is determined as the difference value between the first image and the second image, so that the obtained difference value can accurately reflect the difference between the first image and the second image. Since the distance between the energy matrix is used to obtain the difference between the first image and the second image, the processing speed is fast, and whether there is dirt can be quickly detected in the robot operation process.

[0105] In some embodiments, step 10211 can be implemented by the following steps 102111-102114.

[0106] Step 102111, dividing the first image into N first regions of the same size, N being a positive integer.

[0107] For the first image, the first image can be divided into N first regions of the same size, the N first regions do not overlap, which is equivalent to macroblock division of the first image according to the same size, and each macroblock is a first region.

[0108] Further, the size of the first region can be predetermined, for example, the size of the first region can be 8x8, that is, a square with 8 pixels in length and width.

[0109] Step 102112, obtaining a composite gradient value of each pixel point according to the pixel value of each pixel point in the first image.

[0110] For all pixel points in the first image, the composite gradient value of the pixel point is obtained according to the pixel value of the pixel point and the pixel value of the pixel points around the pixel point. The composite gradient value of the pixel point is used to represent the comprehensive change rate of the pixel point in the x and y directions of the image.

[0111] Further, step 102112 can be implemented by the following way:

[0112] According to the pixel value of each pixel point in the first image and the horizontal convolution kernel, the gradient value of each pixel point in the first image in the horizontal direction is determined.

[0113] According to the pixel value of each pixel point in the first image and the vertical convolution kernel, the gradient value of each pixel point in the first image in the vertical direction is determined.

[0114] For each pixel point in the first image, the composite gradient value of the pixel point is obtained according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction.

[0115] In this embodiment, the horizontal convolution kernel can be used to perform convolution operation on the first image in the horizontal direction to obtain the gradient value of each pixel point in the first image in the horizontal direction. The vertical convolution kernel can be used to perform convolution operation on the first image in the vertical direction to obtain the gradient value of each pixel point in the first image in the vertical direction. For each pixel point in the first image, the composite gradient value of the pixel point can be obtained according to the gradient value in the horizontal direction and the gradient value in the vertical direction.

[0116] Optionally, the convolution kernel shown in FIG. 4A can be used as a horizontal convolution kernel. The horizontal convolution kernel is convolved with each pixel and its surrounding pixels in the first image. For each pixel, the pixel values in the surrounding 3x3 area are multiplied by the horizontal convolution kernel element by element, and the results are added to obtain the horizontal gradient value of the pixel point. The convolution kernel shown in FIG. 4B can be used as a vertical convolution kernel. The vertical convolution kernel is convolved with each pixel and its surrounding pixels in the first image. For each pixel, the pixel values in the surrounding 3x3 area are multiplied by the vertical convolution kernel element by element, and the results are added to obtain the vertical gradient value of the pixel point.

[0117] Optionally, the composite gradient value can be obtained by the sum of the square of the horizontal gradient value and the vertical gradient value of the pixel point. The composite gradient value can be obtained by the following formula (2):

[0118] where gradient_magnitude is the composite gradient value of the pixel point, horizontal_gradient is the horizontal gradient value of the pixel point, vertical_gradient is the vertical gradient value of the pixel point, and sqrt represents the square root function.

[0119] Step 102113, obtaining the energy value of each first region according to the composite gradient value of each pixel point in the first region.

[0120] For each first region, the following processing is performed: obtaining the energy value of the first region according to the composite gradient value of all pixel points in the first region. The energy value of the first region comprehensively reflects the change rate of the pixel points in the first region, and can comprehensively reflect the image energy of the first region.

[0121] Optionally, if the size of the first region is 8x8, the energy value of the first region can be obtained by the following formula (3):

[0122] where E is the energy value of the first region, Ix is the gradient value of the pixel point I in the x direction, Iy is the gradient value of the pixel point I in the y direction, and represents the sum of the composite gradient values of all pixel points in the first region, 1≤i≤8, 1≤j≤8.

[0123] Step 102114, obtaining the energy matrix of the first image according to the energy value of each first region.

[0124] The energy values of all the first regions are arranged according to the positional relationship between the first regions, and an energy matrix of the first image is obtained.

[0125] In this embodiment, the first image is divided into N first regions of the same size, and the energy value of each first region is obtained by comprehensively considering the gradient values of the pixel points, so that the energy value can reflect the change rate and texture features of the first image. The energy matrix of the first image is obtained according to the energy values of all the first regions, so that the energy matrix of the first image can reflect the change rate and texture features of the first image, and the result of determining whether there is dirt according to the energy matrix is more accurate.

[0126] Further, the step 10212 can be implemented by the following steps 102121-102124.

[0127] The step 102121 divides the second image into N second regions of the same size.

[0128] The size of each second region is the same as that of each first region. The second image is divided into second regions in the same way as the first image is divided into first regions, which will not be described here.

[0129] The step 102122 obtains the composite gradient value of each pixel point in the second image according to the pixel value of each pixel point in the second image.

[0130] Further, the step 102122 can be implemented in the following way:

[0131] The gradient value of each pixel point in the second image in the horizontal direction is determined according to the pixel value of each pixel point in the second image and the horizontal convolution kernel.

[0132] The gradient value of each pixel point in the second image in the vertical direction is determined according to the pixel value of each pixel point in the second image and the vertical convolution kernel.

[0133] For each pixel point in the second image, the composite gradient value of the pixel point is obtained according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction.

[0134] The step 102122 and the implementation method and principle of the step 102112 are similar, which will not be described here.

[0135] The step 102123 obtains the energy value of each second region according to the composite gradient value of each pixel point in each second region.

[0136] The step 102123 and the implementation method and principle of the step 102113 are similar, which will not be described here.

[0137] Step 102124, obtaining an energy matrix of the second image according to the energy values of the second regions.

[0138] The step 102124 is similar to the implementation and principle of the step 102114, and details are not repeated here.

[0139] In this embodiment, the second image is divided into N second regions of the same size, and the energy value of each second region is obtained according to the gradient value of each pixel point. The energy value can reflect the change rate and texture features of the second image. The energy matrix of the second image is obtained according to the energy values of all the second regions. Therefore, the energy matrix of the second image can reflect the change rate and texture features of the second image, and the result of determining whether there is dirt according to the energy matrix is more accurate.

[0140] In some embodiments, the step 103 can be followed by the step 104.

[0141] Step 104, outputting the dirt alarm information.

[0142] The dirt alarm information is used to indicate that the first camera has dirt. The dirt alarm information can be voice alarm information, for example, the robot can issue "please clean the camera" or "the camera is blocked, please check" and the like. The dirt alarm information can also be a dirt indicator light that is turned on or flashes. The dirt alarm information can also be voice or text form of dirt alarm information sent to a device connected to the robot. The form of the dirt alarm information is not limited in the present application.

[0143] In this embodiment, when it is detected that there is dirt, the dirt alarm information is issued to prompt the user to clean the dirt in time, thereby improving the efficiency and safety of the robot.

[0144] The embodiment of the present application provides a robot, which comprises a first camera and a processor. The processor is configured to execute the method of any of the above embodiments.

[0145] The embodiment of the present application provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of any of the above embodiments is implemented.

Claims

1. A method of dirt detection, characterized in that The method comprises: acquiring a plurality of image pairs by a binocular camera of a robot; the binocular camera comprises a first camera and a second camera; the image pair comprises a first image acquired by the first camera and a second image acquired by the second camera; determining a number of first marks according to a difference value between the first image and the second image in each image pair; the first mark is obtained when the difference value between the first image and the second image in the image pair is greater than a preset difference threshold; when the number of first marks exceeds a preset dirt threshold, it is determined that the binocular camera is dirty.

2. The method of claim 1, wherein, the image pair is acquired by the binocular camera when the robot is in a working state; the determination of the number of first marks according to the difference value between the first image and the second image in each image pair comprises: in the order of time of acquiring the image pairs, the following processing is sequentially performed on each image pair: obtaining the difference value between the first image in the image pair and the second image in the image pair; if the difference value between the first image and the second image is greater than the preset difference threshold, updating the number of first marks to the current number plus one, and the initial value of the number of first marks is zero; if the difference value between the first image and the second image is less than or equal to the preset difference threshold, updating the number of first marks to zero.

3. The method of claim 1, wherein, the image pair is acquired by the binocular camera at M angles when the robot is in a self-checking state, M being a positive integer; the determination of the number of first marks according to the difference value between the first image and the second image in each image pair comprises: the following processing is performed on the image pair acquired at each of the M angles: obtaining the difference value between the first image in the image pair and the second image in the image pair; if the difference value between the first image and the second image in the image pair is greater than the preset difference threshold, updating the number of first marks to the current number plus one.

4. The method according to claim 2 or 3, characterized in that, the obtaining of the difference value between the first image in the image pair and the second image in the image pair comprises: obtaining an energy matrix of the first image in the image pair; obtaining an energy matrix of the second image in the image pair; obtaining a distance between the first image and the second image according to the energy matrix of the first image and the energy matrix of the second image; determining the distance as the difference value between the first image and the second image.

5. The method of claim 4, wherein, the obtaining of the energy matrix of the first image in the image pair comprises: dividing the first image in the image pair into N first regions of the same size, N being a positive integer; obtaining a composite gradient value of each pixel point in the first image according to a pixel value of each pixel point in the first image; obtaining an energy value of each first region according to the composite gradient value of each pixel point in each first region; obtaining the energy matrix of the first image according to the energy values of the first regions; the obtaining of the energy matrix of the second image in the image pair comprises: divide the second image in the image pair into N second regions of the same size, the size of the second regions being the same as the size of the first regions; obtain a composite gradient value of each pixel point in the second image according to a pixel value of the pixel point in the second image; obtain an energy value of each second region according to the composite gradient value of each pixel point in the second region; obtain an energy matrix of the second image according to the energy values of the second regions.

6. The method of claim 4, wherein, The obtaining of the composite gradient value of each pixel point in the first image according to the pixel value of the pixel point in the first image comprises: determining a gradient value of each pixel point in the first image in a horizontal direction according to the pixel value of the pixel point in the first image and a horizontal convolution kernel; determining a gradient value of each pixel point in the first image in a vertical direction according to the pixel value of the pixel point in the first image and a vertical convolution kernel; for each pixel point in the first image, obtaining a composite gradient value of the pixel point according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction; The obtaining of the composite gradient value of each pixel point in the second image according to the pixel value of the pixel point in the second image comprises: determining a gradient value of each pixel point in the second image in a horizontal direction according to the pixel value of the pixel point in the second image and the horizontal convolution kernel; determining a gradient value of each pixel point in the second image in a vertical direction according to the pixel value of the pixel point in the second image and the vertical convolution kernel; for each pixel point in the second image, obtaining a composite gradient value of the pixel point according to the gradient value of the pixel point in the horizontal direction and the gradient value of the pixel point in the vertical direction.

7. The method of claim 4, wherein, The obtaining of the distance between the first image and the second image according to the energy matrix of the first image and the energy matrix of the second image comprises: obtaining the distance between the first image and the second image according to the following formula: distance = sqrt(sum((A[i][j]-B[i][j])^2)) wherein distance is the distance between the first image and the second image; A[i][j] is an element value in the i-th row and the j-th column in the energy matrix of the first image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, and j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is an element value in the i-th row and the j-th column in the energy matrix of the second image; sum() represents a summation operation within a range of i from 0 to the number of rows of the energy matrix and j from 0 to the number of columns of the energy matrix; and sqrt() represents a square root function.

8. The method of claim 1, wherein, After the determination of the existence of the dirt on the binocular camera, the method further comprises: outputting a dirt alarm information, the dirt alarm information being used for indicating the existence of the dirt on the first camera.

9. A robot, characterized in that The binocular camera and the processor are included, and the processor is configured to execute the method in any one of claims 1 to 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 8.

Images