Dirt detection method, robot and computer readable storage medium

Through binocular cameras, image pairs are collected and the difference value is calculated to detect whether intelligent robot cameras such as mowing robots are blocked by dirt, solving the problem of dirt affecting operation efficiency and safety, and achieving fast and accurate dirt detection and Alarm improves operating efficiency and safety.

CN118982518APending Publication Date: 2024-11-19YOSEMITE SHANGHAI ROBOTICS CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411049067.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

When smart robots such as mowing robots are operating, the binocular camera may be blocked by splashed soil and weeds, resulting in failure to work normally, affecting efficiency and safety, and it is difficult for users to discover and deal with it in a timely manner.

Method used

Multiple image pairs are collected by binocular cameras and the difference value between the image pairs is calculated. If the difference value exceeds the preset threshold, it is determined that the binocular camera is dirty and outputs alarm information or is automatically cleaned.

Benefits of technology

Quickly and accurately detect whether the binocular camera is blocked by dirt, ensure the normal operation of the robot, improve operation efficiency and safety, and do not require increasing hardware costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982518A_ABST
    Figure CN118982518A_ABST
Patent Text Reader

Abstract

The invention provides a dirt detection method, a robot and a computer readable storage medium. The method comprises the steps that a plurality of image pairs are collected through a binocular camera of a robot, a first camera and a second camera in the binocular camera collect images at the same time, the image pairs are obtained, the difference degree value between a first image and a second image in the image pairs is obtained, and according to the difference degree value between the first image and the second image, the first image and the second image are obtained; and recording the first marks, and when the number of the recorded first marks exceeds a preset smudginess threshold value, determining that the binocular camera is smudginess. Therefore, whether the binocular camera is blocked by dirt or not can be quickly and accurately determined, normal operation of the robot is guaranteed, and operation efficiency and safety are improved. In addition, smudginess detection is completed by using an existing binocular camera and processing resources, and hardware cost does not need to be increased.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of robot technology, and in particular to a dirt detection method, a robot and a computer-readable storage medium. Background Art

[0002] Intelligent robots, such as lawn mowing robots and sweeping robots, obtain image information of the surrounding environment through binocular cameras to achieve corresponding functions.

[0003] Take a lawn mower robot as an example. When the lawn mower robot is in operation, the splashed dirt and weeds may block the binocular camera, which will hinder the normal operation of the binocular camera. When dirt blocks the binocular camera, the user needs to find and deal with it in time.

[0004] However, when the lawn mower robot is in operation, users usually do not always check whether there is dirt or other obstructions on the binocular camera. If there is dirt on the binocular camera and it is not discovered and handled in time, it will affect the normal operation of the lawn mower robot and reduce operating efficiency and safety. Summary of the invention

[0005] The present disclosure provides a dirt detection method, a robot and a computer-readable storage medium to solve the problem that the robot's binocular camera is blocked by dirt, thereby affecting the working efficiency and safety.

[0006] In a first aspect, the present disclosure provides a dirt detection method, comprising:

[0007] A plurality of image pairs are collected by a binocular camera of the robot; the binocular camera comprises a first camera and a second camera; the image pairs comprise a first image collected by the first camera and a second image collected by the second camera at the same time;

[0008] Determining the number of first marks according to the difference value between the first image and the second image in each image pair; the first mark is obtained when the difference value between the first image and the second image in the image pair is greater than a preset difference threshold;

[0009] When the number of the first marks exceeds a preset dirt threshold, it is determined that the binocular camera is dirty.

[0010] In one embodiment, the image pairs are collected by the binocular camera when the robot is in an operating state; and determining the number of first marks according to the difference value between the first image and the second image in each image pair includes:

[0011] According to the chronological order of capturing the image pairs, the following processing is performed on each image pair in turn: the difference value between the first image in the image pair and the second image in the image pair is obtained; if the difference value between the first image and the second image is greater than the preset difference threshold, the number of the first marks is updated to the current number plus one, and the initial value of the number of the first marks is zero; if the difference value between the first image and the second image is less than or equal to the preset difference threshold, the number of the first marks is updated to zero.

[0012] In one embodiment, the image pairs are collected by the binocular camera at M angles respectively when the robot is in a self-checking state, where M is a positive integer; and determining the number of first marks according to the difference value between the first image and the second image in each image pair includes:

[0013] The following processing is performed on the image pairs captured at each of the M angles: the difference value between the first image in the image pair and the second image in the image pair is obtained; if the difference value between the first image and the second image in the image pair is greater than a preset difference threshold, the number of the first marks is updated to the current number plus one.

[0014] In one embodiment, obtaining the difference value between the first image in the image pair and the second image in the image pair includes:

[0015] obtaining an energy matrix of a first image in the image pair;

[0016] obtaining an energy matrix of a second image in the image pair;

[0017] Obtaining a distance between the first image and the second image according to an energy matrix of the first image and an energy matrix of the second image;

[0018] The distance is determined as a difference value between the first image and the second image.

[0019] In one embodiment, obtaining the energy matrix of the first image in the image pair includes:

[0020] Dividing the first image in the image pair into N first regions of the same size, where N is a positive integer;

[0021] Obtaining a composite gradient value of each pixel in the first image according to a pixel value of each pixel in the first image;

[0022] Obtaining an energy value of each first region according to a composite gradient value of each pixel point in each first region;

[0023] Obtaining an energy matrix of the first image according to energy values ​​of each first region;

[0024] The acquiring the energy matrix of the second image in the image pair comprises:

[0025] Dividing the second image in the image pair into N second regions of the same size, where the size of the second region is the same as the size of the first region;

[0026] Obtaining a composite gradient value of each pixel in the second image according to a pixel value of each pixel in the second image;

[0027] Obtaining an energy value of each second region according to a composite gradient value of each pixel point in each second region;

[0028] An energy matrix of the second image is obtained according to the energy values ​​of the second regions.

[0029] In one embodiment, obtaining the composite gradient value of each pixel in the first image according to the pixel value of each pixel in the first image includes:

[0030] Determining a gradient value of each pixel in the first image in a horizontal direction according to a pixel value of each pixel in the first image and a horizontal convolution kernel;

[0031] Determining a vertical gradient value of each pixel in the first image according to a pixel value of each pixel in the first image and a vertical convolution kernel;

[0032] For each pixel in the first image, obtaining a composite gradient value of the pixel according to a gradient value of the pixel in a horizontal direction and a gradient value of the pixel in a vertical direction;

[0033] The step of obtaining a composite gradient value of each pixel in the second image according to the pixel value of each pixel in the second image includes:

[0034] Determine the gradient value of each pixel in the second image in the horizontal direction according to the pixel value of each pixel in the second image and the horizontal convolution kernel;

[0035] Determine a gradient value of each pixel in the second image in a vertical direction according to a pixel value of each pixel in the second image and the vertical convolution kernel;

[0036] For each pixel in the second image, a composite gradient value of the pixel is obtained according to the gradient value of the pixel in the horizontal direction and the gradient value of the pixel in the vertical direction.

[0037] In one embodiment, obtaining the distance between the first image and the second image according to the energy matrix of the first image and the energy matrix of the second image includes:

[0038] The distance between the first image and the second image is obtained according to the following formula:

[0039] distance=sqrt(sum((A[i][j]-B[i][j])^2))

[0040] Wherein, distance is the distance between the first image and the second image; A[i][j] is the element value of the i-th row and j-th column in the energy matrix of the first image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, and j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is the element value of the i-th row and j-th column in the energy matrix of the second image; sum() represents the summation operation in the range of i from 0 to the number of rows of the energy matrix and j from 0 to the number of columns of the energy matrix; sqrt() represents the square root function.

[0041] In one embodiment, after determining that the binocular camera is dirty, the method further includes:

[0042] Output dirt alarm information, where the dirt alarm information is used to indicate that the first camera is dirty.

[0043] In a second aspect, the present disclosure provides a robot comprising a binocular camera and a processor, wherein the processor is used to execute the method described in the first aspect.

[0044] In a third aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0045] The present disclosure provides a dirt detection method, a robot and a computer-readable storage medium. A plurality of image pairs are collected by a binocular camera of the robot, wherein a first camera and a second camera in the binocular camera collect images at the same time to obtain an image pair, obtain a difference value between the first image and the second image in the image pair, and record a first mark according to the difference value between the first image and the second image. When the number of recorded first marks exceeds a preset dirt threshold, it is determined that the binocular camera is dirty. Since the scenes seen by the two cameras in the binocular camera of the robot are the same, the pictures of the two images in the image pair collected by the binocular camera are similar. If the binocular camera is blocked by different degrees of dirt, there is a large difference between the two images in the image pair. Therefore, by detecting that the difference value between the two images in the plurality of image pairs collected by the binocular camera is greater than the preset difference threshold, it can be determined that the binocular camera is dirty to different degrees, so that the user can be prompted to clean the dirt, or the robot can automatically clean the dirt. It is possible to quickly and accurately determine whether the binocular camera is blocked by dirt, ensure the normal operation of the robot, and improve the operation efficiency and safety. In addition, dirt detection is completed using existing binocular cameras and processing resources without increasing hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a flow chart of a dirt detection method provided in an embodiment of the present disclosure;

[0047] Figure 2 A schematic diagram of a flow chart of another dirt detection method provided in an embodiment of the present disclosure;

[0048] Figure 3 A schematic diagram of a flow chart of another dirt detection method provided in an embodiment of the present disclosure;

[0049] Figure 4A A schematic diagram of a convolution kernel provided in an embodiment of the present disclosure;

[0050] Figure 4B A schematic diagram of another convolution kernel provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] In the present disclosure, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a alone, b alone, or c alone can represent: a alone, b alone, c alone, a and b in combination, a and c in combination, b and c in combination, or a, b, and c in combination, where a, b, and c can be single or multiple. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0052] The directions or positional relationships indicated by terms such as "center", "longitudinal", "lateral", "up", "down", "left", "right", "front", and "back" are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present disclosure and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present disclosure.

[0053] The terms "connected" and "connected" should be understood in a broad sense. For example, the "connected" or "connected" of a circuit structure may refer to not only a physical connection, but also an electrical connection or a signal connection. For example, it may be a direct connection, i.e. a physical connection, or it may be an indirect connection through at least one intermediate element, as long as the circuit is connected. It may also be the internal connection of two elements. In addition to signal connection through a circuit, signal connection may also refer to signal connection through a media medium, such as radio waves. For those of ordinary skill in the art, the specific meanings of the above terms in this disclosure may be understood according to specific circumstances.

[0054] The present disclosure provides a robot, which includes a binocular camera, and the binocular camera includes a first camera and a second camera. Taking a lawn mowing robot as an example, the binocular camera is usually set in front of the robot. When the robot is working, the binocular camera can obtain environmental information around the robot.

[0055] The present disclosure provides a dirt detection method, which can be applied to the robot introduced above, wherein a plurality of image pairs are collected by the binocular camera of the robot, wherein the first camera and the second camera in the binocular camera collect images at the same time to obtain an image pair, obtain the difference value between the first image and the second image in the image pair, and record the first mark according to the difference value between the first image and the second image, and when the number of the recorded first marks exceeds the preset dirt threshold, it is determined that the binocular camera is dirty. Since the two cameras in the binocular camera of the robot see the same scene, the pictures of the two images in the image pair collected by the binocular camera are similar. If the binocular camera is blocked by different degrees of dirt, there is a large difference between the two images in the image pair. Therefore, by detecting that the difference value between the two images in the plurality of image pairs collected by the binocular camera is greater than the preset difference threshold, it can be determined that the binocular camera is dirty to different degrees, so that the user can be prompted to clean the dirt, or the robot can automatically clean the dirt. It can quickly and accurately determine whether the binocular camera is blocked by dirt, ensure the normal operation of the robot, and improve the operation efficiency and safety. In addition, dirt detection is completed using existing binocular cameras and processing resources without increasing hardware costs.

[0056] The technical solutions and beneficial effects provided by the present disclosure are described in detail below with reference to specific embodiments.

[0057] See also Figure 1 , Figure 1 The following is a flow chart of a dirt detection method provided by an embodiment of the present disclosure. Figure 1 As shown, the method provided in this embodiment is performed by a robot provided with a binocular camera, and the robot may be the robot mentioned in the above embodiment. The method provided in this embodiment may include the following steps 101 to 103.

[0058] Step 101: Collect multiple image pairs through the robot's binocular camera.

[0059] The binocular camera includes a first camera and a second camera. The image pair includes a first image captured by the first camera and a second image captured by the second camera at the same time. In the process of capturing images, the first camera and the second camera simultaneously capture images. The first camera captures the first image and the second camera captures the second image. The first image and the second image captured at the same time are an image pair. It can be understood that the capture time, size, resolution, etc. of the first image and the second image in the image pair are the same.

[0060] Step 102: Determine the number of first marks according to the difference value between the first image and the second image in each image pair.

[0061] The difference value between the first image and the second image refers to a numerical representation of the degree of difference between the image contents of the first image and the second image, and can indicate the difference between the first image and the second image.

[0062] The first mark is obtained when the difference value between the first image and the second image in the image pair is greater than a preset difference threshold. The preset difference threshold is a preset value, and the preset difference threshold may be an empirical value. For example, when the binocular camera has a small amount of dirt that affects the normal operation of the robot, the preset difference threshold is determined as the difference value of the image pair collected by the binocular camera at this time.

[0063] Each image pair is processed separately, and a difference value between the first image in each image pair and the second image in the image pair is obtained to obtain a difference value of the image pair. The difference value is compared with a preset difference threshold. If the difference value is greater than the preset difference threshold, it means that the two images in the current image pair are greatly different, and a first mark can be recorded.

[0064] Step 103: When the number of first marks exceeds a preset dirt threshold, it is determined that the binocular camera is dirty.

[0065] The preset dirt threshold is a preset positive integer, and the preset dirt threshold is used to determine whether the binocular camera is dirty according to the number of the first mark.

[0066] The number of first marks is counted, and if the number of first marks exceeds a preset dirt threshold, it is determined that the binocular camera is dirty. After each image pair is processed in step 102, it is determined whether the current number of first marks exceeds the preset dirt threshold. If the current number of first marks exceeds the preset dirt threshold, it means that the difference between the two images in the multiple image pairs is small, and it is determined that the binocular camera is dirty; if the number of first marks does not reach the preset dirt threshold, the next image pair is processed.

[0067] In this embodiment, a plurality of image pairs are collected by the binocular camera of the robot, wherein the first camera and the second camera in the binocular camera collect images at the same time to obtain an image pair, obtain the difference value between the first image and the second image in the image pair, and record the first mark according to the difference value between the first image and the second image. When the number of the recorded first marks exceeds the preset dirt threshold, it is determined that the binocular camera is dirty. Since the scenes seen by the two cameras in the binocular camera of the robot are the same, the pictures of the two images in the image pair collected by the binocular camera are similar. If the binocular camera is blocked by different degrees of dirt, there is a large difference between the two images in the image pair. Therefore, by detecting that the difference value between the two images in the plurality of image pairs collected by the binocular camera is greater than the preset difference threshold, it can be determined that the binocular camera is dirty to different degrees, so that the user can be prompted to clean the dirt, or the robot can automatically clean the dirt. It is possible to quickly and accurately determine whether the binocular camera is blocked by dirt, ensure the normal operation of the robot, and improve the operation efficiency and safety. In addition, dirt detection is completed using existing binocular cameras and processing resources without increasing hardware costs.

[0068] In some scenarios, the robot is in operation and the corresponding binocular camera is in the state of collecting images. The images collected by the binocular camera can be processed to determine whether the binocular camera is dirty. The following is a detailed description with specific embodiments.

[0069] See also Figure 2 , Figure 2 A flow chart of another dirt detection method provided in an embodiment of the present disclosure. Figure 1 On the basis of the illustrated embodiment, further, the image pairs are collected by the binocular camera when the robot is in the working state. The multiple image pairs in this embodiment can be all the image pairs collected by the binocular camera, or can be obtained by sampling the image pairs collected by the binocular camera, which is not limited by the present disclosure. Accordingly, in step 102, the following steps 1021 to 1024 can be performed for each image pair in sequence according to the time sequence of collecting the image pairs.

[0070] Step 1021: Obtain a difference value between a first image in the image pair and a second image in the image pair.

[0071] Step 1022: Determine whether the difference between the first image and the second image is greater than a preset difference threshold.

[0072] If yes, continue to step 1023, if no, continue to step 1024.

[0073] Step 1023: Update the number of first marks to the current number plus one.

[0074] The first mark is used to mark the first image and the second image whose difference value is greater than the preset difference threshold, so that the first mark is used to mark that the difference between the first image and the second image in the current image pair is small. The initial value of the number of first marks is zero. In the process of processing the acquired image pair, the difference value between the first image and the second image in the current image pair is continuously compared. When the difference value is greater than the preset difference threshold, a first mark is obtained. It can be understood that the number of first marks is a cumulative process.

[0075] Furthermore, the first mark may be the first image, the second image, information of the first image, information of the second image, or numbers, characters, and the like.

[0076] Exemplarily, the first markers may be stored in the form of a queue, and the number of the first markers may be determined according to the length of the queue.

[0077] Step 1024: Update the number of first marks to zero.

[0078] In this embodiment, when the robot is in the working state, the image pairs are collected in real time through the binocular camera. The image pairs collected by the binocular camera have a sequence of collection time. During the processing, the image pairs are processed according to the sequence of collection time, that is, if the binocular camera currently collects an image pair, the currently collected image pair is processed, and a new image pair is collected at the next collection moment, and the new image pair is processed, and so on. In the process of processing each image pair, the difference value between the first image and the second image is obtained. Since the image pairs collected by the binocular camera at the same time usually have similar pictures, if the difference value between the first image and the second image is greater than the preset difference threshold, it means that the difference between the first image and the second image is large, and the current first camera and the second camera may be dirty to different degrees, that is, the current binocular camera may be dirty, and a first mark can be obtained, and the number of the first marks is updated accordingly to the current number plus 1; if the difference value between the first image and the second image is less than or equal to the preset difference threshold, it means that the first image is similar to the second image, and the current binocular camera is not dirty, and the number of the first marks that have been obtained can be cleared. Therefore, the number of first marks is cumulative. When the difference value between the first image and the second image is greater than the preset difference threshold, the number of first marks is accumulated and added by one. Once it is detected that the difference value between the first image and the second image is less than or equal to the preset difference threshold, it may be that the previously existing dirt automatically fell off during the operation of the robot, etc. At this time, the number of first marks that have been obtained can be cleared.

[0079] In this embodiment, when the robot is in operation, multiple image pairs are collected by the robot's binocular camera, and the two images in each image pair are processed in sequence according to the order of the image pair collection time: a difference value representing the difference between the first image and the second image collected at the same time is obtained, and if the difference value between the first image and the second image is greater than the preset difference threshold, the number of first marks is updated to the current number plus one, and the initial value of the number of first marks is zero; if the difference value between the first image and the second image is less than or equal to the preset difference threshold, the number of first marks is updated to zero. When the number of first marks exceeds the preset dirt threshold, it is determined that the binocular camera is dirty. During the robot operation process, since the scenes seen by the robot's binocular camera are the same, the image pairs collected at the same time should be similar. If the binocular camera is blocked by different degrees of dirt, there are large differences between the two images in the image pair. Therefore, by detecting that the difference value between the two images in the image pair collected by the binocular camera is greater than the preset difference threshold within a period of time, it can be determined that the binocular camera is dirty to different degrees, so that the user can be prompted to clean the dirt, or the robot can automatically clean the dirt. During the robot operation, it can quickly and accurately determine whether the binocular camera is blocked by dirt, ensuring the normal operation of the robot and improving the operation efficiency and safety. In addition, dirt detection is completed using existing binocular cameras and processing resources without increasing hardware costs.

[0080] In some scenarios, when the robot is in a non-operating state, for example, when the robot has just completed an operation or is about to start an operation, it is necessary to detect whether the binocular camera of the current robot is dirty.

[0081] See also Figure 3 , Figure 3 A flow chart of another dirt detection method provided in an embodiment of the present disclosure. Figure 1 On the basis of the illustrated embodiment, further, the image pairs are collected by the binocular camera at M angles respectively when the robot is in the self-check state, where M is a positive integer. The self-check state refers to the state in which the binocular camera detects whether there is dirt when the robot is in a non-operating state. The robot can enter the self-check state when a self-check trigger instruction is received or the robot operation is completed, and correspondingly, exit the self-check state when the self-check is completed or a self-check end instruction is received.

[0082] Correspondingly, in step 102, the image pairs collected at M angles may be processed in the following steps 102a to 102d respectively.

[0083] Step 102a: Obtain a difference value between a first image in the image pair and a second image in the image pair.

[0084] Among them, when the robot is in the self-inspection state, the first camera and the second camera of the binocular camera simultaneously capture images, the first camera captures the first image, and the second camera captures the second image, and the image pair includes the first image and the second image captured simultaneously.

[0085] The robot's binocular camera collects image pairs at M angles respectively. The robot can rotate to different angles on the ground with the center of the robot as the midpoint. At each angle, the binocular camera collects the image pair at the current angle. Alternatively, the robot does not move, and the binocular camera rotates to different angles. At each angle, the binocular camera collects the image pair at the current angle. Wherein, M is a positive integer. For example, the robot obtains image pairs when rotating 0 degrees, 90 degrees, 180 degrees, and 270 degrees at the current position. In this scenario, M can be 4.

[0086] Step 102b: determine whether the difference value between the first image in the image pair and the second image in the image pair is greater than a preset difference threshold.

[0087] If yes, then continue to step 102c, if no, then continue to step 102d.

[0088] Step 102c: Update the number of the first mark to the current number plus one.

[0089] In the process of processing the image pair, the difference values ​​between the first image and the second image in the current image pair are compared at M angles, and a first mark is obtained when the difference value is greater than a preset difference threshold.

[0090] Step 102d: Process the image pair at the next angle.

[0091] It should be noted that the execution order of step 101 and step 102 may be to execute step 101 first and then execute step 102. For example, after image pairs at M angles are acquired by step 101, the processing of the image pairs in step 102 may be executed. The execution order of step 101 and step 102 may also be to execute step 101 and step 102 simultaneously. For example, after an image pair at one angle is acquired by step 101, the processing in step 102 may be executed for the image pair.

[0092] In this embodiment, when the robot is in a self-checking state, the binocular camera collects image pairs at M angles, and processes the image pairs at each of the M angles respectively: obtain the difference value between the first image and the second image in the current image pair, if the difference value between the first image and the second image is greater than the preset difference threshold, the current first camera and the second camera may be dirty to varying degrees, that is, the current binocular camera may be dirty, and a first mark can be obtained, and the number of first marks is updated accordingly to be the current number plus one; if the difference value between the first image and the second image is less than or equal to the preset difference threshold, it means that the first image is similar to the second image, and the current binocular camera is not dirty, and the next image pair can be processed. When the total number of first marks exceeds the preset dirt threshold, it is determined that the binocular camera is dirty. Thus, the user can be prompted to clean the dirt, or the robot can automatically clean the dirt. In the non-operating state of the robot, it can automatically, quickly and accurately determine whether the binocular camera is blocked by dirt, ensuring that the subsequent operation of the robot is carried out normally, thereby improving the operation efficiency and safety. In addition, dirt detection is completed using existing binocular cameras and processing resources without increasing hardware costs.

[0093] In some embodiments, step 1021 and step 102a may be implemented by steps 10211 to 10214 as follows.

[0094] Step 10211: Obtain the energy matrix of the first image in the image pair.

[0095] The energy matrix of the first image is used to represent the rate of change or the amount of texture features of the first image. The energy matrix of the first image is usually smaller in size than the first image.

[0096] Step 10212: Obtain the energy matrix of the second image in the image pair.

[0097] The energy matrix of the second image is used to represent the rate of change or the amount of texture features of the second image. The energy matrix of the second image is usually smaller than the energy matrix of the first image. The energy matrix of the first image is in the same form as the energy matrix of the second image.

[0098] It should be noted that there is no order in which step 10211 and step 10212 are executed. Step 10211 may be executed first, and then step 10212; step 10212 may be executed first, and then step 10211; step 10211 and step 10212 may also be executed simultaneously. The present disclosure does not limit the order in which step 10211 and step 10212 are executed.

[0099] Step 10213: Obtain the distance between the first image and the second image according to the energy matrix of the first image and the energy matrix of the second image.

[0100] According to the energy matrix of the first image and the energy matrix of the second image, the distance between the energy matrix of the first image and the energy matrix of the second image is obtained, and the distance between the energy matrix of the first image and the energy matrix of the second image is the distance between the first image and the second image. The distance can be Euclidean distance, Manhattan distance, etc., which is not limited in the present disclosure.

[0101] Furthermore, the distance between the first image and the second image is obtained according to the following formula (1):

[0102] distance = sqrt(sum((A[i][j] - B[i][j])^2)) Formula (1)

[0103] Wherein, distance is the distance between the first image and the second image; A[i][j] is the element value of the i-th row and j-th column in the energy matrix of the first image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, and j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is the element value of the i-th row and j-th column in the energy matrix of the second image; sum() represents the summation operation in the range of i from 0 to the number of rows of the energy matrix and j from 0 to the number of columns of the energy matrix; sqrt() represents the square root function.

[0104] Step 10214: Determine the distance as the difference value between the first image and the second image.

[0105] In this embodiment, the energy matrices of the first image and the second image are obtained respectively, and the texture features of the first image and the second image are obtained respectively. The energy matrix of the first image and the energy matrix of the second image are compared, thereby comparing the difference in the rate of change or the difference in the texture features between the first image and the second image, and obtaining the distance between the first image and the second image. The distance between the first image and the second image is determined as the difference value between the first image and the second image, so that the obtained difference value can accurately reflect the difference between the first image and the second image. Since the difference between the first image and the second image is obtained by the distance between the energy matrices, the processing speed is fast, and the presence of dirt can be quickly detected during the robot operation process.

[0106] In some embodiments, step 10211 can be implemented by the following steps 102111-102114.

[0107] Step 102111: Divide the first image into N first regions of the same size, where N is a positive integer.

[0108] For the first image, the first image may be divided into N first regions of the same size, and the N first regions do not overlap, which is equivalent to dividing the first image into macroblocks of the same size, and each macroblock is a first region.

[0109] Furthermore, the size of the first area may be predetermined. For example, the size of the first area may be 8×8, that is, a square whose length and width are both 8 pixels.

[0110] Step 102112: Obtain a composite gradient value of each pixel according to the pixel value of each pixel in the first image.

[0111] For all pixels in the first image, a composite gradient value of the pixel is obtained according to the pixel value of the pixel and the pixel values ​​of the surrounding pixels, wherein the composite gradient value of the pixel is used to represent the comprehensive change rate of the pixel in the x and y directions of the image.

[0112] Further, step 102112 can be implemented as follows:

[0113] According to the pixel value of each pixel in the first image and the horizontal convolution kernel, the gradient value of each pixel in the first image in the horizontal direction is determined.

[0114] According to the pixel value of each pixel in the first image and the vertical convolution kernel, the gradient value of each pixel in the first image in the vertical direction is determined.

[0115] For each pixel in the first image, a composite gradient value of the pixel is obtained according to the gradient value of the pixel in the horizontal direction and the gradient value of the pixel in the vertical direction.

[0116] In this embodiment, the horizontal convolution kernel may be used to perform a convolution operation on the first image in the horizontal direction to obtain the horizontal gradient value of each pixel in the first image. The vertical convolution kernel may be used to perform a convolution operation on the first image in the vertical direction to obtain the vertical gradient value of each pixel in the first image. For each pixel in the first image, the composite gradient value of the pixel may be obtained based on the horizontal gradient value and the vertical gradient value.

[0117] Optionally, you can Figure 4A The convolution kernel shown is used as the horizontal convolution kernel. The horizontal convolution kernel is convolved with each pixel of the first image and its surrounding pixels. For each pixel, the pixel values ​​in the surrounding 3×3 area are element-wise multiplied with the horizontal convolution kernel, and the results are added to obtain the horizontal gradient value of the pixel. Figure 4BThe convolution kernel shown is used as a vertical convolution kernel. The vertical convolution kernel is convolved with each pixel of the first image and its surrounding pixels. For each pixel, the pixel values ​​in the surrounding 3×3 area are element-wise multiplied with the vertical convolution kernel, and the results are added to obtain the vertical gradient value of the pixel.

[0118] Optionally, the composite gradient value may be obtained by summing the square of the gradient value of the pixel point in the horizontal direction and the gradient value in the vertical direction. The composite gradient value may be obtained by the following formula (2):

[0119] gradient_magnitude=sqrt(horizontal_gradient^2 + vertical_gradient^2) formula (2)

[0120] Among them, gradient_magnitude is the composite gradient value of the pixel, horizontal_gradient is the gradient value of the pixel in the horizontal direction, vertical_gradient is the gradient value of the pixel in the vertical direction, and sqrt represents the square root function.

[0121] Step 102113: Obtain the energy value of each first region according to the composite gradient value of each pixel point in each first region.

[0122] Each first region is processed as follows: the energy value of the first region is obtained according to the conforming gradient values ​​of all pixels in the first region. The energy value of the first region integrates the change rate of the pixels in the first region and can comprehensively reflect the image energy of the first region.

[0123] Optionally, if the size of the first region is 8×8, the energy value of the first region can be obtained by the following formula (3).

[0124]

[0125] Where E is the energy value of the first region, I x is the gradient value of pixel I in the i-th row and j-th column in the x direction, I y is the gradient value of pixel I in the y direction at the i-th row and j-th column, represents the sum of the composite gradient values ​​of all pixels in the first area, 1≤i≤8, 1≤j≤8.

[0126] Step 102114: Obtain an energy matrix of the first image according to the energy values ​​of each first region.

[0127] The energy values ​​of all the first regions are arranged according to the positional relationship between the first regions to obtain an energy matrix of the first image.

[0128] In this embodiment, the first image is divided into N first areas of the same size and the energy value of the first area is comprehensively obtained through the gradient value of the pixel point, so that the energy value can reflect the change rate and texture characteristics of the first image. The energy matrix of the first image is obtained according to the energy values ​​of all the first areas, so that the energy matrix of the first image can reflect the change rate and texture characteristics of the first image, so that the subsequent result of determining whether there is dirt based on the energy matrix is ​​more accurate.

[0129] Further, step 10212 can be implemented through the following steps 102121-102124.

[0130] Step 102121: Divide the second image into N second regions of the same size.

[0131] The size of the second area is the same as the size of the first area. The second image is divided into the second areas in the same manner as the first image is divided into the first areas, which will not be described in detail here.

[0132] Step 102122: Obtain a composite gradient value of each pixel in the second image according to the pixel value of each pixel in the second image.

[0133] Further, step 102122 can be implemented as follows:

[0134] According to the pixel value of each pixel in the second image and the horizontal convolution kernel, the gradient value of each pixel in the second image in the horizontal direction is determined.

[0135] According to the pixel value of each pixel in the second image and the vertical convolution kernel, the gradient value of each pixel in the second image in the vertical direction is determined.

[0136] For each pixel in the second image, a composite gradient value of the pixel is obtained according to the gradient value of the pixel in the horizontal direction and the gradient value of the pixel in the vertical direction.

[0137] Among them, the implementation method and principle of step 102122 are similar to those of the above-mentioned step 102112, and will not be repeated here.

[0138] Step 102123: Obtain the energy value of each second region according to the composite gradient value of each pixel point in each second region.

[0139] Among them, the implementation method and principle of step 102123 are similar to those of the above-mentioned step 102113, and will not be repeated here.

[0140] Step 102124: Obtain an energy matrix of the second image according to the energy values ​​of each second region.

[0141] Among them, the implementation method and principle of step 102124 are similar to those of the above-mentioned step 102114, and will not be repeated here.

[0142] In this embodiment, the second image is divided into N second areas of the same size and the energy value of the second area is comprehensively obtained through the gradient value of the pixel point, so that the energy value can reflect the change rate and texture characteristics of the second image. The energy matrix of the second image is obtained according to the energy values ​​of all the second areas, so that the energy matrix of the second image can reflect the change rate and texture characteristics of the second image, so that the subsequent result of determining whether there is dirt based on the energy matrix is ​​more accurate.

[0143] In some embodiments, step 103 may be followed by step 104.

[0144] Step 104: Output contamination alarm information.

[0145] The dirt alarm information is used to indicate that the first camera is dirty. The dirt alarm information may be an alarm information in the form of voice, for example, the robot may send a message such as "please clean the camera" or "the camera is blocked, please check", etc. The dirt alarm information may also be a dirt indicator light that turns on or flashes, or a dirt alarm information in the form of voice or text sent to a device connected to the robot. The present disclosure does not limit the form of the dirt alarm information.

[0146] In this embodiment, when dirt is detected, a dirt alarm message is issued to prompt the user to clean the dirt in time, thereby improving the operating efficiency and safety of the robot.

[0147] An embodiment of the present disclosure provides a robot, comprising a first camera and a processor, wherein the processor is used to execute the method of any of the above embodiments.

[0148] An embodiment of the present disclosure provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method of any of the above embodiments is implemented.

Claims

1. A dirt detection method, characterized in that: include: Collect multiple image pairs through the robot's binocular camera; The binocular camera includes a first camera and a second camera; the image pair includes a first image captured by the first camera and a second image captured by the second camera at the same time; Determining the number of first marks according to the difference value between the first image and the second image in each image pair; The first mark is obtained when the difference value between the first image and the second image in the image pair is greater than a preset difference threshold; When the number of the first marks exceeds a preset dirt threshold, it is determined that the binocular camera is dirty.

2. The method according to claim 1, characterized in that The image pair is collected by the binocular camera when the robot is in an operating state; The step of determining the number of first marks according to the difference value between the first image and the second image in each image pair comprises: According to the chronological order of capturing the image pairs, the following processing is performed on each image pair in turn: the difference value between the first image in the image pair and the second image in the image pair is obtained; if the difference value between the first image and the second image is greater than the preset difference threshold, the number of the first marks is updated to the current number plus one, and the initial value of the number of the first marks is zero; if the difference value between the first image and the second image is less than or equal to the preset difference threshold, the number of the first marks is updated to zero.

3. The method according to claim 1, characterized in that: The image pairs are collected by the binocular camera at M angles when the robot is in a self-checking state, where M is a positive integer; The step of determining the number of first marks according to the difference value between the first image and the second image in each image pair comprises: The following processing is performed on the image pairs captured at each of the M angles: the difference value between the first image in the image pair and the second image in the image pair is obtained; if the difference value between the first image and the second image in the image pair is greater than a preset difference threshold, the number of the first marks is updated to the current number plus one.

4. The method according to claim 2 or 3, characterized in that: The obtaining of a difference value between a first image in the image pair and a second image in the image pair comprises: obtaining an energy matrix of a first image in the image pair; obtaining an energy matrix of a second image in the image pair; Obtaining a distance between the first image and the second image according to an energy matrix of the first image and an energy matrix of the second image; The distance is determined as a difference value between the first image and the second image.

5. The method according to claim 4, characterized in that The obtaining of the energy matrix of the first image in the image pair comprises: Dividing the first image in the image pair into N first regions of the same size, where N is a positive integer; Obtaining a composite gradient value of each pixel in the first image according to a pixel value of each pixel in the first image; Obtaining an energy value of each first region according to a composite gradient value of each pixel point in each first region; Obtaining an energy matrix of the first image according to energy values ​​of each first region; The acquiring the energy matrix of the second image in the image pair comprises: Dividing the second image in the image pair into N second regions of the same size, where the size of the second region is the same as the size of the first region; Obtaining a composite gradient value of each pixel in the second image according to a pixel value of each pixel in the second image; Obtaining an energy value of each second region according to a composite gradient value of each pixel point in each second region; An energy matrix of the second image is obtained according to the energy values ​​of the second regions.

6. The method according to claim 4, characterized in that The step of obtaining a composite gradient value of each pixel in the first image according to the pixel value of each pixel in the first image includes: Determining a gradient value of each pixel in the first image in a horizontal direction according to a pixel value of each pixel in the first image and a horizontal convolution kernel; Determining a vertical gradient value of each pixel in the first image according to a pixel value of each pixel in the first image and a vertical convolution kernel; For each pixel in the first image, obtaining a composite gradient value of the pixel according to a gradient value of the pixel in a horizontal direction and a gradient value of the pixel in a vertical direction; The step of obtaining a composite gradient value of each pixel in the second image according to the pixel value of each pixel in the second image includes: Determine the gradient value of each pixel in the second image in the horizontal direction according to the pixel value of each pixel in the second image and the horizontal convolution kernel; Determine a gradient value of each pixel in the second image in a vertical direction according to a pixel value of each pixel in the second image and the vertical convolution kernel; For each pixel in the second image, a composite gradient value of the pixel is obtained according to the gradient value of the pixel in the horizontal direction and the gradient value of the pixel in the vertical direction.

7. The method according to claim 4, characterized in that The obtaining the distance between the first image and the second image according to the energy matrix of the first image and the energy matrix of the second image includes: The distance between the first image and the second image is obtained according to the following formula: distance=sqrt(sum((A[i][j]-B[i][j])^2)) Wherein, distance is the distance between the first image and the second image; A[i][j] is the element value of the i-th row and j-th column in the energy matrix of the first image, i is greater than 0 and less than or equal to the number of rows of the energy matrix, and j is greater than 0 and less than or equal to the number of columns of the energy matrix; B[i][j] is the element value of the i-th row and j-th column in the energy matrix of the second image; sum() represents the summation operation in the range of i from 0 to the number of rows of the energy matrix and j from 0 to the number of columns of the energy matrix; sqrt() represents the square root function.

8. The method according to claim 1, characterized in that After determining that the binocular camera is dirty, the method further includes: Output dirt alarm information, where the dirt alarm information is used to indicate that the first camera is dirty.

9. A robot, characterized in that: The method comprises a binocular camera and a processor, wherein the processor is used to execute the method described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Citation Information

Cited By

  • Contamination detection method, and robot and computer-readable storage medium

    WO2026026440A1