Method and device for detecting camera field of view interference, intelligent robot and storage medium

Abstract

The embodiment of the application relates to the field of intelligent robots, and discloses a camera visual field interference detection method and device and an intelligent robot. The method comprises the following steps: acquiring first continuous frame pictures and second continuous frame pictures collected by a first camera and a second camera respectively in the same time period; performing brightness value counting, blur value counting, definition difference calculation and black pixel point counting on the first continuous frame pictures and the second continuous frame pictures respectively, so as to determine whether the first camera and the second camera are faulty and the specific fault reason. The application realizes interference detection on the camera picture brightness, blurriness and shielding condition of the intelligent robot, so as to realize camera fault detection.

Description

Technical Field

[0001] This application relates to the field of intelligent robot technology, and in particular to a method, apparatus, intelligent robot, and storage medium for detecting camera field of view interference. Background Technology

[0002] Currently, cameras are among the most commonly used sensors in autonomous driving applications due to their ability to acquire complete color information of the surrounding environment, high frequency, and low cost. Compared to everyday photographic cameras, intelligent robots used in autonomous driving applications are better adapted and optimized for extreme weather conditions, such as using more advanced waterproofing technology to ensure normal use in rain, and hydrophobic coatings on the lens surface to help water quickly leave the lens surface.

[0003] In implementing the embodiments of this application, the inventors discovered that existing hardware optimization measures cannot guarantee that the camera lens is always clear. Although the time a water droplet spends on the lens surface is significantly shorter than in traditional cameras, the moment the droplet contacts the lens, it still causes blurring of the field of view. Furthermore, the high frequency of camera captures results in multiple frames being blurry and unclear. Blurry and unclear images interfere with existing computer vision technologies to varying degrees, and in severe cases, can lead to algorithm errors. Besides blurring, issues such as camera field of view obstruction or excessive darkness also interfere with the camera's field of view. Moreover, the interference and its accuracy cannot be detected or reported, easily leading to misjudgments in extreme cases and posing safety hazards. Summary of the Invention

[0004] The purpose of this application is to provide a method, device, intelligent robot, and storage medium for detecting camera field of view interference, thereby enabling the detection of interference such as brightness, blur, and occlusion in the camera image of the intelligent robot, and thus enabling camera fault detection.

[0005] To solve the above-mentioned technical problems, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a method for detecting camera field-of-view interference, applied to an intelligent robot. The intelligent robot includes a first camera and a second camera, the first camera and the second camera facing the same direction. The method includes:

[0007] Acquire the first consecutive frame image and the second consecutive frame image captured by the first camera and the second camera respectively within the same time period;

[0008] Calculate the brightness value of each image in the first consecutive frame image and the second consecutive frame image respectively, and count the number of images in the first consecutive frame image whose brightness value is lower than the brightness threshold as the first count, and the number of images in the second consecutive frame image whose brightness value is lower than the brightness threshold as the second count;

[0009] The blur value of each image in the first consecutive frame image and the second consecutive frame image is calculated respectively. The number of images in the first consecutive frame image with blur values ​​lower than the blur threshold is counted as the third count, and the number of images in the second consecutive frame image with blur values ​​lower than the blur threshold is counted as the fourth count.

[0010] The sharpness difference between the images in the first consecutive frame images and the images in the second consecutive frame images at the same time is calculated sequentially.

[0011] If the sharpness difference is greater than the sharpness threshold, then the third count or the fourth count corresponding to the image with lower sharpness in the first consecutive frame image and the second consecutive frame image at the same time is incremented by one;

[0012] The black pixel difference between the images in the first consecutive frame image and the images in the second consecutive frame image at the same time is calculated sequentially. When the black pixel difference is greater than the black pixel threshold, the number of images in the first consecutive frame image with a number of black pixels greater than the corresponding number of black pixels in the second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image with a number of black pixels greater than the corresponding number of black pixels in the first consecutive frame image is counted as the sixth count.

[0013] The faults of the first camera and the second camera are determined based on the first count, the second count, the third count, the fourth count, the fifth count, and the sixth count.

[0014] In some embodiments, the step of determining the fault of the first camera and the second camera based on the first count, the second count, the third count, the fourth count, the fifth count, and the sixth count includes:

[0015] If the first count is greater than the first threshold, then the field of view of the first camera is determined to be too dark;

[0016] If the second count is greater than the first threshold, then the field of view of the second camera is determined to be too dark;

[0017] If the third count is greater than the second threshold, then the first camera is determined to be blurry;

[0018] If the fourth count is greater than the second threshold, then the second camera is determined to be blurry;

[0019] If the fifth count is greater than the third threshold, then it is determined that the first camera is blocked;

[0020] If the sixth count is greater than the third threshold, then it is determined that the second camera is blocked;

[0021] If the first count and the second count are both less than the first threshold, the third count and the fourth count are both less than the second threshold, and the fifth count and the sixth count are both less than the third threshold, then it is determined that both the first camera and the second camera are fault-free.

[0022] In some embodiments, the step of calculating the blur value of each image in the first consecutive frame image and the second consecutive frame image respectively, and counting the number of images in the first consecutive frame image with blur values ​​lower than a blur threshold as a third count, and the number of images in the second consecutive frame image with blur values ​​lower than a blur threshold as a fourth count, includes:

[0023] Edge detection is performed on each image in the first consecutive frame image and the second consecutive frame image respectively, and blur values ​​are obtained;

[0024] The number of images in the first consecutive frames whose blur value is lower than the blur threshold is counted as the third count;

[0025] The number of images in the second consecutive frame that have a blur value below the blur threshold is counted as the fourth count.

[0026] In some embodiments, the step of sequentially calculating the sharpness difference between the images in the first consecutive frame images and the images in the second consecutive frame images at the same time includes:

[0027] Laplacian transform is performed on the images in the first consecutive frame images and the images in the second consecutive frame images at the same time to obtain the first Laplacian transform values ​​of all pixels in the first consecutive frame images and the second Laplacian transform values ​​of all pixels in the second consecutive frame images.

[0028] Calculate the first variance of the first Laplace transform value and the second variance of the second Laplace transform value;

[0029] The difference between the first variance and the second variance is calculated to obtain the sharpness difference.

[0030] In some embodiments, the step of sequentially calculating the black pixel difference between the images in the first consecutive frame image and the images in the second consecutive frame image at the same time, and when the black pixel difference is greater than a black pixel threshold, counting the number of images in the first consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the second consecutive frame image as a fifth count, and counting the number of images in the second consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the first consecutive frame image as a sixth count, includes:

[0031] Calculate the R, G, and B values ​​of each pixel in the first consecutive frame images and the second consecutive frame images at the same time.

[0032] If the R, G, and B values ​​of the pixels in the first or second consecutive frame images are all within the corresponding preset threshold range, then the pixels whose R, G, and B values ​​are within the preset pixel range are determined as black pixels.

[0033] The number of black pixels in the images in the first consecutive frames and the images in the second consecutive frames is counted, and the difference in the number of black pixels in the images in the first consecutive frames and the images in the second consecutive frames at the same time is obtained.

[0034] When the difference in black pixels is greater than the black pixel threshold, the number of images in the first consecutive frame image where the number of black pixels is greater than the number of black pixels in the corresponding second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image where the number of black pixels is greater than the number of black pixels in the corresponding first consecutive frame image is counted as the sixth count.

[0035] Secondly, this application also provides a camera field-of-view interference detection device for use in an intelligent robot, the intelligent robot including a first camera and a second camera, the first camera and the second camera facing the same direction, the device comprising:

[0036] The acquisition module is used to acquire the first consecutive frame image and the second consecutive frame image captured by the first camera and the second camera respectively within the same time period;

[0037] The brightness counting module is used to calculate the brightness value of each image in the first consecutive frame image and the second consecutive frame image respectively, and to count the number of images in the first consecutive frame image whose brightness value is lower than the brightness threshold as the first count, and the number of images in the second consecutive frame image whose brightness value is lower than the brightness threshold as the second count.

[0038] The blur count module is used to calculate the blur value of each image in the first consecutive frame image and the second consecutive frame image respectively, and to count the number of images in the first consecutive frame image whose blur value is lower than the blur threshold as the third count, and the number of images in the second consecutive frame image whose blur value is lower than the blur threshold as the fourth count.

[0039] The sharpness difference calculation module is used to calculate the sharpness difference between the images in the first consecutive frame images and the images in the second consecutive frame images at the same time.

[0040] A sharpness counting module is used to increment the third or fourth count corresponding to the image with lower sharpness in the first consecutive frame image and the second consecutive frame image at the same time if the sharpness difference is greater than the sharpness threshold.

[0041] The black pixel counting module is used to sequentially calculate the black pixel difference between the images in the first consecutive frame image and the images in the second consecutive frame image at the same time. When the black pixel difference is greater than the black pixel threshold, the number of images in the first consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the first consecutive frame image is counted as the sixth count.

[0042] The judgment module is used to determine the faults of the first camera and the second camera based on the first count, the second count, the third count, the fourth count, the fifth count, and the sixth count.

[0043] In some embodiments, the determining module is further configured to:

[0044] If the first count is greater than the first threshold, then the field of view of the first camera is determined to be too dark;

[0045] If the second count is greater than the first threshold, then the field of view of the second camera is determined to be too dark;

[0046] If the third count is greater than the second threshold, then the first camera is determined to be blurry;

[0047] If the fourth count is greater than the second threshold, then the second camera is determined to be blurry;

[0048] If the fifth count is greater than the third threshold, then it is determined that the first camera is blocked;

[0049] If the sixth count is greater than the third threshold, then it is determined that the second camera is blocked;

[0050] If the first count and the second count are both less than the first threshold, the third count and the fourth count are both less than the second threshold, and the fifth count and the sixth count are both less than the third threshold, then it is determined that both the first camera and the second camera are fault-free.

[0051] In some embodiments, the ambiguity counting module is further configured to:

[0052] Edge detection is performed on each image in the first consecutive frame image and the second consecutive frame image respectively, and blur values ​​are obtained;

[0053] The number of images in the first consecutive frames whose blur value is lower than the blur threshold is counted as the third count;

[0054] The number of images in the second consecutive frame that have a blur value below the blur threshold is counted as the fourth count.

[0055] Thirdly, this application also provides an intelligent robot, the intelligent robot comprising:

[0056] First camera;

[0057] A second camera, the first camera and the second camera are facing the same direction;

[0058] At least one processor, and

[0059] A memory, communicatively connected to the processor, stores instructions executable by the at least one processor to implement the steps of the camera field-of-view interference detection method as described in the first aspect above.

[0060] Fourthly, this application also provides a non-volatile computer-readable storage medium storing computer-executable instructions, which, when executed by an intelligent robot, implement the steps of the camera field-of-view interference detection method described in the first aspect above.

[0061] The beneficial effects of this application's embodiments are as follows: Unlike the prior art, the camera field-of-view interference detection method, device, intelligent robot, and storage medium provided in this application's embodiments, after acquiring the first and second consecutive frame images captured by the first and second cameras respectively within the same time period, can count the brightness and blur values ​​of the first and second consecutive frame images since the fields of view of the two cameras are the same; by comparing the sharpness of the first and second consecutive frame images to obtain the sharpness difference, and counting the images with lower sharpness, the blur level of a certain camera can be detected; by judging the difference in the number of black pixels in the first and second consecutive frame images, the degree of occlusion can be counted. Based on the counts of various camera field-of-view interference factors, the malfunction of the first and second cameras can be determined.

[0062] This application can ensure coverage of all possible situations that may cause camera image blurring or occlusion, and can also ensure the sensitivity of the algorithm by adjusting the thresholds of various counts. The thresholds can be flexibly adjusted according to the sensitivity requirements of the algorithm under different conditions. Attached Figure Description

[0063] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0064] Figure 1 This is a schematic diagram of the camera structure of the intelligent robot of this application;

[0065] Figure 2 This is a flowchart illustrating one embodiment of the camera field-of-view interference detection method of this application;

[0066] Figure 3 This is a schematic diagram of an image captured by the camera of the intelligent robot in this application during heavy rain or fog.

[0067] Figure 4 This is a schematic diagram of the structure of one embodiment of the camera field-of-view interference detection device of this application;

[0068] Figure 5 This is a schematic diagram of the hardware structure of the controller in one embodiment of the intelligent robot of this application. Detailed Implementation

[0069] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0072] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0073] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0074] The camera field-of-view interference detection method and apparatus provided in this application can be applied to intelligent robots, which can be unmanned robots. The intelligent robot includes a first camera and a second camera, which are combined to form the intelligent robot's camera. The camera, such as... Figure 1 As shown, the intelligent robot obtains information about its surrounding environment through the first and second cameras, which facilitates the operation of the intelligent robot.

[0075] The first and second cameras can be in the form of binocular cameras, installed at the same location on the intelligent robot. Both cameras acquire images from the same angle simultaneously, producing a first and a second series of consecutive frames. The field of view angles of the first and second consecutive frames are identical. If the higher-order features of the consecutive frames acquired by the first and second cameras differ significantly, it indicates that one camera is experiencing severe visual interference.

[0076] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the camera field-of-view interference detection method applied to this application. The method can be executed by a controller in an intelligent robot 100, and includes steps S201-S207.

[0077] S201: Obtain the first consecutive frame image and the second consecutive frame image captured by the first camera and the second camera respectively within the same time period.

[0078] The intelligent robot includes a first camera and a second camera, which are facing the same direction.

[0079] The first and second consecutive frame images are consecutive frame images acquired by the first and second cameras at the same time and angle.

[0080] By identifying and comparing the first consecutive frame images captured by the first camera and the second consecutive frame images captured by the second camera, it can be determined whether one or both of the two cameras are subject to field of view interference.

[0081] S202: Calculate the brightness value of each image in the first consecutive frame image and the second consecutive frame image respectively, and count the number of images in the first consecutive frame image whose brightness value is lower than the brightness threshold as the first count, and the number of images in the second consecutive frame image whose brightness value is lower than the brightness threshold as the second count.

[0082] Specifically, the existing OpenCV brightness detection technology can be used. This involves calculating the mean and variance of each image in the first and second consecutive frames on the grayscale image. When brightness anomalies exist, the mean will deviate from the mean point (which can be assumed to be 128), and the variance will also be smaller. By calculating the mean and variance of the grayscale image, it is possible to assess whether the image is overexposed or underexposed, thus achieving brightness detection for both the first and second consecutive frames, obtaining the brightness values ​​of each image in the first and second consecutive frames. Then, the number of images in the first consecutive frames with brightness values ​​below a brightness threshold is counted as a first count; that is, the count of images with brightness values ​​below the brightness threshold in the first consecutive frames is incremented by one during the counting process, while if the brightness value is above the brightness threshold, it is not incremented. Similarly, the number of images in the second consecutive frames with brightness values ​​below the brightness threshold is counted as a second count; that is, the count of images in the second consecutive frames with brightness values ​​below the brightness threshold is incremented by one during the counting process, while if the brightness value is above the brightness threshold, it is not incremented.

[0083] S203: Calculate the blur value of each image in the first consecutive frame image and the second consecutive frame image respectively, and count the number of images in the first consecutive frame image whose blur value is lower than the blur threshold as the third count, and the number of images in the second consecutive frame image whose blur value is lower than the blur threshold as the fourth count.

[0084] In some embodiments, the blur value of each image in the first consecutive frame image and the second consecutive frame image is calculated respectively, and the number of images in the first consecutive frame image with blur values ​​lower than a blur threshold is counted as a third count, and the number of images in the second consecutive frame image with blur values ​​lower than a blur threshold is counted as a fourth count, which may include:

[0085] Edge detection is performed on each image in the first consecutive frame image and the second consecutive frame image respectively, and blur values ​​are obtained;

[0086] The number of images in the first consecutive frames whose blur value is lower than the blur threshold is counted as the third count;

[0087] The number of images in the second consecutive frame that have a blur value below the blur threshold is counted as the fourth count.

[0088] Specifically, interference counting is also required for the blurring of the first and second cameras. Edge detection is performed on the first and second consecutive frame images respectively, obtaining the first edge detection result for each image in the first consecutive frame image and the second edge detection result for each image in the second consecutive frame image. The first and second edge detection results are used as blur values. Further, edge detection can be obtained by calculating the Laplacian transform of the first or second consecutive frame images, as shown in Formula 1:

[0089]

[0090] Where I represents each image in the first consecutive frame or each image in the second consecutive frame, x and y represent the x and y coordinates of the pixels in the first consecutive frame or the second consecutive frame, and Laplace(I) represents the Laplacian operator of the first consecutive frame or the second consecutive frame.

[0091] Formula 1 is used to calculate the Laplacian transform values ​​of all pixels in each image of the first consecutive frame or each image of the second consecutive frame. These Laplacian transform values ​​characterize the color change trend of each image relative to pixel coordinates, and therefore can be used for edge detection. Alternatively, the Laplacian transform values ​​of the first or second consecutive frame can be directly calculated using the Laplacian operator function `cv::Lap l acian` in OpenCV, and the variance of these Laplacian transform values ​​is also calculated. The variance of the Laplacian transform values ​​of each image in the first consecutive frame is the first edge detection result, representing the blur value of each image in the first consecutive frame; the variance of the Laplacian transform values ​​of each image in the second consecutive frame is the second edge detection result, representing the blur value of each image in the second consecutive frame.

[0092] The number of images in the first consecutive frames with a blur value lower than the blur threshold is counted as the third count; that is, if the value is lower than the blur threshold, the count is incremented by 1. The number of images in the second consecutive frames with a blur value lower than the blur threshold is counted as the fourth count; that is, if the value is lower than the blur threshold, the count is incremented by 1. This allows us to determine the number of images with lower blur values ​​in both the first and second consecutive frames.

[0093] It should be noted that the first and second counts can be obtained by first verifying the darkness of the first and second consecutive frames and then performing the blur verification to obtain the third and fourth counts; alternatively, the third and fourth counts can be obtained by first verifying the blur of the first and second consecutive frames and then performing the darkness verification to obtain the first and second counts. The order of the two verifications is not limited here.

[0094] S204: Calculate the difference in sharpness between the images in the first consecutive frame images and the images in the second consecutive frame images at the same time.

[0095] In some implementations, sequentially calculating the sharpness difference between images in the first consecutive frames and images in the second consecutive frames at the same time may include:

[0096] Laplacian transform is performed on the images in the first consecutive frame images and the images in the second consecutive frame images at the same time to obtain the first Laplacian transform values ​​of all pixels in the first consecutive frame images and the second Laplacian transform values ​​of all pixels in the second consecutive frame images.

[0097] Calculate the first variance of the first Laplace transform value and the second variance of the second Laplace transform value;

[0098] The difference between the first variance and the second variance is calculated to obtain the sharpness difference.

[0099] Specifically, when comparing the sharpness of images in the first and second consecutive frames at the same time, Formula 1 can be used to perform Laplacian transform on the images in the first and second consecutive frames respectively, to obtain the first Laplacian transform value of all pixels in the first consecutive frame image and the second Laplacian transform value of all pixels in the second consecutive frame image; calculate the first variance of the first Laplacian transform value and the second variance of the second Laplacian transform value; calculate the difference between the first variance and the second variance, and use this difference as the sharpness difference value.

[0100] S205: If the sharpness difference is greater than the sharpness threshold, then the third count or the fourth count corresponding to the image with lower sharpness in the first consecutive frame image and the second consecutive frame image at the same time is incremented by one.

[0101] Specifically, if the sharpness difference is greater than the sharpness threshold, it indicates that the sharpness difference between the two cameras is large, and one of the cameras is experiencing field of view interference. In this case, the third or fourth count corresponding to the image with lower sharpness in the first consecutive frame image and the second consecutive frame image at the same time is incremented by one, that is, the count of the blurrier image is incremented by one. Conversely, if the sharpness difference is less than the sharpness threshold, it indicates that the two cameras are indeed not interfered with in the current situation, and the blur status count of the relevant camera is cleared to zero.

[0102] By comparing the sharpness of two images, we can ensure that some blurry parts of the detected image are included. That is, sometimes only one area of ​​the image is blurred, but that area happens to be where the target to be detected is located. Therefore, we need to ensure that all partially blurred parts are detected.

[0103] S206: Calculate the black pixel difference between the images in the first consecutive frame image and the images in the second consecutive frame image at the same time. When the black pixel difference is greater than the black pixel threshold, count the number of images in the first consecutive frame image with a number of black pixels greater than the corresponding number of black pixels in the second consecutive frame image as the fifth count, and count the number of images in the second consecutive frame image with a number of black pixels greater than the corresponding number of black pixels in the first consecutive frame image as the sixth count.

[0104] Experiments revealed that when a camera's view is obstructed, a large number of black pixels appear in the captured image because light cannot enter. Therefore, the number of black pixels can be used as a criterion for whether a camera is obstructed. Furthermore, since the number of black pixels is higher when the camera observes black objects at night and during the day, the difference in black pixels between the first consecutive frame image of the first camera and the second consecutive frame image of the second camera at the same time can be calculated. When the difference in black pixels exceeds a black pixel threshold, it indicates that the number of black pixels in the images from the two cameras differs significantly, confirming that the camera with more black pixels is obstructed. The number of images in the first consecutive frame image with a greater number of black pixels than the corresponding number in the second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image with a greater number of black pixels than the corresponding number in the first consecutive frame image is counted as the sixth count. In other words, the obstruction count of the camera with a greater number of black pixels is incremented by 1. If the difference in black pixels between two cameras is less than the black pixel threshold, it means that the images from the two cameras are within the occlusion threshold range. In this case, the corresponding occlusion count is reset to zero, thus ensuring that the camera is detected as being occluded.

[0105] Therefore, in some embodiments, the step of sequentially calculating the black pixel difference between the images in the first consecutive frame image and the images in the second consecutive frame image at the same time, and when the black pixel difference is greater than a black pixel threshold, counting the number of images in the first consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the second consecutive frame image as a fifth count, and counting the number of images in the second consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the first consecutive frame image as a sixth count, includes:

[0106] Calculate the R, G, and B values ​​of each pixel in the first consecutive frame images and the second consecutive frame images at the same time.

[0107] If the R, G, and B values ​​of the pixels in the first or second consecutive frame images are all within the corresponding preset threshold range, then the pixels whose R, G, and B values ​​are within the preset pixel range are determined as black pixels.

[0108] The number of black pixels in the images in the first consecutive frames and the images in the second consecutive frames is counted, and the difference in the number of black pixels in the images in the first consecutive frames and the images in the second consecutive frames at the same time is obtained.

[0109] When the difference in black pixels is greater than the black pixel threshold, the number of images in the first consecutive frame image where the number of black pixels is greater than the number of black pixels in the corresponding second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image where the number of black pixels is greater than the number of black pixels in the corresponding first consecutive frame image is counted as the sixth count.

[0110] Specifically, the R, G, and B values ​​of each pixel in the first and second consecutive frame images at the same time are calculated. Since the R, G, and B values ​​of black pixels have a certain threshold range, the upper and lower thresholds of which can be obtained using a computer color wheel. For example, we consider the color in a certain area on the color wheel to be black. Therefore, we can count the range of the R / G / B values ​​in that area as a preset threshold range. Thus, if the R value, G value, and B value of a pixel in the first consecutive frame image are all within the corresponding preset threshold range, then the pixel whose R, G, and B values ​​are within the preset pixel range is determined to be a black pixel. Otherwise, it is not a black pixel. The number of black pixels in each pixel of the first consecutive frame image is counted. Similarly, the number of black pixels in the second consecutive frame image at the same time is counted.

[0111] Then, when the difference in black pixels is greater than the black pixel threshold, the number of images in the first consecutive frame image where the number of black pixels is greater than the corresponding number of black pixels in the second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image where the number of black pixels is greater than the corresponding number of black pixels in the first consecutive frame image is counted as the sixth count. This completes the statistical analysis of the number of black pixels in the first and second consecutive frame images.

[0112] S207: Determine the fault of the first camera and the second camera based on the first count, the second count, the third count, the fourth count, the fifth count, and the sixth count.

[0113] In some embodiments, determining the malfunction of the first camera and the second camera based on the first count, the second count, the third count, the fourth count, the fifth count, and the sixth count may include:

[0114] If the first count is greater than the first threshold, then the field of view of the first camera is determined to be too dark;

[0115] If the second count is greater than the first threshold, then the field of view of the second camera is determined to be too dark;

[0116] If the third count is greater than the second threshold, then the first camera is determined to be blurry;

[0117] If the fourth count is greater than the second threshold, then the second camera is determined to be blurry;

[0118] If the fifth count is greater than the third threshold, then it is determined that the first camera is blocked;

[0119] If the sixth count is greater than the third threshold, then it is determined that the second camera is blocked;

[0120] If the first count and the second count are both less than the first threshold, the third count and the fourth count are both less than the second threshold, and the fifth count and the sixth count are both less than the third threshold, then it is determined that both the first camera and the second camera are fault-free.

[0121] Specifically, the faults include excessive darkness, blurriness, and obstruction. A first count is the number of images in the first consecutive frames whose brightness value is below a brightness threshold. If the first count is greater than the first threshold, the first camera's field of view is determined to be too dark. A second count is the number of images in the second consecutive frames whose brightness value is below a brightness threshold. If the second count is greater than the first threshold, the second camera's field of view is determined to be too dark. This achieves brightness field-of-view interference detection for both the first and second cameras, and when it is determined that either the first or second camera's field of view is too dark, a corresponding fault warning for excessive darkness can be output.

[0122] The third count is the number of images in the first consecutive frames whose blur value is lower than the blur threshold. If the third count is greater than the second threshold, the first camera is determined to be blurry. The fourth count is the number of images in the second consecutive frames whose blur value is lower than the blur threshold. If the fourth count is greater than the second threshold, the second camera is determined to be blurry. This achieves blur detection for both the first and second cameras.

[0123] Experiments revealed that when the first or second camera is blurred by water droplets, the edges of actual objects observed in the first or second consecutive frames become blunt, making it impossible to clearly identify object boundaries. Specifically, this manifests as an excessively low variance in the image edge detection results. Therefore, the relationship between the variance of the Laplacian transform value of the first or second consecutive frames and the blur threshold can be used to determine whether the first or second consecutive frames are blurred. If blurred, the blur count of that pixel in the image can be incremented by 1; otherwise, similar to brightness detection, the blur count is reset to zero.

[0124] By assessing the severity of blur, it's possible to detect situations that severely impact image sharpness, such as images captured under extreme weather conditions like heavy rain or dense fog. For example... Figure 3 As shown, Figure 3 These are images taken by the first and second cameras during heavy rain or fog.

[0125] When the third or fourth count is greater than the second threshold, it is determined that the first or second camera is blurry. It can also output a blur fault warning for the first or second camera, so as to let the user know the blur status of the two cameras.

[0126] The fifth count is the number of images in the first consecutive frame where the number of black pixels is greater than the number of black pixels in the corresponding second consecutive frame. If the fifth count is greater than the third threshold, it is determined that the first camera is occluded. The sixth count is the number of images in the second consecutive frame where the number of black pixels is greater than the number of black pixels in the corresponding first consecutive frame. If the sixth count is greater than the third threshold, it is determined that the second camera is occluded.

[0127] Specifically, if the fifth count is greater than the third threshold (occlusion threshold), it can be determined that the first camera is occluded, resulting in more black pixels and affecting the camera's field of view. In this case, a fault warning for the occluded first camera is output.

[0128] Conversely, if the sixth count is greater than the third threshold (occlusion threshold), it can be determined that the second camera is occluded, resulting in more black pixels and affecting the camera's field of view. In this case, a fault warning for the occluded second camera is output.

[0129] Correspondingly, if the first count and the second count are both less than the first threshold, the third count and the fourth count are both less than the second threshold, and the fifth count and the sixth count are both less than the third threshold, then it is determined that both the first camera and the second camera are fault-free.

[0130] In embodiments of this application, after acquiring first and second consecutive frame images captured by the first and second cameras within the same time period, since the fields of view of the two cameras are identical, brightness and blur values ​​can be counted for the first and second consecutive frame images. A sharpness comparison is performed between the first and second consecutive frame images to obtain the sharpness difference, and the image with lower sharpness is counted to detect the blur level of a particular camera. The difference in the number of black pixels in the first and second consecutive frame images is used to determine the degree of occlusion. Based on the counts of various camera field-of-view interference factors, a fault in the first and second cameras can be determined.

[0131] This application can ensure coverage of all possible situations that may cause camera image blurring or occlusion, and can also ensure the sensitivity of the algorithm by adjusting the thresholds of various counts. The thresholds can be flexibly adjusted according to the sensitivity requirements of the algorithm under different conditions.

[0132] This application also provides a camera field-of-view interference detection device for an intelligent robot. The intelligent robot includes a first camera and a second camera, the first camera and the second camera facing the same direction; please refer to [link to relevant documentation]. Figure 4 This illustration shows the structure of a camera field-of-view interference detection device 400 provided in an embodiment of this application. The camera field-of-view interference detection device 400 includes:

[0133] The acquisition module 401 is used to acquire the first consecutive frame image and the second consecutive frame image captured by the first camera and the second camera respectively within the same time period;

[0134] The brightness counting module 402 is used to calculate the brightness value of each image in the first consecutive frame image and the second consecutive frame image respectively, and to count the number of images in the first consecutive frame image whose brightness value is lower than the brightness threshold as the first count, and the number of images in the second consecutive frame image whose brightness value is lower than the brightness threshold as the second count.

[0135] The blur count module 403 is used to calculate the blur value of each image in the first consecutive frame image and the second consecutive frame image respectively, and to count the number of images in the first consecutive frame image whose blur value is lower than the blur threshold as the third count, and the number of images in the second consecutive frame image whose blur value is lower than the blur threshold as the fourth count.

[0136] The sharpness difference calculation module 404 is used to calculate the sharpness difference between the images in the first consecutive frame images and the images in the second consecutive frame images at the same time.

[0137] The sharpness counting module 405 is used to increment the third count or the fourth count corresponding to the image with lower sharpness in the first consecutive frame image and the second consecutive frame image at the same time if the sharpness difference is greater than the sharpness threshold.

[0138] The black pixel counting module 406 is used to sequentially calculate the black pixel difference between the images in the first consecutive frame image and the images in the second consecutive frame image at the same time. When the black pixel difference is greater than the black pixel threshold, the number of images in the first consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the first consecutive frame image is counted as the sixth count.

[0139] The judgment module 407 is used to determine the faults of the first camera and the second camera based on the first count, the second count, the third count, the fourth count, the fifth count, and the sixth count.

[0140] In embodiments of this application, after acquiring first and second consecutive frame images captured by the first and second cameras within the same time period, since the fields of view of the two cameras are identical, brightness and blur values ​​can be counted for the first and second consecutive frame images. A sharpness comparison is performed between the first and second consecutive frame images to obtain the sharpness difference, and the image with lower sharpness is counted to detect the blur level of a particular camera. The difference in the number of black pixels in the first and second consecutive frame images is used to determine the degree of occlusion. Based on the counts of various camera field-of-view interference factors, a fault in the first and second cameras can be determined.

[0141] This application can ensure coverage of all possible situations that may cause camera image blurring or occlusion, and can also ensure the sensitivity of the algorithm by adjusting the thresholds of various counts. The thresholds can be flexibly adjusted according to the sensitivity requirements of the algorithm under different conditions.

[0142] In some embodiments, the determining module 407 is further configured to:

[0143] If the first count is greater than the first threshold, then the field of view of the first camera is determined to be too dark;

[0144] If the second count is greater than the first threshold, then the field of view of the second camera is determined to be too dark;

[0145] If the third count is greater than the second threshold, then the first camera is determined to be blurry;

[0146] If the fourth count is greater than the second threshold, then the second camera is determined to be blurry;

[0147] If the fifth count is greater than the third threshold, then it is determined that the first camera is blocked;

[0148] If the sixth count is greater than the third threshold, then it is determined that the second camera is blocked;

[0149] If the first count and the second count are both less than the first threshold, the third count and the fourth count are both less than the second threshold, and the fifth count and the sixth count are both less than the third threshold, then it is determined that both the first camera and the second camera are fault-free.

[0150] In some embodiments, the ambiguity counting module 403 is further configured to:

[0151] Edge detection is performed on each image in the first consecutive frame image and the second consecutive frame image respectively, and blur values ​​are obtained;

[0152] The number of images in the first consecutive frames whose blur value is lower than the blur threshold is counted as the third count;

[0153] The number of images in the second consecutive frame that have a blur value below the blur threshold is counted as the fourth count.

[0154] In some embodiments, the sharpness difference calculation module 404 is further configured to:

[0155] Laplacian transform is performed on the images in the first consecutive frame images and the images in the second consecutive frame images at the same time to obtain the first Laplacian transform values ​​of all pixels in the first consecutive frame images and the second Laplacian transform values ​​of all pixels in the second consecutive frame images.

[0156] Calculate the first variance of the first Laplace transform value and the second variance of the second Laplace transform value;

[0157] The difference between the first variance and the second variance is calculated to obtain the sharpness difference.

[0158] In some embodiments, the black pixel counting module 406 is further configured to:

[0159] Calculate the R, G, and B values ​​of each pixel in the first consecutive frame images and the second consecutive frame images at the same time.

[0160] If the R, G, and B values ​​of the pixels in the first or second consecutive frame images are all within the corresponding preset threshold range, then the pixels whose R, G, and B values ​​are within the preset pixel range are determined as black pixels.

[0161] The number of black pixels in the images in the first consecutive frames and the images in the second consecutive frames is counted, and the difference in the number of black pixels in the images in the first consecutive frames and the images in the second consecutive frames at the same time is obtained.

[0162] When the difference in black pixels is greater than the black pixel threshold, the number of images in the first consecutive frame image where the number of black pixels is greater than the number of black pixels in the corresponding second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image where the number of black pixels is greater than the number of black pixels in the corresponding first consecutive frame image is counted as the sixth count.

[0163] It should be noted that the above-described apparatus can execute the method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the apparatus embodiments can be found in the method provided in the embodiments of this application.

[0164] Figure 5 This is a schematic diagram of the hardware structure of the controller of the intelligent robot 100 in one embodiment of the intelligent robot, as shown below. Figure 5 As shown, the controller includes:

[0165] One or more processors 111 and memory 112. Figure 5 The example uses a processor 111 and a memory 112.

[0166] Processor 111 and memory 112 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0167] Memory 112, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the camera field-of-view interference detection method in the embodiments of this application (e.g., attached...). Figure 4 The module shown includes an acquisition module 401, a brightness counting module 402, a blur counting module 403, a sharpness difference calculation module 404, a sharpness counting module 405, a black pixel counting module 406, and a judgment module 407. The processor 111 executes various functions and data processing of the controller by running non-volatile software programs, instructions, and modules stored in the memory 112, thereby implementing the camera field-of-view interference detection method of the above-described embodiment.

[0168] The memory 112 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the personnel access detection device. Furthermore, the memory 112 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 112 may optionally include memory remotely located relative to the processor 111, and these remote memories can be connected to the intelligent robot via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0169] The one or more modules are stored in the memory 112. When executed by the one or more processors 111, they perform the camera field-of-view interference detection method in any of the above method embodiments, for example, performing the above-described... Figure 2 Method steps S201 to S207; implementation Figure 4 The functions of modules 401-407 in the document.

[0170] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0171] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 5 One of the processors 111 can enable the above-described one or more processors to execute the camera field-of-view interference detection method in any of the above method embodiments, for example, to execute the above-described method. Figure 2 Method steps S201 to S207; implementation Figure 4 The functions of modules 401-407 in the document.

[0172] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; under the concept of the present invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the present invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting camera field-of-view interference, applied to an intelligent robot, the intelligent robot comprising a first camera and a second camera, the first camera and the second camera facing the same direction, characterized in that, The method includes: Acquire the first consecutive frame image and the second consecutive frame image captured by the first camera and the second camera respectively within the same time period; Calculate the brightness value of each image in the first consecutive frame image and the second consecutive frame image respectively, and count the number of images in the first consecutive frame image whose brightness value is lower than the brightness threshold as the first count, and the number of images in the second consecutive frame image whose brightness value is lower than the brightness threshold as the second count; The blur value of each image in the first consecutive frame image and the second consecutive frame image is calculated respectively. The number of images in the first consecutive frame image with blur values ​​lower than the blur threshold is counted as the third count, and the number of images in the second consecutive frame image with blur values ​​lower than the blur threshold is counted as the fourth count. The sharpness difference between the images in the first consecutive frame images and the images in the second consecutive frame images at the same time is calculated sequentially. If the sharpness difference is greater than the sharpness threshold, then the third count or the fourth count corresponding to the image with lower sharpness in the first consecutive frame image and the second consecutive frame image at the same time is incremented by one; The black pixel difference between the images in the first consecutive frame image and the images in the second consecutive frame image at the same time is calculated sequentially. When the black pixel difference is greater than the black pixel threshold, the number of images in the first consecutive frame image with a number of black pixels greater than the corresponding number of black pixels in the second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image with a number of black pixels greater than the corresponding number of black pixels in the first consecutive frame image is counted as the sixth count. The faults of the first camera and the second camera are determined based on the first count, the second count, the third count, the fourth count, the fifth count, and the sixth count.

2. The method according to claim 1, characterized in that, The step of determining the faults of the first camera and the second camera based on the first count, the second count, the third count, the fourth count, the fifth count, and the sixth count includes: If the first count is greater than the first threshold, then the field of view of the first camera is determined to be too dark; If the second count is greater than the first threshold, then the field of view of the second camera is determined to be too dark; If the third count is greater than the second threshold, then the first camera is determined to be blurry; If the fourth count is greater than the second threshold, then the second camera is determined to be blurry; If the fifth count is greater than the third threshold, then it is determined that the first camera is blocked; If the sixth count is greater than the third threshold, then it is determined that the second camera is blocked; If the first count and the second count are both less than the first threshold, the third count and the fourth count are both less than the second threshold, and the fifth count and the sixth count are both less than the third threshold, then it is determined that both the first camera and the second camera are fault-free.

3. The method according to claim 1, characterized in that, The step of calculating the blur value of each image in the first consecutive frame image and the second consecutive frame image respectively, and counting the number of images in the first consecutive frame image with blur values ​​lower than the blur threshold as the third count, and the number of images in the second consecutive frame image with blur values ​​lower than the blur threshold as the fourth count, includes: Edge detection is performed on each image in the first consecutive frame image and the second consecutive frame image respectively, and blur values ​​are obtained; The number of images in the first consecutive frames whose blur value is lower than the blur threshold is counted as the third count; The number of images in the second consecutive frame that have a blur value below the blur threshold is counted as the fourth count.

4. The method according to claim 1, characterized in that, The step of sequentially calculating the sharpness difference between the images in the first consecutive frame images and the images in the second consecutive frame images at the same time includes: Laplacian transform is performed on the images in the first consecutive frame images and the images in the second consecutive frame images at the same time to obtain the first Laplacian transform value of all pixels in the first consecutive frame images and the second Laplacian transform value of all pixels in the second consecutive frame images. Calculate the first variance of the first Laplace transform value and the second variance of the second Laplace transform value; The difference between the first variance and the second variance is calculated to obtain the sharpness difference.

5. The method according to any one of claims 1 to 4, characterized in that, The step of sequentially calculating the black pixel difference between the images in the first consecutive frame image and the images in the second consecutive frame image at the same time, and when the black pixel difference is greater than a black pixel threshold, counting the number of images in the first consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the second consecutive frame image as the fifth count, and counting the number of images in the second consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the first consecutive frame image as the sixth count, includes: Calculate the R, G, and B values ​​of each pixel in the first consecutive frame images and the second consecutive frame images at the same time. If the R, G, and B values ​​of the pixels in the first or second consecutive frame images are all within the corresponding preset threshold range, then the pixels whose R, G, and B values ​​are within the preset pixel range are determined as black pixels. The number of black pixels in the images in the first consecutive frames and the images in the second consecutive frames is counted, and the difference in the number of black pixels in the images in the first consecutive frames and the images in the second consecutive frames at the same time is obtained. When the difference in black pixels is greater than the black pixel threshold, the number of images in the first consecutive frame image where the number of black pixels is greater than the number of black pixels in the corresponding second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image where the number of black pixels is greater than the number of black pixels in the corresponding first consecutive frame image is counted as the sixth count.

6. A camera field-of-view interference detection device, applied to an intelligent robot, the intelligent robot comprising a first camera and a second camera, the first camera and the second camera facing the same direction, characterized in that, The device includes: The acquisition module is used to acquire the first consecutive frame image and the second consecutive frame image captured by the first camera and the second camera respectively within the same time period; The brightness counting module is used to calculate the brightness value of each image in the first consecutive frame image and the second consecutive frame image respectively, and to count the number of images in the first consecutive frame image whose brightness value is lower than the brightness threshold as the first count, and the number of images in the second consecutive frame image whose brightness value is lower than the brightness threshold as the second count. The blur count module is used to calculate the blur value of each image in the first consecutive frame image and the second consecutive frame image respectively, and to count the number of images in the first consecutive frame image whose blur value is lower than the blur threshold as the third count, and the number of images in the second consecutive frame image whose blur value is lower than the blur threshold as the fourth count. The sharpness difference calculation module is used to calculate the sharpness difference between the images in the first consecutive frame images and the images in the second consecutive frame images at the same time. The sharpness counting module is used to increment the third count or the fourth count corresponding to the image with lower sharpness in the first consecutive frame images and the second consecutive frame images at the same time if the sharpness difference is greater than the sharpness threshold. The black pixel counting module is used to sequentially calculate the black pixel difference between the images in the first consecutive frame image and the images in the second consecutive frame image at the same time. When the black pixel difference is greater than the black pixel threshold, the number of images in the first consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the second consecutive frame image is counted as the fifth count, and the number of images in the second consecutive frame image with a greater number of black pixels than the corresponding number of black pixels in the first consecutive frame image is counted as the sixth count. The judgment module is used to determine the faults of the first camera and the second camera based on the first count, the second count, the third count, the fourth count, the fifth count, and the sixth count.

7. The apparatus according to claim 6, characterized in that, The judgment module is also used for: If the first count is greater than the first threshold, then the field of view of the first camera is determined to be too dark; If the second count is greater than the first threshold, then the field of view of the second camera is determined to be too dark; If the third count is greater than the second threshold, then the first camera is determined to be blurry; If the fourth count is greater than the second threshold, then the second camera is determined to be blurry; If the fifth count is greater than the third threshold, then it is determined that the first camera is blocked; If the sixth count is greater than the third threshold, then it is determined that the second camera is blocked; If the first count and the second count are both less than the first threshold, the third count and the fourth count are both less than the second threshold, and the fifth count and the sixth count are both less than the third threshold, then it is determined that both the first camera and the second camera are fault-free.

8. The apparatus according to claim 6, characterized in that, The ambiguity counting module is also used for: Edge detection is performed on each image in the first consecutive frame image and the second consecutive frame image respectively, and blur values ​​are obtained; The number of images in the first consecutive frames whose blur value is lower than the blur threshold is counted as the third count; The number of images in the second consecutive frame that have a blur value below the blur threshold is counted as the fourth count.

9. An intelligent robot, characterized in that, The intelligent robot includes: First camera; The second camera is positioned in the same direction as the first camera; At least one processor, and A memory communicatively connected to the processor, the memory storing instructions executable by the at least one processor to perform the steps of the method as described in any one of claims 1 to 5.

10. A non-volatile computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by an intelligent robot, implement the steps of the method as described in any one of claims 1 to 5.

Images