A camera dirt detection method, apparatus, device and medium
By constructing an image pyramid and using adaptive threshold segmentation technology, dirt on the camera sensor is detected, solving the problems of low efficiency and poor accuracy in existing technologies. This achieves efficient and accurate dirt detection and improves the camera's imaging quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG HUARAY TECH CO LTD
- Filing Date
- 2022-12-26
- Publication Date
- 2026-05-29
Smart Images
Figure CN116012322B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to a method, apparatus, device, and medium for detecting dirt in a camera. Background Technology
[0002] Existing technologies for detecting dirt on camera sensor surfaces primarily rely on human observation of the sensor image displayed on a monitor to locate dirt. This method is slow and the results depend on subjective human judgment. Alternatively, correlation heatmaps can be used to determine the location of dirt on the camera sensor, but this method is computationally intensive and unsuitable for detecting dirt of varying sizes. Therefore, both of these methods suffer from low detection efficiency and inaccurate results, failing to adequately meet the needs of camera dirt detection. Summary of the Invention
[0003] This application provides a method, apparatus, device, and medium for detecting camera dirt, which improves the efficiency and accuracy of camera dirt detection to meet the needs of camera dirt detection.
[0004] In a first aspect, embodiments of this application provide a camera dirt detection method, which can be executed by a computer device. The method includes: acquiring a first image captured by a camera; determining a first grayscale image based on the first image; constructing an image pyramid based on the first grayscale image, the image pyramid including at least one second grayscale image, wherein the resolution of each second grayscale image is less than the resolution of the first grayscale image; determining dirt information of each second grayscale image; mapping the dirt information of each second grayscale image onto the first grayscale image to obtain first dirt information of the first grayscale image, wherein the first dirt information includes the position and size of the dirt area; and determining dirt information of the camera sensor based on the first dirt information of the first grayscale image.
[0005] In this scheme, the computer device first determines a first grayscale image based on a first image captured by the camera, obtains the first grayscale image, constructs an image pyramid of the first grayscale image, and identifies the dirt information of the dirty areas on each second grayscale image in the image pyramid. Then, the dirt information from each second grayscale image is mapped onto the first grayscale image to obtain the first dirt information of the dirty areas on the first grayscale image. Since the pixels on the first grayscale image correspond to the camera sensor, the dirt information on the camera sensor can be determined based on the first dirt information of the first grayscale image. By finding dirt on images with lower resolution (i.e., at least one second grayscale image) and then mapping it onto the first grayscale image, the computer device improves the efficiency of dirt detection. Furthermore, this scheme can determine the location and size of the dirt, resulting in relatively accurate detection results that meet the requirements for camera dirt detection.
[0006] Optionally, after determining the first grayscale image based on the first image and before constructing the image pyramid based on the first grayscale image, the method further includes: determining a first type of pixel unit of the first grayscale image according to a preset grayscale value range, wherein the grayscale value of the first type of pixel unit is less than the minimum value in the preset grayscale value range; determining the dirt information of each second grayscale image and mapping the dirt information of each second grayscale image onto the first grayscale image includes: determining a first sub-dirty region of each second grayscale image based on the first type of pixel unit and mapping the first sub-dirty region onto the first grayscale image; And / or, after determining the first grayscale image based on the first image and before constructing the image pyramid based on the first grayscale image, the method further includes: determining a second type of pixel unit of the first grayscale image according to a preset grayscale value range, wherein the grayscale value of the second type of pixel unit is greater than the maximum value in the preset grayscale value range; determining the dirt information of each second grayscale image and mapping the dirt information of each second grayscale image onto the first grayscale image includes: determining a second sub-dirty region of each second grayscale image based on the second type of pixel unit and mapping the second sub-dirty region onto the first grayscale image.
[0007] In this method, the computer device determines the first type of pixel unit and the second type of pixel unit. Based on the first type of pixel unit, the first sub-dirty region of each second grayscale image is determined, and based on the second type of pixel unit, the second sub-dirty region of each second grayscale image is determined. In fact, the first type of pixel unit may be the pixel unit where black dirt is located, and the second type of pixel unit may be the pixel unit where white dirt is located. The dirty region on the first grayscale image can be mapped from the first sub-dirty region, or it can be mapped from the second sub-dirty region, or it can be mapped from both the first and second sub-dirty regions. The scheme has high flexibility.
[0008] Optionally, before determining the first sub-dirty region of each second grayscale image based on the first type of pixel unit, the method further includes: if the second type of pixel unit exists, processing the second type of pixel unit into the first type of image unit; or, before determining the second sub-dirty region of each second grayscale image based on the second type of pixel unit, the method further includes: if the first type of pixel unit exists, processing the first type of pixel unit into the second type of image unit.
[0009] This method processes second-type pixel units as first-type image units or first-type pixel units as second-type image units, so that the computer device only needs to process one type of pixel, reducing the amount of computation and improving the efficiency of dirt detection.
[0010] Optionally, determining the dirt information of each second grayscale image includes: determining the connected components of each second grayscale image based on an adaptive threshold segmentation method, wherein the absolute value of the difference between the grayscale value of each pixel in the connected component and the grayscale value of a pixel outside the connected component is greater than a first threshold, and / or the absolute value of the difference between the grayscale values of pixels in the connected component is less than a second threshold; filtering connected components that meet preset conditions from the connected components of each second grayscale image, wherein the connected components that meet the preset conditions are the dirty regions of each second grayscale image; wherein the preset conditions include one or more of the following: the area of the connected component is within a first preset range, the aspect ratio of the connected component is within a second preset range, and the roundness of the connected component is within a third preset range; determining the dirt information of each second grayscale image based on the dirty regions of each second grayscale image.
[0011] In this method, the computer device determines the connected components that may be dirty areas in each second grayscale image based on an adaptive threshold segmentation method. Then, according to preset conditions, it filters out the connected components that are dirty areas from the connected components of each second grayscale image. Since the dirty areas of each second grayscale image have passed the adaptive threshold segmentation and meet the preset conditions, the dirty areas of each second grayscale image can more realistically reflect the dirt on the camera sensor, thus improving the reliability of the solution.
[0012] Optionally, determining the dirt information of the camera's sensor based on the first dirt information of the first grayscale image includes:
[0013] The first dirt information is mapped to the dirt information of the camera's sensor.
[0014] Using this method, computer equipment can determine the dirt information of the camera sensor based on the first dirt information. The method is simple and the dirt detection efficiency is high.
[0015] Optionally, determining the dirt information of the camera sensor based on the first dirt information of the first grayscale image includes: determining at least one region of interest (ROI) from the first grayscale image based on the first dirt information of the first grayscale image; calculating the grayscale curve of each ROI; determining second dirt information of the first grayscale image based on the grayscale curve of each ROI, the second dirt information including at least one of the location, size, and degree of dirt of the dirt region; and determining the dirt information of the camera sensor based on the second dirt information.
[0016] In this method, the computer device, based on the first dirt information of the first grayscale image, further determines the second dirt information of the first grayscale image through the grayscale curve of each ROI. Since the second dirt information more accurately reflects the dirt information of the dirt area of the first grayscale image, the dirt information on the camera sensor is determined based on the second dirt information, thus improving the accuracy and reliability of the solution.
[0017] Optionally, determining the second dirt information of the first grayscale image based on the grayscale curve of each ROI includes: filtering the grayscale curve of each ROI to obtain the position and size of the dirt area corresponding to each ROI; and determining the degree of dirtiness of the dirt area corresponding to each ROI based on the position and size of the dirt area corresponding to each ROI and the grayscale curve of each ROI.
[0018] In this method, the computer equipment filters the grayscale curve of each ROI, reducing noise interference in the image of the ROI. This makes it easier to obtain the location and size of the dirty area corresponding to each ROI. Furthermore, based on the location, size, and grayscale curve of the dirty area corresponding to each ROI, the degree of dirtiness of the dirty area corresponding to each ROI can be obtained, improving the completeness of dirt information and enhancing the practicality and completeness of the solution.
[0019] Secondly, this application provides a camera dirt detection device, which includes modules / units / technical means for performing the method described in the first aspect or any optional embodiment of the first aspect.
[0020] For example, the device may include:
[0021] The acquisition module is used to acquire the first image captured by the camera;
[0022] A processing module is configured to: determine a first grayscale image based on the first image; construct an image pyramid based on the first grayscale image, the image pyramid including at least one second grayscale image, wherein the resolution of each second grayscale image is less than the resolution of the first grayscale image; determine dirt information of each second grayscale image; map the dirt information of each second grayscale image onto the first grayscale image to obtain first dirt information of the first grayscale image, wherein the first dirt information includes the position and size of the dirt area; and determine the dirt information of the camera sensor based on the first dirt information of the first grayscale image.
[0023] Optionally, the processing module is configured to: determine a first type of pixel unit in the first grayscale image according to a preset grayscale value range, wherein the grayscale value of the first type of pixel unit is less than the minimum value in the preset grayscale value range; determine a first sub-dirty region in each second grayscale image according to the first type of pixel unit, and map the first sub-dirty region onto the first grayscale image; and / or, determine a second type of pixel unit in the first grayscale image according to a preset grayscale value range, wherein the grayscale value of the second type of pixel unit is greater than the maximum value in the preset grayscale value range; determine a second sub-dirty region in each second grayscale image according to the second type of pixel unit, and map the second sub-dirty region onto the first grayscale image.
[0024] Optionally, the processing module is configured to: if the second type of pixel unit exists, process the second type of pixel unit into the first type of image unit; or, before determining the second sub-dirty region of each second grayscale image based on the second type of pixel unit, the method further includes: if the first type of pixel unit exists, process the first type of pixel unit into the second type of image unit.
[0025] Optionally, the processing module is configured to: determine the connected components of each second grayscale image based on an adaptive threshold segmentation method, wherein the absolute value of the difference between the grayscale value of each pixel in the connected component and the grayscale value of a pixel outside the connected component is greater than a first threshold, and / or the absolute value of the difference between the grayscale values of pixels in the connected component is less than a second threshold; filter connected components that meet preset conditions from the connected components of each second grayscale image, wherein the connected components that meet the preset conditions are the dirty areas of each second grayscale image; wherein the preset conditions include one or more of the following: the area of the connected component is within a first preset range, the aspect ratio of the connected component is within a second preset range, and the roundness of the connected component is within a third preset range; determine the dirt information of each second grayscale image based on the dirty areas of each second grayscale image.
[0026] Optionally, the processing module is configured to: map the first dirt information to dirt information of the camera's sensor.
[0027] Optionally, the processing module is configured to: determine at least one region of interest (ROI) from the first grayscale image based on first dirt information of the first grayscale image; calculate the grayscale curve of each ROI; determine second dirt information of the first grayscale image based on the grayscale curve of each ROI, the second dirt information including the location, size, and degree of dirt of the dirt area; and determine the dirt information of the camera sensor based on the second dirt information.
[0028] Optionally, the processing module is used to: filter the grayscale curve of each ROI to obtain the position and size of the dirty area corresponding to each ROI; and determine the degree of dirtiness of the dirty area corresponding to each ROI based on the position and size of the dirty area corresponding to each ROI and the grayscale curve of each ROI.
[0029] Thirdly, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor, by executing the instructions stored in the memory, causes the at least one processor to perform the method described in the first aspect or any optional embodiment of the first aspect.
[0030] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in any one of the first aspects. Attached Figure Description
[0031] Figure 1 This application provides a flowchart of a method for detecting camera dirt in an embodiment;
[0032] Figure 2 A schematic diagram of ROI provided for an embodiment of this application;
[0033] Figure 3 Another ROI diagram provided for an embodiment of this application;
[0034] Figure 4 A schematic diagram of a grayscale curve provided for an embodiment of this application;
[0035] Figure 5 Another grayscale curve diagram provided for an embodiment of this application;
[0036] Figure 6A schematic diagram of a broken line provided in an embodiment of this application;
[0037] Figure 7 A structural diagram of a camera dirt detection device provided in an embodiment of this application;
[0038] Figure 8 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0039] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.
[0040] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.
[0041] The term "and / or" in the embodiments of this application is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0042] To facilitate understanding of the solutions in the embodiments of this application, the possible application scenarios of the embodiments of this application will be introduced below.
[0043] During the production and assembly process, dust and other particles may accumulate on the surface of the camera sensor, forming dirt that can affect the image quality acquired by the camera. For cameras used in high-precision machine vision defect detection systems (such as advanced automotive glass defect detection and precision instrument surface inspection), dirt can significantly interfere with the normal operation of the vision system. Therefore, it is necessary to detect and remove dirt from the camera sensor surface to ensure the image quality of the camera.
[0044] In scenarios involving the detection of dirt on the surface of camera sensors, the low detection efficiency and inaccurate results prevent the system from adequately meeting the requirements for camera dirt detection.
[0045] Therefore, this application provides a technical solution to improve the efficiency and accuracy of camera dirt detection, so as to meet the needs of camera dirt detection.
[0046] Reference Figure 1 The flowchart below shows a method for detecting camera dirt provided in an embodiment of this application.
[0047] This method can be executed by computer devices, such as laptops, desktop computers, and servers, and can also be applied to various devices with computing capabilities, such as mobile phones and other devices with computing capabilities. Furthermore, it can be applied to various chips with computing capabilities, such as processing chips in cameras. The above devices are merely illustrative examples, and this application does not impose any limitations.
[0048] The following example illustrates how this method is executed by a computer device. The method includes:
[0049] S101: Acquire the first image captured by the camera, and determine the first grayscale image based on the first image.
[0050] First, the camera captures a first image under uniform, perpendicularly illuminating parallel light to the camera sensor. The camera type is either a color camera or a grayscale camera. The first image captured by a color camera is a color image, and the first image captured by a grayscale camera is a grayscale image.
[0051] Then, the computer device acquires the first image captured by the camera. If the first image is a color image, the computer device processes the first image into a first grayscale image based on the following formula (1); if the first image is a grayscale image, the computer device processes the first image into a first grayscale image based on the following formula (2).
[0052] I(x,y)=R(x,y)×0.299+ G(x,y)×0.587+B(x,y)×0.114 (1)
[0053] I(x,y)=Gray(x,y) (2)
[0054] Where x is the column coordinate of the first image pixel, y is the row coordinate of the first image pixel, I(x,y) is the gray value of the pixel at coordinate (x,y), R(x,y) is the red brightness of the pixel at coordinate (x,y), G(x,y) is the green brightness of the pixel at coordinate (x,y), B(x,y) is the blue brightness of the pixel at coordinate (x,y), and Gray(x,y) is the gray value of the pixel at coordinate (x,y).
[0055] Optionally, after determining the first grayscale image, the computer device may determine the first type of pixel units and / or the second type of pixel units of the first grayscale image according to a preset grayscale value range.
[0056] In a specific example, the grayscale value of a pixel ranges from 0 to 255, with a preset grayscale value range of 126 to 130. Pixels with grayscale values less than 126 are classified as first-type pixel units, and pixels with grayscale values greater than 130 are classified as second-type pixel units. In reality, first-type pixel units could be the pixel units containing black dirt, and second-type pixel units could be the pixel units containing white dirt.
[0057] Understandably, the preset grayscale value range can be selected according to actual needs, and this application does not impose any restrictions.
[0058] Furthermore, the above examples use black and white dirt as examples. In practical applications, there can be more categories of dirt, and similar methods can be used to filter more categories of pixel units.
[0059] In one possible implementation, if a second type of pixel unit exists, the computer device can process the second type of pixel unit as a first type of image unit; or, if a first type of pixel unit exists, the computer device can process the first type of pixel unit as a second type of image unit.
[0060] Taking the computer device as an example of processing the second type of pixel unit into the first type of image unit, the computer device processes the second type of pixel unit into the first type of image unit based on the following formula (3).
[0061]
[0062] Where I(x,y) is the gray value of the pixel with coordinates (x,y).
[0063] This method processes second-type pixel units as first-type image units or first-type pixel units as second-type image units, so that the computer device only needs to process one type of pixel, reducing the amount of computation and improving the efficiency of dirt detection.
[0064] In one possible implementation, the computer device can also perform grayscale stretching on the first grayscale image based on the following formula (4).
[0065]
[0066] Where a and b are the minimum and maximum gray values in the first grayscale image, and A and B are the minimum and maximum gray values of the stretching result.
[0067] Understandably, the above is merely an example and not a limitation.
[0068] This method enhances the contrast between dirty and non-dirty areas in the first grayscale image by stretching grayscale, which facilitates subsequent dirt detection by computer equipment and improves dirt detection efficiency.
[0069] S102: Construct an image pyramid based on the first grayscale image.
[0070] The image pyramid includes at least one second grayscale image, and the resolution of each second grayscale image in the at least one second grayscale image is less than the resolution of the first grayscale image.
[0071] The computer device uses the following formulas (5)-(6) to continuously downsample the first grayscale image (downsampling is divided into two steps: first, Gaussian kernel convolution (smoothing filtering) is performed on the image, and then all even-numbered rows and columns of the image pixels are deleted for downsampling. The area of the image obtained after downsampling is one-quarter of that before downsampling), to construct an image pyramid. The image pyramid includes at least one second grayscale image, and the resolution of each second grayscale image in the at least one second grayscale image is less than the resolution of the first grayscale image.
[0072] G0(x,y)=I(x,y) (5)
[0073] G j+1 (x,y)=pyDown(G j (x,y)), 0≤j≤J (6)
[0074] Where G0 is the first grayscale image, x is the column coordinate of a pixel in the first grayscale image, y is the row coordinate of a pixel in the first grayscale image, I(x,y) is the grayscale value of the pixel at coordinates (x,y), G j+1 For G j The downsampled image is denoted by j, which is the index of the image in the image pyramid, ranging from 0 to J. The value of J can be selected according to actual needs (for example, the value of J is usually 5 to 7).
[0075] S103: Determine the dirt information of each second grayscale image; map the dirt information of each second grayscale image onto the first grayscale image to obtain the first dirt information of the first grayscale image.
[0076] The method described above for determining the first type of pixel units and / or the second type of pixel units in the first grayscale image using computer equipment is adopted.
[0077] In one possible implementation, the computer device determines a first sub-dirty region in each second grayscale image based on the first type of pixel units, and maps the first sub-dirty region onto the first grayscale image. The specific steps are as follows:
[0078] First, the computer device performs adaptive threshold segmentation on each second grayscale image based on the adaptive threshold segmentation method (when processing the first type of pixel unit, the principle of adaptive threshold segmentation is the following formula (7)), to obtain the connected components of each second grayscale image. The connected components are the regions in each second grayscale image that may contain dirt.
[0079]
[0080] Where T j (x, y) represents pixel point G. j The grayscale threshold for adaptive threshold segmentation of (x,y), B j (x,y) represents the adaptive threshold segmentation result.
[0081] The absolute value of the difference between the gray value of each pixel in the connected component and the gray value of a pixel outside the connected component is greater than a first threshold, and / or the absolute value of the difference between the gray values of each pixel in the connected component is less than a second threshold.
[0082] In a specific example, the gray values of any two pixels in the connected component are a1 and a2, and the gray value of any pixel outside the connected component is b. The first threshold and the second threshold are both 2. Then |a1-a2| < 2 and |a1-b| > 2. It is understandable that |a1-a2| can also be equal to 2.
[0083] It is understandable that the values of the first threshold and the second threshold can be the same or different, and this application does not impose any restrictions.
[0084] Then, the computer device filters out connected components that meet preset conditions from the connected components of each second grayscale image. The connected components that meet the preset conditions are the dirty areas (i.e., the first sub-dirty areas) of each second grayscale image. The preset conditions include one or more of the following: the area of the connected component is within a first preset range, the aspect ratio of the connected component is within a second preset range, and the roundness of the connected component is within a third preset range.
[0085] In a specific example, the first preset range is 0.01 to 0.02 mm. 2 The second preset range is 1 to 1.2, and the third preset range is 0 to 0.3. If the area of a connected region is 0.015 mm², then...2 If the aspect ratio is 1 and the roundness is 0.2, then the connected component is the dirty region (i.e. the first sub-dirty region) of the second grayscale image.
[0086] Understandably, the above is merely an example and not a limitation.
[0087] Next, the computer device determines the dirt information of each second grayscale image based on the dirty area (i.e., the first sub-dirty area) of each second grayscale image. The dirt information of each second grayscale image is the coordinates of the pixels in the dirty area (i.e., the first sub-dirty area) of each second grayscale image.
[0088] Finally, the computer device upsamples each second grayscale image to map the dirt information onto the first grayscale image, obtaining the coordinates of the dirt area on the first grayscale image. By calculating the geometric center and the bounding rectangle of the dirt area on the first grayscale image, the first dirt information of the first grayscale image is obtained, where the first dirt information includes the position and size of the dirt area.
[0089] In a specific example, a dirty region in a first grayscale image contains n pixels, and the coordinates of the i-th pixel in the dirty region are (x, y, y). i ,y i The maximum column coordinate of the bounding rectangle of the dirty area is x. max The smallest column coordinate is x min The maximum row coordinate is y max The smallest row coordinate is y min The first center coordinates (cx, cy) of the contaminated area are: The first radius of the dirty area is r1 = (x max -x min +y max -y min 4. Based on the first center coordinates (cx, cy) and the first radius r1 of the dirty area, the position and size of the dirty area can be determined. The computer device can calculate the position and size of each dirty area on the first grayscale image, and accumulate the first dirt information of the first grayscale image.
[0090] In this method, the computer device determines the connected components that may be contaminated areas in each second grayscale image based on an adaptive threshold segmentation method. Then, according to preset conditions, it filters out the connected components that are contaminated areas from the connected components of each second grayscale image. The contamination information of the contaminated areas in each second grayscale image is mapped onto the first grayscale image to obtain the first contamination information of the first grayscale image. Since the contaminated areas in each second grayscale image pass the adaptive threshold segmentation and meet the preset conditions, the contaminated areas in each second grayscale image can more realistically reflect the contamination on the camera sensor, improving the reliability of the scheme. Furthermore, the computer device finds contamination in images with lower resolution (i.e., at least one second grayscale image) and then maps it onto the first grayscale image, improving the efficiency of contamination detection.
[0091] In another possible implementation, the computer device determines the second sub-dirty region of each second grayscale image based on the second type of pixel units, and maps the second sub-dirty region onto the first grayscale image. Understandably, the method by which the computer device determines the second sub-dirty region of each second grayscale image based on the second type of pixel units and maps the second sub-dirty region onto the first grayscale image is similar to the method by which the computer device determines the first sub-dirty region of each second grayscale image based on the first type of pixel units and maps the first sub-dirty region onto the first grayscale image; the difference lies in the formula for adaptive threshold segmentation. For example, when processing the second type of pixel units, the principle of adaptive threshold segmentation can be the following formula (8).
[0092]
[0093] In this method, the computer device determines the first type of pixel unit and the second type of pixel unit. Based on the first type of pixel unit, the first sub-dirty region of each second grayscale image is determined, and based on the second type of pixel unit, the second sub-dirty region of each second grayscale image is determined. The dirty region on the first grayscale image can be mapped to the first sub-dirty region, or it can be mapped to the second sub-dirty region, or it can be mapped to both the first and second sub-dirty regions, making the scheme highly flexible.
[0094] S104: Determine the dirt information of the camera sensor based on the first dirt information of the first grayscale image.
[0095] In one possible implementation, the computer device directly determines the dirt information of the camera sensor based on the first dirt information of the first grayscale image. There is a correspondence between the first dirt information of the first grayscale image and the dirt information of the camera sensor; the position of each dirt smudge in the first grayscale image can correspond to a position on the sensor.
[0096] In a specific example, the computer device determines the coordinates of a dirt in the first grayscale image as (x1, y1) based on the first dirt information, and then determines that there is dirt on the sensor with coordinates (x1', y1'), where (x1, y1) and (x1', y1') have a mapping relationship. Correspondingly, the size of the dirt in the first grayscale image and its size on the sensor also have a mapping relationship.
[0097] Using this method, computer equipment can determine the dirt information of the camera sensor based on the first dirt information. The method is simple and the dirt detection efficiency is high.
[0098] In another possible implementation, the computer device could further subdivide the dirt information.
[0099] For example, the computer device first determines at least one Region of Interest (ROI) from the first grayscale image based on the first dirt information of the first grayscale image, calculates the grayscale curve of each ROI, and then determines the second dirt information of the first grayscale image based on the grayscale curve of each ROI. The second dirt information includes at least one of the location, size, and degree of dirt of the dirt area. The dirt information of the camera sensor is determined based on the second dirt information, and the second dirt information of the first grayscale image corresponds to the dirt information of the camera sensor.
[0100] Optionally, the computer equipment can also filter the grayscale curve of each ROI to obtain the location and size of the dirty area corresponding to each ROI; based on the location and size of the dirty area corresponding to each ROI and the grayscale curve of each ROI, the degree of dirtiness of the dirty area corresponding to each ROI can be determined.
[0101] In a specific example, a dirty area in the first grayscale image can correspond to one ROI or multiple ROIs (e.g., Figure 2 and Figure 3 For each of the two ROIs corresponding to the dirty area (here, we take two as an example, but in reality, there can be fewer or more than two), the computer device can obtain a set of third dirt information based on the grayscale curve of an ROI. Thus, a dirty area in the first grayscale image can correspond to a set of third dirt information or multiple sets of third dirt information. The steps for the computer device to obtain a set of third dirt information based on the grayscale curve of an ROI are as follows (1) to (4):
[0102] (1) See Figure 2 The computer device first determines a rectangular region of interest (ROI) from the first grayscale image based on the first dirt information of the first grayscale image, and uses the first center coordinates (cx, cy) of the dirt area where the ROI is located as the center coordinates of the ROI, where the ROI is a rectangular area with height h and width w.
[0103] (2) See Figure 4 The computer device projects the image within the ROI along its height h to obtain the grayscale curve f(t). 0≤t≤w, where I(t+cx-0.5w,y) is the gray value of the pixel at coordinates (t+cx-0.5w,y).
[0104] (3) See Figure 5 , Figure 5 The smoothed curve is the processed grayscale curve g(t). The computer equipment performs two consecutive Gaussian smoothing filters on the grayscale curve f(t) to obtain the processed grayscale curve g(t). 0≤t≤w, where Gauss(i) is the normalized discrete Gaussian kernel weight, and r1 is the first radius of the dirty region where the ROI is located.
[0105] (4) The computer device differentiates the processed grayscale curve g(t), the derivative function d(t) = g(t+1) - g(t), 0≤t≤w, and determines the maximum value d of the derivative function d(t). max With minimum value d min and d max The corresponding x-coordinate t max With minimum value d min The corresponding x-coordinate t min ,For example Figure 6 As shown, Figure 6 The concave portion below the mid-line represents the dirty area in the first grayscale image. A third set of dirty information includes t corresponding to a ROI. max With t min , t max With t min Represents the boundary of a dirty area.
[0106] Next, the computer equipment can determine the second center coordinates (x, y) of the dirty area based on the third dirt information. c ,y c The second center coordinates (x, r2), the second radius r2, and the degree of soiling DN are at least one of the following: c ,y c The second radius r2 and the degree of dirtiness DN belong to the second dirtiness information of the first grayscale image.
[0107] When the computer equipment determines the second center coordinates (x) of the dirty area c ,y c When performing ROI 1 and ROI2, two special ROIs are needed. ROI1 is parallel to the horizontal axis of the coordinate system (e.g., Figure 2Rotate ROI1 by 90° around the first center coordinates (cx, cy) of the contaminated area to obtain ROI2. A third set of contamination information obtained from ROI1 includes t1 corresponding to ROI1. max With t1 min Second center coordinates (x c ,y c ) column coordinates A third set of contamination information obtained based on ROI2 includes t2 corresponding to ROI2. max With t2 min Second center coordinates (x c ,y c Row coordinates
[0108] The second radius of the dirty area The degree of dirtiness in the soiled area Since the dirty area can correspond to one or more sets of third dirty information, the computer device can also calculate multiple dirty degree DNs and multiple second radii r2, and then calculate the average or vote on the multiple dirty degree DNs and multiple second radii r2.
[0109] Finally, the computer equipment uses the second center coordinate (x) of the dirty area. c ,y c The sensor is determined to have dirt on the pixel corresponding to the dirty region by at least one of the following: the second radius r2, the degree of dirt DN, and the center coordinates of the dirt on the sensor are consistent with the second center coordinates (x, y). c ,y c There is a mapping relationship between the radius and the second radius r2, and the degree of dirtiness is the degree of dirtiness DN.
[0110] It is understandable that the shape of the ROI can be rectangular or other shapes. When the shape of the ROI is rectangular, the values of the height h and width w of the ROI can be selected according to actual needs, and this application does not impose any restrictions.
[0111] Optionally, the computer device can also determine whether a technician needs to handle the dirt based on a preset dirt handling threshold. For example, when the dirt level DN of a dirty area is greater than the preset dirt handling threshold thresh, the computer device sends a notification to the technician device, instructing the technician to handle the dirt in that area of the camera sensor; when the dirt level DN of another dirty area is less than or equal to the preset dirt handling threshold thresh, the computer device does not send a notification to the technician device.
[0112] In this method, the computer device, based on the first dirt information of the first grayscale image, further determines the second dirt information of the first grayscale image through the grayscale curve of each ROI. Since the second dirt information more accurately reflects the dirt information of the dirt area of the first grayscale image, the dirt information on the camera sensor is determined based on the second dirt information, thus improving the accuracy of the solution.
[0113] It is understood that the above-described embodiments can be implemented individually or in combination, and this application does not impose any restrictions.
[0114] In the above scheme, the computer device first determines a first grayscale image based on a first image captured by the camera, obtains the first grayscale image, constructs an image pyramid of the first grayscale image, and determines the dirt information of the dirty area on each second grayscale image in the image pyramid. Then, the dirt information on each second grayscale image is mapped onto the first grayscale image to obtain the first dirt information of the dirty area on the first grayscale image. Since the pixels on the first grayscale image correspond to the camera sensor, the dirt information on the camera sensor can be determined based on the first dirt information of the first grayscale image. The computer device finds dirt on images with lower resolution (i.e., at least one second grayscale image) and then maps it onto the first grayscale image, improving the dirt detection efficiency. Furthermore, this scheme can determine the location and size of the dirt, resulting in relatively accurate detection results and meeting the camera's dirt detection requirements.
[0115] The methods provided in the embodiments of this application have been described above. The apparatus provided in the embodiments of this application will be described below.
[0116] See Figure 7 This is a structural diagram of a camera dirt detection device provided in an embodiment of this application. The device includes modules / units / technical means for performing the methods executed by computer devices in the above-described method embodiments.
[0117] For example, the device 700 includes:
[0118] The acquisition module 701 is used to acquire the first image captured by the camera;
[0119] Processing module 702 is configured to: determine a first grayscale image based on the first image; construct an image pyramid based on the first grayscale image, the image pyramid including at least one second grayscale image, wherein the resolution of each second grayscale image is less than the resolution of the first grayscale image; determine dirt information of each second grayscale image; map the dirt information of each second grayscale image onto the first grayscale image to obtain first dirt information of the first grayscale image, wherein the first dirt information includes the position and size of the dirt area; and determine the dirt information of the camera sensor based on the first dirt information of the first grayscale image.
[0120] As one possible product form of the aforementioned device, see [link to product description]. Figure 8 This application also provides an electronic device 800, comprising:
[0121] At least one processor 801; and a communication interface 803 communicatively connected to the at least one processor 801; the at least one processor 801 causes the electronic device 800 to execute the method steps performed by any device in the above method embodiments through the communication interface 803 by executing instructions stored in the memory 802.
[0122] Optionally, the memory 802 is located outside the electronic device 800.
[0123] Optionally, the electronic device 800 includes the memory 802, which is connected to the at least one processor 801, and stores instructions executable by the at least one processor 801. (Appendix) Figure 8 The dashed line indicates that memory 802 is optional for electronic device 800.
[0124] The processor 801 and the memory 802 can be coupled through an interface circuit or integrated together; no restriction is imposed here.
[0125] This application embodiment does not limit the specific connection medium between the processor 801, memory 802, and communication interface 803. This application embodiment... Figure 8 The processor 801, memory 802, and communication interface 803 are connected via a bus 804. Figure 8 The connections between other components are shown in bold and are for illustrative purposes only, not as limiting information. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0126] It should be understood that the processor mentioned in the embodiments of this application can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.
[0127] For example, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0128] It should be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct RAM (DR RAM).
[0129] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) can be integrated into the processor.
[0130] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.
[0131] As another possible product form, this application embodiment also provides a computer-readable storage medium for storing instructions that, when executed, cause a computer to perform the method steps performed by the first device in the above method example.
[0132] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0133] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0136] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method for detecting dirt in a camera, characterized in that, include: Acquire a first image captured by the camera, and determine a first grayscale image based on the first image; An image pyramid is constructed based on the first grayscale image, the image pyramid including at least one second grayscale image, and the resolution of each second grayscale image in the at least one second grayscale image is less than the resolution of the first grayscale image; Determine the dirt information of each of the second grayscale images; The dirt information of each second grayscale image is mapped onto the first grayscale image to obtain the first dirt information of the first grayscale image, wherein the first dirt information includes the position and size of the dirt area; Based on the first dirt information of the first grayscale image, determine the dirt information of the camera's sensor; The step of determining the dirt information of each second grayscale image includes: The connected components of each second grayscale image are determined based on an adaptive threshold segmentation method, wherein the absolute value of the difference between the grayscale value of each pixel in the connected component and the grayscale value of the pixel outside the connected component is greater than a first threshold, and / or the absolute value of the difference between the grayscale values of each pixel in the connected component is less than a second threshold. Connected components that meet preset conditions are selected from the connected components of each second grayscale image, and the connected components that meet the preset conditions are the dirty areas of each second grayscale image. The dirt information of each second grayscale image is determined based on the dirty areas of each second grayscale image.
2. The method as described in claim 1, characterized in that, After determining the first grayscale image based on the first image and before constructing the image pyramid based on the first grayscale image, the method further includes: determining a first type of pixel unit in the first grayscale image according to a preset grayscale value range, wherein the grayscale value of the first type of pixel unit is less than the minimum value in the preset grayscale value range; determining the dirt information of each second grayscale image and mapping the dirt information of each second grayscale image onto the first grayscale image includes: determining a first sub-dirty region of each second grayscale image based on the first type of pixel unit and mapping the first sub-dirty region onto the first grayscale image; and / or, After determining the first grayscale image based on the first image and before constructing the image pyramid based on the first grayscale image, the method further includes: determining a second type of pixel unit of the first grayscale image according to a preset grayscale value range, wherein the grayscale value of the second type of pixel unit is greater than the maximum value in the preset grayscale value range; determining the dirt information of each second grayscale image and mapping the dirt information of each second grayscale image onto the first grayscale image includes: determining a second sub-dirty region of each second grayscale image based on the second type of pixel unit and mapping the second sub-dirty region onto the first grayscale image.
3. The method as described in claim 2, characterized in that, Before determining the first sub-dirty region of each second grayscale image based on the first type of pixel units, the method further includes: If the second type of pixel unit exists, process the second type of pixel unit as the first type of pixel unit; or... Before determining the second sub-dirty region of each second grayscale image based on the second type of pixel units, the method further includes: If the first type of pixel unit exists, the first type of pixel unit is processed into the second type of pixel unit.
4. The method as described in claim 1, characterized in that, The preset conditions include one or more of the following: the area of the connected region is within a first preset range, the aspect ratio of the connected region is within a second preset range, and the roundness of the connected region is within a third preset range.
5. The method according to any one of claims 1-4, characterized in that, The step of determining the dirt information of the camera sensor based on the first dirt information of the first grayscale image includes: At least one region of interest (ROI) is determined from the first grayscale image based on the first dirt information of the first grayscale image; Calculate the grayscale curve of each ROI in the at least one ROI; The second dirt information of the first grayscale image is determined based on the grayscale curve of each ROI, and the second dirt information includes at least one of the location, size and degree of dirt of the dirt area; The dirt information of the camera's sensor is determined based on the second dirt information.
6. The method as described in claim 5, characterized in that, The second dirt information of the first grayscale image is determined based on the grayscale curve of each ROI, including: The grayscale curve of each ROI is filtered to obtain the location and size of the dirty area corresponding to each ROI; The degree of dirtiness of the dirt area corresponding to each ROI is determined based on the location and size of the dirt area corresponding to each ROI and the grayscale curve of each ROI.
7. A camera dirt detection device, characterized in that, include: The acquisition module is used to acquire the first image captured by the camera; The processing module is used to determine a first grayscale image based on the first image; An image pyramid is constructed based on the first grayscale image, the image pyramid including at least one second grayscale image, and the resolution of each second grayscale image in the at least one second grayscale image is less than the resolution of the first grayscale image; Determine the dirt information of each of the second grayscale images; The dirt information of each second grayscale image is mapped onto the first grayscale image to obtain the first dirt information of the first grayscale image, wherein the first dirt information includes the position and size of the dirt area; based on the first dirt information of the first grayscale image, the dirt information of the camera sensor is determined. Wherein, when determining the dirt information of each second grayscale image, the processing module is used to: The connected components of each second grayscale image are determined based on an adaptive threshold segmentation method, wherein the absolute value of the difference between the grayscale value of each pixel in the connected component and the grayscale value of the pixel outside the connected component is greater than a first threshold, and / or the absolute value of the difference between the grayscale values of each pixel in the connected component is less than a second threshold. Connected components that meet preset conditions are selected from the connected components of each second grayscale image, and the connected components that meet the preset conditions are the dirty areas of each second grayscale image. The dirt information of each second grayscale image is determined based on the dirty areas of each second grayscale image.
8. The apparatus as claimed in claim 7, characterized in that, When the processing module determines the dirt information of the camera sensor based on the first dirt information of the first grayscale image, it is specifically used for: At least one region of interest (ROI) is determined from the first grayscale image based on the first dirt information of the first grayscale image; Calculate the grayscale curve of each ROI in the at least one ROI; The second dirt information of the first grayscale image is determined based on the grayscale curve of each ROI, and the second dirt information includes at least one of the location, size and degree of dirt of the dirt area; The dirt information of the camera's sensor is determined based on the second dirt information.
9. An electronic device, characterized in that, include: At least one processor; And a memory and a communication interface that are communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, by executing the instructions stored in the memory, causes the electronic device to perform the method as described in any one of claims 1-6 through the communication interface.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1-6.