An image processing-based stain recognition detection system

Abstract

The application relates to the technical field of image recognition and discloses a stain identification and detection system based on image processing, which comprises an image acquisition module, an environment perception module, an optimization and adjustment module and a stain identification module; wherein: the image acquisition module is used for acquiring a ground image; the environment perception module is used for acquiring an environment illumination condition; the optimization and adjustment module controls illumination compensation of the acquisition of the ground image based on the environment illumination condition; the optimization and adjustment module also identifies and corrects a shadow of the ground image based on the environment illumination condition; the stain identification module stores reference images of grounds with different materials; the stain identification module identifies and detects stains based on the ground image and the reference images of the grounds with corresponding materials; the application improves the stain identification and detection efficiency of the ground image and reduces the image processing cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to a stain recognition and detection system based on image processing. BACKGROUND

[0002] Currently, some driving-type scrubbers begin to try to use cameras to collect ground images to assist in judging the cleaning effect. Some simple vision systems can recognize obvious stains on the ground and roughly judge the change of the ground state before and after cleaning, providing certain data support for cleaning work. This improves the pertinence of cleaning to some extent, enabling the operator to find areas that are not cleaned well in time and perform secondary cleaning. However, the existing stain recognition and detection system still has many shortcomings. In terms of light adaptability, the existing stain recognition system faces challenges. Different working environment light conditions differ greatly, from various indoor light illuminations to outdoor natural light changes, and the light intensity and direction are unstable. Under strong light directivity, the collected ground image is prone to overexposure, resulting in loss of stain details and inaccurate recognition; and in a dimly lit environment, the image becomes unclear and the noise increases, also affecting the judgment of stains. The current technology is difficult to automatically and accurately adjust the image acquisition parameters according to the real-time changes of light, so that under complex light conditions, the accuracy and stability of stain recognition are greatly reduced. The shadow interference problem also affects the reliability of the cleanliness monitoring. The shadow generated by the driving-type scrubber during operation is inevitable. The existing stain recognition system is difficult to accurately distinguish between shadow areas and stain areas, often misjudging shadows as stains or missing the detection of real stains in shadow-covered areas. For cleanliness monitoring of different material surfaces, the existing technology has obvious shortcomings. Ground materials are diverse, such as ceramic tiles, marble, cement, etc., each of which has different stain absorption capacity, stain presentation and surface texture. The existing stain recognition system lacks self-adaptability to different material surfaces, and when switching between multiple material surfaces, it cannot accurately adjust the parameters of the recognition algorithm, resulting in decreased monitoring accuracy. In addition, when the ground has complex patterns or textures, the difficulty of stain recognition is further increased. These patterns and textures will interfere with the judgment of stains, making the system prone to misjudging patterns as stains or ignoring real stains hidden in patterns, causing misjudgment.

[0003] The patent application with the publication number CN114723767A discloses a stain detection method, device, electronic equipment and robot sweeper system, the method comprises: acquiring a to-be-detected image containing a ground; grouping each pixel point in the to-be-detected image based on the similarity between target attributes of the pixel points, to obtain a segmentation image; wherein, the pixel points belonging to different groups in the segmentation image are marked with different colors; the target attributes include color values of the pixel points; determining the edges of objects in the segmentation image based on the differences between pixel values of each pixel point in the segmentation image; in the case that the edges of one determined object meet the preset stain edge characteristics, it is determined that there is a stain on the ground corresponding to the object. In this way, the use complexity of the robot sweeper can be reduced, and the storage space of the robot sweeper can be reduced.

[0004] The patent application with the publication number CN115049733A discloses a stain intelligent cleaning system and method based on a depth camera, the system comprises: a robot sweeper, a depth camera, a stain identification module, a navigation module and a cleaning device; the depth camera acquires a color image and a depth image, and acquires a color point cloud in a camera coordinate system according to the color image and the depth image, so as to obtain a color image containing only the ground and send it to the stain identification module; the stain identification module extracts the color and contour features of the stain itself from the color image containing only the ground through a GM M algorithm, and generates stain coordinates in a world coordinate system and sends them to the navigation module; the navigation module generates a navigation route of the robot sweeper according to the stain coordinates in the world coordinate system, and guides the robot sweeper to the stain position to clean the stain using the cleaning device.

[0005] The above prior art has the problem proposed in the background: when there is shadow interference and complex ground pattern interference, the accuracy of stain recognition and detection is difficult to guarantee.

[0006] The information disclosed in this background section is only intended to increase the understanding of the overall background of the present invention, and should not be regarded as acknowledging or implying in any form that the information constitutes prior art known to those of ordinary skill in the art. SUMMARY

[0007] The technical problem to be solved by the present application is to overcome the defects of the prior art, provide a stain recognition and detection system based on image processing, improve the efficiency of stain recognition and detection in a ground image, and reduce the cost of image processing.

[0008] To solve the above technical problems, the present application provides the following technical solutions:

[0009] A stain recognition and detection system based on image processing, comprising an image acquisition module, an environment perception module, an optimization and adjustment module, and a stain recognition module; wherein:

[0010] The image acquisition module is configured to acquire the ground image;

[0011] The environment perception module is configured to obtain the environment lighting condition;

[0012] The optimization adjustment module is configured to perform lighting compensation control on the acquisition of the ground image based on the environment lighting condition; the optimization adjustment module is also configured to perform shadow identification and correction on the ground image based on the environment lighting condition;

[0013] The stain identification module stores reference images of ground surfaces of different materials; the stain identification module is configured to perform stain identification detection based on the ground image and the reference images of ground surfaces of corresponding materials.

[0014] As a preferred scheme of the stain identification detection system based on image processing, the image acquisition module comprises a camera unit, a light supplement unit, and a control unit; wherein:

[0015] The camera unit is configured with a camera; the camera is configured to acquire the ground image based on set camera parameters; the camera parameters comprise exposure time, gain coefficient, and aperture size;

[0016] The light supplement unit is configured with a light supplement lamp; the light supplement lamp is configured to supplement light when acquiring the ground image;

[0017] The control unit is configured to control the camera parameters of the camera unit and the light supplement intensity of the light supplement lamp.

[0018] As a preferred scheme of the stain identification detection system based on image processing, the environment lighting condition comprises camera light intensity;

[0019] The environment perception module comprises a first perception unit; the first perception unit is configured to obtain the camera light intensity;

[0020] The first perception unit comprises a light intensity sensor arranged at the camera; the first perception unit is configured to obtain the environment lighting intensity at the camera as the camera light intensity based on the light intensity sensor;

[0021] As a preferred scheme of the stain identification detection system based on image processing, the environment lighting condition further comprises a dominant lighting direction; the environment perception module further comprises a second perception unit; the second perception unit is configured to obtain the dominant lighting direction; the second perception unit comprises light intensity sensors installed at different positions on the driver-type scrubber; the light intensity sensor installed at any position is configured to acquire the environment lighting intensity in one direction only;

[0022] The second sensing unit is configured with a light intensity difference threshold value; the method for obtaining the dominant light direction is as follows: collecting light intensity of each direction; if the difference between the light intensities of the two directions with the maximum light intensity is greater than the light intensity difference threshold value, then the direction corresponding to the larger light intensity is the dominant light direction; otherwise, there is no dominant light direction.

[0023] As a preferred scheme of the stain recognition and detection system based on image processing, the optimization adjustment module comprises an illumination compensation unit; the illumination compensation unit controls illumination compensation of the ground image based on the camera light intensity, specifically comprising:

[0024] establishing a mapping relationship between the camera light intensity, the camera parameter, the supplementary light intensity and the average brightness of the ground image;

[0025] setting a target brightness of the ground image;

[0026] collecting the camera light intensity; calculating the optimal solution of the camera parameter and the supplementary light intensity based on the mapping relationship and the camera light intensity;

[0027] sending the optimal solution of the camera parameter and the supplementary light intensity to the control unit; the control unit controls the camera parameter of the camera unit and the supplementary light intensity of the supplementary light unit based on the optimal solution.

[0028] As a preferred scheme of the stain recognition and detection system based on image processing, the optimization adjustment module further comprises a shadow correction unit; the shadow correction unit performs shadow recognition and correction on the ground image based on the dominant light direction, specifically comprising:

[0029] if the dominant light direction exists, performing shadow recognition and correction on the ground image;

[0030] obtaining a direction included angle between the dominant light direction and the driving direction of the driving type scrubber;

[0031] pre-marking a shadow area in the ground image based on the direction included angle;

[0032] converting the ground image from RGB space to HSV space and obtaining the lightness of each pixel point;

[0033] performing shadow detection based on the lightness of the pixel point by referring to the pre-marked shadow area; dividing the ground image into a shadow area and a non-shadow area;

[0034] performing compensation correction on the shadow area based on the average lightness of the non-shadow area;

[0035] converting the ground image after compensation correction back to RGB space.

[0036] As a preferred scheme of the image processing-based stain recognition and detection system, the shadow detection based on the brightness of the pixel points specifically comprises:

[0037] M pixel points are selected from the pre-marked shadow area, and the average brightness of the M pixel points is calculated; a brightness difference threshold range is set, and the difference between the brightness of each pixel point and the average brightness of the M pixel points is calculated in sequence; if the difference between the brightness of any pixel point and the average brightness of the M pixel points is within the brightness difference threshold range, the pixel point is marked as a shadow point; and the shadow points are connected to form a shadow area.

[0038] As a preferred scheme of the image processing-based stain recognition and detection system, the compensation and correction of the shadow area based on the average brightness of the non-shadow area specifically comprises: calculating the average brightness of all pixel points in the non-shadow area as a correction brightness; and compensating and correcting the shadow area based on the correction brightness, specifically comprising: adjusting the pixel value of each pixel point in the shadow area to the correction brightness.

[0039] As a preferred scheme of the image processing-based stain recognition and detection system, the stain recognition module comprises a comparison unit; and the comparison unit is configured to perform stain recognition and detection.

[0040] The comparison unit stores reference images of ground surfaces of different materials; and the reference image of the ground surface of any material is a ground surface image collected by the camera unit under the condition that there is no stain, no shadow, and uniform illumination, and the average brightness of the reference image is the target brightness.

[0041] The stain recognition and detection specifically comprises:

[0042] The ground surface material corresponding to the real-time collected ground surface image is recognized.

[0043] The reference image of the ground surface of the corresponding material is obtained.

[0044] The real-time collected ground surface image is aligned with the reference image based on a feature point matching algorithm.

[0045] The stain area in the ground surface image is recognized based on the aligned ground surface image and the reference image.

[0046] As a preferred scheme of the image processing-based stain recognition and detection system, the recognition of the stain area in the ground surface image based on the aligned ground surface image and the reference image specifically comprises: calculating the pixel difference of the pixel points at the corresponding positions in the ground surface image and the reference image point by point; the comparison unit is further configured with a pixel difference threshold; in the ground surface image, the pixel points with the pixel difference greater than the pixel difference threshold are marked as stain points; and the stain points in the ground surface image are connected to form a stain area.

[0047] As a preferred scheme of the image processing based stain recognition detection system, the aligned ground image and the reference image are used to recognize the stain area in the ground image, and the method further comprises:

[0048] The aligned ground image and the reference image are converted to HSV space, and the brightness of each pixel in the ground image and the reference image is recorded.

[0049] The brightness of the corresponding pixels in the ground image and the reference image is compared point by point, and a brightness residual matrix is constructed; the brightness residual matrix has the same dimension as the pixel matrix of the ground image; any element in the brightness residual matrix corresponds to the pixel in the same position in the pixel matrix of the ground image, and the element value is the difference between the brightness of the corresponding pixels in the ground image and the reference image.

[0050] The stain area in the ground image is recognized based on the element value in the brightness residual matrix.

[0051] As a preferred scheme of the image processing based stain recognition detection system, the stain area in the ground image is recognized based on the element value in the brightness residual matrix, and the method specifically comprises:

[0052] At least N sub-matrices are obtained from the brightness residual matrix by sliding window sliding.

[0053] The variance of all element values in each sub-matrix is calculated.

[0054] If the variance of all element values in all sub-matrices is less than a preset variance threshold, there is no stain area in the ground image; if the variance of the element values in at least one sub-matrix is greater than the preset variance threshold, the corresponding sub-matrix is marked as a potential stain area.

[0055] The gradient direction of the element values in the potential stain area is determined.

[0056] The gradient direction concentration of the potential stain area is calculated based on the gradient direction of the element values; if the gradient direction concentration is greater than a preset concentration threshold, the potential stain area is a non-stain area; if the gradient direction concentration is less than or equal to the preset concentration threshold, the potential stain area is a stain area.

[0057] Compared with the prior art, the present application has the following beneficial effects:

[0058] The environment sensing module acquires the environment light condition, the optimization adjustment module establishes a mapping relationship according to the environment light condition, and calculates the optimal solution of the camera parameter and the light compensation parameter, so as to realize light compensation control, ensure that clear and high-quality ground images can be acquired under various light conditions, and improve the stability and accuracy of stain identification and detection.

[0059] The shadow in the ground image is identified and corrected based on the dominant light direction. The shadow area is pre-marked, the shadow is detected and segmented in the HSV space, and the shadow is compensated and corrected according to the average brightness of the non-shadow area, and then converted back to the RGB space, so as to avoid that the shadow is misjudged as a stain and improve the accuracy of visual identification.

[0060] By pre-acquiring reference images of ground surfaces of different materials, the ground material is identified and the corresponding reference image is acquired, the reference image is aligned with the real-time ground image through a feature point matching algorithm, and then the pixel difference is compared point by point, so that the stain area can be accurately identified, the interference of complex patterns on stain identification is reduced, and the detection result of the stain area is more reliable. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0062] Figure 1 The structure diagram of the stain identification and detection system based on image processing provided by the present application is shown.

[0063] Figure 2 The example diagram of aligning the ground image with the reference image based on feature point matching provided by the present application is shown. DETAILED DESCRIPTION

[0064] The technical solutions of the present application will be described in detail below with the help of the drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solutions of the present application, but not limitations of the technical solutions of the present application. In the case of no conflict, the technical features in the embodiments of the present application and the embodiments can be combined with each other.

[0065] The present embodiment introduces a stain identification and detection system based on image processing, which is referred to Figure 1 The system includes an image acquisition module, an environment sensing module, an optimization adjustment module, and a stain identification module.

[0066] The image acquisition module is used for acquiring ground images.

[0067] The image acquisition module includes a camera unit, a supplementary lighting unit, and a control unit; wherein: the camera unit is equipped with a camera; the camera acquires ground images based on set camera parameters; the camera parameters include exposure time, gain coefficient, and aperture size;

[0068] The supplementary lighting unit is equipped with a supplementary light; the supplementary light is used to provide supplementary lighting when acquiring ground images; the supplementary light is an adjustable brightness LED light with a fast response characteristic, which can quickly adjust the light brightness according to control; the position and angle of the supplementary light should be designed and tested to avoid shadows and reflections, and to ensure that the light is evenly illuminated on the ground area where images need to be acquired.

[0069] The control unit is used to control the camera parameters of the camera unit and the illumination intensity of the fill light.

[0070] The environmental sensing module is used to obtain ambient lighting conditions;

[0071] The ambient lighting conditions include camera light intensity and dominant lighting direction;

[0072] The environmental perception module includes a first perception unit and a second perception unit; wherein, the first perception unit is used to acquire the camera light intensity;

[0073] The first sensing unit includes a light intensity sensor disposed at the camera; the first sensing unit obtains the ambient light intensity at the camera based on the light intensity sensor, and uses it as the camera light intensity.

[0074] The second sensing unit is used to acquire the dominant light direction; the second sensing unit includes light intensity sensors installed at different locations on the ride-on floor scrubber, and the light intensity sensor installed at any location only collects the ambient light intensity in one direction; for example, four light intensity sensors (excluding the light intensity sensor of the first sensing unit) are installed on the ride-on floor scrubber to collect the ambient light intensity in front of, behind, to the left and to the right of the ride-on floor scrubber (with the forward direction of the ride-on floor scrubber as the front); each light intensity sensor blocks ambient light from other directions by shading;

[0075] The second sensing unit is configured with a light intensity difference threshold; the method by which the second sensing unit obtains the dominant illumination direction is as follows: collect the light intensity of each direction; if the difference between the light intensity of the two directions with the largest light intensity is greater than the light intensity difference threshold, then the direction corresponding to the larger light intensity is the dominant illumination direction; otherwise, there is no dominant illumination direction.

[0076] The optimization and adjustment module performs illumination compensation control on the acquired ground image based on the ambient lighting conditions; the optimization and adjustment module also performs shadow recognition and correction on the ground image based on the ambient lighting conditions;

[0077] The optimization adjustment module comprises an illumination compensation unit; the illumination compensation unit controls illumination compensation of collection of the ground image based on the camera light intensity, and specifically comprises:

[0078] A mapping relationship between the camera light intensity, the camera parameter, the supplementary light intensity and the average brightness of the ground image is established;

[0079] For example, a linear regression model is constructed to record the mapping relationship by taking the camera light intensity, the camera parameter (including exposure time, gain coefficient, aperture size) and the supplementary light intensity as independent variables and taking the average brightness of the ground image as a dependent variable. A series of value schemes of the camera parameter and the supplementary light intensity are set under different camera light intensities, and an experiment of collecting the ground image is performed, and experimental data are recorded; wherein the average brightness of the ground image is calculated as follows: the ground image is converted into a gray image; the average value of the gray values of all pixels in the gray image is calculated as the average brightness of the ground image. The linear regression model is fitted by the experimental data, and the mapping relationship is obtained.

[0080] A target brightness of the ground image is set; it is crucial to keep the brightness of the ground image collected each time consistent and stable. Under different ambient light, if no parameter adjustment is performed, the image can be too bright or too dark, resulting in loss of details. By establishing the mapping relationship, setting the target brightness and adjusting the camera parameter and the supplementary light intensity, the brightness of the ground image can always be in a suitable range, such as reducing exposure under strong light to avoid over-brightness, increasing exposure or supplementary light under weak light to prevent over-darkness, and ensuring that the overall brightness of the image is uniform and can clearly display the content.

[0081] The camera light intensity is collected; the optimal solution of the camera parameter and the supplementary light intensity is calculated based on the mapping relationship and the camera light intensity; a linear quadratic programming algorithm or the like can be used to calculate the optimal solution of the camera parameter and the supplementary light intensity. On the basis that the linear regression model has been fitted, the camera light intensity is known and the target brightness is given, the linear quadratic programming can find the optimal solution of the camera parameter and the supplementary light intensity, so that when the camera parameter and the supplementary light intensity are set according to the optimal solution, the brightness of the ground image collected can be ensured to be the target brightness set.

[0082] The optimal solution of the camera parameter and the supplementary light intensity is sent to a control unit; the control unit controls the camera parameter of the camera unit and the supplementary light intensity of the supplementary light unit based on the optimal solution. By real-time analysis of the surrounding light conditions and automatic adjustment of the camera parameter, clear and high-quality ground images can be obtained under various illumination conditions, and the stability and accuracy of the stain area monitoring are improved.

[0083] The optimization adjustment module further comprises a shadow correction unit; the shadow correction unit performs shadow identification and correction on the ground image based on the dominant light direction, specifically comprising:

[0084] If the dominant light direction exists, the ground image is identified and corrected for shadow;

[0085] Obtain the direction angle between the dominant light direction and the driving direction of the driving scrubber;

[0086] Pre-mark the shadow area in the ground image based on the direction angle; through experiments, the range of the shadow appearing in the ground image corresponding to the angle between the dominant light direction and the driving direction is determined, and the corresponding relationship between the angle and the shadow area is established; in actual application, based on the above corresponding relationship, the approximate area of the shadow appearing is determined according to the direction angle. For example, when the scrubber drives forward, and the dominant light is incident from the front, the shadow generated by the scrubber itself will appear on the cleaning path behind the machine body.

[0087] Convert the ground image from RGB space to HSV space, and obtain the brightness of each pixel point;

[0088] Referring to the pre-marked shadow area, the brightness of the pixel point is used for shadow detection; the ground image is divided into a shadow area and a non-shadow area; for example, M pixel points are selected from the pre-marked shadow area, and the average brightness of the M pixel points is calculated; M is a positive integer; a brightness difference threshold range is set, and the difference between the brightness of each pixel point and the average brightness of the M pixel points is calculated in turn, if the difference is within the brightness difference threshold range, the pixel point is marked as a shadow point; the shadow points are connected into a shadow area, and the isolated shadow points are removed; the pixel points outside the shadow area constitute a non-shadow area.

[0089] The shadow area is compensated and corrected based on the average brightness of the non-shadow area; specifically, the average brightness of all pixel points in the non-shadow area is calculated as a correction brightness; the shadow area is compensated and corrected based on the correction brightness, specifically including: adjusting the pixel value of each pixel point in the shadow area to the correction brightness.

[0090] Convert the ground image after compensation and correction back to RGB space.

[0091] In the RGB color space, the color of each pixel is composed of three components of R, G and B. When the intensity of the ambient light increases, the values of the three color components will theoretically increase, resulting in the overall image becoming brighter; the shadow blockage will make the value of each color component decrease, and the image becomes dark. However, the influence of light and shadow on R, G and B components is nonlinear, so it is difficult to directly perform accurate shadow compensation in the RGB space. The HSV space describes each pixel by lightness, saturation and hue, wherein the change of the ambient light mainly significantly affects the lightness, and theoretically has no obvious effect on the saturation and hue. Therefore, in the HSV space, the shadow compensation is performed by lightness, which can more accurately eliminate the influence of the shadow on the ground image. The RGB space can provide more rich color information and is more suitable for stain identification, so the ground image is converted back to the RGB space after the shadow compensation.

[0092] When there is a dominant light direction, it means that the intensity of the ambient light in a certain direction is obviously stronger than that in other directions for the driver-type scrubber, and in this case, the shadow of the scrubber is likely to be left on the ground image. When the driver-type scrubber is working, the shadow generated by itself may be misjudged as a stain. Through shadow detection and compensation, the interference of the shadow on the stain judgment can be avoided, and the accuracy of visual recognition is improved.

[0093] The stain identification module stores reference images of ground surfaces of different materials; the stain identification module performs stain identification detection based on the ground image and the reference image of the ground surface of the corresponding material.

[0094] The stain identification module includes a comparison unit; the comparison unit is used for stain identification detection;

[0095] The comparison unit stores reference images of ground surfaces of different materials; the reference image of the ground surface of any material is a ground image collected by the camera unit under the condition of no stain, no shadow and uniform light, and the average brightness of the reference image is the target brightness;

[0096] The stain identification detection specifically includes:

[0097] Identify the ground material corresponding to the real-time collected ground image;

[0098] Obtain the reference image of the ground surface of the corresponding material;

[0099] Based on the feature point matching algorithm, the real-time collected ground image is aligned with the reference image; first, the feature points are detected and marked in the ground image and the reference image respectively through the feature point detection algorithm, and then the feature points in the ground image and the reference image are matched based on the feature point matching algorithm; the ground image and the reference image are aligned based on the matched feature points, so as to compare the pixel points between them one by one, thereby judging the cleanliness of the corresponding ground position of each pixel point.

[0100] The aligned ground image and the reference image are used to identify the stain area in the ground image, specifically including:

[0101] The pixel difference of the pixel points at the corresponding positions in the ground image and the reference image is calculated point by point; the comparison unit is also provided with a pixel difference threshold; in the ground image, the pixels with the pixel difference greater than the pixel difference threshold are marked as stain points; the stain points in the ground image are connected to form a stain area. The isolated stain points that cannot be connected with other stain points are removed.

[0102] Preferably, the embodiment provides another way to identify the stain area in the ground image based on the aligned ground image and the reference image, specifically including:

[0103] The aligned ground image and the reference image are converted to HSV space, and the brightness of each pixel point in the ground image and the reference image is recorded;

[0104] The brightness of the pixel points at the corresponding positions in the ground image and the reference image is compared point by point, and a brightness residual matrix is constructed; the brightness residual matrix has the same dimension as the pixel matrix of the ground image; any element in the brightness residual matrix corresponds to the pixel point at the same position in the pixel matrix of the ground image, and the element value is the difference between the brightness of the pixel points at the corresponding positions in the ground image and the reference image;

[0105] The stain area in the ground image is identified based on the element value in the brightness residual matrix; specifically including:

[0106] At least N sub-matrices are slid and cut from the brightness residual matrix through a sliding window, N being a positive integer;

[0107] The variance of all element values in each sub-matrix is calculated;

[0108] If the variance of all element values of all sub-matrices is less than a preset variance threshold, there is no stain area in the ground image; if the variance of the element values of at least one sub-matrix is greater than the preset variance threshold, the corresponding sub-matrix is marked as a potential stain area; the position of the potential stain area in the brightness residual matrix is recorded;

[0109] The gradient direction of the element value in the potential stain area is determined;

[0110] a gradient direction concentration of the potential stain region is calculated based on the gradient direction of the element values; if the gradient direction concentration is greater than a preset concentration threshold, the potential stain region is a non-stain region; if the gradient direction concentration is less than or equal to the preset concentration threshold, the potential stain region is a stain region, and a position of the corresponding stain region in the ground image is determined based on a position of the potential stain region in the luminance residual matrix.

[0111] Preferably, the embodiment provides a calculation formula of the gradient direction concentration, as follows:

[0112]

[0113] wherein C represents the gradient direction concentration of the potential stain region; N1 represents a number of element values with gradient directions in the potential stain region; θi represents a gradient direction of the i-th element value with a gradient direction, recorded in the form of an angle; i wherein θi represents a gradient direction of the i-th element value with a gradient direction, recorded in the form of an angle; wherein θ represents an average gradient direction of the N1 element values with gradient directions; cos(·) represents a cosine value.

[0114] Since the cosine value ranges from 0 to 1, according to the calculation formula, it is obvious that the gradient direction concentration C also ranges from 0 to 1. The greater the value of C is, the more regular and consistent the gradient directions of the element values in the corresponding sub-matrix of the potential stain region are; the smaller the value of C is, the more scattered and random the gradient directions of the different element values are. When C is greater than the preset concentration threshold, it indicates that the reason for the large variance of the element values in the corresponding sub-matrix is the overall luminance deviation caused by ground reflection, shadow overcorrection or insufficient correction, etc., which is not caused by stains. When C is less than or equal to the preset concentration threshold, it indicates that the reason for the large variance of the element values in the corresponding sub-matrix is the local luminance irregular mutation caused by the absorption and scattering of stains to the ambient light, and thus the corresponding sub-matrix (i.e. the potential stain region) contains stains.

[0115] The stain detection method based on the regularity analysis of the luminance deviation between the ground image and the reference image provided by the embodiment can distinguish the characteristics of regular deviation (such as reflection and shadow correction error) and local mutation deviation (stains), reduce the misjudgment of the stain region, and improve the detection reliability.

[0116] The present application provides an example image in which the ground image and the reference image are aligned based on feature point matching, as shown in Figure 2 The ground of a certain material has complex patterns, which obviously interfere with the identification of the stain region. Figure 2In the two images, two square marks or two triangular marks or two solid dots represent a group of matching feature points. The three groups of matching feature points can be used as a reference system to overlap the two images, and the excess part of the reference image (i.e., the part of the reference image larger than the ground image) is cut off; if necessary, the coordinate system between the ground image and the reference image can also be aligned through affine transformation (suitable for extreme cases where the camera is disturbed and shaken unexpectedly). When the two images are aligned through feature point matching, it can be determined which pixel points in the ground image correspond to the pattern and which position is the stain, thereby reducing false positives.

[0117] Different materials of the ground (such as ceramic tiles, marble, cement ground, etc.) have different absorption and presentation of stains. The existing visual recognition system has a decrease in cleaning degree monitoring accuracy when switching between various material grounds. In addition, the ground with complex patterns or textures also causes great interference to visual recognition. The present application sets a reference image for each material ground, aligns the reference image with the real-time collected ground image, and then detects the stain area by point-by-point comparison, so that the cleaning degree evaluation is more accurate and reliable.

[0118] Preferably, the stain recognition module further comprises a stain recognition unit; the stain recognition unit is configured to identify the stain type in the ground image.

[0119] The stain recognition unit is configured with an image segmentation algorithm and a trained target detection model; the stain recognition unit segments the stain area from the ground image through the image segmentation algorithm; and then identifies the stain type through the target detection algorithm.

[0120] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0121] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above specific embodiments, which are merely illustrative and not limiting. Those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose and scope of the present application, which are all within the protection of the present application.

Claims

1. A stain recognition and detection system based on image processing, characterized in that: It includes an image acquisition module, an environmental perception module, an optimization and adjustment module, and a stain recognition module; among which: The image acquisition module is used to acquire ground images; The environmental sensing module is used to acquire ambient lighting conditions; the ambient lighting conditions include camera light intensity and dominant lighting direction; The environmental perception module is configured with a light intensity difference threshold; the method for obtaining the dominant illumination direction is as follows: collect the light intensity in each direction; if the difference between the light intensity of the two directions with the largest light intensity is greater than the light intensity difference threshold, then the direction corresponding to the larger light intensity is the dominant illumination direction; otherwise, there is no dominant illumination direction. The optimization and adjustment module performs illumination compensation control on the acquired ground image based on the ambient lighting conditions; the optimization and adjustment module also performs shadow recognition and correction on the ground image based on the ambient lighting conditions; The optimization and adjustment module includes a shadow correction unit; the shadow correction unit performs shadow recognition and correction on the ground image based on the dominant illumination direction, specifically including: If a dominant lighting direction exists, shadow identification and correction are performed on the ground image; Obtain the angle between the dominant lighting direction and the forward direction of the ride-on floor scrubber; Based on the directional angle, pre-mark the shaded area in the ground image; Convert the ground image from RGB space to HSV space and obtain the brightness of each pixel; Based on the pre-labeled shadow region, shadow detection is performed according to the brightness of the pixels; the ground image is segmented into shadow regions and non-shadow regions. The shadow area is compensated and corrected based on the average brightness of the non-shadow area; Convert the compensated and corrected ground image back to RGB space; The stain recognition module stores reference images of different floor materials; the stain recognition module performs stain recognition and detection based on the floor images and the corresponding reference images of the floor materials.

2. The stain recognition and detection system based on image processing as described in claim 1, characterized in that: The image acquisition module includes a camera unit, a supplementary lighting unit, and a control unit; wherein: The camera unit is equipped with a camera; the camera acquires ground images based on set camera parameters; the camera parameters include exposure time, gain coefficient, and aperture size; The supplementary lighting unit is equipped with a supplementary light; the supplementary light is used to provide supplementary lighting when acquiring ground images; The control unit is used to control the camera parameters of the camera unit and the illumination intensity of the fill light.

3. The stain recognition and detection system based on image processing as described in claim 2, characterized in that: The environmental perception module obtains the ambient light intensity at the camera location through a light intensity sensor installed at the camera, which is used as the camera light intensity. The environmental perception module also includes light intensity sensors installed at different locations on the ride-on floor scrubber. Each light intensity sensor installed at any location only collects the ambient light intensity in one direction.

4. The stain recognition and detection system based on image processing as described in claim 3, characterized in that: The optimization and adjustment module includes a lighting compensation unit; The illumination compensation unit performs illumination compensation control based on the captured ground image intensity, specifically including: Establish a mapping relationship between camera light intensity, camera parameters, supplementary lighting intensity, and the average brightness of the ground image; Set the target brightness for the ground image; Acquire the camera light intensity; calculate the optimal solution for camera parameters and supplementary lighting intensity based on the mapping relationship and camera light intensity; The optimal solution for the camera parameters and the supplementary lighting intensity is sent to the control unit; the control unit controls the camera parameters of the camera unit and the supplementary lighting intensity of the supplementary lighting unit based on the optimal solution.

5. The stain recognition and detection system based on image processing as described in claim 4, characterized in that: The shadow detection based on pixel brightness specifically includes: selecting M pixels from a pre-marked shadow region and calculating the average brightness of the M pixels; setting a brightness difference threshold range and sequentially calculating the difference between the brightness of each pixel and the average brightness of the M pixels; if the difference between the brightness of any pixel and the average brightness of the M pixels is within the brightness difference threshold range, then marking the pixel as a shadow point; and connecting the shadow points to form a shadow region. The compensation and correction of the shadow area based on the average brightness of the non-shadow area specifically includes: calculating the average brightness of all pixels in the non-shadow area as the correction brightness; and compensating and correcting the shadow area based on the correction brightness, specifically including: adjusting the pixel value of each pixel in the shadow area to the correction brightness.

6. The stain recognition and detection system based on image processing as described in claim 5, characterized in that: The stain recognition module includes a comparison unit; the comparison unit is used for stain recognition and detection. The comparison unit stores reference images of ground surfaces of different materials; the reference image of any ground surface material is a ground image captured by the camera unit under conditions of no stains, no shadows, and uniform lighting, and the average brightness of the reference image is the target brightness; The specific steps for stain identification and detection include: Identify the ground material corresponding to the real-time acquired ground images; Obtain a reference image of the ground with the corresponding material; Based on the feature point matching algorithm, the real-time acquired ground image is aligned with the reference image; Identifying stained areas in ground images based on aligned ground images and reference images.

7. The stain recognition and detection system based on image processing as described in claim 6, characterized in that: The method of identifying stain areas in a ground image based on an aligned ground image and a reference image specifically includes: calculating the pixel difference between corresponding pixels in the ground image and the reference image point by point; the comparison unit is also configured with a pixel difference threshold; in the ground image, pixels with a pixel difference greater than the pixel difference threshold are marked as stain points; and the stain points in the ground image are connected to form stain areas.

8. The stain recognition and detection system based on image processing as described in claim 7, characterized in that: The method of identifying stain areas in a ground image based on an aligned ground image and a reference image also includes: The aligned ground image and reference image are converted to HSV space, and the brightness of each pixel in the ground image and reference image is recorded. The brightness of corresponding pixels in the ground image and the reference image are compared point by point, and a brightness residual matrix is ​​constructed. The brightness residual matrix has the same dimension as the pixel matrix of the ground image. Each element in the brightness residual matrix corresponds to a pixel at the same position in the pixel matrix of the ground image, and the element value is the difference in brightness between the corresponding pixels in the ground image and the reference image. The stain areas in the ground image are identified based on the element values ​​in the brightness residual matrix.

9. The stain recognition and detection system based on image processing as described in claim 8, characterized in that: Identifying stain areas in a ground image based on element values ​​in the brightness residual matrix specifically includes: At least N sub-matrices are extracted from the brightness residual matrix by sliding a window; Calculate the variance of all element values ​​within each submatrix; If the variance of the element values ​​of all submatrices is less than the preset variance threshold, then there is no stained area in the ground image; if the variance of the element values ​​of at least one submatric is greater than the preset variance threshold, then the corresponding submatric is marked as a potential stained area. Determine the gradient direction of the element values ​​in the potential stain region; The gradient direction concentration of the potential stain area is calculated based on the gradient direction of the element value; if the gradient direction concentration is greater than a preset concentration threshold, the potential stain area is a non-stain area; if the gradient direction concentration is less than or equal to the preset concentration threshold, the potential stain area is a stain area.

Images