Method for detecting contamination of an electronic rearview mirror system and related devices

Abstract

The application provides a dirty detection method and related device of an electronic rearview mirror system, and the method comprises the following steps: obtaining a first image, the first image being a rearview mirror sensing image which is uniformly optimized in size and quality, and the rearview mirror sensing image being an image from a sensing device; determining a dirty type and a dirty position based on the first image, the dirty type representing the physical characteristics of the dirt; determining a coincidence degree parameter based on the dirty position, the first image and a second image, the second image being a frame image of the first image; determining a dirty existence condition based on the coincidence degree parameter and the dirty type, and controlling the dirty existence condition and the dirty position to be visually displayed on a display device. In this way, the dirty type and the dirty position are determined through intelligent analysis of the first image, the coincidence degree parameter is determined based on the dirty condition and the frame image, the confidence is confirmed, and the credibility and accuracy of the dirty condition are improved.

Description

Technical Field

[0001] This invention relates to the field of electronic rearview mirror technology, and in particular to a method and related apparatus for detecting dirt in an electronic rearview mirror system. Background Technology

[0002] With the rapid development of advanced driver assistance systems (ADAS), onboard optical sensors, such as forward-looking and surround-view cameras, have become core components for environmental perception. Their imaging quality directly affects the reliability of critical functions such as target detection, lane recognition, and traffic sign identification. However, during actual driving, camera lenses are highly susceptible to contamination from rain, mud, frost, oil, or insects, leading to blurred images, distorted features, or information loss. This can cause system misjudgments or functional degradation, seriously threatening driving safety.

[0003] Current methods for detecting dirt in electronic rearview mirrors primarily involve directly detecting whether there is obstruction in the current frame image, or determining the type of deposit based on the humidity and temperature of the lens protective film, such as low-temperature solid, non-low-temperature solid, or liquid, and then directly controlling the protective film to remove the obstruction based on the type. However, current detection technologies still suffer from low reliability and vague judgment conclusions. Summary of the Invention

[0004] This application provides a method and related apparatus for detecting dirt in an electronic rearview mirror system, which can improve the reliability and accuracy of dirt detection.

[0005] In a first aspect, embodiments of this application provide a method for detecting dirt in an electronic rearview mirror system, applied to an electronic rearview mirror chip processor in the electronic rearview mirror system, the electronic rearview mirror system further including a sensing device and a display device, the method comprising:

[0006] Acquire a first image, which is a rearview mirror sensing image that is uniformly optimized in terms of size and image quality, and the rearview mirror sensing image is an image from the sensing device;

[0007] Based on the first image, the type and location of the dirt are determined, wherein the type of dirt characterizes the physical properties of the dirt;

[0008] The overlap parameter is determined based on the location of the dirt, the first image, and the second image, where the second image is a neighboring frame of the first image.

[0009] Based on the overlap parameter and the type of dirt, the presence of dirt is determined, and the presence and location of dirt are displayed on the display device.

[0010] In one possible embodiment, the second image is a frame preceding the first image, and the step of determining the overlap parameter based on the location of the dirt, the first image, and the second image includes:

[0011] Retrieve the second image and the dirt information of the previous frame corresponding to the second image, wherein the dirt information of the previous frame includes the location and type of dirt in the previous frame.

[0012] The intersection-union ratio is determined based on the second image and the first image;

[0013] The overlap parameter is determined based on the crossover ratio and the location of the dirt in the previous frame. The overlap parameter is used to characterize the degree of overlap of the dirt locations.

[0014] In one possible embodiment, the second image is a subsequent frame image of the first image, and the step of determining the overlap parameter based on the location of the dirt, the first image, and the second image includes:

[0015] Acquire a second image, and determine the type and location of dirt in the subsequent frame based on the second image;

[0016] The intersection-union ratio is determined based on the second image and the first image;

[0017] The overlap parameter is determined based on the intersection-to-union ratio and the location of the contamination in the subsequent frame. The overlap parameter is used to characterize the degree of overlap of the contamination locations.

[0018] In one possible embodiment, determining the presence of dirt based on the overlap parameter and the type of dirt includes:

[0019] An assessment strategy is determined based on the type of dirt, and each assessment strategy corresponds to one type of dirt.

[0020] The confidence level of the result is determined based on the evaluation strategy and the overlap parameter.

[0021] The presence of dirt is determined based on the result confidence level and the result confidence threshold, wherein the presence of dirt includes at least one of dirt type information and the probability of dirt presence.

[0022] In one possible embodiment, prior to the step of determining the presence of dirt based on the overlap parameter and the type of dirt, the method further includes:

[0023] Obtain vehicle environmental parameters, including vehicle speed;

[0024] Based on the vehicle environmental parameters, an environmental impact factor is determined, which is used to characterize the degree of influence of at least one of vibration and / or airflow caused by vehicle speed.

[0025] The confidence threshold of the results is adjusted based on the environmental impact factors.

[0026] In one possible embodiment, determining the type and location of dirt based on the first image includes:

[0027] Based on the first image and a pre-trained dirt detection model, the type and location of dirt are determined. The process of determining the type and location of dirt using the pre-trained dirt detection model includes:

[0028] Based on the first image, unclear areas and unclear regions are determined. The unclear area is the relative or absolute area of ​​the unclear region relative to the overall area. The unclear region represents the location and / or shape of the dirt.

[0029] Determine boundary parameters and region image parameters based on the unclear region;

[0030] The type of dirt is determined based on the boundary parameters and / or the unclear area and / or region image parameters;

[0031] The location of the dirt is determined based on the unclear area and the unclear region.

[0032] In one possible embodiment, the boundary parameters include boundary strength parameters, which characterize the realism of dirt, and the region image parameters include texture characteristics and / or contrast characteristics.

[0033] Determining the type of contamination based on the boundary parameters and / or the unclear area and / or region image parameters includes:

[0034] Determine the boundary gradient parameters, and determine the boundary strength parameters based on the boundary gradient parameters. The boundary strength parameters are used to characterize the real and virtual state of the dirt.

[0035] The degree of dirt coverage is determined based on the unclear area, and the degree of dirt coverage represents the size of the area of ​​the entire image covered by the dirt;

[0036] The type of dirt is determined based on the dirt coverage and / or texture characteristics and / or contrast characteristics and / or boundary gradient parameters.

[0037] Secondly, embodiments of this application provide a dirt detection device for an electronic rearview mirror system, applied to an electronic rearview mirror chip processor in the electronic rearview mirror system. The electronic rearview mirror system further includes a sensing device and a display device. The device includes:

[0038] An image determination module is used to acquire a first image, wherein the first image is a rearview mirror sensing image that is uniformly optimized in terms of size and image quality, and the rearview mirror sensing image is an image from the sensing device;

[0039] The first determining module is used to determine the type and location of dirt based on the first image, wherein the type of dirt characterizes the physical properties of the dirt;

[0040] The second determining module is used to determine the overlap parameter based on the location of the dirt, the first image, and the second image, wherein the second image is a neighboring frame image of the first image;

[0041] The conclusion determination module is used to determine the presence of dirt based on the overlap parameter and the type of dirt, and to control the presence and location of dirt to be visually displayed on the display device. Thirdly, embodiments of this application provide a computer-readable storage medium storing a dirt detection program for an electronic rearview mirror system. The dirt detection program for the electronic rearview mirror system includes execution instructions. When a processor executes the execution instructions stored in the memory, the processor performs some or all of the steps described in the first aspect.

[0042] Fourthly, embodiments of this application provide an electronic device, including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and when the processor executes the one or more programs, the processor executes some or all of the instructions of the steps described in the first aspect of the embodiments of this application.

[0043] Fifthly, embodiments of this application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in the first aspect of embodiments of this application. The computer program product may be a software installation package.

[0044] By implementing the embodiments of this application, a first image is acquired. The first image is a rearview mirror sensing image optimized for uniform size and image quality, and the rearview mirror sensing image is an image from the sensing device. Based on the first image, the type and location of dirt are determined, where the type of dirt characterizes the physical properties of the dirt. Based on the location of the dirt, the first image, and a second image, an overlap parameter is determined, where the second image is a neighboring frame of the first image. Based on the overlap parameter and the type of dirt, the presence of dirt is determined, and the presence and location of dirt are controlled to be visually displayed on the display device. Thus, by intelligently analyzing the first image to determine the type and location of dirt, and by determining the overlap parameter based on the dirt condition and neighboring frames, confidence level confirmation is achieved, improving the reliability and accuracy of the dirt condition assessment. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the background art, the accompanying drawings used in the embodiments of the present invention or the background art will be described below.

[0046] Figure 1a This is a schematic diagram of the architecture of an electronic rearview mirror system provided in an embodiment of this application;

[0047] Figure 1b This is a schematic diagram of an electronic rearview mirror chip architecture provided in an embodiment of this application;

[0048] Figure 1c This is a schematic diagram of another electronic rearview mirror chip architecture provided in an embodiment of this application;

[0049] Figure 2 This is a schematic flowchart of a dirt detection method for an electronic rearview mirror system provided in an embodiment of this application;

[0050] Figure 3 This is a flowchart illustrating a method for determining the type of dirt provided in an embodiment of this application;

[0051] Figure 4a This is a schematic diagram of the first image overlap parameter determination process provided in the embodiments of this application;

[0052] Figure 4b This is a schematic diagram of the second image overlap parameter determination process provided in the embodiments of this application;

[0053] Figure 4c This is a schematic diagram of the third image overlap parameter determination process provided in the embodiments of this application;

[0054] Figure 5 This is a schematic diagram of the structure of a dirt detection device for an electronic rearview mirror system according to an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0056] Figure 7 This is a schematic diagram of the structure of a dirt detection device for another electronic rearview mirror system provided in this application embodiment. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0058] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or electronic device that includes a series of steps or units is not limited to the listed steps or units, but in an alternative example also includes steps or units not listed, or in an alternative example also includes other steps or units inherent to these processes, methods, products, or electronic devices.

[0059] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0060] With the rapid development of advanced driver assistance systems (ADAS), onboard optical sensors (such as forward-looking and surround-view cameras) have become core components for environmental perception. Their imaging quality directly affects the reliability of key functions such as target detection, lane recognition, and traffic sign identification. However, during actual driving, camera lenses are highly susceptible to contamination from rain, mud, frost, oil, or insects, leading to blurred images, distorted features, or information loss. This can cause system misjudgments or functional degradation, seriously threatening driving safety.

[0061] Current methods for detecting dirt in electronic rearview mirrors primarily involve directly detecting whether there is obstruction in the current frame image, or determining the type of deposit (low-temperature solid, non-low-temperature solid, liquid) based on the humidity and temperature of the lens protective film, and then directly controlling the protective film to remove the obstruction based on the type. However, current detection technologies still suffer from low reliability and vague judgment conclusions.

[0062] To address the aforementioned issues, this application provides a method and related apparatus for detecting dirt in an electronic rearview mirror system. The method uses a first image for intelligent image analysis to determine the type and location of dirt, and determines overlap parameters based on the dirt condition and adjacent frame images to confirm confidence levels, thereby improving the reliability and accuracy of the dirt condition assessment.

[0063] This application provides a method for detecting dirt in an electronic rearview mirror system, which can be applied to, for example... Figure 1a Please refer to the electronic rearview mirror system shown. Figure 1a , Figure 1a This is a schematic diagram of the architecture of an electronic rearview mirror system provided in an embodiment of this application. The electronic rearview mirror system 100 includes an electronic rearview mirror chip 110, an electronic rearview mirror chip processor 111, a sensing device 120, and a display device 130. The sensing device 120 and the display device 130 can communicate with the electronic rearview mirror chip processor 111. The display device 130 refers to a display device used by the user, such as an electronic rearview mirror display screen.

[0064] Preferably, in this solution, the electronic rearview mirror chip processor 111 includes an image signal processor (ISP) and a neural network processing unit (NPU), which are part of the electronic rearview mirror chip 110. The ISP is used for real-time processing and optimization of image or video signals, specifically, it is responsible for high-quality processing of the raw images input from the sensing device 120. The NPU is used to execute various mathematical operations in the neural network algorithm using its specially designed hardware structure, specifically, it is used to implement functions such as dirt detection, rain removal, fog removal, and halo removal. The electronic rearview mirror chip processor 111 can also be used to collect data during the model's use, facilitating subsequent model optimization.

[0065] Specifically, the NPU stores a dirt detection model, which is used to perform large-scale model analysis based on the first image to determine the type and location of dirt.

[0066] In this design, the sensing device 120 is an image acquisition module, which may include a lens and a high-sensitivity image sensor, and is used to realize the function of an electronic rearview mirror. The display device 130 can be a display screen used to display the image data acquired by the sensing device 120, thus realizing the function of a rearview mirror.

[0067] Preferably, please refer to Figure 1b , Figure 1b This is a schematic diagram of an electronic rearview mirror chip architecture provided in an embodiment of this application, such as... Figure 1b As shown, in this solution, the Longquan 560 series chip can be used as the electronic rearview mirror chip 110. Figure 1b The example is LQ560 200M1120324050101500263502010933-CN.

[0068] For an example, please refer to Figure 1c , Figure 1c This is a schematic diagram of another electronic rearview mirror chip architecture provided in an embodiment of this application, such as... Figure 1c As shown, the electronic rearview mirror chip 110 can be a Longquan 560 mini chip, supporting multi-channel video input, ISP image processing, NPU AI model processing, and video encoding / decoding. Specifically, the camera deserializer is a bridge chip that deserializes image data from multiple cameras or image sensors included in the sensing device 120 for subsequent ISP image processing. The MCU is a microcontroller unit (MCU) that integrates key processors such as the ISP and NPU. The CAN Transceiver is used to connect and acquire other vehicle operating data, such as vehicle speed. The processor interface MIPI (Mobile Industry Processor Interface) is an interface protocol used for on-chip image data transmission, such as transmitting the output of the camera deserializer to the ISP via MIPI. Of course, other integrated chips containing ISP and NPU can also be selected to implement the dirt detection method of the electronic rearview mirror system; the specific chip is not limited here.

[0069] Communication between the ISP and NPU is achieved through an ultra-high-speed on-chip bus within the chip. The sensing device 120 can be connected to the electronic rearview mirror chip 110 via a high-speed serial differential input pin pair, for example, a low-voltage differential signaling (LVDS) pin. The display device 130 can be connected to the electronic rearview mirror chip 110 via a high-speed serial differential output pin pair, specifically, an I2C / SPI pin. In some cases, vehicle status signals, such as vehicle speed, received from the body controller can be acquired via the Controller Area Network (CAN) bus and transmitted to the internal MCU of the electronic rearview mirror chip 110 via the CAN Transceiver.

[0070] Based on this, this application provides a method for detecting dirt in an electronic rearview mirror system, which will be described in detail below with reference to the accompanying drawings.

[0071] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for detecting dirt in an electronic rearview mirror system according to an embodiment of this application. The method is applied to the electronic rearview mirror chip processor of the electronic rearview mirror system. The electronic rearview mirror system also includes a sensing device and a display device, such as... Figure 2 As shown, the method includes the following steps:

[0072] S210, acquire a first image, the first image being a rearview mirror sensing image optimized for uniform size and image quality, the rearview mirror sensing image being an image from the sensing device.

[0073] First, a rearview mirror sensor image from the sensing device is acquired. This image is the original image, and the first image is the current frame image of the current detection cycle. The rearview mirror sensor image is then processed by the ISP to optimize the size and image quality of the original image data, resulting in the first image. The image quality optimization process may include, but is not limited to, defect pixel correction, noise reduction, de-mosaicing, automatic white balance, color correction and enhancement, wide dynamic range processing, sharpening, and contrast enhancement. The first image obtained after processing can be better used for image analysis. Size optimization is used to adjust the rearview mirror sensor image to be suitable for the model processing of the dirt detection model used by the NPU. The dirt detection model is a neural network model, and the number of neurons in the input layer of the neural network model is fixed. Therefore, it is necessary to optimize the image size to uniformly scale the rearview mirror sensor images of different resolutions to the fixed size preset by the neural network model used in the NPU.

[0074] Optionally, upon receiving a rearview mirror sensor image from the sensing device, the image is denoised to remove random noise generated by photoelectric conversion or circuitry. Then, the missing color channel values ​​of the red, green, and blue pixels surrounding the pixels in the image are calculated to obtain a complete RGB three-channel image. Color correction and color enhancement conversion are then performed based on the image content to obtain a more accurate image, improving edge sharpness and overall contrast, making object outlines more distinct. Finally, the processed image is output for subsequent dirt detection.

[0075] S220, based on the first image, determine the type and location of the dirt, wherein the type of dirt characterizes the physical properties of the dirt.

[0076] The physical characteristics represented by the type of dirt can include, for example, texture characteristics, edge conditions, light transmittance, temperature, and humidity. The location of the dirt is the position of the dirt in the first image or the rearview mirror sensor image. Furthermore, since the location of the dirt in the image can be inferred from the location of the dirt on the sensor device, the location of the dirt can also be the location of the dirt on the sensor device.

[0077] Optionally, the first image is input into a pre-trained neural network model. This neural network model performs multi-level feature extraction on the image; the backbone network of the neural network model performs multi-level feature extraction, the shallow network captures low-level features such as texture and edges, for example, the curved edges of dirt, and the deep network integrates the contextual information of the first image to determine the overall shape of the dirt and its relationship with the background environment. Based on the feature information, a probability distribution vector for a predefined dirt type is determined.

[0078] In one possible embodiment, determining the type and location of dirt based on the first image includes:

[0079] The dirt type and location are determined based on the first image and a pre-trained dirt detection model. The process of determining the dirt type and location using the pre-trained dirt detection model includes: determining unclear area and unclear region based on the first image, where the unclear area is the relative or absolute area of ​​the unclear region relative to the overall region, and the unclear region represents the location and / or shape of the dirt; determining boundary parameters and region image parameters based on the unclear region; determining the dirt type based on the boundary parameters and / or the unclear area and / or region image parameters; and determining the dirt location based on the unclear area and the unclear region.

[0080] The process of determining the unclear area and unclear region based on the first image includes: calculating a sharpness evaluation value using a local window centered on each pixel; performing adaptive threshold segmentation on the sharpness evaluation value to determine the unclear region and unclear area; wherein, calculating the sharpness evaluation value using a local window centered on each pixel can be determined by methods such as gradient magnitude, Laplacian variance, and frequency domain energy, which are not limited here. The unclear regions are then binarized to obtain binarized regions, and morphological operations are performed on these binarized regions to connect adjacent small regions, smooth boundaries, and eliminate noise points. Examples of binarization operations include closing operations, dilation followed by erosion, etc., which are not limited here. Finally, one or more connected binary masks of unclear regions are obtained. The total number of white pixels in the mask is calculated based on the binary mask, divided by the total number of pixels in the image to obtain the relative area ratio, thus determining the unclear area. Furthermore, the sequence of contour points in the binary mask can describe the specific shape.

[0081] In this process, a sampling line is formed by extending a certain number of pixels both inside and outside the unclear region along its contour line. The change curve of the sharpness evaluation value on this sampling line is calculated to determine the boundary gradient decay rate. Furthermore, the roundness of the unclear region's contour is calculated, and the boundary shape regularity is determined based on the roundness and a preset roundness threshold, where the preset roundness threshold includes 1. The roundness can be determined using a method such as 4π × area / perimeter². Alternatively, the curvature statistical characteristics of individual points on the contour line can be calculated, such as the curvature variance, and the boundary shape regularity is determined based on the curvature variance and a preset curvature variance threshold.

[0082] Specifically, the image parameters of the unclear area are determined by performing texture analysis and / or optical property analysis. Specifically, this includes calculating the texture uniformity within the unclear area; calculating the standard deviation of the grayscale value or color channel within the unclear area; and / or calculating the average hue and saturation within the unclear area, where the average hue and saturation can be calculated in a red-green-blue (RGB) color space or a hue, saturation, and hue (HSV) color space; and / or calculating the average brightness ratio of the unclear area to other surrounding areas. The aforementioned image parameters include at least one of the following: texture uniformity, standard deviation of grayscale value or color channel, average hue and saturation, and average brightness ratio.

[0083] Determining the type of dirt based on the boundary parameters and / or the unclear area and / or region image parameters includes: inputting the extracted boundary parameters and / or the unclear area and / or region image parameters into a classification decision module to determine the type of dirt. This classification decision module can be a module within a dirt detection model, performing rule mapping or logical judgment based on all parameters included in the boundary parameters, the unclear area, and the region image parameters to obtain the dirt type. The classification decision module is a pre-trained model and can adaptively optimize based on feedback.

[0084] The types of dirt include solid, smear, and water mist. The dirt detection model is pre-trained with mapping rules or judgment logic based on the above types of dirt, which can quickly determine the type of dirt.

[0085] For example, if the boundary gradient decay rate is within the low decay rate threshold range (meaning the boundary is smooth), and the average brightness ratio is close to or greater than the brightness ratio threshold, and the boundary shape regularity is within the high regularity threshold range, then it is classified as a water mist type. If the boundary gradient decay rate is within the high decay rate threshold range (meaning the boundary is sharp and irregular), and the average brightness ratio is lower than the brightness ratio threshold, and the internal color deviation is small, then it is classified as a solid type. If the internal texture uniformity is high but the saturation variance in the internal color deviation is large, meaning color spots appear, then it is classified as a smear type. The above examples are for illustrative purposes only and are not intended to be limiting.

[0086] Determining the location of the contaminant based on the unclear area and the unclear region includes: performing geometric and topological analysis on the mask to convert pixel coordinates into a location description with clear physical or relative meaning. The location description includes the coordinates determined by the mask based on the unclear region, plus the unclear area, to obtain the location of the contaminant. Specifically, if the intrinsic and extrinsic parameters of the sensing device are known, the image coordinates or contour points can be mapped back to the actual physical coordinates of the glass surface of the camera module of the sensing device through an inverse perspective transformation model. The intrinsic parameters include focal length and principal point, and the extrinsic parameters include the installation angle of the camera module in the sensing device.

[0087] As can be seen, by calculating parameters with clear physical and image meanings on the first image, quantitative information such as category, location, area, and shape is output. The structure is clear, the implementation is strong, and it is easy to debug and verify, thereby further improving the accuracy of dirt detection.

[0088] In one possible embodiment, the boundary parameters include boundary strength parameters, which characterize the realism or abstraction of the dirt; the region image parameters include texture characteristics and / or contrast characteristics; determining the type of dirt based on the boundary parameters and / or the unclear area and / or region image parameters includes:

[0089] Determine the boundary gradient parameters, and determine the boundary strength parameters based on the boundary gradient parameters. The boundary strength parameters are used to characterize the real and virtual state of the dirt.

[0090] The degree of dirt coverage is determined based on the unclear area, and the degree of dirt coverage represents the size of the area of ​​the entire image covered by the dirt;

[0091] The type of dirt is determined based on the dirt coverage and / or texture characteristics and / or contrast characteristics and / or boundary gradient parameters.

[0092] Boundary strength is used to quantify the sharpness of the transition between dirty and clean areas, reflecting whether the dirt is solid or diffuse, such as water mist or smear.

[0093] Optionally, determining the boundary gradient parameters, and determining the boundary strength parameters based on the boundary gradient parameters, includes: taking a narrow band region on both the inner and outer sides of the mask along the boundary normal direction of the unclear region, for example, each 5 pixels wide, and calculating the decay curve of the image sharpness evaluation function value within the narrow band along the normal direction, for example, the image sharpness evaluation function value is the average gradient magnitude. The boundary strength parameter can be the average negative slope of the decay curve, or the ratio of the sharpness values ​​on the inner and outer sides, etc. A boundary strength parameter greater than the boundary strength threshold indicates a sharp difference in sharpness between the inner and outer sides of the boundary, a steep transition, characterized as a solid boundary, corresponding to solid dirt; a boundary strength parameter less than the boundary strength threshold indicates a gentle change in sharpness between the inner and outer sides of the boundary, a blurred transition, characterized as a virtual boundary, corresponding to liquid smears, oil films, or water mist dirt.

[0094] Optionally, the local variance map of the grayscale image within the unclear area can be calculated, and statistical features can be extracted. These statistical features can be, for example, the mean variance or the variance distribution entropy, and are not limited here. If the variance is large, the texture is relatively coarse, which can correspond to solid or smear-like textures; if the variance is small, the texture is relatively smooth and uniform, which can correspond to water mist textures.

[0095] Optionally, determining the degree of dirt coverage based on the unclear area includes: directly using the relative proportion of the unclear area as a quantitative indicator to determine whether the dirt is localized or diffuse. Specifically, if the relative proportion of the unclear area is less than a first relative proportion threshold, it is considered localized coverage; if the relative proportion of the unclear area is greater than the first relative proportion threshold, it is considered global coverage. The average brightness ratio of the unclear area to other surrounding areas is calculated; for water mist or smear-like substances, the brightness ratio is greater than a first brightness ratio threshold, while for solid substances, the brightness ratio is less than a first brightness ratio threshold.

[0096] For an example, please refer to Figure 3 , Figure 3This is a flowchart illustrating a method for determining the type of dirt provided in an embodiment of this application, such as... Figure 3 As shown, determining the dirt type based on the dirt coverage and / or texture characteristics and / or contrast characteristics and / or boundary gradient parameters includes: S31, determining whether the boundary intensity parameter is less than the boundary intensity threshold; wherein, if not, the dirt type is determined to be solid dirt; if yes, it is smear-type or water mist-type, proceeding to S32, determining whether the texture uniformity is within the texture uniformity threshold range; if yes, proceeding to S321, determining whether the average brightness ratio is within the brightness ratio threshold range; if yes... If the value is clean, it is a water mist; otherwise, it is a dirty water mist. If not, proceed to step 322 to determine if the average brightness ratio is within the brightness ratio threshold range. If yes, it is a liquid water mist, such as water droplets. If no, proceed to step S323 to determine if the dirt coverage is global or local. If it is global, it is a uniform film water mist; if it is local, it is a smear, such as fingerprints or local smear stains.

[0097] It is evident that by calculating parameters with clear physical and image meanings on the first image, the original image feature parameters can be mapped to dirt types with clear physical meanings, outputting quantitative information such as category, location, area, and shape. The structure is clear, the implementation is strong, and it is easy to debug and verify, further improving the accuracy of dirt detection.

[0098] S230, based on the location of the dirt, the first image and the second image, determine the overlap parameter, wherein the second image is a neighboring frame of the first image.

[0099] The second image can be either a frame preceding or following the first image; in some cases, it may include both. The second image can be used to distinguish between static dirt and temporary dynamic dirt. Static dirt includes, for example, fog-like or smear-like dirt, while dynamic dirt includes, for example, splashed water droplets or moving objects. Specifically, an overlap parameter is used to determine the overlap. This parameter is used for continuity verification and is obtained by performing overlap analysis on the target positions in both the first and second images.

[0100] In some possible embodiments, please refer to Figure 4a , Figure 4a This is a schematic diagram of the first image overlap parameter determination process provided in the embodiments of this application, as shown below. Figure 4aAs shown, the second image is the preceding frame and the following frame of the first image. The step of determining the overlap parameter based on the location of the dirt, the first image, and the second image includes: retrieving the dirt information of the second image and the preceding and following frames corresponding to the second image, wherein the preceding and following frame dirt information includes the location of the dirt in the preceding frame, the location of the dirt in the following frame, the type of dirt in the preceding frame, and the type of dirt in the following frame; determining the multi-frame cross-union ratio based on the preceding frame, the following frame, and the first image, and using the multi-frame cross-union ratio as the overlap parameter. The method for determining the multi-frame cross-union ratio can be to perform cross-union ratio calculations with the first image for the preceding frame and the following frame, respectively, to obtain the cross-union ratio of the preceding frame and the following frame, and to determine the multi-frame cross-union ratio based on the cross-union ratio of the preceding frame and the following frame. Examples include calculating the average value, taking the maximum value, or other mathematical statistical operations, which are not limited here.

[0101] The number of the aforementioned preceding and following frames can be multiple consecutive frames or a single frame, such as 2 preceding frames and 2 following frames.

[0102] The calculation involves determining the intersection area (number of overlapping pixels) and union area (total number of covered pixels) of two regions. The Intersection over Union (IoU) value is calculated, ranging from 0 to 1. A value closer to 1 indicates a higher degree of overlap, while a value of 0 indicates no overlap, directly reflecting the spatial consistency of the dirty regions in adjacent frames. For example, the IoU value can be determined in the following ways:

[0103] ; ; ;

[0104] in, This refers to the location or area of ​​dirt in the first image. This refers to the location or area of ​​dirt in the previous frame image. This refers to the location or area of ​​dirt in the subsequent frame image. Based on... and Sure , You can also use the mean or median of IoU within a sliding window.

[0105] It should be noted that if there are multiple previous and subsequent frames, the mean or median IoU within the sliding window can be used to reduce the impact of single-frame fluctuations.

[0106] As can be seen, in this embodiment, by acquiring the previous frame and the next frame images, the overlap parameter of the dirt detected in the first image is determined to verify the dirt detection situation of the first image, thereby improving the accuracy and reliability of the dirt situation on the sensing device of the electronic rearview mirror.

[0107] In one possible embodiment, please refer to Figure 4b , Figure 4b This is a schematic diagram of the second image overlap parameter determination process provided in the embodiments of this application, as shown below. Figure 4b As shown, the second image is the preceding frame image of the first image. The step of determining the overlap parameter based on the location of the dirt, the first image, and the second image includes: retrieving the dirt information of the second image and the preceding frame corresponding to the second image, wherein the preceding frame dirt information includes the location and type of dirt in the preceding frame; determining the intersection-union ratio based on the second image and the first image; and determining the overlap parameter based on the intersection-union ratio and the location of the dirt in the preceding frame, wherein the overlap parameter is used to characterize the degree of overlap of the dirt locations.

[0108] In this context, the preceding frame of the first image refers to the frame immediately preceding or several frames prior to the current frame in the time series. Since dirt detection is a continuous process, the preceding frame dirt information represents the dirt detected and the results stored during system operation. The concept of the preceding frame dirt location corresponds to the concept of the dirt location in the first image; it refers to the specific location of the dirty area in the second image. The specific definition and determination method can refer to the definition and determination method of the dirt location in the first image, and will not be repeated here. The preceding frame dirt type refers to the identified dirt category, such as water mist, smear, or solid. The specific definition and determination method can refer to the definition and determination method of the dirt type in the first image, and will not be repeated here. This historical data can be retrieved from a cache or storage.

[0109] The calculation involves determining the intersection area (number of overlapping pixels) and union area (total number of covered pixels) of two regions. The Intersection over Union (IoU) value is calculated, ranging from 0 to 1. A value closer to 1 indicates a higher degree of overlap, while a value of 0 indicates no overlap, directly reflecting the spatial consistency of dirty regions in adjacent frames. The IoU value is determined using the following methods:

[0110] ;

[0111] in, This refers to the location or area of ​​dirt in the first image. This refers to the location or area of ​​dirt in the previous frame image. It should be noted that if there are multiple previous frames, the mean or median IoU within a sliding window can be used to reduce the impact of fluctuations in a single frame.

[0112] As can be seen, in this embodiment, by acquiring the previous frame image and determining the overlap parameter of the dirt detected in the first image, the dirt detection situation of the first image is verified, thereby improving the accuracy and reliability of the dirt situation on the sensing device of the electronic rearview mirror.

[0113] In one possible embodiment, please refer to Figure 4c , Figure 4c This is a schematic diagram of the third image overlap parameter determination process provided in the embodiments of this application, as shown below. Figure 4c As shown, the second image is a subsequent frame image of the first image. The step of determining the overlap parameter based on the location of the dirt, the first image, and the second image includes: acquiring the second image; determining the type and location of the dirt in the subsequent frame based on the second image; determining the intersection-union ratio based on the second image and the first image; and determining the overlap parameter based on the intersection-union ratio and the location of the dirt in the subsequent frame. The overlap parameter is used to characterize the degree of overlap of the dirt locations.

[0114] In this embodiment, the second image is the frame following the first image. Lookahead verification is used, utilizing information from future frames to confirm the reliability of the current detection result. After completing the initial dirt detection of the first image (current frame), the image immediately following the current frame is obtained from the image acquisition buffer. This requires the system to have a buffer capacity of at least one frame, which introduces a processing cycle delay. Then, the same dirt detection process as the first image is performed on the second image, including image preprocessing, feature extraction, type classification, and region localization. For details, please refer to the processing procedure of the first image, which will not be repeated here.

[0115] The dirt type in the subsequent frame refers to the type of dirt detected in the second image, including, for example, water mist, smears, and solids. The specific determination process is similar to that in the first image; please refer to the implementation steps of the determination process for the first image, which will not be repeated here. The dirt location in the subsequent frame is the precise location of the dirt area in the second image. The specific determination process is similar to that in the first image; please refer to the implementation steps of the determination process for the first image, which will not be repeated here.

[0116] The process of determining the Intersection over Union (IoU) ratio includes calculating the intersection area (number of overlapping pixels) and union area (total number of covered pixels) of the two regions. The IoU value is calculated, ranging from 0 to 1. A value closer to 1 indicates a higher degree of overlap, while a value of 0 indicates no overlap, directly reflecting the spatial consistency of dirty regions in adjacent frames. The IoU ratio is determined using the following methods:

[0117] ;

[0118] in, This refers to the location or area of ​​dirt in the first image. This refers to the location or area of ​​dirt in the subsequent frame image. It should be noted that if there are multiple preceding frames, the mean or median IoU within a sliding window can be used to reduce the impact of single-frame fluctuations.

[0119] As can be seen, in this embodiment, by using the information of the following frame to verify the current detection, false alarms caused by instantaneous interference are reduced, and the continuity is confirmed by comparing with the following frame. This does not rely on historical frames that may contain erroneous detections, thus improving the accuracy and reliability of dirt detection.

[0120] S240, based on the overlap parameter and the type of dirt, determine the presence of dirt, and control the presence and location of dirt to be displayed visually on the display device.

[0121] The overlap parameter is a quantitative value. Different types of contamination have different overlap parameters and decision branches based on their physical characteristics. For example, a contamination area classified as solid will have a high confidence level if its overlap parameter is high, such as 0.9; however, if its overlap parameter is average, such as 0.6, the confidence level will be significantly reduced because it does not meet the expectation of high stability for solid contamination.

[0122] The process of visually displaying the presence and location of dirt on the display device can involve using a semi-transparent color block, such as a striking color like red or yellow, to outline or cover the area identified as dirty on the real-time video screen. Different colors can be used to distinguish the type of dirt, or text or graphic labels can be used to indicate the presence and location of the dirt.

[0123] In one possible embodiment, determining the presence of dirt based on the overlap parameter and the dirt type includes: determining an evaluation strategy based on the dirt type, where each evaluation strategy corresponds to one dirt type; determining a result confidence level based on the evaluation strategy and the overlap parameter; and determining the presence of dirt based on the result confidence level and a result confidence threshold, wherein the presence of dirt includes at least one of dirt type information and the probability of dirt presence.

[0124] The evaluation strategies include a first evaluation strategy, a second evaluation strategy, and a third evaluation strategy. The evaluation strategy is determined based on the type of dirt: if it is a water mist, the first evaluation strategy is used; if it is a smear, the second evaluation strategy is used; and if it is a solid, the third evaluation strategy is used. The process of determining the confidence level involves placing the quantified inter-unit overlap (IoU) parameter into the selected strategy framework for comprehensive evaluation, calculating a confidence score. Specifically, the IoU parameter can be used as the base confidence level and adjusted according to the type of dirt to obtain the final confidence level.

[0125] If the substance is water mist, it is determined as the first evaluation strategy. Since water mist needs to tolerate moderate diffusion, contraction, or uniform changes, and because the water film / mist is affected by airflow and evaporation, the expected value of the overlap parameter can be slightly lower. Based on the evaluation strategy and the overlap parameter, the confidence level of the result is determined, including: using the overlap parameter directly as the basic confidence level, determining the smoothness and persistence of area change; adding points if it meets the criteria, and deducting points otherwise, to obtain a corrected value for the overlap parameter. The specific rules for adding and deducting points can be pre-established mapping rules, which are not limited here; then, normalization is performed based on the corrected value of the overlap parameter to obtain the confidence level of the result.

[0126] If the texture is a smear, the second evaluation strategy is adopted. Since smear textures or stripes are regular, the expected value of the overlap parameter can be moderate. The confidence level of the result is determined based on the evaluation strategy and the overlap parameter, including: using the overlap parameter directly as the basic confidence level to determine the stability of the texture direction. If it is consistent, a point is added; otherwise, a point is deducted, and a correction value for the overlap parameter is obtained. The specific rules for adding and deducting points can be pre-established mapping rules, which are not limited here. Then, the result confidence level is obtained by normalization based on the correction value of the overlap parameter.

[0127] If the object is a solid, the third evaluation strategy is adopted. Since the shape and texture of solid objects are highly stable, the expected value of the overlap parameter can be relatively high. The confidence level of the result is determined based on the evaluation strategy and the overlap parameter, including: using the overlap parameter directly as the basic confidence level, determining the boundary sharpness and texture consistency, adding points if they are met, and deducting points otherwise, to obtain the overlap parameter correction value. The specific rules for adding and deducting points can be pre-established mapping rules, which are not limited here; then, the overlap parameter correction value is normalized to obtain the result confidence level.

[0128] The system sets one or more confidence thresholds for each type of dirt. A confirmed presence threshold is defined as follows: for example, for solid dirt, this includes a threshold of 0.85; for smear-like dirt, 0.75; and for water mist dirt, 0.65. If the level exceeds this threshold, the dirt is considered present. A suspected presence threshold is defined as the level below the confirmed presence threshold but above the suspected presence threshold, requiring continued observation or prompting the user. If the level is below the suspected presence threshold, the dirt is directly determined to be absent or a temporary disturbance.

[0129] As can be seen, in this embodiment, precise and differentiated decision-making for different levels of dirt is achieved through strategy differentiation, and multi-dimensional judgment fusion is achieved through confidence calculation, thereby improving the accuracy and reliability of dirt detection.

[0130] In one possible embodiment, prior to the step of determining the presence of dirt based on the overlap parameter and the type of dirt, the method further includes: acquiring vehicle environmental parameters, including vehicle speed; determining an environmental impact factor based on the vehicle environmental parameters, the environmental impact factor being used to characterize the degree of influence of at least one of vibration and / or airflow caused by vehicle speed; and adjusting the confidence threshold of the result based on the environmental impact factor.

[0131] It should be noted that changes in vehicle speed directly affect the camera's image quality and the characteristics of dirt detection during vehicle movement. For example, high-speed driving causes vehicle vibration, resulting in camera shake and slight shifts in the position of the same dirt in consecutive frames. High-speed airflow may also agitate liquid dirt (such as water droplets or oil films) on the lens surface, altering its shape or position. Therefore, a vehicle speed parameter is introduced to dynamically adjust the confidence threshold for dirt detection. This allows the detection system to adapt to different driving conditions, avoiding overly lenient thresholds at low speeds that might mistakenly identify temporary interference as dirt, and overly strict thresholds at high speeds that might miss actual dirt.

[0132] This process involves acquiring real-time vehicle speed data and ensuring that the speed data is synchronized with the image acquisition timestamp. Environmental impact factors are then determined based on the vehicle speed data to characterize the degree of interference from the current vehicle speed on dirt detection.

[0133] For example, the environmental impact factor can be calculated based on vehicle speed using a piecewise or continuous function to obtain the first adjustment factor. Then, based on different types of pollution, this calculated first adjustment factor is adjusted to obtain the final environmental impact factor. For example, if a piecewise function is used, speed levels are divided based on vehicle speed, and the base value of the environmental impact factor is adjusted by increasing or decreasing based on the speed level to obtain the first adjustment factor. When the vehicle speed is less than the first speed threshold, the environmental impact factor is equal to the base value; when the vehicle speed is greater than the first speed threshold but less than the second speed threshold, the environmental impact factor is the base value + the first adjustment factor; when the vehicle speed is greater than the second speed threshold, the environmental impact factor is the base value + the second adjustment factor; the calculation of the adjustment factor is related to vehicle speed. For example, the adjustment factor calculation process can be: Adjustment factor = Environmental impact factor base value + F(v, V) 阈值 ), the above F(v, V 阈值 (This is related to vehicle speed v and speed threshold V) 阈值 The function, V 阈值 This includes either a first speed threshold or a second speed threshold. If it is a continuous function, an example could be the determination method for an exponential function.

[0134] For example, the base value of the environmental impact factor is m. Assuming that water mist-type pollution occurs at a vehicle speed of 100km / h, the calculated environmental impact factor is b (indicating significant interference). After adjustment, the threshold is m / b≈n. The calculated n is less than m, which is equivalent to relaxing the judgment condition.

[0135] It should be noted that, to avoid over-adjustment of the threshold (e.g., too high leading to missed reports or too low leading to false reports), upper and lower limits are set to constrain the dynamic adjustment range of the threshold. When the confidence threshold adjusted based on environmental impact factors is lower than the lower confidence limit, the confidence threshold is determined as the lower confidence limit; when the confidence threshold adjusted based on environmental impact factors is greater than the upper confidence limit, the confidence threshold is determined as the upper confidence limit.

[0136] Thus, in this embodiment, considering the different impacts of vehicle speed on different types of dirt, the judgment strictness can be automatically adjusted according to driving conditions without manual intervention. This compensates for the different sensitivity of different types of dirt to vehicle speed, ensuring stable dirt detection under various operating conditions.

[0137] As can be seen, in this embodiment, a first image is acquired, which is a rearview mirror sensing image optimized for uniform size and image quality, and the rearview mirror sensing image is an image from the sensing device; based on the first image, the type and location of dirt are determined, and the type of dirt characterizes the physical properties of the dirt; based on the location of the dirt, the first image, and a second image, an overlap parameter is determined, and the second image is a neighboring frame image of the first image; based on the overlap parameter and the type of dirt, the presence of dirt is determined, and the presence and location of dirt are controlled to be visually displayed on the display device. Thus, by intelligently analyzing the first image to determine the type and location of dirt, and by determining the overlap parameter based on the dirt condition and the neighboring frame image for confidence confirmation, the reliability and accuracy of the dirt condition are improved.

[0138] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a dirt detection device for an electronic rearview mirror system according to an embodiment of this application. The dirt detection device 500 for the electronic rearview mirror system includes: an image determination module 510, a first determination module 520, a second determination module 530, and a conclusion determination module 540, wherein...

[0139] Image determination module 510 is used to acquire a first image, the first image being a rearview mirror sensing image that is uniformly optimized in terms of size and image quality, and the rearview mirror sensing image being an image from the sensing device;

[0140] The first determining module 520 is used to determine the type and location of dirt based on the first image, wherein the type of dirt characterizes the physical properties of the dirt.

[0141] The second determining module 530 is used to determine the overlap parameter based on the location of the dirt, the first image, and the second image, wherein the second image is a neighboring frame image of the first image.

[0142] The conclusion determination module 540 is used to determine the presence of dirt based on the overlap parameter and the type of dirt, and to control the presence of dirt and the location of dirt to be displayed visually on the display device.

[0143] In one possible embodiment, the first determining module 520, in determining the type and location of dirt based on the first image, is specifically configured to:

[0144] Based on the first image and a pre-trained dirt detection model, the type and location of dirt are determined. The process of determining the type and location of dirt using the pre-trained dirt detection model includes:

[0145] Based on the first image, unclear areas and unclear regions are determined. The unclear area is the relative or absolute area of ​​the unclear region relative to the overall area. The unclear region represents the location and / or shape of the dirt.

[0146] Determine boundary parameters and region image parameters based on the unclear region;

[0147] The type of dirt is determined based on the boundary parameters and / or the unclear area and / or region image parameters;

[0148] The location of the dirt is determined based on the unclear area and the unclear region.

[0149] In one possible embodiment, the boundary parameters include boundary strength parameters, which characterize the realism of the dirt, and the region image parameters include texture characteristics and / or contrast characteristics; the first determining module 520, in determining the type of dirt based on the boundary parameters and / or the unclear area and / or the region image parameters, is specifically configured to:

[0150] Determine the boundary gradient parameters, and determine the boundary strength parameters based on the boundary gradient parameters. The boundary strength parameters are used to characterize the real and virtual state of the dirt.

[0151] The degree of dirt coverage is determined based on the unclear area, and the degree of dirt coverage represents the size of the area of ​​the entire image covered by the dirt;

[0152] The type of dirt is determined based on the dirt coverage and / or texture characteristics and / or contrast characteristics and / or boundary gradient parameters.

[0153] In one possible embodiment, the second image is a frame preceding the first image; the second determining module 530, in determining the overlap parameter based on the location of the dirt, the first image, and the second image, is specifically used for:

[0154] Retrieve the second image and the dirt information of the previous frame corresponding to the second image, wherein the dirt information of the previous frame includes the location and type of dirt in the previous frame.

[0155] The intersection-union ratio is determined based on the second image and the first image;

[0156] The overlap parameter is determined based on the crossover ratio and the location of the dirt in the previous frame. The overlap parameter is used to characterize the degree of overlap of the dirt locations.

[0157] In one possible embodiment, the second image is a subsequent frame of the first image; the second determining module 530, in determining the overlap parameter based on the location of the dirt, the first image, and the second image, is specifically used for:

[0158] Acquire a second image, and determine the type and location of dirt in the subsequent frame based on the second image;

[0159] The intersection-union ratio is determined based on the second image and the first image;

[0160] The overlap parameter is determined based on the intersection-to-union ratio and the location of the contamination in the subsequent frame. The overlap parameter is used to characterize the degree of overlap of the contamination locations.

[0161] In one possible embodiment, the conclusion determination module 540, in determining the presence of dirt based on the overlap parameter and the dirt type, is specifically used for:

[0162] An assessment strategy is determined based on the type of dirt, and each assessment strategy corresponds to one type of dirt.

[0163] The confidence level of the result is determined based on the evaluation strategy and the overlap parameter.

[0164] The presence of dirt is determined based on the result confidence level and the result confidence threshold, wherein the presence of dirt includes at least one of dirt type information and the probability of dirt presence.

[0165] In one possible embodiment, prior to the step of determining the presence of dirt based on the overlap parameter and the type of dirt, the conclusion determination module 540 is further configured to:

[0166] Obtain vehicle environmental parameters, including vehicle speed;

[0167] Based on the vehicle environmental parameters, an environmental impact factor is determined, which is used to characterize the degree of influence of at least one of vibration and / or airflow caused by vehicle speed.

[0168] The confidence threshold of the results is adjusted based on the environmental impact factors.

[0169] It is worth noting that the specific functional implementation of the dirt detection device 500 in the electronic rearview mirror system is described above. Figure 2 The description of the dirt detection method for the illustrated electronic rearview mirror system includes, for example, the image determination module 510 for implementing the relevant content of S210, the first determination module 520 for implementing the relevant content of S220, the second determination module 530 for implementing the relevant content of S230, and the conclusion determination module 540 for implementing the relevant content of S240. Each unit or module in the dirt detection device 500 of the electronic rearview mirror system can be individually or entirely merged into one or more other units or modules, or some of the units or modules can be further divided into multiple functionally smaller units or modules. This achieves the same operation without affecting the technical effect of the embodiments of the present invention. The above-mentioned units or modules are based on logical function division. In practical applications, the function of one unit (or module) is implemented by multiple units (or modules), or the function of multiple units (or modules) is implemented by one unit (or module).

[0170] As can be seen, the dirt detection device of the electronic rearview mirror system described in this application acquires a first image, which is a rearview mirror sensing image optimized for both size and image quality, and the rearview mirror sensing image is an image from the sensing device; determines the type and location of dirt based on the first image, where the type of dirt characterizes the physical properties of the dirt; determines an overlap parameter based on the location of the dirt, the first image, and a second image, where the second image is a neighboring frame of the first image; determines the presence of dirt based on the overlap parameter and the type of dirt, and controls the presence and location of dirt to be visually displayed on a display device. Thus, by intelligently analyzing the first image to determine the type and location of dirt, and by determining the overlap parameter based on the dirt condition and the neighboring frame image for confidence confirmation, the reliability and accuracy of the dirt condition are improved.

[0171] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device proposed in an embodiment of this application, as shown below. Figure 6As shown, the electronic device 600 includes a processor 610, a memory 620, a communication interface 630, and one or more programs 621. The one or more programs 621 are stored in the memory 620 and configured to be executed by the processor 610. The processor 610, the memory 620, and the communication interface 630 are interconnected and perform communication between them.

[0172] The memory 620 can be volatile memory such as dynamic random access memory (DRAM) or non-volatile memory such as a hard disk drive (HDD). The memory 620 stores a set of executable program code, and the processor 610 calls one or more programs 621 stored in the memory 620 to execute programs such as... Figure 2 Some or all of the steps of the dirt detection method for any electronic rearview mirror system described in the embodiments.

[0173] Among them, electronic devices 600 may include dashcams, in-vehicle electronic devices, mobile internet devices (MIDs), etc. The above are just examples and not an exhaustive list, including but not limited to the above electronic devices.

[0174] As can be seen, the electronic device 600 acquires a first image, which is a rearview mirror sensor image optimized for uniform size and image quality, and the rearview mirror sensor image is an image from the sensing device; based on the first image, it determines the type and location of dirt, where the type of dirt characterizes the physical properties of the dirt; based on the location of the dirt, the first image, and a second image, it determines an overlap parameter, where the second image is a neighboring frame of the first image; based on the overlap parameter and the type of dirt, it determines the presence of dirt, and controls the presence and location of dirt to be visually displayed on a display device. Thus, by intelligently analyzing the first image to determine the type and location of dirt, and by determining the overlap parameter based on the dirt condition and the neighboring frame image for confidence confirmation, the reliability and accuracy of the dirt condition are improved.

[0175] In the case of using integrated units, please refer to Figure 7 , Figure 7 This is a schematic diagram of the structure of a dirt detection device for another electronic rearview mirror system provided in this application embodiment, as shown below. Figure 7As shown, the dirt detection device 500 of the electronic rearview mirror system includes a processing module 502 and a communication module 501. The processing module 502 controls and manages the operation of the dirt detection device 500, for example, executing the steps of the image determination module 510, the first determination module 520, the second determination module 530, and the conclusion determination module 540, and / or performing other processes of the technology described herein. The communication module 501 is used for interaction between the dirt detection device 500 and other devices. Figure 7 As shown, the dirt detection device 500 of the electronic rearview mirror system may also include a storage module 503, which is used to store the program code and data of the dirt detection device 500 of the electronic rearview mirror system.

[0176] The processing module 502 can be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The communication module 501 can be a transceiver, RF circuitry, or a communication interface, etc. The storage module 503 can be a memory.

[0177] All relevant content in the various scenarios involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here. The dirt detection device 500 of the above-mentioned electronic rearview mirror system can perform the above-mentioned... Figure 2 The method for detecting dirt in the electronic rearview mirror system shown.

[0178] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.

[0179] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may include an electronic device.

[0180] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0181] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0183] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0185] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer electronic device (which may be a personal computer, electronic device, or network electronic device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0186] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0187] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting dirt in an electronic rearview mirror system, characterized in that, An electronic rearview mirror chip processor for use in an electronic rearview mirror system, the electronic rearview mirror chip processor including an image signal processor and a neural network processor, the electronic rearview mirror system further including a sensing device and a display device, the method including: Acquire a first image, which is a rearview mirror sensing image that has been uniformly optimized in terms of size and image quality by the image signal processor, and the rearview mirror sensing image is an image from the sensing device; The neural network processor determines the type and location of dirt based on the first image. The type of dirt characterizes the physical properties of the dirt, including solid, smeared, and water mist types. The overlap parameter is determined based on the location of the dirt, the first image, and the second image, where the second image is a neighboring frame of the first image. Determining the presence of dirt based on the overlap parameter and the type of dirt includes: An assessment strategy is determined based on the type of dirt, and each assessment strategy corresponds to one type of dirt. The confidence level of the result is determined based on the evaluation strategy and the overlap parameter. The presence of dirt is determined based on the result confidence level and the result confidence threshold, wherein the presence of dirt includes at least one of dirt type information and the probability of dirt presence; The presence and location of the dirt are displayed visually on the display device.

2. The method according to claim 1, characterized in that, The second image is the preceding frame of the first image. The step of determining the overlap parameter based on the location of the dirt, the first image, and the second image includes: Retrieve the second image and the dirt information of the previous frame corresponding to the second image, wherein the dirt information of the previous frame includes the location and type of dirt in the previous frame. The intersection-union ratio is determined based on the second image and the first image; The overlap parameter is determined based on the crossover ratio and the location of the dirt in the previous frame. The overlap parameter is used to characterize the degree of overlap of the dirt locations.

3. The method according to claim 1, characterized in that, The second image is a frame following the first image. The step of determining the overlap parameter based on the location of the dirt, the first image, and the second image includes: Acquire a second image, and determine the type and location of dirt in the subsequent frame based on the second image; The intersection-union ratio is determined based on the second image and the first image; The overlap parameter is determined based on the intersection-to-union ratio and the location of the contamination in the subsequent frame. The overlap parameter is used to characterize the degree of overlap of the contamination locations.

4. The method according to claim 1, characterized in that, Before the step of determining the presence of dirt based on the overlap parameter and the type of dirt, the method further includes: Obtain vehicle environmental parameters, including vehicle speed; Based on the vehicle environmental parameters, an environmental impact factor is determined, which is used to characterize the degree of influence of at least one of vibration and / or airflow caused by vehicle speed. The confidence threshold of the results is adjusted based on the environmental impact factors.

5. The method according to claim 1, characterized in that, Determining the type and location of dirt based on the first image includes: Based on the first image and a pre-trained dirt detection model, the type and location of dirt are determined. The process of determining the type and location of dirt using the pre-trained dirt detection model includes: Based on the first image, unclear areas and unclear regions are determined. The unclear area is the relative or absolute area of ​​the unclear region relative to the overall area. The unclear region represents the location and / or shape of the dirt. Determine boundary parameters and region image parameters based on the unclear region; The type of dirt is determined based on the boundary parameters and / or the unclear area and / or region image parameters; The location of the dirt is determined based on the unclear area and the unclear region.

6. The method according to claim 5, characterized in that, The boundary parameters include boundary strength parameters, which are used to characterize the real or virtual state of dirt; the region image parameters include texture characteristics and / or contrast characteristics. Determining the type of contamination based on the boundary parameters and / or the unclear area and / or region image parameters includes: Determine the boundary gradient parameters, and determine the boundary strength parameters based on the boundary gradient parameters. The boundary strength parameters are used to characterize the real and virtual state of the dirt. The degree of dirt coverage is determined based on the unclear area, and the degree of dirt coverage represents the size of the area of ​​the entire image covered by the dirt; The type of dirt is determined based on the dirt coverage and / or texture characteristics and / or contrast characteristics and / or boundary gradient parameters.

7. A dirt detection device for an electronic rearview mirror system, characterized in that, An electronic rearview mirror chip processor for use in an electronic rearview mirror system, the electronic rearview mirror chip processor including an image signal processor and a neural network processor, the electronic rearview mirror system further including a sensing device and a display device, the device including: An image determination module is used to acquire a first image, which is a rearview mirror sensing image that has been uniformly optimized in terms of size and image quality by the image signal processor, and the rearview mirror sensing image is an image from the sensing device. The first determining module is used by the neural network processor to determine the type and location of dirt based on the first image. The type of dirt characterizes the physical properties of the dirt, and the type of dirt includes solid, smear, and water mist. The second determining module is used to determine the overlap parameter based on the location of the dirt, the first image, and the second image, wherein the second image is a neighboring frame image of the first image; The conclusion determination module is used to determine the presence of dirt based on the overlap parameter and the type of dirt, including: An assessment strategy is determined based on the type of dirt, and each assessment strategy corresponds to one type of dirt. The confidence level of the result is determined based on the evaluation strategy and the overlap parameter. The presence of dirt is determined based on the result confidence level and the result confidence threshold, wherein the presence of dirt includes at least one of dirt type information and the probability of dirt presence; The presence and location of the dirt are displayed visually on the display device.

8. A computer-readable storage medium, characterized in that, The device stores a dirt detection program for an electronic rearview mirror system, including execution instructions, which, when executed by a processor of the electronic device, perform the method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes a processor, memory, a communication interface, and one or more programs, which are stored in the memory and configured to be executed by the processor; When the processor executes the one or more programs stored in the memory, the processor performs the method as described in any one of claims 1 to 6.

Images