Image processing method of vehicle and vehicle

Abstract

The embodiment of the invention provides a vehicle image processing method and a vehicle, and the method comprises the steps: obtaining an environment image collected by an image collection device, and the image content of the environment image is used for representing the environment where the vehicle is located; acquiring first feature information of the environment image and second feature information of the image acquisition equipment; based on the first feature information and the second feature information, detecting the image acquisition equipment to obtain a detection result; in response to the detection result indicating that the target substance is attached to the image acquisition equipment and the state information of the target substance exceeds a state information threshold, adjusting the environment image to obtain an adjusted environment image, the state information being used for indicating the degree of influence of the target substance on the degree of sharpness and the degree of sharpness of the adjusted environment image, and the state information being used for indicating the degree of sharpness of the adjusted environment image. The definition degree is greater than that of the environment image before adjustment. The technical problem of vehicle image processing accuracy is solved.

Description

Vehicle image processing methods and vehicles Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image processing method for a vehicle and a vehicle. Background Technology

[0002] Currently, the cleanliness of a vehicle's image acquisition equipment, and the presence of target substances on or in the environment, directly affect the clarity of the images captured and the accuracy of the vehicle's driving decisions. In related technologies, if target substances are present on the image acquisition equipment, physical methods, such as ultrasonic cleaning technology or defogging algorithms, can be used to attempt to remove them.

[0003] However, the above methods can only clean the image acquisition equipment to ensure the clarity of subsequently acquired environmental images. They cannot process environmental images that are already affected by target substances and have low clarity. Furthermore, they cannot remove target substances that cannot be cleaned from the image acquisition equipment or that are present in the environment. Therefore, technical issues regarding the accuracy of vehicle image processing remain.

[0004] There is currently no good solution to the above problems. Summary of the Invention

[0005] This application provides an image processing method and a vehicle to at least solve the technical problem of image processing accuracy for vehicles.

[0006] According to one aspect of the embodiments of this application, an image processing method for a vehicle is provided, wherein the vehicle includes an image acquisition device. The method may include: acquiring an environmental image acquired by the image acquisition device, wherein the image content of the environmental image is used to represent the environment in which the vehicle is located; acquiring first feature information of the environmental image and second feature information of the image acquisition device, wherein the first feature information is used to represent the clarity of the environmental image and the second feature information is used to represent the degree of influence of the image acquisition device on the clarity; detecting the image acquisition device based on the first feature information and the second feature information to obtain a detection result; and adjusting the environmental image in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold to obtain an adjusted environmental image, wherein the state information is used to represent the degree of influence of the target substance on the clarity, and the clarity of the adjusted environmental image is greater than the clarity of the environmental image before adjustment.

[0007] Furthermore, acquiring the first feature information of the environmental image includes: analyzing the environmental image using the vehicle's image analysis unit to obtain the first feature information; acquiring the second feature information of the image acquisition device includes: acquiring the second feature information using the vehicle's physical sensor unit.

[0008] Furthermore, the vehicle's image analysis unit is used to analyze the environmental image to obtain first feature information, including:

[0009] Using an image analysis unit, the grayscale value and pixel count of an environmental image are acquired, and gradient information of the environmental image is determined based on the grayscale value and pixel count. The gradient information represents the richness of image content and / or the clarity of image edges in the environmental image. Using the image analysis unit, color channels and region information of the environmental image are acquired. Region information represents the brightness difference between different pixels in a preset local region of the environmental image. Channel information of the environmental image is determined based on the color channels and region information. Channel information represents the severity of the influence of the target substance on the environmental image. First feature information is determined based on the gradient information and channel information. Alternatively, second feature information is acquired using a vehicle's physical sensor unit, including: acquiring transmission information and humidity information of the image acquisition device using the physical sensor unit. Transmission information characterizes the cleanliness of the image acquisition device, and humidity information represents the moisture level of the image acquisition device. Second feature information is determined based on the humidity information and transmission information.

[0010] Furthermore, based on the first feature information and the second feature information, the image acquisition device is detected to obtain the detection result, including: using a machine learning model to fuse the first feature information and the second feature information to obtain fused feature information; using the fused feature information to detect the image acquisition device to obtain the detection result.

[0011] Furthermore, the detection results include a first detection result and a second detection result. The image acquisition device is detected using fused feature information to obtain the detection results, including: using a support vector machine model, based on fused feature information, to determine the first detection result and the second detection result, wherein the first detection result is used to represent the probability of a target substance being attached to the image acquisition device, and the second detection result is used to represent the type of the target substance.

[0012] Furthermore, the state information threshold includes a first state information threshold and a second state information threshold, wherein the second state information threshold is less than the first state information threshold. The method further includes: determining the first state information threshold based on the vehicle's operating state information, wherein the operating state information is used to represent the vehicle's operating state during the process of the image acquisition device acquiring environmental images; responding to a first detection result that the state information exceeds the first state information threshold and a second detection result that the type is a first preset type, controlling the vehicle to perform a cleaning operation on the image acquisition device, wherein the cleaning difficulty of the target substance of the first preset type is less than the cleaning difficulty threshold; after performing the cleaning operation on the image acquisition device, responding to a state information exceeding the second state information threshold, or a second detection result that the type is a second preset type, controlling the vehicle to issue a prompt message to the driver, wherein the cleaning difficulty of the target substance of the second preset type is greater than or equal to the cleaning difficulty threshold, and the prompt message is used to instruct the driver and passengers in the vehicle to take over the perception of the environment from the image acquisition device.

[0013] Furthermore, in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds the state information threshold, the environmental image is adjusted to obtain an adjusted environmental image, including: in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds the state information threshold, determining an image degradation model of the environmental image; and adjusting the environmental image based on the image degradation model to obtain the adjusted environmental image.

[0014] Furthermore, in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, an image degradation model for the environmental image is determined, including: in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, determining atmospheric transmission information and material transmission information of the environmental image, wherein atmospheric transmission information is used to represent the degree of light attenuation caused by atmospheric conditions reflected in the environmental image, and material transmission information is used to represent the degree of light attenuation caused by the target substance; and determining an image degradation model based on the fused transmission information of atmospheric transmission information and material transmission information, the light value information of the environmental image, and the number of pixels in the environmental image, wherein the light value information is used to represent the global brightness level of the target substance in the environmental image.

[0015] Furthermore, based on the image degradation model, the environmental image is adjusted to obtain the adjusted environmental image, including: using guided filtering, adjusting the fused transmission information based on the environmental image to obtain the adjusted fused transmission information, wherein the matching degree between the adjusted fused transmission information and the environmental image is greater than the matching degree between the fused transmission information and the environmental image before adjustment; using the image degradation model corresponding to the adjusted fused transmission information to adjust the environmental image to obtain the adjusted environmental image.

[0016] Furthermore, the method also includes: identifying environmental objects in the environment from the adjusted environmental image, wherein the environmental objects are used to characterize the degree of safety affecting the safety of the vehicle driving process.

[0017] According to one aspect of the embodiments of this application, an image processing apparatus for a vehicle is provided, wherein the apparatus may include: a first acquisition module, configured to acquire an environmental image acquired by an image acquisition device, wherein the image content of the environmental image is used to represent the environment in which the vehicle is located; a second acquisition module, configured to acquire first feature information of the environmental image and second feature information of the image acquisition device, wherein the first feature information is used to represent the clarity of the environmental image and the second feature information is used to represent the degree of influence of the image acquisition device on the clarity; a detection module, configured to detect the image acquisition device based on the first feature information and the second feature information, and obtain a detection result; and an adjustment module, configured to adjust the environmental image in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, and obtain an adjusted environmental image, wherein the state information is used to represent the degree of influence of the target substance on the clarity, and the clarity of the adjusted environmental image is greater than the clarity of the environmental image before adjustment.

[0018] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0019] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0020] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0021] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0022] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0023] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0024] In this embodiment of the invention, by combining the first feature information of the environmental image and the second feature information of the image acquisition device for joint detection, the presence of a target substance affecting the image acquisition device and the presence of a target substance on the image acquisition device, as well as whether the state information of the target substance exceeds a state information threshold, are detected. Based on these detection results, the environmental image is adjusted, effectively improving the clarity of the environmental image by addressing the target substance. This method overcomes the limitations of related technologies that rely solely on physical cleaning and cannot handle environmental images with or without attached target substances. It provides an intelligent and adaptive solution for image acquisition device state detection and image clarity restoration, thereby solving the technical problem of image processing accuracy for vehicles and achieving the technical effect of improving vehicle image processing accuracy. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 is a flowchart of a vehicle image processing method according to an embodiment of this application;

[0027] Figure 2 is a schematic diagram of a multimodal stain detection and image restoration system according to an embodiment of this application;

[0028] Figure 3 is a flowchart of a multimodal stain detection module according to an embodiment of this application;

[0029] Figure 4 is a flowchart of an adaptive image restoration algorithm according to an embodiment of this application;

[0030] Figure 5 is a flowchart of a vehicle cooperative response control logic according to an embodiment of this application;

[0031] Figure 6 is a schematic diagram of an image processing apparatus for a vehicle according to an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0034] According to an embodiment of this application, an embodiment of a vehicle image processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] This embodiment provides a vehicle image processing method. Figure 1 is a flowchart of a vehicle image processing method according to an embodiment of this application. As shown in Figure 1, the method may include the following steps.

[0036] Step S102: Obtain the environmental image captured by the image acquisition device.

[0037] In the technical solution provided by step S102 of the embodiments of this application, the image content of the environmental image is used to represent the environment in which the vehicle is located.

[0038] Optionally, the vehicle's image acquisition equipment can refer to equipment installed on the vehicle to capture visual feature information of the road and surrounding environment in real time. For example, the aforementioned image acquisition equipment can be a camera deployed in front of the vehicle. The aforementioned image acquisition equipment can also be some auxiliary sensing elements other than the aforementioned camera, such as infrared thermal imagers, surround-view cameras, and specific band filter cameras. Among them, the vehicle with the aforementioned image acquisition equipment can be a vehicle with autonomous driving function, or a vehicle with driver assistance function, or a vehicle that requires a driver to drive itself.

[0039] Optionally, the environmental image can be used to represent visual feature information of the environment in which the vehicle is located. For example, the visual feature information in the environmental image may include: road features during vehicle movement, the status of traffic participants, traffic signals and signs, weather and visibility conditions, surrounding obstacles, and specific scenes.

[0040] In this embodiment, if image processing of the vehicle is required, an environmental image captured by an image acquisition device can be obtained. The image acquisition device receives light from the surrounding environment of the vehicle, and its photosensitive element converts the light signal into a digital image signal, thereby obtaining the environmental image.

[0041] In this embodiment, the vehicle's image acquisition device can adapt to different lighting conditions in the vehicle's environment, automatically adjusting exposure time and gain to obtain clear and accurate environmental images. For example, when driving at night, the infrared night vision function of the image acquisition device is activated to enhance image quality in low-light environments; in strong sunlight, the image acquisition device automatically controls exposure to avoid overexposure and maintain visible details. Furthermore, the image acquisition device can be equipped with special filters, such as polarizing filters, to reduce the impact of reflections from water or road surfaces on environmental image acquisition, improving the clarity of environmental images when driving in rain or on slippery roads, thus obtaining clearer environmental images.

[0042] Step S104: Obtain the first feature information of the environmental image and the second feature information of the image acquisition device.

[0043] In the technical solution provided by step S104 of the embodiments of this application, the first feature information can be used to represent the clarity of the environmental image. The second feature information can be used to represent the degree to which the image acquisition device affects the clarity.

[0044] Optionally, the first feature information can be a visual feature index of the environmental image, which can be used to quantify the sharpness of the environmental image. The aforementioned visual feature index may include, but is not limited to, image gradient, dark channel, contrast, sharpness, and color saturation of the environmental image.

[0045] Optionally, the second feature information can be environmental perception data acquired by a physical sensor built into the image acquisition device. This physical sensor may include, but is not limited to, optical transmittance sensors, capacitive droplet sensors, and temperature sensors. The environmental perception data can be used to assess the degree of influence of the target substance on the image acquisition device, thereby quantifying the impact on the clarity of the environmental image. If the physical sensor is an optical transmittance sensor, the acquired environmental perception data can be transmittance; if the physical sensor is a capacitive droplet sensor, the acquired environmental perception data can be capacitance; and if the physical sensor is a temperature sensor, the acquired environmental perception data can be ambient temperature.

[0046] In this embodiment, after acquiring the environmental image captured by the image acquisition device, the first feature information of the environmental image can be acquired, and the second feature information of the image acquisition device can also be acquired.

[0047] Optionally, during the acquisition of the first feature information, visual feature indices can be directly calculated from the environmental image, and the first feature information can be determined based on these indices. The image gradient can be obtained from the richness of edge and texture information in the environmental image; for example, in fog-free conditions, the low-pixel portions of the color channels in the environmental image can be used as the dark channel. In foggy environments, the low-pixel portions of the color channels in the environmental image are brightened by the fog, and the dark channel is obtained by performing a minimum value operation on each pixel in the environmental image.

[0048] Optionally, during the acquisition of the second feature information, environmental perception data can be collected using physical sensors built into the image acquisition device, and the second feature information can be determined based on this data. For example, the attenuation of light passing through the mirror of the image acquisition device can be continuously monitored using an optical transmittance sensor built into the device, and compared with the readings when the mirror is clean, to calculate the transmittance of the image acquisition device. This transmittance can represent the degree to which the mirror of the image acquisition device is affected by the target substance, and can be used to quantify the impact of the mirror on the clarity of the environmental image.

[0049] In this embodiment, by acquiring the first feature information of the environmental image and the second feature information of the image acquisition device in real time for comprehensive analysis, it is possible to evaluate and process environmental images that have been affected by the target substance and have suffered a decrease in clarity. Compared with related technologies that only target the cleaning strategy of the image acquisition device, this solves the technical problem that environmental images with the target substance attached cannot be accurately processed in the current technology, and achieves the technical effect of improving the accuracy of processing environmental images with the target substance attached.

[0050] Step S106: Based on the first feature information and the second feature information, the image acquisition device is detected to obtain the detection result.

[0051] In the technical solution provided by step S106 of the embodiments of this application, the detection result can be a numerical value representing the degree of influence of the target substance on the image acquisition device, and a classification label representing the type of target substance attached to the image acquisition device. For example, the detection result can be a numerical value between 0 and 1, representing the degree of influence of the target substance on the image acquisition device; or, the detection result can be a classification label, such as "no pollution", "dust", "water droplets", "mud spots", etc., used to indicate the type of target substance attached to the image acquisition device.

[0052] In this embodiment, after acquiring the first feature information of the environmental image and the second feature information of the image acquisition device, the system can detect whether the image acquisition device is attached to a target substance and the state information of the attached target substance based on the first feature information and the second feature information, and obtain the detection result.

[0053] Optionally, by fusing the first feature information of the environmental image and the second feature information of the image acquisition device, fused information is obtained. This fused information is then input into a machine learning model for detection. The machine learning model can output a detection result representing a numerical value indicating the degree of influence of the target substance on the image acquisition device; or, the machine learning model can output a classification label indicating the type of target substance. For example, the machine learning model outputs a value between 0 and 1, where 0 represents no pollution and 1 represents severe impact, and also provides a specific pollution type label, such as "no pollution," "dust," "water droplets," or "mud spots."

[0054] In this embodiment, by analyzing and integrating the first feature information of the environmental image and the second feature information of the image acquisition device, it is possible to determine whether the image acquisition device is attached to a target substance, the type of the target substance, and the degree of influence of the target substance on the environmental image. A detection result is then output, including the degree of influence of the target substance on the environmental image and a classification label for the type of target substance. This achieves intelligent and comprehensive detection of the image acquisition device, effectively solving the problem of image processing accuracy in the prior art.

[0055] Step S108: In response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds the state information threshold, the environmental image is adjusted to obtain the adjusted environmental image.

[0056] In the technical solution provided by step S108 of the embodiments of this application, the state information can be used to represent the degree of influence of the target substance on the clarity. The clarity of the adjusted environmental image is greater than the clarity of the unadjusted environmental image.

[0057] Optionally, the target substance can refer to external substances that obstruct the mirror surface of the image acquisition device or affect the imaging quality of the image acquisition device. The target substance can be substances adhering to the image acquisition device, such as soil, oil stains, water droplets, or dirt, or substances in the environment where the image acquisition device is located, such as smog or dust. The aforementioned state information includes, but is not limited to: transmittance attenuation coefficient, pollution distribution density map, and pollution degree quantification score. It should be noted that the above state information is only illustrative and is not specifically limited here. Any state information that can be used to describe the type, coverage area, and obstruction effect of the target substance is within the protection scope of this application's embodiments.

[0058] Optionally, the state information threshold can be a preset limit value used to assess the impact of the state of the target substance on the environmental image. The state information threshold can be set considering vehicle driving conditions, weather conditions, and the type of target substance. For example, in rainy weather, considering that raindrops may instantly adhere to a large area of ​​the image acquisition device, causing a significant decrease in environmental image quality, the state information threshold can be set as a raindrop coverage threshold. For example, a coverage threshold of 5%; when the target substance coverage on the image acquisition device exceeds this 5% threshold, environmental image adjustment can be triggered. Conversely, in sunny and dry weather conditions, the likelihood of target substance adhering to the image acquisition device is low, and the state information threshold can be set to a higher value. For example, the state information threshold can be a coverage threshold of 15%; when the target substance coverage on the image acquisition device exceeds this 15% threshold, image adjustment can be triggered. When the state information of the target substance reaches or exceeds the state information threshold, the clarity of the environmental image is considered to have been affected, and the environmental image can be adjusted to obtain an adjusted environmental image.

[0059] In this embodiment, after obtaining the detection result, it can be determined whether the state information of the target substance attached to the image acquisition device exceeds the state information threshold. If the state information of the target substance exceeds the state information threshold, the environmental image is adjusted to obtain the adjusted environmental image.

[0060] Optionally, if the detection results indicate that a target substance, such as water droplets or dust, is attached to the mirror surface of the image acquisition device, and the state information of this target substance exceeds a preset state information threshold, the environmental image will be adjusted. The impact level obtained from the detection results is 0.8, while the state information threshold is set to 0.7. This indicates that the target substance has reduced the clarity of the environmental image. For water droplets, a composite image enhancement algorithm based on an atmospheric scattering model will be applied to adjust the environmental image, resulting in an adjusted environmental image.

[0061] In steps S102 to S108 of this embodiment, by combining the first feature information of the environmental image and the second feature information of the image acquisition device for joint detection, the presence of a target substance affecting the image acquisition device and the detection results of a target substance adhering to the image acquisition device and a target substance state information exceeding a state information threshold are obtained. Based on the detection results, the environmental image is adjusted, effectively adjusting the environmental image with an attached target substance, thereby significantly improving the clarity of the environmental image. This method overcomes the limitations of related technologies that rely solely on physical cleaning and cannot process environmental images with attached target substances or those that cannot be cleaned. It achieves an intelligent and adaptive image acquisition device state detection and image clarity restoration scheme, thus solving the technical problem of vehicle image processing accuracy and improving the technical effect of vehicle image processing accuracy.

[0062] The embodiments of this application will be described in detail below with reference to the steps described above.

[0063] As an optional implementation, step S104, obtaining the first feature information of the environmental image, includes: using the vehicle's image analysis unit to analyze the environmental image to obtain the first feature information.

[0064] In this embodiment, during the acquisition of the first feature information of the environmental image, the vehicle's image analysis unit can be used to analyze the environmental image and obtain the first feature information. The image analysis unit can refer to software or hardware components deployed on the vehicle for analyzing and processing environmental images. The image analysis unit can continuously receive environmental images acquired in real time from the image acquisition device, extract visual feature information from the acquired environmental images, and convert the extracted visual feature information into a series of quantitative indicators, such as the standard deviation of the gradient distribution, the average value of the dark channel prior, and the recognition rate of key visual information, which constitute the first feature information.

[0065] Optionally, the image analysis unit can continuously receive environmental images from the image acquisition device, and use the image analysis unit to perform grayscale conversion on the environmental images, simplifying the RGB driving environment images into grayscale images for subsequent gradient analysis and dark channel calculation. By calculating the standard deviation of the gradient distribution of the grayscale image, the image analysis unit calculates the average value of the dark channel prior. In addition to the aforementioned standard deviation of the gradient distribution and the average value of the dark channel prior, the image analysis unit may also be involved in the calculation of other visual information, which may include, but is not limited to, traffic signs, lane lines, pedestrians, or obstacles. The image analysis unit can comprehensively extract the visual feature information of the environmental images, transforming the visual feature information into a series of quantitative indicators to form the first feature information.

[0066] As an optional implementation, step S104, acquiring the second feature information of the image acquisition device, includes: acquiring the second feature information using the vehicle's physical sensor unit.

[0067] In this embodiment, during the acquisition of the second feature information of the environmental image, the vehicle's physical sensor unit can be used to acquire the second feature information. The physical sensor unit refers to a collection of hardware devices integrated on the vehicle for directly monitoring the physical state of the image acquisition device. The physical sensor unit may include, but is not limited to, the following types of sensors: optical transmittance sensors, capacitive droplet sensors, temperature and humidity sensors, ultrasonic sensors, pressure sensors, etc. The image analysis unit provides data on the adhesion of the target substance by detecting the physical parameters on the image acquisition device; this data constitutes the second feature information.

[0068] Optionally, the physical sensor unit directly monitors and quantifies the physical state of the mirror surface of the image acquisition device. The physical sensor unit includes various types of sensors. Each sensor in the physical sensor unit precisely measures different physical parameters to ensure the adhesion of target substances to the image acquisition device. For example, an optical transmittance sensor is deployed around the lens of the image acquisition device to monitor the attenuation of light passing through the lens in real time. In a clean state, the transmittance is close to 100%, meaning almost no light loss; however, when the mirror surface is covered with dust, water droplets, or other contaminants, the transmittance decreases. The optical transmittance sensor converts this decrease in transmittance into a visually intuitive value as part of the second feature information. Data collected by various types of sensors, after real-time conversion by the physical sensor unit, is summarized into second feature information reflecting the adhesion of target substances. The aforementioned clean state can be a state where no target substances adhere to the image acquisition device, or a state where no target substances are present in the surrounding environment of the image acquisition device.

[0069] In this embodiment, the first feature information and the second feature information are obtained by the image analysis unit and the physical sensor unit, so as to realize multimodal detection of the environmental image and the image acquisition device to remove the target substance. This solves the technical problem in the related technology that the target substance cleaning operation is started when any type of target substance is detected, and achieves the technical effect of improving the processing efficiency of the target substance.

[0070] As an optional implementation, the vehicle's image analysis unit is used to analyze the environmental image to obtain first feature information, including: using the image analysis unit to obtain the grayscale value and the number of pixels of the environmental image, and determining the gradient information of the environmental image based on the grayscale value and the number of pixels; using the image analysis unit to obtain the color channels of the environmental image, and the region information of the environmental image; determining the channel information of the environmental image based on the color channels and the region information; and determining the first feature information based on the gradient information and the channel information.

[0071] In this embodiment, during the process of analyzing the environmental image using the vehicle's image analysis unit to obtain the first feature information, the image analysis unit can be used to identify the environmental image and obtain the grayscale value and number of pixels of the environmental image.

[0072] Optionally, grayscale values ​​can be used to represent the brightness and contrast information of an environmental image, and can be used by the image analysis unit to evaluate the sharpness of the environmental image. Grayscale values ​​can be the brightness level of each pixel after converting an RGB environmental image to a grayscale image, typically ranging from 0 (representing black, i.e., the lowest brightness) to 255 (representing white, i.e., the highest brightness). In the field of image processing, the distribution and changes in grayscale values ​​can be used to evaluate the sharpness and contrast of an environmental image, thereby reflecting the degree to which the image acquisition device is affected by the target substance. For example, the aforementioned grayscale values ​​can be image grayscale values, which can be represented by I(x).

[0073] Optionally, the number of pixels can be used to represent the resolution of the image acquisition device and the size of the environmental image, and can be used to determine the sharpness of the environmental image. The aforementioned number of pixels can be the total number of pixels, represented by N. A higher number of pixels means that the image acquisition device can capture more information from the environmental image.

[0074] Optionally, the received RGB format environmental image is converted into a grayscale image by an image analysis unit. The conversion is based on a linear combination rule, such as Y = 0.2126R + 0.7152G + 0.0722B, where Y can represent the grayscale value, and R, G, and B can represent the pixel values ​​of the red, green, and blue color channels, respectively. The RGB color information of each pixel is compressed into a grayscale value between 0 and 255 to obtain the grayscale value of the environmental image. The width and height of the environmental image are calculated by the image analysis unit, and the number of pixels is obtained by multiplying the width and height. If the number of pixels is greater than a pixel count threshold, it indicates that the environmental image has more detailed information.

[0075] In this embodiment, after obtaining the grayscale value and the number of pixels, gradient information can be determined based on the grayscale value and the number of pixels. The gradient information is used to represent the richness of image content in the environmental image and / or the sharpness of image edges in the environmental image.

[0076] Optionally, gradient information can refer to the difference in grayscale value between each pixel and its neighboring pixels in the environmental image. Gradient information includes, but is not limited to, horizontal gradient, vertical gradient, gradient magnitude, and gradient direction. Gradient information can be used to represent edge details in the environmental image, corresponding to edges or boundaries in the image. For example, the horizontal gradient can be used to represent the rate of change of grayscale values ​​along the horizontal direction in the environmental image, helping to detect and emphasize edges in the horizontal direction, such as road markings and the horizontal edges of buildings. For instance, the gradient information mentioned above can be the average gradient of the environmental image.

[0077] Optionally, an environmental image with a resolution of 640×480 pixels is acquired from the image acquisition device via an image analysis unit, and the environmental image is converted into a grayscale image. Since the environmental image resolution is 640×480, the total number of pixels is N=640. 480 = 307200. Gradient information is determined by calculating the difference in grayscale values ​​between each pixel and its neighboring pixels. For example, the horizontal gradient is obtained by comparing the grayscale values ​​of each pixel with its two left and right adjacent pixels.

[0078] For example, if the gradient information is the average gradient, it can be determined using the following formula:

[0079] G_avg=(1 / N) Σ| I(x)|

[0080] Here, G_avg can be used to represent the average gradient, and I(x) can be used to represent the image grayscale value. It can be used to represent gradient operators, and N can be used to represent the total number of pixels.

[0081] In this embodiment, an image analysis unit can be used to acquire the color channels of an environmental image and the region information of the environmental image. The region information represents the brightness difference between different pixels in a preset local area of ​​the environmental image, and the channel information represents the severity of the environmental image being affected by the target substance.

[0082] Optionally, color channels can be used to represent the distribution and intensity of different color information in an environmental image, as well as the spatial layout of color information within the image. A color channel can refer to a monochrome image layer obtained after decomposing a color image in computer image processing. A monochrome image layer includes three basic color channels: Red (R), Green (G), and Blue (B). Additional channels, such as transparency channels, can also be included. If the color channels of the aforementioned environmental image include R, G, and B channels, then the environmental image can be an RGB image. Each pixel in the RGB image is determined by the values ​​of these three color channels, and analyzing each channel individually can reveal the characteristics and variations of the environmental image across different color dimensions. For example, the aforementioned color channels can be RGB color channels.

[0083] Optionally, region information refers to visual feature data extracted from a preset local region or sub-image in an environmental image. Visual feature data can reflect the environmental image attributes and changes in a specific region. Region information includes, but is not limited to, mean brightness, standard deviation of brightness, contrast, and pixel density. For example, the aforementioned region information can be a local block centered on a pixel in the environmental image, where a pixel in the environmental image can be represented by x, and the region information can be represented by Ω(x).

[0084] Optionally, the image analysis unit extracts the R, G, and B component values ​​of each pixel in the environmental image, decomposing the RGB format environmental image into three independent grayscale image channels: R (red), G (green), and B (blue), which are also color channels. The image analysis unit performs statistical analysis on each color channel, calculating the distribution and intensity of color information, as well as the spatial layout of color information. For example, by calculating the average brightness and standard deviation of the R channel, the proportion of red areas in the environment and the brightness variation can be quantified. The image analysis unit segments the environmental image into multiple preset local regions or sub-images; for example, it can be divided into 10×10 sub-regions according to a grid layout. For each local region, the image analysis unit can analyze the statistical characteristics of the local region or sub-image, including parameters such as the mean brightness, standard deviation of brightness, and contrast, to obtain the regional information of the environmental image.

[0085] In this embodiment, after obtaining color channel and region information, the channel information of the environmental image can be determined using the color channel and region information.

[0086] Optionally, channel information can refer to a set of data about the clarity, visual features, and degree of influence from target substances in each color channel of the environmental image. Channel information includes, but is not limited to, mean and distribution of brightness, contrast and clarity, saturation and chromaticity variations, etc. For example, the aforementioned channel information can be a dark channel, which can be used to represent the lowest brightness value of each pixel in the environmental image across the three color channels.

[0087] Optionally, the color channel characteristics of each preset local region can be statistically analyzed to construct the channel information of an environmental image. For example, in a road area, the mean and distribution of brightness, contrast and sharpness, and changes in saturation and chroma of the R, G, and B channels can be analyzed. In a vegetation area, the characteristics of the G and B channels are analyzed in detail, because vegetation typically has stronger signals in the green and blue channels.

[0088] For example, if the channel information is a dark channel, it can be determined using the following formula:

[0089] J_dark(x)=min_{c∈{r,g,b}}(min_{y∈Ω(x)}(J_c(y)))

[0090] Here, J_dark(x) can be used to represent the dark channel, J_c can be used to represent the RGB color channel, Ω(x) can be used to represent a local block centered at x, and J_c(y) represents the brightness value of the color channel at pixel y.

[0091] In this embodiment, after obtaining gradient information and channel information, the second feature information can be determined using the gradient information and channel information.

[0092] Optionally, an image analysis unit is used to extract gradient and channel information. The extracted information includes, but is not limited to, dark channel, average brightness, contrast, and saturation. This information is organized into a feature vector, which is then input into a machine learning model. The machine learning model outputs the first feature information.

[0093] For example, the feature vector can be determined by the following formula:

[0094] V_image=[G_avg,J_dark_mean,J_dark_var,...]

[0095] Here, G_avg can be used to represent the average gradient, J_dark_mean can be used to represent the dark channel mean, and J_dark_var can be used to represent the dark channel variance.

[0096] In this embodiment, an image analysis unit acquires the grayscale value, pixel count, color channels, and region information of an environmental image, and then calculates gradient information and channel information to form first feature information. This first feature information is used to evaluate the clarity of the environmental image and to determine whether a target substance is causing a decrease in the clarity of the environmental image. This solves the technical problems of low detection accuracy of vehicle image acquisition devices when encountering target substances attached to mirror surfaces and the inability to effectively process environmental images already covered by target substances in current technologies. It achieves the technical effect of improving the detection efficiency of vehicle image acquisition devices when encountering target substances attached to mirror surfaces and effectively processing environmental images already covered by target substances.

[0097] As an optional implementation, the second feature information is obtained using the vehicle's physical sensor unit, including: obtaining transmission information of the image acquisition device and humidity information of the image acquisition device using the physical sensor unit, and determining the second feature information based on the humidity information and transmission information.

[0098] In this embodiment, during the process of acquiring the second feature information using the vehicle's physical sensor unit, the physical sensor unit can also acquire the transmission information and humidity information of the image acquisition device. The transmission information characterizes the cleanliness of the image acquisition device, and the humidity information indicates the moisture level of the image acquisition device.

[0099] Optionally, the transmission information can be the transmittance value directly measured by an optical transmittance sensor integrated into the external structure of the image acquisition device. Transmission information includes, but is not limited to, optical transmittance, transmission spectral characteristics, mirror temperature, and humidity. The aforementioned transmission information can be a value from the optical transmittance sensor and can be represented by T_measured.

[0100] Optionally, transmission information can be obtained by directly measuring the transmittance value using an optical transmittance sensor integrated into the external structure of the image acquisition device. This sensor consists of an infrared LED emitter and a photodetector. The infrared LED emitter emits infrared light into the image acquisition device, and the photodetector measures the intensity of the light reflected by the device. If the image acquisition device is free of any target material, the reflected light intensity is close to the sensor's maximum value. When a target material is present on the device, the reflected light intensity decreases due to absorption and scattering. By calculating the ratio between the reflected light intensity and the maximum value, the cleanliness level of the image acquisition device can be determined, typically represented by a value between 0 and 1, where 1 represents complete cleanliness, and a smaller value indicates a lower level of cleanliness.

[0101] Optionally, the humidity information can be an air humidity value directly measured by a humidity sensor integrated into or near the image acquisition device. Humidity information includes, but is not limited to, the humidity of the mirror surface, moisture distribution patterns, and relative humidity percentage. This humidity information can be a reading from a capacitive droplet sensor and can be represented by C_value.

[0102] Alternatively, humidity information can be obtained by directly measuring air humidity values ​​using a humidity sensor integrated into or near the image acquisition device. Utilizing the relationship between capacitance and the thickness of a target substance, when a target substance, such as water, is present on the image acquisition device, the effective capacitance between the image acquisition device and the sensor increases because water has a higher dielectric constant than air. By monitoring changes in capacitance, the thickness of water on the image acquisition device can be indirectly measured, reflecting the humidity level of the image acquisition device in the current environment.

[0103] In this embodiment, after acquiring transmission information and humidity information, the second feature information can be determined based on the transmission information and humidity information.

[0104] Optionally, the acquired transmission and humidity information can be analyzed. For example, the average and standard deviation of transmittance can be calculated to determine whether the transmittance of the current image acquisition device's mirror deviates from the set normal range. Simultaneously, humidity information can be analyzed to determine whether the current environment is conducive to the deposition or evaporation of the target substance on the image acquisition device's mirror. Based on the analysis results, the cleanliness and humidity level of the image acquisition device's mirror are evaluated, and the evaluation results are converted into an input vector. Second feature information is derived based on the analysis of the input vector.

[0105] For example, the input vector can be determined by the following formula:

[0106] [V_image,T_measured,C_value]

[0107] Among them, V_image can be used to represent visual feature information of environmental images, T_measured can be used to represent transmission information, and C_value can be used to represent humidity information.

[0108] In this embodiment, the transmission and humidity information acquired by the physical sensor unit on the vehicle are used to determine the second feature information, thereby achieving the assessment and judgment of the degree of influence of the target substance on the image acquisition device. This is something that existing technologies cannot achieve by simply relying on physical cleaning of the target substance. This solves the technical problem of the detection accuracy of the vehicle's image acquisition device when encountering a target substance adhered to a mirror surface, and achieves the technical effect of improving the detection accuracy of the vehicle's image acquisition device when encountering such a substance.

[0109] As an optional implementation, step S106, based on the first feature information and the second feature information, detects the image acquisition device to obtain a detection result, including: using a machine learning (ML) model to fuse the first feature information and the second feature information to obtain fused feature information; using the fused feature information to detect the image acquisition device to obtain a detection result.

[0110] In this embodiment, during the process of detecting the image acquisition device based on the first feature information and the second feature information to obtain the detection result, a machine learning model can be used to fuse the first feature information and the second feature information to obtain fused feature information.

[0111] Alternatively, a machine learning model can refer to a pre-trained support vector machine model, deep neural network model, random forest model, or other machine learning algorithm suitable for multi-feature fusion and classification. Machine learning models are used for information fusion.

[0112] Optionally, the first and second feature information are standardized into a single feature vector, which is then input into a support vector machine (SVM) model. Through internal operations within the SVM model, the input feature vector can be transformed into fused feature information. The aforementioned SVM model is a type of machine learning model.

[0113] In this embodiment, after obtaining the fused feature information, the fused feature information can be used to detect the image acquisition device and obtain the detection result.

[0114] Optionally, the fused feature information is decomposed into multiple sub-features related to the cleanliness of the image acquisition device's mirror and the ambient humidity. These sub-features may include: average mirror transmittance, transmittance coefficient of variation, mirror relative humidity, mirror surface humidity, the average value of the brightest pixel in the dark current image, and the rate of change in image contrast. A rule base is queried, containing a series of rules and strategies based on these sub-features, to map the fused feature information to specific detection results. Detection results are generated based on the rules in the rule base that match the current fused feature information.

[0115] In this embodiment, the collected feature information is fused using a machine learning model, which not only improves the accuracy of target substance detection but also overcomes the limitation of existing technologies in effectively processing low-resolution environmental images. This enables effective recovery of environmental images even when target substances are present in the image acquisition device. The detection method that fuses feature information can more accurately determine the type of target substance and its impact on the clarity of the environmental image, thereby enabling selective cleaning operations on the target substance and avoiding significant resource waste.

[0116] As an optional implementation, step S106, the detection result includes a first detection result and a second detection result. The image acquisition device is detected using fused feature information to obtain the detection result, including: using a support vector machine model, based on fused feature information, to determine the first detection result and the second detection result, wherein the first detection result is used to represent the probability of a target substance being attached to the image acquisition device, and the second detection result is used to represent the type of the target substance.

[0117] In this embodiment, during the process of detecting the image acquisition device using fused feature information and obtaining the detection result, a support vector machine model can be used to determine the first detection result and the second detection result based on the fused feature information. The detection result includes the first detection result and the second detection result, wherein the first detection result is used to represent the probability of a target substance adhering to the image acquisition device, and the second detection result is used to represent the type of the target substance.

[0118] Optionally, the first detection result may refer to one or more probability values ​​output by the machine learning model, which represent the likelihood of a specific type of target substance being present on the image acquisition device. Each probability value corresponds to one of the target substance types predefined by the machine learning model, representing the probability that the image acquisition device is affected by a certain type of target substance. For example, if the machine learning model defines three pollution states of the target substance (no pollution, slight pollution, severe pollution), the first detection result may be a three-dimensional probability vector, such as [0.1, 0.8, 0.1], where 0.8 indicates that the machine learning model judges that the image acquisition device has an 80% probability of being in a slight pollution state. For example, if the target substance is a stain, the first detection result may be a stain confidence score, using a continuous value between 0 and 1 to represent the probability of the presence of a stain.

[0119] Optionally, the second detection result represents the classification result of the target substance attached to the image acquisition device by the machine learning model through the fusion of feature information. The classification result represents the type of target substance, such as "no pollution," "dust," "water droplets," "soil," and "smog." By classifying the target substance, the second detection result provides guidance for subsequent cleaning strategies. For example, if the target substance is a stain, the second detection result can be the stain type, which can be a discrete classification result, such as {0: no pollution, 1: dust, 2: water droplets, 3: mud}.

[0120] Optionally, the generated fusion feature information is converted into an input vector. The input vector is then fed into a pre-trained support vector machine (SVM) model using a fusion decision unit. After nonlinear transformations within the SVM model, one or more probability values ​​are output, forming the first detection result. The SVM model is then used to analyze the fusion feature information to classify the type of target material attached to the image acquisition device, generating a second detection result.

[0121] In this application embodiment, based on fused feature information, the probability and type of the target substance are accurately determined, solving the technical problem of the accuracy of target substance detection on image acquisition devices in the prior art. The prior art only performs physical cleaning on the target substance, while the technical solution of this application realizes a dual detection mechanism, which not only assesses the probability of the target substance adhering to the image acquisition device, but also further identifies the type of the target substance, thereby providing accurate guidance for the subsequent processing of the target substance.

[0122] As an optional implementation, in step S108, the state information threshold includes a first state information threshold and a second state information threshold, wherein the second state information threshold is less than the first state information threshold. The method further includes: determining the first state information threshold based on the vehicle's operating state information; controlling the vehicle to perform a cleaning operation on the image acquisition device in response to a first detection result indicating that the state information exceeds the first state information threshold and a second detection result indicating that the type is a first preset type; and after performing the cleaning operation on the image acquisition device, controlling the vehicle to issue a prompt message to the driver in response to the state information exceeding the second state information threshold or the second detection result indicating that the type is a second preset type.

[0123] In this embodiment, the method can also determine a first state information threshold based on the vehicle's operating state information. The operating state information represents the vehicle's operating state during the process of the image acquisition device acquiring environmental images.

[0124] Optionally, the operational status information can indicate the specific operating conditions or environmental conditions under which the vehicle is located during the process of the image acquisition device acquiring environmental images. Operational status information includes, but is not limited to, vehicle speed, weather conditions, driving direction, driving mode, vehicle's geographical location, and road type. For example, the aforementioned operational status information could be the current vehicle status.

[0125] Optionally, the first state information threshold can refer to the critical condition that triggers the cleaning operation, used to determine when to start the cleaning process. For example, the first state information threshold can be a dynamic safety threshold.

[0126] Optionally, based on the operational status information, the response control module retrieves dynamic safety thresholds corresponding to the current vehicle status from the database. The database includes threshold settings for different driving environments and conditions, such as "cleaning is initiated when the transmittance drops to 0.85 at a speed of 100 km / h in rainy weather" or "cleaning is triggered when the relative humidity of the image acquisition device's mirror exceeds 80% in urban slow-moving mode." Based on the retrieved database and specific operational status information, the first status information threshold can be dynamically adjusted. For example, when the vehicle is driving in rainy weather, the threshold for the transmittance of the image acquisition device's mirror can be lowered from 0.9 to 0.75 to detect target substances earlier and initiate the cleaning process promptly.

[0127] In this embodiment, after obtaining the first state information threshold, if the first detection result is that the state information exceeds the first state information threshold, and the second detection result is that the type is a first preset type, the vehicle is controlled to perform a cleaning operation on the image acquisition device.

[0128] Optionally, the target substance of the first preset type can refer to a target substance that is relatively easy to clean and can be effectively removed by the image acquisition device, such as fine dust, smog, water droplets, and other cleanable substances. The impact on the clarity of the environmental images acquired by the image acquisition device can be quickly eliminated by the vehicle's built-in cleaning devices (such as windshield wipers and water spray systems) without human intervention.

[0129] Optionally, the system checks whether the first detection result exceeds a first state information threshold dynamically set based on vehicle operating status information. Assume the current threshold is set to a specular transmittance of the image acquisition device below 0.85. If the second detection result is "water droplets," and if the first detection result indicates that the state information is indeed below the first state information threshold, and the second detection result confirms the contamination type as "water droplets," a conditional cleaning operation will be triggered. For example, the vehicle's cleaning equipment, such as the built-in mini wipers and washer system, will be activated to perform a "spray + wipe" cycle.

[0130] In this embodiment, after cleaning the image acquisition device, if the status information exceeds the second status information threshold, or if the second detection result is of the second preset type, the vehicle is controlled to issue a prompt message to the driver.

[0131] Optionally, the second preset type of target substance can refer to target substances that are difficult to clean or effectively remove, such as oil stains, mud, and soil. The impact on the clarity of the environmental images captured by the image acquisition equipment cannot be eliminated by the vehicle's built-in cleaning devices (such as windshield wipers and water spray systems) and requires manual intervention. By setting the second preset type, it is possible to distinguish which target substances require more complex cleaning solutions and which target substances can be cleaned automatically.

[0132] Optionally, the second state information threshold can refer to the critical value at which the target material still affects the image acquisition device after the cleaning operation is completed. For example, even if the overall transmittance of the mirror reaches 0.90, if there are local areas where the transmittance is lower than 0.70 due to oil stains, it may still be considered as exceeding the second state information threshold, because the scattering and absorption properties of oil stains are different from those of water droplets, making them more difficult to remove automatically.

[0133] Optionally, the status information is compared with a second status information threshold. If the detected status information is lower than the second status information threshold, it means that the cleanliness of the image acquisition device's mirror surface does not meet the requirements for safe operation. If the second detection result type is a second preset type, a prompt message will be issued to the driver. The prompt message may include "Camera cleanliness is substandard, manual intervention required," "Oil stains on the mirror surface, affecting autonomous driving function," "Audiovisual takeover request + "Severe degradation warning of forward camera performance," etc., to ensure that the driver is quickly aware of the potential risks.

[0134] In this embodiment, by dynamically adjusting the state information threshold, target substances with different cleaning difficulties can be identified and distinguished. For target substances of the first preset type, cleaning operations are initiated promptly to ensure the clarity of the environmental image. For target substances of the second preset type, a prompt message is issued to the driver to ensure that vehicle driving safety is maintained even when automatic cleaning operations cannot be performed. This solves the technical problem in related technologies of being unable to remove target substances under complex conditions, and achieves the technical effect of removing target substances under complex conditions.

[0135] As an optional implementation, in response to a detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, the environmental image is adjusted to obtain an adjusted environmental image, including: in response to a detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, determining an image degradation model of the environmental image; and adjusting the environmental image based on the image degradation model to obtain the adjusted environmental image.

[0136] In this embodiment, when the detection result indicates that a target substance is attached to the image acquisition device and the state information of the target substance exceeds the state information threshold, the environmental image is adjusted to obtain the adjusted environmental image. If the detection result indicates that a target substance is attached to the image acquisition device and the state information of the target substance exceeds the state information threshold, the image degradation model of the environmental image is determined.

[0137] Alternatively, the image degradation model can be an algorithm for restoring the sharpness of an environmental image, used to describe the specific impact of a target substance on the sharpness of an environmental image.

[0138] Optionally, if the detection results indicate that the image acquisition device is covered with target material, and the state information of the target material exceeds a state information threshold, an image degradation model for the environmental image is determined. For example, if the image acquisition device is covered with dust, the image analysis unit calculates features such as the dark channel and average gradient; simultaneously, an optical transmittance sensor monitors the transmittance of the mirror surface of the image acquisition device. If the calculated average value of the dark channel is much higher than the standard for a clean environmental image, and the transmittance sensor reading shows that the transmittance is lower than a preset threshold (e.g., 95%), an image degradation model for dust pollution can be selected.

[0139] In this embodiment, after obtaining the image degradation model, the environmental image is adjusted based on the image degradation model to obtain the adjusted environmental image.

[0140] Optionally, when the state information of the target substance exceeds the state information threshold, the environmental image features, such as light, color, and contrast, are analyzed through an image degradation model to adjust the environmental image and generate an adjusted environmental image. The above-mentioned environmental image is processed to reduce the clarity problems of the environmental image caused by the target substance, such as blurring, decreased contrast, or color deviation.

[0141] Optionally, the parameters in the image degradation model can be adjusted based on the detected dust state information, such as the specific value of transmittance and the statistical characteristics of the dark channel. For example, if the transmittance is found to be only 90%, the atmospheric light in the image degradation model needs to be re-estimated. The image degradation model can then restore the image details damaged by atmospheric scattering and dust occlusion in the dust-affected environmental image and output the adjusted environmental image.

[0142] In this embodiment, by constructing an image degradation model, effective restoration of environmental images affected by target substances is achieved. This solves the technical problem in the prior art of not being able to effectively process environmental images already affected by target substances. This solution not only improves the imaging quality of image acquisition equipment in complex environments and the safety of vehicle driving, but also achieves the technical effect of accurately processing environmental images already affected by target substances.

[0143] As an optional implementation, in response to a detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, an image degradation model for the environmental image is determined, including: in response to a detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, atmospheric transmission information and material transmission information of the environmental image are determined, and an image degradation model is determined based on the fused transmission information of the atmospheric transmission information and material transmission information, the light value information of the environmental image, and the number of pixels in the environmental image.

[0144] In this embodiment, during the process of determining the image degradation model of the environmental image when the detection result indicates that a target substance is attached to the image acquisition device and the state information of the target substance exceeds the state information threshold, the atmospheric transmission information and material transmission information of the environmental image are determined if the detection result indicates that a target substance is attached to the image acquisition device and the state information of the target substance exceeds the state information threshold. The atmospheric transmission information represents the degree of light attenuation caused by the atmospheric conditions reflected in the environmental image, and the material transmission information represents the degree of light attenuation caused by the target substance.

[0145] Optionally, atmospheric transmission information can refer to the degree of attenuation of light as it travels from distant objects to the image acquisition device in an environmental image. Atmospheric transmission information can be represented as a function or a numerical value, and is related to the clarity of the environment. In a clear environment, atmospheric transmission information is close to 1, indicating minimal light attenuation; however, in dense fog or heavily polluted environments, atmospheric transmission information will significantly decrease, meaning that light transmission is severely hindered. For example, the aforementioned atmospheric transmission information can be atmospheric transmittance, which can be represented by t_atm(x).

[0146] For example, if the atmospheric transmission information is atmospheric transmittance, it can be determined using the following formula:

[0147] t_atm(x)=1-ω J_dark(x) / A

[0148] Where t_atm(x) can be used to represent atmospheric transmittance, ω can be used to represent an adjustable parameter (which can be 0.95), J_dark(x) can be used to represent the dark channel, and A can be used to represent the estimated atmospheric light value.

[0149] Optionally, material transmission information can refer to a dataset of light attenuation caused by the target material. This dataset represents the degree to which the target material attached to the mirror of the image acquisition device affects light transmittance. Material transmission information can be a transmittance map, where the value of each pixel reflects the light transmittance at that point; lower transmittance means more light is absorbed or scattered, corresponding to the presence of a target material on the mirror of the image acquisition device. For example, if the target material is a stain, the above material transmission information can be the stain transmittance, which can be represented by t_dirt(x).

[0150] Optionally, if the detected mirror transmittance is below 80% and the dark channel mean exceeds 0.3, the state information of the target material exceeds a set threshold, meaning the mirror is severely contaminated. If the target material is water, after the detection results indicate severe water droplet adhesion to the mirror, atmospheric transmittance information, i.e., the degree of atmospheric scattering and absorption of light, is estimated based on current weather conditions and dark channel analysis. For example, on rainy days, atmospheric transmittance may be low because water droplets suspended in the air scatter light, making distant objects appear more blurry. The distribution, size, and density of water droplets, as well as their absorption and scattering characteristics for specific frequencies of light, are analyzed to obtain material transmittance information. By observing local transmittance changes in the detection results, particularly the decrease in transmittance in areas covered by water droplets, the light-blocking effect of these water droplets can be estimated. For example, water droplets may cause the transmittance in some local areas to drop to around 50%, while other areas maintain higher transmittance.

[0151] In this embodiment, after acquiring atmospheric transmission information and material transmission information, fused transmission information can be determined based on atmospheric transmission information and material transmission information.

[0152] Optionally, the fused transmission information is used to describe the overall light attenuation and scattering along the entire light propagation path from the scene to the image acquisition device.

[0153] Optionally, atmospheric transmission information and material transmission information can be combined to form a transmittance map or function reflecting the causes of environmental image degradation. Mathematical operations or logical processing are then performed on the atmospheric and material transmission information to generate fused transmission information. For example, multiplication can be used to combine atmospheric and material transmission information, where the transmittance of the atmosphere and matter at each pixel multiplies the final light intensity.

[0154] For example, the fused transmission information can be determined using the following formula:

[0155] t_total(x) = t_atm(x) t_dirt(x)

[0156] Among them, t_total(x) can be used to represent fused transmission information, t_atm(x) can be used to represent atmospheric transmission information, and t_dirt(x) can be used to represent material transmission information.

[0157] In this embodiment, after determining the fused transmission information, an image degradation model can be determined based on the fused transmission information, the light value information of the environmental image, and the number of pixels in the environmental image. The light value information represents the global brightness level of the target material in the environmental image.

[0158] Optionally, the light value information can be used to represent the influence of a target substance on the ambient light in an image. The light value information can consist of an average brightness value, a histogram of brightness distribution, or statistical parameters of brightness variation. For example, the aforementioned light value information can be an estimated atmospheric light value, which can be represented by A.

[0159] Optionally, an image degradation model is determined based on the fused transmission information, the light value information of the environmental image, and the number of pixels in the environmental image.

[0160] For example, an image degradation model can be determined using the following formula:

[0161] I(x)=J(x) t_total(x)+A (1-t_total(x))+N

[0162] Where A can be used to represent the estimated atmospheric light value, t_total(x) can be used to represent the fused transmission information, I(x) can be used to represent the original degraded image, J(x) can be used to represent the restored clear image, and N can be used to represent the total number of pixels.

[0163] As an optional implementation, the environmental image is adjusted based on an image degradation model to obtain an adjusted environmental image, including: using guided filtering to adjust the fused transmission information based on the environmental image to obtain adjusted fused transmission information, wherein the matching degree between the adjusted fused transmission information and the environmental image is greater than the matching degree between the fused transmission information and the environmental image before adjustment; and using the image degradation model corresponding to the adjusted fused transmission information to adjust the environmental image to obtain the adjusted environmental image.

[0164] In this embodiment, during the process of adjusting the environmental image based on the image degradation model to obtain the adjusted environmental image, guided filtering can be used to adjust the fused transmission information based on the environmental image to obtain the adjusted fused transmission information. The matching degree between the adjusted fused transmission information and the environmental image is greater than the matching degree between the fused transmission information and the environmental image before adjustment.

[0165] Alternatively, guided filtering is an image processing technique that can be used to manipulate environmental images to improve their clarity.

[0166] Optionally, guided filtering is used to adjust and optimize the pixel values ​​in the fused transmission information according to the content characteristics of the environmental image, ensuring that the edges of the adjusted fused transmission information are consistent with the edges of the environmental image content, and reducing local non-uniformity caused by atmospheric scattering and specular target material.

[0167] In this embodiment, after obtaining the adjusted fused transmission information, the environmental image can be adjusted using the image degradation model corresponding to the adjusted fused transmission information to obtain the adjusted environmental image.

[0168] Optionally, the adjusted fused transmission information is input into the corresponding image degradation model, and the environmental image is adjusted using the image degradation model to obtain the adjusted environmental image.

[0169] For example, the adjusted environmental image can be determined using the following formula:

[0170] J(x)=(I(x)-A) / max(t_total(x),t0)+A

[0171] Where J(x) can be used to represent the restored clear image, I(x) can be used to represent the original degraded image, A can be used to represent the estimated atmospheric light value, t_total(x) can be used to represent the fused transmission information, and t0 is a lower limit threshold (such as 0.1) to prevent the denominator from being too small and causing noise amplification.

[0172] In this embodiment, guided filtering technology is introduced to adjust the fused transmission information, and combined with an image degradation model, effective restoration of environmental images affected by target substances is achieved. Unlike existing technologies that rely solely on physical cleaning, this solves the technical problem of improving the accuracy of environmental image processing affected by target substances, and achieves the technical effect of improving the accuracy of environmental image processing affected by target substances.

[0173] As an optional implementation, the method further includes: identifying environmental objects in the environment from the adjusted environmental image.

[0174] In this embodiment, the environmental object is used to characterize the degree of safety that affects the safety of the vehicle's driving process.

[0175] Optionally, environmental objects can refer to entities such as obstacles, pedestrians, vehicles, traffic signs, traffic lights, road conditions, and lane markings that appear in the road environment where the vehicle is traveling. Environmental objects serve as the basis for the vehicle to perform operations such as path planning, obstacle avoidance decisions, and speed control. The safety level of environmental objects is related to whether the vehicle can travel safely.

[0176] Optionally, from the adjusted environmental image, identify entities in the environment that appear in the road environment where the vehicle is traveling, such as obstacles, pedestrians, vehicles, traffic signs, traffic lights, road conditions, and traffic lines.

[0177] In this embodiment of the application, by identifying environmental objects from the adjusted environmental image, the problem of reduced vehicle environmental perception caused by low clarity of the environmental image under the influence of target substances is solved, while resource utilization efficiency is optimized and vehicle driving safety is enhanced.

[0178] The methods of the embodiments of this application will be further illustrated below.

[0179] In the field of vehicle image processing, the clarity of environmental images is the cornerstone of ensuring driving safety. However, the presence of target substances adhering to the mirror surface of image acquisition equipment can severely reduce the clarity of environmental images. Traditional image acquisition equipment cleaning mechanisms mostly rely on physical removal of target substances, which presents technical problems in processing environmental images affected by target substances and in terms of the accuracy of processing environmental images with target substances adhering to them.

[0180] To address the aforementioned issues, this application proposes a multimodal intelligent stain detection and image restoration technology. Environmental images are acquired in real-time using an image acquisition device, and then features of these images are extracted as first feature information. Simultaneously, second feature information is acquired using physical sensors integrated around the image acquisition device. By inputting these two types of feature information into a deep learning model, the presence and state information of target substances are identified, thereby determining whether an automatic cleaning function needs to be activated. When the state information of a target substance exceeds a preset threshold, indicating that the target substance has severely affected the clarity of the environmental image, an image restoration algorithm is activated to specifically compensate for the impact of different types of target substances on the clarity of the environmental image, ensuring that the clarity of the output adjusted environmental image is higher than the original environmental image.

[0181] Figure 2 is a schematic diagram of a multimodal stain detection and image restoration system according to an embodiment of this application. As shown in Figure 2, the image restoration system may include an in-vehicle camera 201, a multimodal stain detection module 202, and a decision core 208. The multimodal stain detection module 202 may include an image analysis unit 203, a physical sensor unit 204, a fusion decision unit (ML model) 205, an adaptive image restoration module 206, and a vehicle cooperative response control module 207. The advantage of this system is that it has modular functionality, can be attached to the original image processing model architecture, and only intervenes in image processing, which has strong model architecture compatibility and shortens the development cycle.

[0182] Optionally, as shown in Figure 2, an onboard camera is used to acquire environmental images and transmit them to other components in the multimodal stain detection and image restoration system for processing and analysis. The onboard camera 201 is mounted at the front of the vehicle, such as the upper part of the windshield, to capture visual feature information in front of the vehicle. The multimodal stain detection module 202 is used to identify and quantify target substances on the image acquisition device. The decision core 208 is responsible for summarizing and analyzing information from the image analysis unit 203, the physical sensor unit 204, and the fusion decision unit ML model 205, thereby directing the adaptive image restoration module 206 and the vehicle cooperative response control module 207 to take appropriate actions. Its design follows a modular principle, allowing seamless integration into existing vehicle image processing architectures. Only specific functional modules need to be added or adjusted without major modifications to the entire system, significantly simplifying the development process and time.

[0183] Optionally, the image analysis unit 203 analyzes the image data captured by the vehicle-mounted camera 201 to identify any signs of degradation in the image, especially those caused by the target substance. The image analysis unit 203 employs computer vision technology and image processing algorithms, such as edge detection, dark channel prior, and color analysis, to perform in-depth analysis of the environmental image, quantifying the clarity of the environmental image and accurately locating the position, type, and severity of the target substance. The physical sensor unit 204 directly monitors the cleanliness of the vehicle-mounted camera mirror at a physical level. By integrating sensor technologies, such as optical transmittance sensors and capacitive droplet sensors, it captures and evaluates the physical conditions of the mirror in real time, including but not limited to the mirror's transmittance and the presence of water films or droplets. The fusion decision unit ML model 205 plays a crucial role in decision fusion in the multimodal stain detection and image restoration system. It serves as a bridge connecting the image analysis unit 203 and the physical sensor unit 204, and also acts as the intelligent engine guiding the actions of the adaptive image restoration module 206 and the vehicle cooperative response control module 207. Model 205 makes full use of support vector machines to comprehensively analyze environmental image features and sensor data to obtain detection results, including the type of target substance, severity, and potential environmental factors.

[0184] Optionally, the adaptive image restoration module 206 is a component of the multimodal stain detection and image restoration system, used to restore image degradation caused by mirror-like target substances and environmental factors, ensuring that the vehicle's visual perception system can acquire clear and reliable images under any circumstances. Module 206, based on advanced image processing theory and technology, dynamically adjusts algorithm parameters by combining target substance type and severity information obtained from the fusion decision unit ML model 205 to achieve image degradation restoration. The vehicle cooperative response control module 207, a component of the multimodal target substance detection and image restoration system, formulates and executes corresponding vehicle response strategies based on the output of the fusion decision unit ML model 205, i.e., the target substance detection result and type, and the vehicle's real-time operating status. Module 207 not only coordinates the activation of the adaptive image restoration module 206 but also links with the decision system to ensure that the vehicle can take safe and reasonable actions when target substances appear on the camera mirror, avoiding potential driving risks.

[0185] Figure 3 is a flowchart of a multimodal stain detection module according to an embodiment of this application. As shown in Figure 3, the method may include the following steps.

[0186] Step S301: Use the image analysis unit to extract feature average gradient, dark channel, etc.

[0187] In this embodiment, the image analysis unit performs depth analysis on the images captured by the vehicle-mounted camera, extracting key image features such as the average gradient and dark channel. The average gradient is calculated using the formula G_avg=(1 / N). Σ| I(x) calculates the average gradient, where G_avg can be used to represent the average gradient, and I(x) can be used to represent the image grayscale value. The gradient operator can be used, and N can represent the total number of pixels. The average gradient reflects the sharpness of edges or details in the environmental image. The dark channel is calculated using the formula J_dark(x)=min_{c∈{r,g,b}}(min_{y∈Ω(x)}(J_c(y))), where J_dark(x) represents the dark channel, J_c represents the RGB color channel, Ω(x) represents the local block centered at x, and J_c(y) represents the brightness value of the color channel at pixel y. The mean and variance of the dark channel of the entire environmental image are calculated and combined with the average gradient to form a feature vector. The above feature vector can be represented by V_image=[G_avg,J_dark_mean,J_dark_var,...], where G_avg represents the average gradient, J_dark_mean represents the dark channel mean, and J_dark_var represents the dark channel variance. The dark channel is used to detect the presence of fog or haze by analyzing the distribution of the darkest pixels in an environmental image to determine the degree of scattering in the atmosphere.

[0188] Step S302: Use the physical sensor unit to obtain readings such as transmittance and capacitance.

[0189] In this embodiment, the physical sensor unit obtains information about the cleanliness of the camera mirror by directly measuring physical parameters such as optical transmittance and capacitance. These physical parameters include transmission information and humidity information. The optical transmittance sensor can read the light transmission capability of the mirror, while the capacitive droplet sensor can detect the presence of a water film, providing a different perspective from the features extracted by the image analysis unit—that is, judging whether the mirror is contaminated from a physical level. The physical parameters are converted into an input vector, which can be represented by [V_image, T_measured, C_value]. Here, V_image can be used to represent the visual features of the environmental image, T_measured can be used to represent the transmission information, and C_value can be used to represent the humidity information.

[0190] Step S303: Input the features and readings into the fusion decision unit machine learning model.

[0191] In this embodiment, the feature vector extracted in step S301 and the input vector obtained in step S302 are input together into the fusion decision unit (ML) model. The machine learning model, trained on a large amount of data beforehand, can comprehensively analyze multimodal information, identify the correlation between features and readings, and the relationship with the state of mirror stains. Information fusion technology improves the accuracy of stain detection and can effectively distinguish between atmospheric scattering effects and genuine mirror contamination.

[0192] Step S304: Output the stain confidence level and type through the model, such as: water droplets, confidence level 90%.

[0193] In this embodiment, after receiving image features and physical readings, the fusion decision unit (ML) model performs calculations and analysis using internal algorithms to output a confidence level regarding the presence of stains on the mirror surface and determines the type of stain. For example, "water droplets, 90% confidence" indicates that the model determines with a 90% probability that water droplets are present on the camera mirror surface. The output not only provides information on the presence or absence of stains but also classifies the type of target substance, providing guidance for subsequent image restoration and vehicle response.

[0194] Figure 4 is a flowchart of an adaptive image restoration algorithm according to an embodiment of this application.

[0195] Step S401: Estimate the initial stain transmission map based on the multimodal detection results.

[0196] In this embodiment, the stain confidence level and target substance type information obtained by the multimodal stain detection module, as well as the transmittance and droplet sensor readings, are used as inputs to the recovery algorithm. This provides a basis for parameter selection and image recovery strategies in subsequent steps, ensuring that the algorithm can process specific stain types and severity.

[0197] Step S402: Combine the dark channel prior to estimate the atmospheric transmission map and atmospheric light value.

[0198] In this embodiment, by analyzing the dark channel information in the environmental image, the transmittance map and atmospheric light value A of atmospheric fog or haze on the environmental image are estimated. The dark channel prior theory is based on the observation that in environmental images of fog-free outdoor scenes, most pixels not containing the sky have very low brightness values ​​in at least one color channel. This allows for a preliminary estimate of the impact of atmospheric fog or haze on the environmental image, laying the foundation for subsequent environmental image restoration. By utilizing the dark channel, the atmospheric transmittance is initially estimated using t_atm(x) = 1 - ω. The calculation of J_dark(x) / A uses t_atm(x) to represent atmospheric transmittance, ω to represent an adjustable parameter (usually 0.95), J_dark(x) to represent the dark channel, and A to represent the estimated atmospheric light value.

[0199] Step S403: Merge T_dirt(x) and T_atm(x).

[0200] In this embodiment, according to the fusion formula t_total(x)=t_atm(x) t_dirt(x), where t_total(x) can be used to represent fused transmission information, t_atm(x) can be used to represent atmospheric transmission information, and t_dirt(x) can be used to represent material transmission information.

[0201] The local transmittance attenuation map T_dirt(x) caused by stains is combined with the atmospheric transmittance map T_atm(x) to construct a comprehensive transmittance map t_total(x). T_dirt(x) is dynamically generated by the fusion decision unit ML model based on stain type and confidence level, while T_atm(x) is obtained based on dark channel prior estimation in step S402. The image degradation model formula I(x) = J(x) is then used. t_total(x)+A (1-t_total(x))+N. Where A can be used to represent the estimated atmospheric light value, t_total(x) can be used to represent the fused transmission information, I(x) can be used to represent the original degraded image, J(x) can be used to represent the restored sharp image, and N can be used to represent the total number of pixels. This yields the image degradation model.

[0202] Step S404: Calculate according to the restoration formula.

[0203] In this embodiment, the restoration formula J(x) = (I(x) - A) / max(t_total(x), t0) + A is used, where J(x) represents the restored clear image, I(x) represents the original degraded image, A represents the estimated atmospheric light value, t_total(x) represents the fused transmission information, and t0 is a lower limit threshold (e.g., 0.1) to prevent noise amplification due to an excessively small denominator. Using this formula, the restored clear image J(x) is sent to the autonomous driving decision-making system for perception tasks such as target detection and lane line recognition.

[0204] Figure 5 is a flowchart of a vehicle cooperative response control logic according to an embodiment of this application.

[0205] Step S501: Read the stain confidence level C.

[0206] In this embodiment, the stain confidence C output by the fusion decision unit ML model is a value that quantifies the probability that there may be stains on the camera lens.

[0207] Step S502: Determine whether C is greater than the dynamic threshold TH.

[0208] In this embodiment, the stain confidence level C is compared with a pre-set dynamic threshold TH. The value of TH can be dynamically adjusted according to the vehicle's environment and driving mode to adapt to different scenarios. If C is greater than TH, it indicates that the stain on the camera lens may have a sufficient impact on visual perception, and measures need to be taken. If C > dynamic threshold TH, step S503 can be executed. Conversely, if C is less than or equal to dynamic threshold TH, step S501 can be executed.

[0209] Step S503: Determine the stain type selection and respond to the branch.

[0210] In this embodiment, if the confidence level of the target substance exceeds a threshold, the next step is to determine the type of the target substance. Based on the type of target substance, the system will move to a specific response branch, as different stain types require different cleaning or restoration strategies. Step S504: Activate the cleaning system.

[0211] In this embodiment, once cleaning is confirmed to be necessary, a cleaning system, such as a miniature wiper and a spray nozzle, will be activated. Depending on the type of target substance, the cleaning system may have different activation conditions or cleaning intensities to ensure effective removal of the target substance.

[0212] Step S505: After cleaning, perform another test.

[0213] In this embodiment, after cleaning is completed, the response process does not end immediately. Instead, multimodal stain detection is performed again to check the cleanliness of the image acquisition device mirror and ensure that the target substance has been effectively removed.

[0214] Step S506: Determine whether C is still greater than the threshold.

[0215] In this embodiment, if the second detection shows that the stain confidence level C is still higher than a set threshold, it means that the first cleaning may not have completely solved the problem, or the target substance type is difficult to clean. If C is still greater than the threshold, step S503 can be executed. Conversely, if C is less than or equal to the threshold, step S501 can be executed.

[0216] Step S507: Issue a performance degradation warning to the decision system.

[0217] In this embodiment, a warning will be issued to the decision-making system, indicating that the performance of the image acquisition device has been severely degraded due to the target substance, which may affect the normal perception and operation of the vehicle.

[0218] Step S508: Determine whether it is an L3 system.

[0219] In this embodiment, the level of the current multimodal dirt detection and image restoration system is determined. If the vehicle is equipped with Level 3, then the scenario of driver takeover needs to be considered. If it is an Level 3 system, step S509 can be executed. Otherwise, if it is not an Level 3 system, step S510 can be executed.

[0220] Step S509 prompts the driver to take over.

[0221] In this embodiment, for Level 3 vehicles, once the performance of the image acquisition equipment degrades to a level that may endanger driving safety, the multimodal dirt detection and image restoration system will send a takeover request to the driver, prompting him to prepare to regain control of the vehicle.

[0222] Step S510: Instruct the vehicle to implement a conservative strategy to reduce speed and restrict functions.

[0223] In this embodiment, regardless of the level of the multimodal stain detection and image restoration system, when the image acquisition device is severely affected by the target substance, the vehicle cooperative response control module will instruct the vehicle to execute a series of conservative strategies, such as reducing driving speed, restricting automatic lane changing or overtaking, to ensure driving safety until the camera is cleaned or the driver takes over control.

[0224] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0225] According to another aspect of the embodiments of this application, corresponding to the embodiments of the above-described vehicle image processing method, this specification also provides a vehicle image processing apparatus.

[0226] Figure 6 is a schematic diagram of a vehicle image processing device according to an embodiment of the present application. As shown in Figure 6, the vehicle image processing device 600 may include: a first acquisition module 601, a second acquisition module 604, a detection module 606, and an adjustment module 608.

[0227] The first acquisition module 602 is used to acquire environmental images collected by the image acquisition device, wherein the image content of the environmental images is used to represent the environment in which the vehicle is located.

[0228] The second acquisition module 604 is used to acquire first feature information of the environmental image and second feature information of the image acquisition device, wherein the first feature information is used to represent the clarity of the environmental image and the second feature information is used to represent the degree of influence of the image acquisition device on the clarity.

[0229] The detection module 606 is used to detect the image acquisition device based on the first feature information and the second feature information, and obtain the detection result.

[0230] The adjustment module 608 is used to adjust the environmental image in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds the state information threshold, so as to obtain an adjusted environmental image. The state information is used to indicate the degree of influence of the target substance on the clarity. The clarity of the adjusted environmental image is greater than the clarity of the environmental image before adjustment.

[0231] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application when it runs.

[0232] Embodiments of this application also provide an electronic device, including a memory storing an executable program; and a processor for running the program, wherein the program executes the methods of various embodiments of this application during runtime.

[0233] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0234] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0235] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0236] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of this application.

[0237] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0239] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0241] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0242] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for image processing of a vehicle, characterized in that, The vehicle includes an image acquisition device, and the method includes: acquiring an environmental image acquired by the image acquisition device, wherein the image content of the environmental image is used to represent the environment in which the vehicle is located; acquiring first feature information of the environmental image and second feature information of the image acquisition device, wherein the first feature information is used to represent the clarity of the environmental image, and the second feature information is used to represent the degree of influence of the image acquisition device on the clarity; detecting the image acquisition device based on the first feature information and the second feature information to obtain a detection result; in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, adjusting the environmental image to obtain an adjusted environmental image, wherein the state information is used to represent the degree of influence of the target substance on the clarity, and the clarity of the adjusted environmental image is greater than the clarity of the environmental image before adjustment.

2. The method according to claim 1, characterized in that, Acquiring first feature information of the environmental image includes: analyzing the environmental image using the image analysis unit of the vehicle to obtain the first feature information; acquiring second feature information of the image acquisition device includes: acquiring the second feature information using the physical sensor unit of the vehicle.

3. The method according to claim 2, characterized in that, Using the vehicle's image analysis unit, the environmental image is analyzed to obtain the first feature information, including: using the image analysis unit to obtain the grayscale value and the number of pixels in the environmental image, and determining the gradient information of the environmental image based on the grayscale value and the number of pixels, wherein the gradient information is used to represent the richness of image content in the environmental image and / or the clarity of image edges in the environmental image; using the image analysis unit to obtain the color channels of the environmental image and the region information of the environmental image, wherein the region information is used to represent the brightness difference between different pixels in a preset local region of the environmental image, and based on... The color channel and the region information are used to determine the channel information of the environmental image, wherein the channel information is used to represent the severity of the environmental image being affected by the target substance; the first feature information is determined based on the gradient information and the channel information; or, the second feature information is obtained using the vehicle's physical sensor unit, including: obtaining the transmission information of the image acquisition device and the humidity information of the image acquisition device using the physical sensor unit, wherein the transmission information is used to characterize the cleanliness of the image acquisition device and the humidity information is used to represent the moisture level of the image acquisition device; the second feature information is determined based on the humidity information and the transmission information.

4. The method according to claim 1, characterized in that, Based on the first feature information and the second feature information, the image acquisition device is detected to obtain a detection result, including: using a machine learning model to fuse the first feature information and the second feature information to obtain fused feature information; using the fused feature information to detect the image acquisition device to obtain a detection result.

5. The method according to claim 4, characterized in that, The detection results include a first detection result and a second detection result. The image acquisition device is detected using fused feature information to obtain the detection results, including: using a support vector machine model to determine the first detection result and the second detection result based on the fused feature information, wherein the first detection result is used to represent the probability that the target substance is attached to the image acquisition device, and the second detection result is used to represent the type of the target substance.

6. The method according to claim 5, characterized in that, The state information threshold includes a first state information threshold and a second state information threshold, wherein the second state information threshold is less than the first state information threshold. The method further includes: determining the first state information threshold based on the vehicle's operating state information, wherein the operating state information is used to represent the vehicle's operating state during the process of the image acquisition device acquiring the environmental image; responding to the first detection result that the state information exceeds the first state information threshold and the second detection result that the type is a first preset type, controlling the vehicle to perform a cleaning operation on the image acquisition device, wherein the cleaning difficulty of the target substance of the first preset type is less than a cleaning difficulty threshold; after performing the cleaning operation on the image acquisition device, responding to the state information exceeding the second state information threshold, or the second detection result that the type is a second preset type, controlling the vehicle to issue a prompt message to the driver, wherein the cleaning difficulty of the target substance of the second preset type is greater than or equal to the cleaning difficulty threshold, and the prompt message is used to instruct the driver and passengers in the vehicle to take over the perception of the environment from the image acquisition device.

7. The method according to any one of claims 1 to 6, characterized in that, In response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, the environmental image is adjusted to obtain an adjusted environmental image, including: in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, determining an image degradation model of the environmental image; and adjusting the environmental image based on the image degradation model to obtain the adjusted environmental image.

8. The method according to claim 7, characterized in that, In response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, an image degradation model for the environmental image is determined, including: in response to the detection result indicating that a target substance is attached to the image acquisition device and the state information of the target substance exceeds a state information threshold, determining atmospheric transmission information and material transmission information of the environmental image, wherein the atmospheric transmission information is used to represent the degree of light attenuation caused by atmospheric conditions reflected in the environmental image, and the material transmission information is used to represent the degree of light attenuation caused by the target substance; and determining the image degradation model based on the fused transmission information of the atmospheric transmission information and the material transmission information, the light value information of the environmental image, and the number of pixels in the environmental image, wherein the light value information is used to represent the global brightness level of the target substance in the environmental image.

9. The method according to claim 8, characterized in that, The method adjusts the environmental image based on the image degradation model to obtain the adjusted environmental image, including: using guided filtering to adjust the fused transmission information based on the environmental image to obtain the adjusted fused transmission information, wherein the matching degree between the adjusted fused transmission information and the environmental image is greater than the matching degree between the fused transmission information and the environmental image before adjustment; using the image degradation model corresponding to the adjusted fused transmission information to adjust the environmental image to obtain the adjusted environmental image; or, the method further includes: identifying environmental objects in the environment from the adjusted environmental image, wherein the environmental objects are used to characterize the safety level affecting the safety of the vehicle driving process.

10. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 9.

Images