A Deep Learning-Based Multi-Camera Vehicle Washing Detection Method

Abstract

This invention discloses a deep learning-based multi-camera vehicle washing detection method, belonging to the field of intelligent control technology. The method includes acquiring multi-view RGB images and multi-view point cloud images; fusing the multi-view point cloud images using an iterative nearest-point algorithm to generate a 3D point cloud image; extracting features from the 3D point cloud image using PointNet++ and combining it with a ViT visual Transformer network to identify the distribution of dirt on the vehicle surface; detecting the type and degree of dirt adhesion in different areas of the vehicle surface, calculating a cleaning difficulty score, prioritizing the dirt in different areas of the vehicle surface, and generating a list of washing areas; and constructing a DQN+PPO reinforcement learning model to dynamically adjust the nozzle angle, water pressure, and washing time. This invention utilizes a reinforcement learning model to dynamically adjust the nozzle angle, water pressure, and washing time, and uses a millimeter-wave sensor to detect dirt removal, thus completing the vehicle cleanliness score.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to a multi-camera vehicle washing detection method based on deep learning. Background Technology

[0002] Traditional vehicle washing and inspection methods mainly rely on a single vision sensor or simple infrared detection equipment, which makes it difficult to accurately identify the distribution of stains in complex environments. Most current automatic washing devices still use a fixed nozzle control mode, which lacks an adaptive adjustment mechanism, resulting in serious waste of washing water resources and poor cleaning effect on complex stains.

[0003] Existing 3D point cloud fusion methods mostly use direct stitching or simple feature matching, which leads to large alignment errors in point cloud data and affects the accurate positioning of stained areas. Most of them use fixed nozzle control strategies with preset parameters, which cannot adaptively optimize rinsing for different types and degrees of stains, resulting in water waste and uneven cleaning effects. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a deep learning-based multi-camera vehicle washing detection method to solve the problems of inaccurate stain positioning caused by point cloud alignment errors and the inability to adaptively optimize washing.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a multi-camera vehicle washing detection method based on deep learning, which includes acquiring multi-view RGB images and multi-view point cloud images, and using an iterative nearest point algorithm to fuse the multi-view point cloud images to generate a three-dimensional point cloud image.

[0008] PointNet++ is used to extract features from 3D point cloud images, and ViT vision Transformer network is combined to identify the distribution of dirt on the vehicle body surface;

[0009] The system detects the type and degree of stain adhesion in different areas of the vehicle body surface, calculates the cleaning difficulty score, prioritizes the stains in different areas of the vehicle body surface, and generates a list of washing areas.

[0010] A DQN+PPO reinforcement learning model is constructed to dynamically adjust the nozzle angle, water pressure, and rinsing time. A millimeter-wave sensor is used to detect the removal of dirt and to score the cleanliness of the vehicle.

[0011] As a preferred embodiment of the deep learning-based multi-camera vehicle washing detection method of the present invention, the acquisition of multi-view RGB images and multi-view point cloud images includes the following steps.

[0012] Multi-view RGB images are acquired by synchronously shooting with an industrial camera, and time synchronization is performed using PTP to align the frames of the multi-view RGB images.

[0013] Multi-view point cloud images are acquired using a LiDAR sensor, and stability correction is performed on the multi-view point cloud images using an inertial measurement unit and LiDAR-SLAM.

[0014] As a preferred embodiment of the deep learning-based multi-camera vehicle washing detection method of the present invention, the method includes the following steps: fusing point cloud images from multiple perspectives using an iterative nearest-point algorithm to generate a 3D point cloud image.

[0015] A rotation matrix and a translation matrix are used to unify the multi-view point cloud images into a coordinate system and align the multi-view point cloud images.

[0016] The iterative nearest-point algorithm is used to fuse the aligned multi-view point cloud images to obtain a 3D point cloud image.

[0017] As a preferred embodiment of the deep learning-based multi-camera vehicle washing detection method of the present invention, the method includes the following steps: extracting 3D point cloud image features using PointNet++ and combining it with a ViT visual Transformer network to identify the distribution of dirt on the vehicle surface.

[0018] PointNet++ is used to extract features from 3D point cloud images, obtaining local and global point cloud image features, and calculating the rate of change of point cloud density.

[0019] Based on the rate of change of point cloud density, ViT vision is used to identify changes in vehicle body color and mark areas of dirt on the vehicle body.

[0020] By marking the areas of dirt on the vehicle body, a Transformer network is used to identify the distribution of dirt on the vehicle body surface.

[0021] As a preferred embodiment of the deep learning-based multi-camera vehicle washing detection method of the present invention, the steps of detecting the type and degree of stain adhesion in different areas of the vehicle body surface and calculating the cleaning difficulty score include the following steps.

[0022] The mean hue of the stain distribution is extracted from the RGB image, and the color variance is calculated.

[0023] The stain colors are classified based on color variance to obtain the stain types in different areas of the vehicle body surface;

[0024] The point cloud density of the stained area was calculated using a KD-Tree.

[0025] The roughness of the stained area is calculated using the point cloud normal vector change calculation method for the stained area.

[0026] The degree of stain adhesion is obtained based on the point cloud density and roughness of the stain area;

[0027] Based on the type and degree of stain adhesion in different areas of the vehicle body surface, a weighted method is used to calculate the cleaning difficulty score.

[0028] As a preferred embodiment of the deep learning-based multi-camera vehicle washing detection method of the present invention, the method includes the following steps: prioritizing the dirt on the vehicle surface in different areas to generate a list of washing areas.

[0029] The weight of the vehicle cleaning area is set according to the roughness of the vehicle stain area, and combined with the cleaning difficulty score to obtain the rinsing priority score;

[0030] Based on the rinsing priority score, a rinsing priority threshold is set, and all stained areas are sorted from high to low.

[0031] When the flushing priority score is greater than the flushing priority threshold, the area is considered to be high-cleaning-difficulty.

[0032] When the flushing priority score equals the flushing priority threshold, the cleaning difficulty zone is defined as follows:

[0033] When the flushing priority score is less than the flushing priority threshold, the area is considered a low-cleaning difficulty zone.

[0034] Organize the stained areas and prioritize them to generate a list of rinsing areas.

[0035] As a preferred embodiment of the deep learning-based multi-camera vehicle washing detection method of the present invention, the following steps are included in constructing a DQN+PPO reinforcement learning model and dynamically adjusting the nozzle angle, water pressure, and washing time:

[0036] A PPO reinforcement learning model is constructed based on the list of washing areas, a state space is defined, and a preliminary vehicle washing plan is generated.

[0037] The initial vehicle washing plan was optimized using DQN and updated in stages using PPO, dynamically adjusting nozzle angle, water pressure and washing time.

[0038] As a preferred embodiment of the deep learning-based multi-camera vehicle washing detection method of the present invention, the method includes the following steps: detecting stain removal using a millimeter-wave sensor and scoring the cleanliness of the vehicle.

[0039] Millimeter-wave sensors are used to scan the vehicle surface to acquire point cloud data of dirt residue.

[0040] Based on the point cloud data of stain residue, set the stain data before rinsing and the data after rinsing, and calculate the stain removal rate;

[0041] Based on the stain removal rate, the cleanliness score of the vehicle's stained area before and after rinsing is calculated using IoU.

[0042] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the deep learning-based multi-camera vehicle washing detection method as described in the first aspect of the present invention.

[0043] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the deep learning-based multi-camera vehicle washing detection method described in the first aspect of the present invention.

[0044] The beneficial effects of this invention are as follows: By acquiring and fusing multi-view RGB images and point cloud images, combined with deep learning and reinforcement learning technologies, intelligent detection and efficient cleaning of vehicle surface stains are achieved. Industrial cameras and LiDAR sensors are used to acquire multi-view images, and a high-precision 3D point cloud model is generated through an iterative nearest-point algorithm, providing an accurate data foundation for subsequent detection. PointNet++ is used to extract point cloud features, and the ViT visual Transformer network is combined to identify stain distribution, achieving precise localization of stain areas. Based on RGB images and point cloud data, stain type, adhesion degree, and cleaning difficulty score are calculated through color variance, point cloud density, and roughness, and a priority list of rinsing areas is generated. The DQN+PPO reinforcement learning model is used to dynamically adjust nozzle angle, water pressure, and rinsing time, and a millimeter-wave sensor is used to detect stain removal, completing the vehicle cleanliness score. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of the deep learning-based multi-camera vehicle washing detection method in Example 1.

[0047] Figure 2 This is a flowchart of reinforcement learning control and cleaning in Example 1.

[0048] Figure 3 This is a flowchart of the multi-view data acquisition and fusion process in Example 1.

[0049] Figure 4 This is a flowchart of the stain identification and cleaning priority ranking process in Example 1. Detailed Implementation

[0050] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0051] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0053] Example 1, referring to Figures 1-4 This is the first embodiment of the present invention, which provides a multi-camera vehicle washing detection method based on deep learning, including the following steps:

[0054] S1. Acquire multi-view RGB images and multi-view point cloud images.

[0055] S1.1. Multi-view RGB images are acquired by synchronous shooting with an industrial camera, and time synchronization is performed using PTP to align the frame numbers of the multi-view RGB images.

[0056] Furthermore, after the vehicle enters the inspection area, multiple industrial cameras (such as high frame rate CMOS sensors) are used to simultaneously capture RGB images from different angles to ensure full coverage of the vehicle surface. To prevent image misalignment caused by frame rate errors or time delays between cameras, the PTP time synchronization protocol is adopted, enabling all cameras to complete image acquisition within nanosecond-level time accuracy. PTP ensures that the time base of different cameras is consistent through the clock synchronization mechanism of the master clock and slave clock, thereby aligning the frame numbers of the acquired multi-view RGB images and avoiding deviations in vehicle surface images caused by time asynchrony.

[0057] S1.2 Acquire multi-view point cloud images using a LiDAR sensor, and perform stability correction on the multi-view point cloud images using an inertial measurement unit and LiDAR-SLAM.

[0058] Furthermore, LiDAR (Light Detection and Ranging) sensors are used to collect multi-view point cloud data from multiple angles to construct a complete 3D point cloud model. Due to slight vehicle shaking and environmental vibration, the point cloud data may drift or become distorted. In conjunction with an inertial measurement unit (IMU), acceleration and angular velocity data are acquired in real time to compensate for errors caused by vehicle motion. The LiDAR-SLAM (LiDAR Simultaneous Localization and Mapping) algorithm is used to register and perform stability correction on the multi-view point cloud data. Feature matching and loop closure detection between consecutive frames are used to optimize the point cloud data, aligning the multi-view point cloud data to the global coordinate system, thereby improving the robustness and accuracy of the point cloud.

[0059] S2. The iterative nearest point algorithm is used to fuse point cloud images from multiple perspectives to generate a 3D point cloud image.

[0060] S2.1. Use rotation and translation matrices to unify the multi-view point cloud images into a coordinate system and align the multi-view point cloud images.

[0061] Furthermore, multi-view point cloud images are acquired by LiDAR sensors from the top, bottom, and sides of the vehicle. After acquiring multi-view point cloud images, since the LiDAR sensors are installed at different locations and have different coordinate systems, coordinate unification processing is required. Based on the extrinsic parameter calibration of the LiDAR sensors, the rotation and translation matrices of each LiDAR are calculated, and all point cloud data are transformed into the global coordinate system.

[0062] S2.2. The aligned multi-view point cloud images are fused using the iterative nearest point algorithm to obtain a three-dimensional point cloud image.

[0063] Furthermore, after completing the coordinate alignment, the multi-view point cloud data needs to be fused to generate a complete 3D point cloud image. The Iterative Closest Point (ICP) algorithm is used to perform high-precision registration and fusion of point clouds from different views. By iteratively optimizing the transformation relationship between nearest neighbor pairs, the mean square error between the two sets of point clouds is minimized to obtain the 3D point cloud image.

[0064] S3. Use PointNet++ to extract features from 3D point cloud images and combine it with the ViT Vision Transformer network to identify the distribution of dirt on the vehicle body surface.

[0065] S3.1 Use PointNet++ to extract features from the 3D point cloud image, obtain local point cloud image features and global point cloud image features, and calculate the rate of change of point cloud density.

[0066] Furthermore, after generating a complete 3D point cloud image, PointNet++ (a deep learning point cloud feature extraction network) is used to perform hierarchical feature extraction on the point cloud data to identify the surface structure and stain areas of the vehicle body. PointNet++ uses a multi-scale sampling and local feature aggregation method to first extract local point cloud features (such as curvature, normal vector, and geometric structure), and then extract global point cloud features over a larger area.

[0067] S3.2 uses ViT vision to identify changes in vehicle body color based on the rate of change of point cloud density and marks areas of dirt on the vehicle body.

[0068] Furthermore, based on the 3D point cloud features extracted by PointNet++, the RGB images are further processed using ViT (VisionTransformer) to analyze color changes on the vehicle body surface, identify stain areas, and mark the stain areas on the vehicle body.

[0069] S3.3 identifies the distribution of stains on the vehicle body surface by marking the areas of stains on the vehicle body and using a Transformer network.

[0070] Furthermore, after obtaining the stain-marked areas, the Transformer network is used for deep feature learning to improve the accuracy of stain recognition. This allows the system to capture the correlation between different stain areas on the vehicle surface and reduce the false detection rate.

[0071] S4. Detect the type and degree of stain adhesion in different areas of the vehicle body surface, and calculate the cleaning difficulty score.

[0072] S4.1 Extracts the mean hue of the stain distribution from the RGB image and calculates the color variance, expressed as follows:

[0073]

[0074] Among them, C var Let C be the color variance, N be the total number of pixels in the stained area, and C be the color variance. i Let C be the color value of the i-th pixel. mean is the average color of the stained area, and i is the pixel index;

[0075] Furthermore, to evaluate the color changes in the stained areas of the vehicle body, the stained areas are first extracted from the RGB image, and their mean hue and color variance are calculated. var Let C be the color variance, reflecting the degree of color change, with a value range of [0, 255]. i Let be the color value of the i-th pixel, the mean of the RGB channels, in the range [0, 255].

[0076] S4.2 Classify stain colors based on color variance to obtain stain types in different areas of the vehicle body surface.

[0077] Furthermore, after calculating the color variance, stain types are classified according to the color variance range [500, 2000]. The stain type is determined by the color variance, C. var A value >2000 indicates mud or dust-like contamination, suggesting drastic color changes in the stained area, indicating a deposited stain. A value ≤500 indicates a stain of similar color. var ≤2000 General stains, such as oil stains and leaf residue, C var <500C indicates transparent or low-adhesion stains, such as water stains or light dust. S4.3, Calculate the point cloud density of the stained area using a KD-Tree, the expression is:

[0078]

[0079] Where D is the point cloud density of the stained area, U is the area with thick stained adhesion, and V is the area with shallow stained adhesion.

[0080] Furthermore, in order to evaluate the thickness distribution of stains, a KD-Tree is used for spatial indexing to accelerate the calculation of point cloud density in the stain area. The nearest neighbor point is retrieved in the stain area by KD-Tree to divide the high-density attachment area and the low-density attachment area. D is the point cloud density of the stain area, with a value range of [0,1], which represents the proportion of stain thickness relative to the standard vehicle body surface.

[0081] S4.4 Calculate the roughness of the stained area using the point cloud normal vector change calculation method. The expression is:

[0082]

[0083] Where R is the roughness of the stained area, n bLet n be the normal vector of the b-th boundary point. b+1 Let b be the normal vector of the (b+1)th boundary point, where b is the index of the boundary point;

[0084] Furthermore, a point cloud normal vector change calculation method is adopted. The roughness of the stained area is calculated by analyzing the degree of change of the normal vector of the stained boundary point. The set of boundary points of the stained area is determined based on KD-Tree nearest neighbor search, and the normal vector of each boundary point is calculated. All boundary points are traversed, and the difference between the normal vectors of adjacent points is calculated. R is the roughness of the stained area, and the value range R∈[0,1] represents the smoothness of the stained surface.

[0085] S4.5. The degree of stain adhesion is obtained based on the point cloud density and roughness of the stain area.

[0086] Furthermore, in order to quantify the difficulty of stain adhesion, the degree of stain adhesion is calculated by combining the point cloud density and roughness of the stain area. The point cloud density reflects the thickness of the stain, while the roughness reflects the adhesion characteristics of the stain surface, which can effectively distinguish the adhesion of different types of stains.

[0087] S4.6. Based on the type and degree of stain adhesion in different areas of the vehicle body surface, a weighted method is used to calculate the cleaning difficulty score, expressed as follows:

[0088] S = αA + βU + γR;

[0089] Where S is the cleaning difficulty score, A is the stain area, α is the weight of the stain area, β is the weight of the stain adhesion degree, and γ is the weight of the roughness of the stain area.

[0090] Furthermore, after obtaining the stain type and degree of stain adhesion, a cleaning difficulty score is calculated to determine the required water pressure, nozzle angle, and rinsing time. The cleaning difficulty score is calculated based on a weighted average of stain area, stain adhesion degree, and stain roughness. S is the cleaning difficulty score, with a value range of S∈[0,10], A is the stain area, with a range of A≥0, affecting the size of the rinsing area, and U is the thick area of ​​stain adhesion, with a range of U∈[0,1], representing the difficulty of stain removal.

[0091] S5. Prioritize the stains on the vehicle body surface in different areas and generate a list of washing areas.

[0092] S5.1. Set the weight of the vehicle cleaning area according to the roughness of the vehicle stain area, and combine it with the cleaning difficulty score to obtain the rinsing priority score.

[0093] Furthermore, the weight of vehicle cleaning areas is set based on the roughness of the stained areas to distinguish the washing needs of different parts of the vehicle. Areas such as wheels and chassis are prone to accumulating mud and dirt, so they have a higher weight, while areas such as the roof and windows have relatively less dirt and a lower weight. This weight is combined with the cleaning difficulty score to calculate the washing priority score.

[0094] S5.2. Based on the rinsing priority score, set the rinsing priority threshold and sort all stained areas from high to low.

[0095] Furthermore, after calculating the rinsing priority score, a rinsing priority threshold is set, and all stained areas are sorted from high to low. The priority threshold means that areas with scores higher than the average are processed first. The stained areas are sorted from high to low to form a preliminary rinsing execution order.

[0096] S5.3 When the flushing priority score is greater than the flushing priority threshold, the area is considered to be in a high-cleaning-difficulty zone.

[0097] Furthermore, areas with a washing priority score higher than the set threshold are marked as high-difficulty cleaning areas, such as wheels and chassis (where there is more accumulated mud and sand with high adhesion), door gaps (where oil stains are difficult to clean), and front and rear bumpers (areas that are easily contaminated).

[0098] S5.4 When the flushing priority score equals the flushing priority threshold, the cleaning difficulty zone is medium.

[0099] Furthermore, for areas with a washing priority score close to the threshold (marked as medium cleaning difficulty areas, typically including the sides of the vehicle (where dust easily accumulates but has low adhesion), rearview mirrors, and window edges (where rainwater stains are present but easily removed).

[0100] S5.5 When the flushing priority score is less than the flushing priority threshold, the area is considered a low-cleaning-difficulty zone.

[0101] Furthermore, areas with a washing priority score below the threshold are marked as low-difficulty cleaning areas, including the roof (less dirt, only slight dust) and the window surfaces (water stains and dust that are easier to clean).

[0102] S5.6 Organize the stained areas and prioritize them to generate a list of rinsing areas.

[0103] Furthermore, based on the priority of the stained areas, a final list of rinsing areas is generated, including: area coordinates, rinsing priority score, and rinsing parameters (water pressure, nozzle angle, rinsing time).

[0104] S6. Construct a DQN+PPO reinforcement learning model to dynamically adjust the nozzle angle, water pressure, and rinsing time.

[0105] S6.1. Construct a PPO reinforcement learning model based on the list of washing areas, define the state space, and generate a preliminary vehicle washing plan, expressed as follows:

[0106] O = {A, U, R, θ, P, T};

[0107] Where O represents the initial vehicle washing plan, θ represents the current nozzle angle, P represents the current water pressure, and T represents the current washing time.

[0108] Furthermore, a PPO (Proximal Policy Optimization) reinforcement learning model is constructed based on the list of rinsing areas. The state space includes A, where A is the stain area and the value range is A∈[0,5000cm²]. 2 U represents the thick area of ​​stain adhesion, with a value range of U∈[0,1], R represents the roughness of the stain area, with a value range of R∈[0,1], θ represents the current nozzle angle, with a value range of θ∈[0°,90°], P represents the current water pressure, with a value range of P∈[1,3]MPa, and T represents the current rinsing time, with a value range of T∈[1,10]s. This completes the construction of the state space, thereby generating a preliminary vehicle rinsing plan.

[0109] S6.2 Optimize the initial vehicle washing plan using DQN, and combine it with PPO to perform gradient updates of the initial vehicle washing plan, dynamically adjusting the nozzle angle, water pressure, and washing time. The expression is:

[0110] E = {Δθ, ΔP, ΔT};

[0111] Where E represents the optimized initial vehicle washing plan, Δθ represents the adjusted nozzle angle, ΔP represents the adjusted water pressure, and ΔT represents the adjusted washing time.

[0112] Furthermore, after generating the initial flushing plan, DQN (Deep Q-Network) is used for reinforcement learning optimization, and PPO (Policy Gradient Optimization) is combined to adjust the flushing parameters to improve cleaning efficiency and reduce water waste. Δθ is the adjusted nozzle angle with a value range of Δθ∈[-15°,+15°], ΔP is the adjusted water pressure with a value range of ΔP∈[-0.5,+0.5]MPa, and ΔT is the adjusted flushing time with a value range of ΔT∈[-2,+2]s.

[0113] S7 uses millimeter-wave sensors to detect stain removal and score the cleanliness of the vehicle.

[0114] S7.1. Use millimeter-wave sensors to scan the vehicle surface and obtain point cloud data of stain residue.

[0115] Furthermore, millimeter-wave sensors are used to scan the vehicle surface to obtain point cloud data of stain residue. The millimeter-wave sensor constructs a three-dimensional point cloud model of the vehicle surface by emitting electromagnetic waves and measuring the phase change and reflection intensity of the returned signal. The sensor captures the stain area before and after washing and generates high-precision point cloud data of stain residue.

[0116] S7.2. Based on the stain residue point cloud data, set the stain data before rinsing and the data after rinsing, and calculate the stain removal rate. The expression is as follows:

[0117]

[0118] Where M is the stain removal rate, K is the number of stains before rinsing, and Z is the number of stains after rinsing;

[0119] Furthermore, based on the stain point cloud data before and after rinsing, the stain removal rate is calculated by comparing the changes in the number of stain point clouds. M is the stain removal rate, M∈[0,1], which represents the proportion of stain removal. The larger the value, the better the stain removal effect. K is the number of stain points before rinsing, K≥0, which represents the total number of stain point clouds detected before rinsing. Z is the number of stain points after rinsing, Z≥0.

[0120] S7.3. Based on the stain removal rate, the cleanliness score of the vehicle's stained areas before and after washing is calculated using the IoU (Interval of Units). The expression is as follows:

[0121]

[0122] Where H is the cleanliness score, Q is the area of ​​residual stains after rinsing, and G is the number of stains before rinsing.

[0123] Furthermore, H is the cleanliness score, H∈[0,100]%, which represents the overall cleaning effect of the vehicle. The higher the value, the better the cleaning effect. G is the number of stains before rinsing, G≥0, which represents the total number of stain point clouds detected before rinsing. Q is the area of ​​residual stains after rinsing, Q≥0.

[0124] This embodiment also provides a computer device applicable to the deep learning-based multi-camera vehicle washing detection method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the deep learning-based multi-camera vehicle washing detection method proposed in the above embodiment.

[0125] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0126] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the deep learning-based multi-camera vehicle washing detection method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0127] In summary, this invention achieves intelligent detection and efficient cleaning of vehicle surface stains by acquiring and fusing multi-view RGB images and point cloud images, combined with deep learning and reinforcement learning technologies. It utilizes industrial cameras and LiDAR sensors to acquire multi-view images and generates a high-precision 3D point cloud model through an iterative nearest-point algorithm, providing an accurate data foundation for subsequent detection. PointNet++ is used to extract point cloud features, and the ViT visual Transformer network is combined to identify stain distribution, achieving precise localization of stain areas. Based on RGB images and point cloud data, stain type, adhesion degree, and cleaning difficulty score are calculated through color variance, point cloud density, and roughness, generating a priority list of rinsing areas. A DQN+PPO reinforcement learning model is used to dynamically adjust nozzle angle, water pressure, and rinsing time, and a millimeter-wave sensor is used to detect stain removal, completing the vehicle cleanliness score.

[0128] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A deep learning based multi-camera vehicle wash detection method, characterized in that: include, Multi-view RGB images and multi-view point cloud images are acquired, and the multi-view point cloud images are fused using the iterative nearest point algorithm to generate a three-dimensional point cloud image. PointNet++ was used to extract features from 3D point cloud images, and the ViT visual Transformer network was combined to identify the distribution of dirt on the vehicle body surface. Includes the following steps, PointNet++ is used to extract features from 3D point cloud images, obtaining local and global point cloud image features, and calculating the rate of change of point cloud density. Based on the rate of change of point cloud density, ViT vision is used to identify changes in vehicle body color and mark areas of dirt on the vehicle body. By marking the areas of dirt on the vehicle body, a Transformer network is used to identify the distribution of dirt on the vehicle body surface; The process involves detecting the type and degree of stain adhesion in different areas of the vehicle's surface, calculating a cleaning difficulty score, prioritizing the stains in different areas, and generating a list of areas to be washed. This includes the following steps: The weight of the vehicle cleaning area is set according to the roughness of the vehicle stain area, and combined with the cleaning difficulty score to obtain the rinsing priority score; Based on the rinsing priority score, a rinsing priority threshold is set, and all stained areas are sorted from high to low. When the rinsing priority score is greater than the rinsing priority threshold, the stained area is considered a high-difficulty cleaning area. When the rinsing priority score equals the rinsing priority threshold, the stained area belongs to the medium cleaning difficulty area. When the rinsing priority score is less than the rinsing priority threshold, the stained area is considered a low-difficulty cleaning area. Organize the stained areas and prioritize them to generate a list of rinsing areas; A DQN+PPO reinforcement learning model is constructed to dynamically adjust nozzle angle, water pressure, and rinsing time, including the following steps. A PPO reinforcement learning model is constructed based on the list of washing areas, a state space is defined, and a preliminary vehicle washing plan is generated. The initial vehicle washing plan was optimized using DQN and updated in stages using PPO, dynamically adjusting nozzle angle, water pressure and washing time. Millimeter-wave sensors are used to detect the removal of stains and to score the cleanliness of the vehicle. 2.The deep learning-based multi-camera vehicle wash detection method of claim 1, wherein: Acquiring multi-view RGB images and multi-view point cloud images includes the following steps. Multi-view RGB images are acquired by synchronously shooting with an industrial camera, and time synchronization is performed using PTP to align the frames of the multi-view RGB images. Multi-view point cloud images are acquired using a LiDAR sensor, and stability correction is performed on the multi-view point cloud images using an inertial measurement unit and LiDAR-SLAM. 3.The deep learning-based multi-camera vehicle wash detection method of claim 2, wherein: The process of fusing point cloud images from multiple perspectives using the iterative nearest-point algorithm to generate a 3D point cloud image includes the following steps: A rotation matrix and a translation matrix are used to unify the multi-view point cloud images into a coordinate system and align the multi-view point cloud images. The iterative nearest-point algorithm is used to fuse the aligned multi-view point cloud images to obtain a 3D point cloud image. 4.The deep learning-based multi-camera vehicle wash detection method of claim 3, wherein: The process of detecting the type and degree of stain adhesion in different areas of the vehicle's surface and calculating the cleaning difficulty score includes the following steps. The mean hue of the stain distribution is extracted from the RGB image, and the color variance is calculated. Classify the color of the stain based on color variance to obtain the stain type of different areas on the vehicle body surface; KD-Tree is used to calculate the point cloud density of the stain area; The roughness of the stain area is calculated using the stain area point cloud normal vector change calculation method; Based on the point cloud density of the stain area and the roughness of the stain area, the stain adhesion degree is obtained; Based on the stain type of different areas on the vehicle body surface and the stain adhesion degree, the cleaning difficulty score is calculated using a weighted method. 5.The deep learning-based multi-camera vehicle wash detection method of claim 4, wherein, The millimeter wave sensor detects the stain removal condition and scores the cleanliness of the vehicle, including the following steps, The millimeter wave sensor is used to scan the vehicle surface to obtain stain residue point cloud data, Based on the stain residue point cloud data, set the pre-washing stain data and post-washing data, and calculate the stain removal rate; Based on the stain removal rate, the IoU is used to calculate the cleanliness score of the vehicle before and after washing the stain area. 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The processor executes the computer program to implement the steps of the deep learning-based multi-camera vehicle washing detection method of any one of claims 1-5.

7. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the deep learning-based multi-camera vehicle washing detection method of any one of claims 1-5.

Images