A cleaning control method for a sweeper vehicle based on multimodal perception

Abstract

The present invention discloses a cleaning control method for a sweeper and washer based on multimodal perception, comprising: acquiring road pollutant image data at different angles, fusing the road pollutant image data at different angles to obtain a first fused image, and identifying a first pollutant type and its pollution degree through the first fused image; acquiring road pollutant odor data, fusing the first fused image with the road pollutant odor data according to the first pollution type to obtain fused data features; determining a second pollutant type and its pollution degree through the fused data features, and generating a cleaning strategy corresponding to the first pollutant type and the second pollutant type according to the first pollutant type and the second pollutant type.

Description

Technical Field

[0001] The present invention relates to the technical field of cleaning control of a washing and sweeping vehicle, and in particular to a cleaning control method of a washing and sweeping vehicle based on multimodal perception. Background Art

[0002] A washing and sweeping vehicle is a multifunctional sanitation equipment that combines cleaning, flushing and vacuuming functions. It is mainly used to clean garbage, dust and other dirt from urban roads, public squares, highways and other places.

[0003] Traditional sweepers mostly use fixed cleaning parameters (such as water spray pressure and brush rotation speed). When faced with different pollutant types (such as solid waste, particulate dust, and liquid pollutants), relying solely on mechanical cleaning devices operating in fixed modes makes it difficult to adjust cleaning strategies based on real-time cleaning needs.

[0004] Some existing cleaning vehicles use cameras installed around the vehicle to perceive the surrounding environment and adjust their cleaning strategies accordingly. However, in practice, image recognition relies on the visual characteristics of pollutants (such as color, shape, and texture), making it difficult to detect transparent or difficult-to-identify pollutants (such as colorless oil stains, chemical liquid residues, and other liquid pollutants). In addition, some pollutants are obscured (such as liquid waste covered by leaves, sewage hidden under solid waste), and single-image recognition is difficult to capture these obscured pollutants. This can lead to subsequent cleaning strategies focusing solely on visible waste cleaning without targeted pollutant removal, resulting in incomplete cleaning and potentially even the spread of pollutants.

[0005] The purpose of this invention is to design a cleaning control method for a washing and sweeping vehicle based on multimodal perception to solve the above-mentioned problems in the prior art. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to propose a cleaning control method for a washing and sweeping vehicle based on multimodal perception, which can solve the above-mentioned problems.

[0007] The present invention provides a cleaning control method for a washing and sweeping vehicle based on multimodal perception, and a cleaning control system for a washing and sweeping vehicle based on multimodal perception, comprising:

[0008] Camera groups are installed on the body of the cleaning and sweeping vehicle to collect image data of road pollutants at different angles;

[0009] Odor sensor groups are installed on both sides of the washing and sweeping vehicle to monitor the odor data of road pollutants;

[0010] a controller, connected to the camera group and the odor sensor group, respectively, for generating a cleaning control strategy based on the road pollution image data and the road pollutant odor data;

[0011] The cleaning mechanism is connected to the controller and is used to execute the cleaning control strategy issued by the controller;

[0012] The method comprises:

[0013] Acquire road surface pollutant image data at different angles, fuse the road surface pollution image data at different angles to obtain a first fused image, and identify the first pollutant type and its pollution degree through the first fused image;

[0014] Acquiring road pollutant odor data, fusing the first fused image with the road pollutant odor data according to a first pollution type to obtain fused data features;

[0015] The second pollutant type and its pollution degree are determined by fusing the data features, and a cleaning strategy corresponding to the pollutant type is generated according to the first pollutant type and the second pollutant type.

[0016] Furthermore, the camera group includes:

[0017] The front camera is located at the front of the vehicle and is used to obtain images of the front side of the vehicle in the direction of travel;

[0018] The left camera is located on the left rearview mirror and is used to obtain the left image of the vehicle;

[0019] The right camera is installed on the right rearview mirror and is used to obtain the right image of the vehicle body;

[0020] The rear camera is installed at the rear of the vehicle and is used to obtain the rear image of the rear side of the vehicle.

[0021] Furthermore, the acquiring of road surface pollutant image data at different angles, fusing the road surface pollutant image data at different angles to obtain a first fused image, and identifying the first pollutant type and its pollution degree through the first fused image include:

[0022] Acquire a front image, a left image, and a right image at the same time point, and fuse the front image, the left image, and the right image at the same time point to obtain a first fused image;

[0023] The first fused image is input into a pre-trained first pollutant detection model to obtain a first pollutant type and a pollution degree thereof. The pollutant types include: solid waste, particulate dust, and liquid pollutants. The pollution degrees include: high pollution, moderate pollution, and low pollution.

[0024] Furthermore, fusing the front image, the left image, and the right image at the same time point to obtain a first fused image includes:

[0025] The deep features of the front image, left image, and right image are extracted respectively through the pre-trained convolutional neural network;

[0026] The deep features of the front image, left image, and right image are weightedly fused using the attention mechanism, and the decoder is used to reconstruct the weighted fused features into the first fused image.

[0027] Furthermore, the road pollutant odor data is obtained, and the first fused image and the road pollutant odor data are fused according to the first pollution type to obtain fused data features including:

[0028] Extract visual features of the first fused image through a pre-trained convolutional neural network ;

[0029] Input the road pollution odor data into the long short-term memory network model to extract the odor characteristics ;

[0030] Use the embedded features of the first pollutant type as the query vector , visual features are calculated separately through the attention mechanism and odor characteristics The corresponding weight is calculated as follows:

[0031] ,

[0032] ,

[0033] in, For visual features The corresponding weight, Odor characteristics The corresponding weight, is the attention mechanism function;

[0034] According to visual features and odor characteristics The corresponding weights will be visual features and odor characteristics Fusion is performed to obtain fused data features , the calculation formula is as follows:

[0035] .

[0036] Furthermore, the cleaning mechanism includes:

[0037] Cleaning devices are respectively arranged on the left and right sides of the washing and sweeping vehicle, and are used to sweep and / or wash away road pollutants;

[0038] The sewage suction device is respectively installed on the left and right sides of the sweeper and the rear side of the cleaning device to absorb the pollutants into the garbage bin loaded on the sweeper;

[0039] The spraying devices are respectively arranged on the left and right sides of the washing and sweeping vehicle and in front of the cleaning device, and are used to spray water to soften and dilute pollutants.

[0040] Furthermore, the second pollutant type and its pollution degree are obtained by fusing the data features, and a cleaning strategy corresponding to the first pollutant type and the second pollutant type is generated according to the first pollutant type and the second pollutant type, including:

[0041] Inputting the fused data features into a second pollutant detection model to obtain the second pollutant type and its pollution degree;

[0042] If the first pollutant type and the second pollutant type are the same, the operation of the cleaning mechanism is controlled according to the corresponding pollutant type and the maximum pollution level;

[0043] If the first pollutant type and the second pollutant type are different, a mixed pollutant cleaning strategy is executed.

[0044] Furthermore, if the first pollutant type and the second pollutant type are the same, controlling the operation of the cleaning mechanism according to the corresponding pollutant type and the maximum pollution level includes:

[0045] A reinforcement learning model is constructed using the pollutant type and pollution degree as state variables, and the sweeping brush speed adjustment value of the cleaning device, the sweeping brush lift adjustment value of the cleaning device, the water pump speed adjustment value of the spraying device, and the fan speed adjustment value of the suction device as action variables.

[0046] A reward function is constructed based on the cleaning effect value and the cleaning time. When the reward value of the reward function is higher than the reward threshold, it is marked as a positive reward signal, otherwise it is marked as a negative reward signal. The reinforcement learning model is optimized based on the reward signal.

[0047] The corresponding pollutant type and maximum pollution level are input into the reinforcement learning model to obtain the sweeping brush speed adjustment amount of the cleaning device, the sweeping brush lifting adjustment amount of the cleaning device, the water pump speed adjustment amount of the spraying device, and the fan speed adjustment amount of the suction device.

[0048] Furthermore, constructing a reward function based on cleaning effect and cleaning time includes:

[0049] Acquire a rear image, a left image, and a right image at the same time point, and fuse the rear image, the left image, and the right image at the same time point to obtain a second fused image;

[0050] Inputting the second fused image into the pre-trained first pollutant detection model to obtain the degree of contamination after cleaning;

[0051] Degree of contamination after cleaning and the highest contamination level before cleaning Calculate the cleaning effect C, the calculation formula is as follows:

[0052] ;

[0053] The reward function R is constructed by the cleaning effect C and the cleaning time t. The calculation formula is as follows:

[0054] ,

[0055] in, is the weight of the cleaning effect, is the weight of cleaning time.

[0056] Furthermore, if the first pollutant type and the second pollutant type are different, the pollutant type is marked as a mixed pollutant, and executing the mixed pollutant cleaning strategy includes:

[0057] If the types of mixed pollutants are solid waste and particulate dust, the cleaning strategies for solid waste and particulate dust are obtained respectively through the reinforcement learning model, and the cleaning strategy for solid waste is executed first and then the cleaning strategy for particulate dust;

[0058] If the types of mixed pollutants are solid waste and liquid pollutants, the cleaning strategies for solid waste and liquid pollutants are obtained respectively through the reinforcement learning model, and the cleaning strategy for solid waste is executed first and then the cleaning strategy for liquid pollutants;

[0059] If the types of mixed pollutants are particulate dust and liquid pollutants, the cleaning strategies for particulate dust and liquid pollutants are obtained respectively through the reinforcement learning model, and the cleaning strategy for particulate dust is executed first and then the cleaning strategy for liquid pollutants.

[0060] Beneficial effects of the present invention:

[0061] First, images of road pollutants from different perspectives (e.g., front, left, and right) are fused to form a complete description of the scene's characteristics. Image fusion preserves key information from each perspective, more comprehensively reflecting the shape, color, and distribution of pollutants while reducing the risk of occlusion or information loss. Inputting the fused first image into a pre-trained pollutant detection model effectively extracts the pollutant's deep characteristics, accurately identifying the type of the first pollutant (e.g., solid waste, liquid pollutants, particulate matter, dust) and its degree of contamination.

[0062] Second, by fusing image and odor data, the limitations of single-modal data are overcome. Images provide information on the spatial distribution, appearance, shape, and occlusion effects of pollutants, while odors complement chemical properties (such as toxicity, harmfulness, or corrosiveness). Multimodal fusion enables a more comprehensive representation of pollutant characteristics, providing a more accurate reference for pollution identification and cleaning strategy generation. Based on the characteristics of the primary pollutant type, the fusion weights of image and odor data are dynamically adjusted to ensure a matching of importance.

[0063] Third, by integrating data features, we can provide richer feature judgments for secondary pollutant detection, helping the model to further uncover hidden information from a single pollutant type. This step ensures comprehensive identification of all pollutant types and their levels, whether in areas of single or mixed pollution, laying a reliable data foundation for the subsequent generation of cleaning strategies. Once the cleaning strategy is generated, the cleaning device's operating parameters (brush speed, lift height, water pump flow rate, etc.) are precisely controlled to maximize cleaning effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0065] Figure 1 It is a flow chart of the method of this embodiment. DETAILED DESCRIPTION

[0066] To facilitate understanding by those skilled in the art, the structure of the present invention will now be further described in detail with reference to the embodiments and accompanying drawings. It should be understood that the steps mentioned in this embodiment, unless otherwise specified, can be adjusted in sequence according to actual needs, and can even be executed simultaneously or partially simultaneously.

[0067] Example 1

[0068] This embodiment provides a cleaning control system for a washing and sweeping vehicle based on multimodal perception, including:

[0069] Camera groups are installed on the body of the cleaning and sweeping vehicle to collect image data of road pollutants at different angles;

[0070] The camera group includes:

[0071] The front camera is located at the front of the vehicle and is used to obtain images of the front side of the vehicle in the direction of travel;

[0072] The left camera is located on the left rearview mirror and is used to obtain the left image of the vehicle;

[0073] The right camera is installed on the right rearview mirror and is used to obtain the right image of the vehicle body;

[0074] The rear camera is installed at the rear of the vehicle and is used to obtain the rear image of the rear side of the vehicle.

[0075] Odor sensor groups are installed on both sides of the washing and sweeping vehicle to monitor the odor data of road pollutants;

[0076] a controller, connected to the camera group and the odor sensor group, respectively, for generating a cleaning control strategy based on the road pollution image data and the road pollutant odor data;

[0077] The cleaning mechanism is connected to the controller and is used to execute the cleaning control strategy issued by the controller;

[0078] Wherein, the cleaning mechanism includes:

[0079] Cleaning devices are respectively arranged on the left and right sides of the washing and sweeping vehicle, and are used to sweep and / or wash away road pollutants;

[0080] The sewage suction device is respectively installed on the left and right sides of the sweeper and the rear side of the cleaning device to absorb the pollutants into the garbage bin loaded on the sweeper;

[0081] The spraying devices are respectively arranged on the left and right sides of the washing and sweeping vehicle and in front of the cleaning device, and are used to spray water to soften and dilute pollutants.

[0082] In this embodiment, multiple cameras capture road surface images from different directions, allowing a road sweeper to accurately identify the location and distribution of pollutants. While cameras can only detect visual information (such as shape and distribution) of road pollutants, some pollutants, such as chemical liquids, oil stains, or decomposed garbage residues, may be more easily detected simply by odor. Odor sensors, by sensing the chemical vapors emitted by pollutants, can provide important supplementary sensory information. Combining image and odor data allows for a more comprehensive identification of pollutant type and severity.

[0083] The odor sensor group is mainly used to detect volatile chemical gases or characteristic odors of road pollutants. Therefore, the odor sensor can use a combination of multiple odor sensors, such as electrochemical gas sensors (which can detect ammonia and hydrogen sulfide gases produced by food spoilage in the market), metal oxide semiconductor gas sensors (which can detect volatile gases of chemical liquids and grease pollutants on the road), and PID photoionization gas sensors (which can detect volatile gases of paint, chemical cleaning agents, oil stains, etc.).

[0084] The spraying device, located at the front of the vehicle, dissolves dried dust particles or solidified dirt on the ground, making it easier for the cleaning device to brush away. It also reduces the stickiness of liquid contaminants (such as oil and organic residues), making them easier for the subsequent suction device to absorb. Adding water spray before the cleaning process can effectively suppress secondary dust pollution caused by dried dust. The cleaning device, located in the middle of the vehicle body, directly sweeps and collects household garbage and large dust particles near the suction device through a mechanical brushing action. Multiple sweeps and simultaneous water spraying can deeply scrub and clean residual contaminants. Raising and lowering the cleaning device can reduce or increase the scrubbing force. The suction device, located at the rear of the vehicle's rear section, removes dust, liquid stains, or softened, dissolved contaminants not thoroughly removed by the cleaning device. The specific mechanical structures of the spraying, cleaning, and suction devices are all prior art. This application does not involve improvements to these mechanical structures and does not provide a detailed description of these mechanical structures.

[0085] Example 2

[0086] like Figure 1 As shown, this embodiment provides a cleaning control method for a washing and sweeping vehicle based on multimodal perception, including:

[0087] S1 acquires road surface pollutant image data at different angles, fuses the road surface pollutant image data at different angles to obtain a first fused image, and identifies a first pollutant type and its pollution degree through the first fused image;

[0088] S101: acquiring a front image, a left image, and a right image at the same time point, and fusing the front image, the left image, and the right image at the same time point to obtain a first fused image;

[0089] S1011 extracts deep features of the front image, left image, and right image respectively through pre-trained convolutional neural networks;

[0090] S1012 performs weighted fusion of the deep features of the front image, the left image, and the right image using an attention mechanism, and uses a decoder to reconstruct the weighted fused features into a first fused image.

[0091] S102: inputting the first fused image into a pre-trained first pollutant detection model to obtain a first pollutant type and a rectangular bounding box thereof, wherein the pollutant type includes: solid waste, particulate dust, and liquid pollutants;

[0092] S103 calculates the percentage of pollutant pixels based on the number of pixels actually covered by pollutants and the total number of all pixels within the rectangular boundary box, and obtains the corresponding pollution level based on the percentage of pollutant pixels. The pollution level includes: high pollution, moderate pollution, and low pollution.

[0093] In this step, image data from a single angle may have blind spots or incomplete information (e.g., due to occlusion or lighting), leading to inaccurate pollutant detection. By acquiring road surface image data from multiple angles, including the front, left, and right sides, and using deep feature extraction and an attention mechanism for weighted fusion, a first fused image with a global view is generated. Image data from different angles may have different viewpoints, resolutions, or inconsistent features, and direct fusion may result in information loss or redundancy. Deep features are extracted using a pretrained convolutional neural network and weighted fused using an attention mechanism to highlight key features and suppress irrelevant information.

[0094] Traditional methods struggle to simultaneously accurately identify multiple pollutant types (solid waste, particulate matter, and liquid pollutants) and their respective levels (high, medium, and low), especially in complex scenarios. This solution uses a pre-trained YOLO model as the primary pollutant detection model. Through training, the YOLO model can distinguish between solid waste, particulate matter, and liquid pollutants, and returns the results as specific pollutant types using a class confidence score.

[0095] When detecting pollutants, the YOLO model will output a bounding box of the target, which describes the rectangular range of the area where the pollutant is located. The total number of pixels in the bounding box The number of pixels covered by actual pollutants It can be used as an indicator to determine the degree of pollution. The calculation formula is: Specific: Highly polluted: Percentage of polluted pixels ≥80%, moderate pollution: 40%≤proportion of polluted pixels <80%, low pollution: percentage of polluted pixels <40%.

[0096] The ratio of polluted pixels to bounding box pixels is used as the basis for judgment, avoiding direct reliance on the absolute value of the polluted area. This is because the extent of road pollution can be affected by viewing angle and distance, while the ratio is relatively stable. Whether it is solid waste (evenly distributed lumps) or liquid pollutants (which may have irregular shapes or be scattered), the ratio as a criterion is universal.

[0097] S2 obtains road pollutant odor data, and fuses the first fused image and the road pollutant odor data according to the first pollution type to obtain fused data features;

[0098] In this step, traditional detection schemes that rely on visual data are significantly affected by environmental factors (such as insufficient lighting and obstructions) and may not accurately detect pollutant types and characteristics. By integrating odor data, we can supplement characteristics that visual information cannot capture or observe. For example, some odor data can effectively identify putrefactive pollutants or liquid contamination. In complex pollutant types, different pollutants rely on visual and odor characteristics to varying degrees (for example, solid waste is more noticeable visually, while putrefactive liquid contamination is more sensitive to odor). By extracting and weighting visual and odor features separately, and applying an attention mechanism, the final fused feature preserves the spatial information of the image while capturing the temporal dynamics of odors. The resulting fused feature is a multidimensional, integrated information that simultaneously characterizes the visual appearance and odor characteristics of the pollutant. This multimodal fusion provides a precise basis for subsequent pollutant type detection.

[0099] S201 extracts visual features of the first fused image through a pre-trained convolutional neural network ;

[0100] S202 inputs the road pollution odor data into the long short-term memory network model to extract the odor characteristics ;

[0101] In this step, the data type collected by the odor sensor is usually an analog or digital quantity reflecting the characteristics of a specific gas. This odor data is usually time series data formed by sampling from multiple sensors at different time points. The LSTM (long short-term memory network model) is good at processing this time series and can capture the dynamic characteristics of pollutant odors.

[0102] S203 uses the embedded features of the first pollutant type as a query vector , visual features are calculated separately through the attention mechanism and odor characteristics The corresponding weight is calculated as follows:

[0103] ,

[0104] ,

[0105] in, For visual features The corresponding weight, Odor characteristics The corresponding weight, is the attention mechanism function;

[0106] S204 Based on visual features and odor characteristics The corresponding weights will be visual features and odor characteristics Fusion is performed to obtain fused data features , the calculation formula is as follows:

[0107] .

[0108] In this step, the importance of fusing visual features and odor features varies depending on the type of pollutant. In pollutant detection tasks involving multimodal features (visual and odor), different types of pollutants have different degrees of dependence on visual and odor features. For example, for solid garbage (such as paper scraps, plastic bottles), the visual features are very obvious (shape, outline, color), and the visual features contribute more to recognition, while the odor features may be irrelevant and have a lower weight. For liquid pollutants (such as oil stains, sewage), vision may only capture the area of ​​the pollutant, while the odor features supplement its composition information (such as whether there is a foul smell), and the odor features contribute more to judging the type of pollutant and the degree of pollution. In the attention mechanism, the query vector It is possible to dynamically focus on which parts of the visual and odor features are more important to this type of pollutant, thereby improving the accuracy of the fused data.

[0109] S3 determines the second pollutant type and its pollution degree by fusing the data features, and generates a cleaning strategy corresponding to the pollutant type according to the first pollutant type and the second pollutant type.

[0110] S301 inputs the fused data features into a second pollutant detection model to obtain the second pollutant type and its pollution degree;

[0111] In this step, after the fused features have been deeply encoded by CNN and LSTM and converted into a fixed-length fused vector, a multi-layer perceptron (MLP) is used as the second pollutant detection model. During training, the fused data features of manually labeled pollution types and pollution levels are used as samples to train the MLP model.

[0112] The pollution type is labeled as one-hot encoding, and the pollution degree is obtained by calculating the proportion of polluted pixels (the ratio of the number of pixels in the polluted area to the total number of pixels in the image) through an automatic segmentation algorithm. In addition, the pollution degree labeling in this step also needs to be combined with the odor intensity to standardize the collected data. For example: VOC concentration: 65 ppm (the maximum value is set to 100 ppm, and G = 0.65 after standardization). Then, a weighted combination is performed based on the previous proportion of polluted pixels. For example: the proportion of polluted pixels is 0.2. For liquid pollution, the weight of odor data is 0.6, and the proportion of image data is 0.4. The final calculation is 0.4⋅0.2+0.6⋅0.65=0.47. The final pollution degree is divided in the same way as step S1, and the pollution degree belongs to moderate pollution.

[0113] The model's input is a fused multimodal feature set (combining visual and odor features), and its output is pollution type (classification) and pollution severity (regression). Multi-task learning achieved through joint optimization improves both classification accuracy and pollution severity quantification precision.

[0114] S302: If the first pollutant type and the second pollutant type are the same, control the operation of the cleaning mechanism according to the corresponding pollutant type and the maximum pollution level;

[0115] S3021 uses the pollutant type and pollution degree as state variables, and the sweeping brush speed adjustment value of the cleaning device, the sweeping brush lift adjustment value of the cleaning device, the water pump speed adjustment value of the spraying device, and the fan speed adjustment value of the suction device as action variables to build a reinforcement learning model;

[0116] S3022 constructs a reward function based on the cleaning effect value and the cleaning time. When the reward value of the reward function is higher than the reward threshold, it is marked as a positive reward signal; otherwise, it is marked as a negative reward signal. The reinforcement learning model is optimized according to the reward signal.

[0117] Among them, the reward function constructed by cleaning effect value and cleaning time includes:

[0118] Acquire a rear image, a left image, and a right image at the same time point, and fuse the rear image, the left image, and the right image at the same time point to obtain a second fused image;

[0119] Inputting the second fused image into the pre-trained first pollutant detection model to obtain the degree of contamination after cleaning;

[0120] Degree of contamination after cleaning and the highest contamination level before cleaning Calculate the cleaning effect C, the calculation formula is as follows:

[0121] ;

[0122] The reward function R is constructed by the cleaning effect C and the cleaning time t. The calculation formula is as follows:

[0123] ,

[0124] in, is the weight of the cleaning effect, is the weight of cleaning time.

[0125] S3023 inputs the corresponding pollutant type and the maximum pollution level into the reinforcement learning model to obtain the sweeping brush speed adjustment amount of the cleaning device, the sweeping brush lifting adjustment amount of the cleaning device, the water pump speed adjustment amount of the spraying device, and the fan speed adjustment amount of the suction device.

[0126] In this step, different pollutant types and their corresponding pollution levels have different requirements for the operating parameters of the cleaning mechanism (such as sweeping brush speed, lifting height, water pump speed, etc.). Furthermore, the cleaning mechanism has multiple adjustable parameters (such as sweeping brush speed, sweeping brush lifting height, water pump speed, fan speed, etc.), and these parameters may have complex synergistic relationships. Using a reinforcement learning model, the parameters of the cleaning mechanism are dynamically adjusted according to pollutant type and pollution level, enabling the cleaning mechanism to optimize its operation for different pollution situations. By dynamically adjusting the action parameters, the cleaning effect is maximized and the cleaning time is minimized.

[0127] Simply relying on cleaning effect (such as the reduction of pollution level) may ignore the efficiency problem, while simply relying on efficiency (such as cleaning time) may sacrifice the cleaning effect. Construct a reward function R based on cleaning effect and cleaning time, and use the weight and Balancing the priorities of cleaning effectiveness and time. Cleaning effectiveness assessment relies on comparing the contamination level before and after cleaning, which requires extraction from multi-view images (rear, left, and right). Using image fusion technology, the rear, left, and right images are fused into a second fused image. This image is then input into a pre-trained first contaminant detection model (YOLO model) to determine the post-cleaning contamination level. This multi-view fusion reduces occlusions or omissions that may occur with a single viewpoint, improving the accuracy of contamination assessment.

[0128] S303: If the first pollutant type and the second pollutant type are different, a mixed pollutant cleaning strategy is executed.

[0129] S3031 If the types of the mixed pollutants are solid waste and particulate dust, respectively, a cleaning strategy for the solid waste and a cleaning strategy for the particulate dust are obtained through a reinforcement learning model, and the cleaning strategy for the solid waste is executed first and then the cleaning strategy for the particulate dust;

[0130] S3032 If the types of the mixed pollutants are solid waste and liquid pollutants, respectively, a cleaning strategy for the solid waste and a cleaning strategy for the liquid pollutants are obtained through a reinforcement learning model, and the cleaning strategy for the solid waste is executed first and then the cleaning strategy for the liquid pollutants;

[0131] S3033 If the types of mixed pollutants are particulate dust and liquid pollutants, respectively, the cleaning strategy for particulate dust and the cleaning strategy for liquid pollutants are obtained respectively through the reinforcement learning model, and the cleaning strategy for particulate dust is executed first and then the cleaning strategy for liquid pollutants.

[0132] In this step, in actual scenarios, different types of pollutants (such as solid waste, particulate dust, and liquid pollutants) often exist at the same time, forming mixed pollution. Cleaning different types of pollutants requires different operating parameters, and the different cleaning orders of mixed pollutants will directly affect the cleaning effect. For example, cleaning liquid pollutants first may cause solid waste to be compacted, increasing the difficulty of subsequent cleaning. For mixed pollutants, cleaning strategies are formulated according to the type of pollutant, and cleaning order priority rules are designed based on the physical properties of the pollutant types: Solid waste first: Because solid waste has less mutual interference with cleaning equipment, priority cleaning can avoid equipment blockage. Particulate dust is second: Cleaning the dust can reduce the adhesion of liquid pollution during diffusion. Liquid pollutants are last: They can quickly restore the ground to a clean state, while avoiding mixing with dust and solid waste to create mud that is difficult to clean.

[0133] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0137] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third etc. does not indicate any order. These words may be interpreted as names.

[0138] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0139] Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the claims and their equivalents, the present invention is intended to include such modifications and variations.

[0140] In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connect," "fixed," etc. should be understood broadly. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0141] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

Claims

1. A cleaning control method for a sweeper vehicle based on multimodal perception, characterized in that: A cleaning control system for a washing and sweeping vehicle based on multimodal perception includes: Camera groups are installed on the body of the cleaning and sweeping vehicle to collect image data of road pollutants at different angles; Odor sensor groups are installed on both sides of the washing and sweeping vehicle to monitor the odor data of road pollutants; a controller, connected to the camera group and the odor sensor group, respectively, for generating a cleaning control strategy based on the road pollution image data and the road pollutant odor data; The cleaning mechanism is connected to the controller and is used to execute the cleaning control strategy issued by the controller; The method comprises: Acquire road surface pollutant image data at different angles, fuse the road surface pollution image data at different angles to obtain a first fused image, and identify the first pollutant type and its pollution degree through the first fused image; Obtain road pollutant odor data, fuse the first fused image and the road pollutant odor data according to the first pollution type, and obtain fused data features, specifically: Extract visual features of the first fused image through a pre-trained convolutional neural network ; Input the road pollution odor data into the long short-term memory network model to extract the odor characteristics ; Use the embedded features of the first pollutant type as the query vector , visual features are calculated separately through the attention mechanism and odor characteristics The corresponding weight is calculated as follows: ， ， in, For visual features The corresponding weight, Odor characteristics The corresponding weight, is the attention mechanism function; According to visual features and odor characteristics The corresponding weights will be visual features and odor characteristics Fusion is performed to obtain fused data features , the calculation formula is as follows: ； The second pollutant type and its pollution degree are determined by fusion data features. If the first pollutant type and the second pollutant type are the same, a cleaning strategy for the corresponding pollutant type is generated. Specifically: Inputting the fused data features into a second pollutant detection model to obtain the second pollutant type and its pollution degree; If the first pollutant type and the second pollutant type are the same, the operation of the cleaning mechanism is controlled according to the corresponding pollutant type and the maximum pollution level; If the first pollutant type and the second pollutant type are different, the mixed pollutant cleaning strategy is executed. Specifically: If the types of mixed pollutants are solid waste and particulate dust, the cleaning strategies for solid waste and particulate dust are obtained respectively through the reinforcement learning model, and the cleaning strategy for solid waste is executed first and then the cleaning strategy for particulate dust; If the types of mixed pollutants are solid waste and liquid pollutants, the cleaning strategies for solid waste and liquid pollutants are obtained respectively through the reinforcement learning model, and the cleaning strategy for solid waste is executed first and then the cleaning strategy for liquid pollutants; If the types of mixed pollutants are particulate dust and liquid pollutants, the cleaning strategies for particulate dust and liquid pollutants are obtained respectively through the reinforcement learning model, and the cleaning strategy for particulate dust is executed first and then the cleaning strategy for liquid pollutants.

2. The cleaning control method for a washing and sweeping vehicle based on multimodal perception according to claim 1 is characterized in that: The camera group includes: The front camera is located at the front of the vehicle and is used to obtain images of the front side of the vehicle in the direction of travel; The left camera is located on the left rearview mirror and is used to obtain the left image of the vehicle; The right camera is installed on the right rearview mirror and is used to obtain the right image of the vehicle body; The rear camera is installed at the rear of the vehicle and is used to obtain the rear image of the rear side of the vehicle.

3. The cleaning control method for a washing and sweeping vehicle based on multimodal perception according to claim 2 is characterized in that: The acquiring of road surface pollutant image data at different angles, fusing the road surface pollutant image data at different angles to obtain a first fused image, and identifying the first pollutant type and its pollution degree through the first fused image includes: Acquire a front image, a left image, and a right image at the same time point, and fuse the front image, the left image, and the right image at the same time point to obtain a first fused image; The first fused image is input into a pre-trained first pollutant detection model to obtain a first pollutant type and a pollution degree thereof. The pollutant types include: solid waste, particulate dust, and liquid pollutants. The pollution degrees include: high pollution, moderate pollution, and low pollution.

4. The cleaning control method for a washing and sweeping vehicle based on multimodal perception according to claim 3 is characterized in that: The fusing of the front image, the left image, and the right image at the same time point to obtain the first fused image includes: The deep features of the front image, left image, and right image are extracted respectively through the pre-trained convolutional neural network; The deep features of the front image, left image, and right image are weightedly fused using the attention mechanism, and the decoder is used to reconstruct the weighted fused features into the first fused image.

5. The cleaning control method for a washing and sweeping vehicle based on multimodal perception according to claim 1 is characterized in that: The cleaning mechanism comprises: Cleaning devices are respectively arranged on the left and right sides of the sweeper to sweep and / or wash away road pollutants; The sewage suction device is respectively installed on the left and right sides of the sweeper and the rear side of the cleaning device to absorb the pollutants into the garbage bin loaded on the sweeper; The spraying devices are respectively arranged on the left and right sides of the washing and sweeping vehicle and in front of the cleaning device, and are used to spray water to soften and dilute pollutants.

6. The cleaning control method for a washing and sweeping vehicle based on multimodal perception according to claim 5 is characterized in that: If the first pollutant type and the second pollutant type are the same, controlling the operation of the cleaning mechanism according to the corresponding pollutant type and the maximum pollution level includes: A reinforcement learning model is constructed using the pollutant type and pollution degree as state variables, and the sweeping brush speed adjustment value of the cleaning device, the sweeping brush lift adjustment value of the cleaning device, the water pump speed adjustment value of the spraying device, and the fan speed adjustment value of the suction device as action variables. A reward function is constructed based on the cleaning effect value and the cleaning time. When the reward value of the reward function is higher than the reward threshold, it is marked as a positive reward signal, otherwise it is marked as a negative reward signal. The reinforcement learning model is optimized based on the reward signal. The corresponding pollutant model and the maximum pollution level are input into the reinforcement learning model to obtain the sweeping brush speed adjustment amount of the cleaning device, the sweeping brush lifting adjustment amount of the cleaning device, the water pump speed adjustment amount of the spraying device, and the fan speed adjustment amount of the suction device.

7. The cleaning control method for a washing and sweeping vehicle based on multimodal perception according to claim 6 is characterized in that: The reward function constructed by the cleaning effect and the cleaning time includes: Acquire a rear image, a left image, and a right image at the same time point, and fuse the rear image, the left image, and the right image at the same time point to obtain a second fused image; Inputting the second fused image into the pre-trained first pollutant detection model to obtain the degree of contamination after cleaning; Degree of contamination after cleaning and the highest contamination level before cleaning Calculate the cleaning effect C, the calculation formula is as follows: ； The reward function R is constructed by the cleaning effect C and the cleaning time t. The calculation formula is as follows: ， in, is the weight of the cleaning effect, is the weight of cleaning time.

Images