Cleaning and sweeping vehicle cleaning control method based on multi-mode perception

Abstract

The invention discloses a cleaning and sweeping vehicle cleaning control method based on multi-modal perception, and the method comprises the steps: obtaining road surface pollutant image data at different angles, fusing the road surface pollutant image data at different angles to obtain a first fusion image, and carrying out the recognition of the first fusion image, so as to obtain a first pollutant type and the pollution degree of the first pollutant type; obtaining road surface pollutant smell data, and fusing the first fusion image and the road surface pollutant smell data according to the first pollution type to obtain a fusion data feature; the second pollutant type and the pollution degree thereof are obtained through fusion data feature judgment, and a cleaning strategy corresponding to the pollutant type is generated according to the first pollutant type and the second pollutant type.

Description

Technical Field

[0001] The present invention relates to the technical field of cleaning control of a washing and sweeping vehicle, and in particular to a cleaning control method of a washing and sweeping vehicle based on multimodal perception. Background Art

[0002] A washing and sweeping vehicle is a multifunctional sanitation equipment that combines cleaning, flushing and vacuuming functions. It is mainly used to clean garbage, dust and other dirt from urban roads, public squares, highways and other places.

[0003] Traditional cleaning vehicles mostly use fixed cleaning parameters (such as water spray pressure, brush rotation speed, etc.). When faced with different types of pollutants (such as solid garbage, particulate dust, liquid pollutants, etc.), they only rely on mechanical cleaning devices to operate in fixed modes, making it difficult to adjust the cleaning strategy according to real-time cleaning needs.

[0004] Some existing cleaning vehicles use cameras around the vehicle to sense the surrounding environment, and then adjust the cleaning strategy according to the actual environmental conditions. However, in actual applications, image recognition relies on the visual characteristics of pollutants (such as color, shape, texture, etc.), and it is difficult to detect transparent or difficult to visually identify pollutants (such as colorless oil stains, chemical liquid residues and other liquid pollutants). In addition, some pollutants are obscured (such as liquid garbage covered by leaves, sewage hidden under solid garbage), and a single image recognition is difficult to capture these covered pollution. This will result in the subsequent cleaning strategy, which only performs a single visual garbage cleaning without targeted cleaning of pollutants, resulting in incomplete cleaning and may even cause the spread of pollutants.

[0005] The purpose of this invention is to design a cleaning control method for a washing and sweeping vehicle based on multimodal perception in view of the above-mentioned problems in the prior art. Summary of the invention

[0006] In view of this, an object of the present invention is to propose a cleaning control method for a washing and sweeping vehicle based on multimodal perception, which can solve the above-mentioned problems.

[0007] The present invention provides a cleaning control method for a washing and sweeping vehicle based on multimodal perception, and a cleaning control system for a washing and sweeping vehicle based on multimodal perception, comprising: Camera groups are respectively arranged on the body of the cleaning and sweeping vehicle to collect image data of road pollutants at different angles; Odor sensor groups are installed on both sides of the washing and sweeping vehicle to monitor the odor data of road pollutants; A controller, connected to the camera group and the odor sensor group, respectively, for generating a cleaning control strategy according to the road pollution image data and the road pollution odor data; A cleaning mechanism, connected to the controller, for executing the cleaning control strategy issued by the controller; The method comprises: Acquire road surface pollutant image data at different angles, fuse the road surface pollutant image data at different angles to obtain a first fused image, and identify the first pollutant type and its pollution degree through the first fused image; Acquire road pollutant odor data, fuse the first fused image and the road pollutant odor data according to the first pollution type, and obtain fused data features; The second pollutant type and its degree of pollution are determined by fusing the data features, and a cleaning strategy corresponding to the pollutant type is generated according to the first pollutant type and the second pollutant type.

[0008] Furthermore, the camera group includes: The front camera is arranged at the front side of the vehicle and is used to obtain the front side image in the direction of the vehicle body; The left camera is arranged on the left rearview mirror and is used to obtain the left image of the left side of the vehicle body; The right camera is arranged on the right rearview mirror and is used to obtain the right image of the right side of the vehicle body; The rear camera is arranged at the rear of the vehicle and is used to obtain the rear image of the rear side of the vehicle.

[0009] Further, the acquiring of road surface pollutant image data at different angles, fusing the road surface pollutant image data at different angles to obtain a first fused image, and identifying the first pollutant type and its pollution degree through the first fused image include: Acquire a front image, a left image, and a right image at the same time point, and fuse the front image, the left image, and the right image at the same time point to obtain a first fused image; The first fused image is input into a pre-trained first pollutant detection model to obtain a first pollutant type and a pollution degree thereof, wherein the pollutant types include solid waste, particulate dust, and liquid pollutants, and the pollution degrees include high pollution, medium pollution, and low pollution.

[0010] Further, fusing the front image, the left image, and the right image at the same time point to obtain a first fused image includes: The deep features of the front image, left image, and right image are extracted through the pre-trained convolutional neural network. The deep features of the front image, the left image, and the right image are weightedly fused using an attention mechanism, and the weighted fused features are reconstructed into the first fused image using a decoder.

[0011] Furthermore, the obtaining of the road pollutant odor data comprises fusing the first fused image and the road pollutant odor data according to the first pollution type to obtain the fused data features including: Extract visual features of the first fused image through a pre-trained convolutional neural network ; The road pollution odor data is input into the long short-term memory network model to extract the odor characteristics. ; Use the embedded features of the first pollutant type as the query vector , respectively calculate the visual features through the attention mechanism and odor characteristics The corresponding weight is calculated as follows: , , in, For visual features The corresponding weights, Odor characteristics The corresponding weights, is the attention mechanism function; Based on visual features and odor characteristics The corresponding weights will be visual features and odor characteristics Fusion is performed to obtain fusion data features , the calculation formula is as follows: .

[0012] Further, the cleaning mechanism comprises: Cleaning devices are respectively arranged on the left and right sides of the cleaning and sweeping vehicle, and are used to clean and / or wash road pollutants; The sewage suction device is respectively arranged on the left and right sides of the cleaning and sweeping vehicle and the rear side of the cleaning device, and is used to absorb the pollutants into the garbage bin loaded on the cleaning vehicle; The spraying devices are respectively arranged on the left and right sides of the washing and sweeping vehicle and in front of the cleaning device, and are used for spraying water to soften and dilute pollutants.

[0013] Furthermore, the second pollutant type and its pollution degree are obtained by fusing the data features, and a cleaning strategy corresponding to the pollutant type is generated according to the first pollutant type and the second pollutant type, including: Inputting the fused data features into a second pollutant detection model to obtain the second pollutant type and its pollution degree; If the first pollutant type and the second pollutant type are the same, the operation of the cleaning mechanism is controlled according to the corresponding pollutant type and the maximum pollution degree; If the first pollutant type and the second pollutant type are different, a mixed pollutant cleaning strategy is performed.

[0014] Furthermore, if the first pollutant type and the second pollutant type are the same, controlling the operation of the cleaning mechanism according to the corresponding pollutant type and the highest pollution degree includes: The pollutant type and pollution degree are taken as state variables, and the sweeping brush speed adjustment amount of the cleaning device, the sweeping brush lifting adjustment amount of the cleaning device, the water pump speed adjustment amount of the spraying device, and the fan speed adjustment amount of the sewage suction device are taken as action variables to construct a reinforcement learning model. A reward function is constructed by the cleaning effect value and the cleaning time. When the reward value of the reward function is higher than the reward threshold, it is marked as a positive reward signal, otherwise it is marked as a negative reward signal. The reinforcement learning model is optimized according to the reward signal. The corresponding pollutant type and the maximum pollution degree are input into the reinforcement learning model to obtain the sweeping brush speed adjustment amount of the cleaning device, the sweeping brush lifting adjustment amount of the cleaning device, the water pump speed adjustment amount of the spraying device, and the fan speed adjustment amount of the suction device.

[0015] Further, the reward function constructed by the cleaning effect and the cleaning time includes: Acquire a rear image, a left image, and a right image at the same time point, and fuse the rear image, the left image, and the right image at the same time point to obtain a second fused image; Inputting the second fused image into the pre-trained first pollutant detection model to obtain the degree of contamination after cleaning; Degree of contamination after cleaning and the highest contamination level before cleaning Calculate the cleaning effect C, the calculation formula is as follows: ; The reward function R is constructed by the cleaning effect C and the cleaning time t. The calculation formula is as follows: , in, is the weight of the cleaning effect, The weight of the cleaning time.

[0016] Furthermore, if the first pollutant type and the second pollutant type are different, the pollutant type is marked as a mixed pollutant, and the mixed pollutant cleaning strategy is executed including: If the types of mixed pollutants are solid waste and particulate dust, respectively, the cleaning strategies of solid waste and particulate dust are obtained through the reinforcement learning model, and the cleaning strategy of solid waste is executed first and then the cleaning strategy of particulate dust; If the types of mixed pollutants are solid waste and liquid pollutants, respectively, the cleaning strategies of solid waste and liquid pollutants are obtained through the reinforcement learning model, and the cleaning strategy of solid waste is executed first and then the cleaning strategy of liquid pollutants; If the types of mixed pollutants are particulate dust and liquid pollutants, respectively, the cleaning strategies for particulate dust and liquid pollutants are obtained respectively through the reinforcement learning model, and the cleaning strategy for particulate dust is executed first and then the cleaning strategy for liquid pollutants.

[0017] Beneficial effects of the present invention: First, the road pollutant images from different perspectives (such as the front side, left side, and right side) are fused to form a complete description of the scene characteristics. Image fusion retains the key information of each perspective, more comprehensively reflects the shape, color, and distribution characteristics of pollutants, and reduces the risk of occlusion or information loss. The fused first fused image is input into the pre-trained pollutant detection model to effectively extract the deep characteristics of the pollutants. Accurately identify the type of the first pollutant (such as solid waste, liquid pollutants, particulate dust) and its degree of pollution.

[0018] Second, the limitations of single-modal data are eliminated by fusing image and odor data. Images provide information such as the spatial distribution, appearance shape, and occlusion effect of pollutants; odors supplement the characteristics of chemical properties (such as toxicity, harmfulness, or corrosiveness). After multimodal fusion, the characteristics of pollutants can be expressed more comprehensively, providing a more accurate reference for pollution identification and cleaning strategy generation. According to the characteristics of the first pollutant type, the fusion weights of image data and odor data are dynamically adjusted to ensure importance matching.

[0019] Third, by fusing data features, it can provide richer feature judgments for the second pollutant detection, helping the model to further explore hidden information from a single type of pollutant. Whether it is a single pollutant area or a mixed polluted area, this step can ensure the comprehensive identification of all pollutant types and their pollution levels, laying a reliable data foundation for the generation of subsequent cleaning strategies. After the cleaning strategy is generated, the cleaning effect is maximized by accurately controlling the operating parameters of the cleaning device (brush speed, lifting height, water pump flow rate, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1It is a flow chart of the method of this embodiment. DETAILED DESCRIPTION

[0022] To facilitate understanding by those skilled in the art, the structure of the present invention is now further described in detail with reference to the embodiments and the accompanying drawings. It should be understood that the steps mentioned in the present embodiment, unless otherwise specified in their order, can be adjusted in order according to actual needs, and can even be executed simultaneously or partially simultaneously.

[0023] Embodiment 1 This embodiment provides a cleaning control system for a washing and sweeping vehicle based on multimodal perception, including: Camera groups are respectively arranged on the body of the cleaning and sweeping vehicle to collect image data of road pollutants at different angles; Wherein, the camera group includes: The front camera is arranged at the front side of the vehicle and is used to obtain the front side image in the direction of the vehicle body; The left camera is arranged on the left rearview mirror and is used to obtain the left image of the left side of the vehicle body; The right camera is arranged on the right rearview mirror and is used to obtain the right image of the right side of the vehicle body; The rear camera is arranged at the rear of the vehicle and is used to obtain the rear image of the rear side of the vehicle.

[0024] Odor sensor groups are installed on both sides of the washing and sweeping vehicle to monitor the odor data of road pollutants; A controller, connected to the camera group and the odor sensor group, respectively, for generating a cleaning control strategy according to the road pollution image data and the road pollution odor data; A cleaning mechanism, connected to the controller, for executing the cleaning control strategy issued by the controller; Wherein, the cleaning mechanism comprises: Cleaning devices are respectively arranged on the left and right sides of the cleaning and sweeping vehicle, and are used to clean and / or wash road pollutants; The sewage suction device is respectively arranged on the left and right sides of the cleaning and sweeping vehicle and the rear side of the cleaning device, and is used to absorb the pollutants into the garbage bin loaded on the cleaning vehicle; The spraying devices are respectively arranged on the left and right sides of the washing and sweeping vehicle and in front of the cleaning device, and are used for spraying water to soften and dilute pollutants.

[0025] In this embodiment, by using multiple cameras to obtain road images in different directions, the cleaning and sweeping vehicle can accurately identify the location and distribution range of pollutants. Cameras can only detect the "visual information" of road pollutants (such as shape, distribution, etc.), but some pollutants, such as chemical liquids, oil stains, or decomposed garbage residues, may be easier to detect only by smell. Odor sensors can provide important supplementary perception information by sensing the chemical gases volatilized by pollutants. Combining images and odors can more comprehensively identify the type and degree of pollution of pollutants.

[0026] The odor sensor group is mainly used to detect volatile chemical gases or characteristic odors of road pollutants. Therefore, the odor sensor can use a combination of multiple odor sensors, such as electrochemical gas sensors (which can detect ammonia and hydrogen sulfide gases produced by food corruption in the market, etc.), metal oxide semiconductor gas sensors (which can detect volatile gases of chemical liquids and grease pollutants on the road), and PID photoionization gas sensors (which can detect volatile gases of paint, chemical cleaning agents, oil stains, etc.).

[0027] The spraying device is arranged on the front side of the vehicle body, which can dissolve the particulate dust or solidified dirt dried on the ground, making it easier for the cleaning device to brush off, reducing the viscosity of liquid pollutants (such as oil stains and organic residues), so that the subsequent suction device can adsorb more easily. Adding water spray before the cleaning process can effectively suppress the secondary dust pollution caused by dry dust. The cleaning device is located in the middle section of the vehicle body, and can directly sweep and organize domestic garbage and large particles of dust to the vicinity of the suction device through mechanical brushing action; in addition, multiple sweeping and spraying of water at the same time can deeply scrub and clean residual pollutants, and the cleaning device can be raised and lowered to reduce or increase the scrubbing force; the suction device is arranged in the last part of the washing and sweeping vehicle, on the rear side of the cleaning device, and is used to suck away dust, liquid stains or softened and dissolved pollutants that are not thoroughly cleaned by the cleaning device. The specific mechanical structures of the spraying device, the cleaning device, and the suction device are all prior art. This application does not involve the improvement of the mechanical structure, and this application does not specifically describe this part of the mechanical structure.

[0028] Embodiment 2 like Figure 1 As shown, this embodiment provides a cleaning control method for a washing and sweeping vehicle based on multimodal perception, comprising: S1 acquires road surface pollutant image data at different angles, fuses the road surface pollutant image data at different angles to obtain a first fused image, and identifies a first pollutant type and its pollution degree through the first fused image; S101: acquiring a front image, a left image, and a right image at the same time point, and fusing the front image, the left image, and the right image at the same time point to obtain a first fused image; S1011 extracts deep features of the front image, left image, and right image through a pre-trained convolutional neural network; S1012 performs weighted fusion of the deep features of the front image, the left image, and the right image using an attention mechanism, and uses a decoder to reconstruct the weighted fused features into a first fused image.

[0029] S102: input the first fused image into a pre-trained first pollutant detection model to obtain a first pollutant type and a rectangular boundary box thereof, wherein the pollutant type includes: solid waste, particulate dust, and liquid pollutants; S103 calculates the percentage of pollutant pixels according to the number of pixels actually covered by pollutants and the total number of all pixels in the rectangular boundary box, and obtains the corresponding pollution degree according to the percentage of pollutant pixels, and the pollution degree includes: high pollution, medium pollution, and low pollution.

[0030] In this step, image data from a single angle may have blind spots or incomplete information (such as occlusion or light influence), resulting in inaccurate pollutant detection. By acquiring road image data from multiple angles, including the front, left, and right sides, and using deep feature extraction and attention mechanism for weighted fusion, a first fused image with a global view is generated. Image data from different angles may have differences in perspective, resolution, or features, and direct fusion may result in information loss or redundancy. Deep features are extracted through pre-trained convolutional neural networks, and weighted fusion of features is performed in combination with attention mechanism to highlight key features and suppress irrelevant information.

[0031] Traditional methods have difficulty in simultaneously achieving accurate identification of multiple pollutant types (solid waste, particulate dust, liquid pollutants) and their pollution levels (high, medium, low), especially in complex scenarios. This solution uses a pre-trained YOLO model as the first pollutant detection model. Through training and learning, the YOLO model can identify three types of solid waste, particulate dust, and liquid pollutants, and return the results as specific pollutant types through category confidence scores.

[0032] When detecting pollutants, the YOLO model will output a bounding box of the target, which describes the rectangular range of the area where the pollutants are located. The total number of pixels in the bounding box The number of pixels covered with actual contamination It can be used as an indicator to determine the degree of pollution. The calculation formula is: Specific: Highly polluted: Polluted pixel ratio ≥80%, moderate pollution: 40%≤proportion of polluted pixels <80%, low pollution: percentage of polluted pixels <40%.

[0033] The ratio of pollutant pixels to bounding box pixels is used as the basis to avoid relying directly on the absolute value of the pollutant area, because the scope of road pollution may be affected by viewing angle and distance, while the ratio is relatively stable. Whether it is solid waste (evenly distributed lumps) or liquid pollutants (which may be irregularly shaped or splashed), the ratio as a standard is universal.

[0034] S2 obtains road pollutant odor data, and fuses the first fused image and the road pollutant odor data according to the first pollution type to obtain fused data features; In this step, the traditional detection scheme that relies on visual data is greatly affected by environmental factors (such as insufficient light and occlusion), and may not be able to accurately detect the type of pollutant and its characteristics. By combining odor data, the characteristics that cannot be covered or observed by visual information are supplemented. For example, some odor data can effectively identify corrupt pollutants or liquid pollution; in the case of complex pollutant types, different types of pollutants have different degrees of dependence on visual and odor characteristics (such as solid garbage is more obvious to vision, and corrupt liquid pollution is more sensitive to odor). By extracting and weighting the visual features and odor features separately, and using the attention mechanism to calculate, the final fusion feature retains the spatial information of the image and captures the time series dynamic characteristics of the odor. The final fusion feature is a multi-dimensional comprehensive information that can simultaneously characterize the visual performance and odor characteristics of the pollutant. Through this multimodal fusion, an accurate judgment basis is provided for subsequent pollutant type detection.

[0035] S201 extracts the visual features of the first fused image through a pre-trained convolutional neural network ; S202 inputs the road pollution odor data into the long short-term memory network model to extract the odor characteristics ; In this step, the data type collected by the odor sensor is usually an analog or digital quantity reflecting the characteristics of a specific gas. These odor data are usually time series data formed by sampling from multiple sensors at different time points. LSTM (Long Short-Term Memory Network Model) is good at processing this kind of time series and can capture the dynamic characteristics of pollutant odors.

[0036] S203 uses the embedded features of the first pollutant type as a query vector , respectively calculate the visual features through the attention mechanism and odor characteristics The corresponding weight is calculated as follows: , , in, For visual features The corresponding weights, Odor characteristics The corresponding weights, is the attention mechanism function; S204 Based on visual features and odor characteristics The corresponding weights will be visual features and odor characteristics Fusion is performed to obtain fusion data features , the calculation formula is as follows: .

[0037] In this step, the importance of fusing visual features and odor features varies depending on the type of pollutant. In pollutant detection tasks involving multimodal features (visual and odor), different types of pollutants rely on visual and odor features to different degrees. For example, for solid waste (such as paper scraps and plastic bottles), the visual features are very obvious (shape, outline, color), and the visual features contribute more to recognition, while the odor features may be irrelevant and have a lower weight. For liquid pollutants (such as oil stains and sewage), vision may only be able to capture the area of ​​the pollutant, while the odor features supplement its composition information (such as whether there is a foul smell). The odor features contribute more when judging the type of pollutant and the degree of pollution. In the attention mechanism, the query vector It is possible to dynamically focus on which parts of the visual and odor features are more important for this type of pollutant, improving the accuracy of the fused data.

[0038] S3 determines the second pollutant type and its pollution degree by fusing the data features, and generates a cleaning strategy corresponding to the pollutant type according to the first pollutant type and the second pollutant type.

[0039] S301 inputs the fused data features into a second pollutant detection model to obtain a second pollutant type and its pollution degree; In this step, when the fused features have been deeply encoded by CNN and LSTM and converted into a fixed-length fused vector, a multi-layer perceptron (MLP) is used as the second pollutant detection model. During training, the fused data features of the manually labeled pollution types and their pollution degrees are used as samples to train the multi-layer perceptron model.

[0040] The pollution type is labeled as one-hot encoding, and the pollution degree is obtained by calculating the proportion of polluted pixels (the ratio of the number of pixels in the polluted area to the total number of pixels in the image) through the automatic segmentation algorithm. In addition, the pollution degree marking in this step also needs to be combined with the odor intensity to standardize the collected data. For example: VOC concentration: 65 ppm (the maximum value is set to 100 ppm, and G = 0.65 after standardization), then the previous proportion of polluted pixels is weighted. For example: the proportion of polluted pixels is 0.2. For liquid pollution, the weight of odor data is 0.6, and the proportion of image data is 0.4. The final calculation is 0.4⋅0.2+0.6⋅0.65=0.47. The final classification of pollution degree is the same as step S1, and the pollution degree belongs to moderate pollution.

[0041] The input of the model is the fused multimodal features (combining visual and odor features), and the output is the pollution type (classification task) and pollution degree (regression task). Multi-task learning is achieved through joint optimization, which improves the classification accuracy and the precision of pollution degree quantification.

[0042] S302 If the first pollutant type and the second pollutant type are the same, control the operation of the cleaning mechanism according to the corresponding pollutant type and the highest pollution degree; S3021 takes the pollutant type and pollution degree as state variables, and takes the sweeping brush speed adjustment amount of the cleaning device, the sweeping brush lifting adjustment amount of the cleaning device, the water pump speed adjustment amount of the spraying device, and the fan speed adjustment amount of the sewage suction device as action variables to construct a reinforcement learning model; S3022 constructs a reward function based on the cleaning effect value and the cleaning time. When the reward value of the reward function is higher than the reward threshold, it is marked as a positive reward signal, otherwise it is marked as a negative reward signal, and the reinforcement learning model is optimized according to the reward signal; Among them, the reward function constructed by the cleaning effect value and the cleaning time includes: Acquire a rear image, a left image, and a right image at the same time point, and fuse the rear image, the left image, and the right image at the same time point to obtain a second fused image; Inputting the second fused image into the pre-trained first pollutant detection model to obtain the degree of contamination after cleaning; Degree of contamination after cleaning and the highest contamination level before cleaning Calculate the cleaning effect C, the calculation formula is as follows: ; The reward function R is constructed by the cleaning effect C and the cleaning time t. The calculation formula is as follows: , in, is the weight of the cleaning effect, The weight of the cleaning time.

[0043] S3023 inputs the corresponding pollutant type and the maximum pollution degree into the reinforcement learning model to obtain the sweeping brush speed adjustment amount of the cleaning device, the sweeping brush lifting adjustment amount of the cleaning device, the water pump speed adjustment amount of the spraying device, and the fan speed adjustment amount of the suction device.

[0044] In this step, different pollutant types and their corresponding pollution levels have different requirements for the operating parameters of the cleaning mechanism (such as sweeping brush speed, lifting height, water pump speed, etc.). At the same time, the cleaning mechanism has multiple adjustable parameters (such as sweeping brush speed, sweeping brush lifting height, water pump speed, fan speed, etc.), and there may be complex synergistic relationships between these parameters. Through the reinforcement learning model, the parameters of the cleaning device are dynamically adjusted according to the pollutant type and pollution level, so that the cleaning mechanism can optimize its operating state according to different pollution conditions, and maximize the cleaning effect and minimize the cleaning time by dynamically adjusting the action parameters.

[0045] Simply relying on cleaning effect (such as the reduction of pollution level) may ignore the efficiency problem, while simply relying on efficiency (such as cleaning time) may sacrifice the cleaning effect. Construct a reward function R based on cleaning effect and cleaning time, and use the weight and Balance the priority of cleaning effect and time. The evaluation of cleaning effect needs to rely on the comparison of the degree of contamination before and after cleaning, and the degree of contamination needs to be extracted from multi-view (back, left, right) images. Through image fusion technology, the back image, left image and right image are fused into the second fused image, which is input into the pre-trained first pollutant detection model (YOLO model) to obtain the degree of contamination after cleaning. This multi-view fusion can reduce the occlusion or omission problems that may exist in a single view and improve the accuracy of contamination assessment.

[0046] S303: If the first pollutant type and the second pollutant type are different, a mixed pollutant cleaning strategy is executed.

[0047] S3031 If the types of the mixed pollutant objects are solid waste and particulate dust, respectively, a cleaning strategy for the solid waste and a cleaning strategy for the particulate dust are obtained through a reinforcement learning model, and the cleaning strategy for the solid waste is executed first and then the cleaning strategy for the particulate dust; S3032 If the types of the mixed pollutant objects are solid waste and liquid pollutants, respectively, a cleaning strategy for the solid waste and a cleaning strategy for the liquid pollutants are obtained through a reinforcement learning model, and the cleaning strategy for the solid waste is executed first and then the cleaning strategy for the liquid pollutants; S3033 If the types of mixed pollutants are particulate dust and liquid pollutants, respectively, the cleaning strategy for particulate dust and the cleaning strategy for liquid pollutants are obtained respectively through the reinforcement learning model, and the cleaning strategy for particulate dust is executed first and then the cleaning strategy for liquid pollutants.

[0048] In this step, in actual scenarios, different types of pollutants (such as solid waste, particulate dust, and liquid pollutants) often exist at the same time, forming mixed pollution. Cleaning different types of pollutants requires different operating parameters, and different cleaning orders of mixed pollutants will directly affect the cleaning effect. For example, cleaning liquid pollutants first may cause solid waste to be compacted, increasing the difficulty of subsequent cleaning. For mixed pollutants, cleaning strategies are formulated according to the types of pollutants, and cleaning order priority rules are designed according to the physical properties of the pollutant types: Solid waste is prioritized: Because solid waste has less mutual interference with cleaning equipment, priority cleaning can avoid equipment blockage. Particulate dust is second: After cleaning the dust, the adhesion of liquid pollution can be reduced during diffusion. Liquid pollutants are last: The ground can be quickly restored to a clean state, while avoiding mixing with dust and solid waste to cause difficult-to-clean mud.

[0049] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0050] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0051] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0052] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0053] It should be noted that in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0054] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0055] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0056] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0057] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

Claims

1. A cleaning control method for a washing and sweeping vehicle based on multimodal perception, characterized in that: According to a cleaning control system of a washing and sweeping vehicle based on multimodal perception, the invention comprises: Camera groups are respectively arranged on the body of the cleaning and sweeping vehicle to collect image data of road pollutants at different angles; Odor sensor groups are installed on both sides of the washing and sweeping vehicle to monitor the odor data of road pollutants; A controller, connected to the camera group and the odor sensor group, respectively, for generating a cleaning control strategy according to the road pollution image data and the road pollution odor data; A cleaning mechanism, connected to the controller, for executing the cleaning control strategy issued by the controller; The method comprises: Acquire road surface pollutant image data at different angles, fuse the road surface pollutant image data at different angles to obtain a first fused image, and identify the first pollutant type and its pollution degree through the first fused image; Acquire road pollutant odor data, fuse the first fused image and the road pollutant odor data according to the first pollution type, and obtain fused data features; The second pollutant type and its degree of pollution are determined by fusing the data features, and a cleaning strategy corresponding to the pollutant type is generated according to the first pollutant type and the second pollutant type.

2. The cleaning control method of a washing and sweeping vehicle based on multimodal perception according to claim 1 is characterized in that: The camera group includes: The front camera is arranged at the front side of the vehicle and is used to obtain the front side image in the direction of the vehicle body; The left camera is arranged on the left rearview mirror and is used to obtain the left image of the left side of the vehicle body; The right camera is arranged on the right rearview mirror and is used to obtain the right image of the right side of the vehicle body; The rear camera is arranged at the rear of the vehicle and is used to obtain the rear image of the rear side of the vehicle.

3. A cleaning control method for a washing and sweeping vehicle based on multimodal perception according to claim 2, characterized in that: The acquiring of road surface pollutant image data at different angles, fusing the road surface pollutant image data at different angles to obtain a first fused image, and identifying the first pollutant type and its pollution degree through the first fused image includes: Acquire a front image, a left image, and a right image at the same time point, and fuse the front image, the left image, and the right image at the same time point to obtain a first fused image; The first fused image is input into a pre-trained first pollutant detection model to obtain a first pollutant type and a pollution degree thereof, wherein the pollutant types include solid waste, particulate dust, and liquid pollutants, and the pollution degrees include high pollution, medium pollution, and low pollution.

4. The cleaning control method of a washing and sweeping vehicle based on multimodal perception according to claim 3 is characterized in that: The fusing of the front image, the left image, and the right image at the same time point to obtain a first fused image comprises: The deep features of the front image, left image, and right image are extracted through the pre-trained convolutional neural network. The deep features of the front image, the left image, and the right image are weightedly fused using an attention mechanism, and the weighted fused features are reconstructed into the first fused image using a decoder.

5. The cleaning control method of a washing and sweeping vehicle based on multimodal perception according to claim 3 is characterized in that: The obtaining of the road pollutant odor data comprises fusing the first fused image and the road pollutant odor data according to the first pollution type to obtain the fused data features, including: Extract visual features of the first fused image through a pre-trained convolutional neural network ; The road pollution odor data is input into the long short-term memory network model to extract the odor characteristics. ; Use the embedded features of the first pollutant type as the query vector , respectively calculate the visual features through the attention mechanism and odor characteristics The corresponding weight is calculated as follows: ， ， in, For visual features The corresponding weight, Odor characteristics The corresponding weight, is the attention mechanism function; Based on visual features and odor characteristics The corresponding weights will be visual features and odor characteristics Fusion is performed to obtain fusion data features , the calculation formula is as follows: 。 6. The cleaning control method of a washing and sweeping vehicle based on multimodal perception according to claim 5 is characterized in that: The cleaning mechanism comprises: Cleaning devices are respectively arranged on the left and right sides of the cleaning and sweeping vehicle, and are used to clean and / or wash road pollutants; The sewage suction device is respectively arranged on the left and right sides of the cleaning and sweeping vehicle and the rear side of the cleaning device, and is used to absorb the pollutants into the garbage bin loaded on the cleaning vehicle; The spraying devices are respectively arranged on the left and right sides of the washing and sweeping vehicle and in front of the cleaning device, and are used for spraying water to soften and dilute pollutants.

7. The cleaning control method of a washing and sweeping vehicle based on multimodal perception according to claim 6 is characterized in that: The determining of the second pollutant type and its degree of pollution by fusing the data features, and generating a cleaning strategy corresponding to the pollutant type according to the first pollutant type and the second pollutant type include: Inputting the fused data features into a second pollutant detection model to obtain the second pollutant type and its pollution degree; If the first pollutant type and the second pollutant type are the same, the operation of the cleaning mechanism is controlled according to the corresponding pollutant type and the maximum pollution degree; If the first pollutant type and the second pollutant type are different, a mixed pollutant cleaning strategy is performed.

8. The cleaning control method of a washing and sweeping vehicle based on multimodal perception according to claim 7 is characterized in that: If the first pollutant type and the second pollutant type are the same, controlling the operation of the cleaning mechanism according to the corresponding pollutant type and the highest pollution degree includes: The pollutant type and pollution degree are taken as state variables, and the sweeping brush speed adjustment amount of the cleaning device, the sweeping brush lifting adjustment amount of the cleaning device, the water pump speed adjustment amount of the spraying device, and the fan speed adjustment amount of the sewage suction device are taken as action variables to construct a reinforcement learning model. A reward function is constructed by the cleaning effect value and the cleaning time. When the reward value of the reward function is higher than the reward threshold, it is marked as a positive reward signal, otherwise it is marked as a negative reward signal. The reinforcement learning model is optimized according to the reward signal. The corresponding pollutant type and the maximum pollution degree are input into the reinforcement learning model to obtain the sweeping brush speed adjustment amount of the cleaning device, the sweeping brush lifting adjustment amount of the cleaning device, the water pump speed adjustment amount of the spraying device, and the fan speed adjustment amount of the suction device.

9. The cleaning control method of a washing and sweeping vehicle based on multimodal perception according to claim 8, characterized in that: The reward function constructed by the cleaning effect and the cleaning time includes: Acquire a rear image, a left image, and a right image at the same time point, and fuse the rear image, the left image, and the right image at the same time point to obtain a second fused image; Inputting the second fused image into the pre-trained first pollutant detection model to obtain the degree of contamination after cleaning; Degree of contamination after cleaning and the highest contamination level before cleaning Calculate the cleaning effect C, the calculation formula is as follows: ； The reward function R is constructed by the cleaning effect C and the cleaning time t. The calculation formula is as follows: ， in, is the weight of the cleaning effect, The weight of the cleaning time.

10. The cleaning control method of a washing and sweeping vehicle based on multimodal perception according to claim 9, characterized in that: If the first pollutant type and the second pollutant type are different, the pollutant type is marked as a mixed pollutant, and the mixed pollutant cleaning strategy is executed including: If the types of mixed pollutants are solid waste and particulate dust, respectively, the cleaning strategies of solid waste and particulate dust are obtained through the reinforcement learning model, and the cleaning strategy of solid waste is executed first and then the cleaning strategy of particulate dust; If the types of mixed pollutants are solid waste and liquid pollutants, respectively, the cleaning strategies of solid waste and liquid pollutants are obtained through the reinforcement learning model, and the cleaning strategy of solid waste is executed first and then the cleaning strategy of liquid pollutants; If the types of mixed pollutants are particulate dust and liquid pollutants, respectively, the cleaning strategies for particulate dust and liquid pollutants are obtained respectively through the reinforcement learning model, and the cleaning strategy for particulate dust is executed first and then the cleaning strategy for liquid pollutants.

Images