Intelligent pollution cleaning method and system based on multi-modal deep learning

Abstract

The application provides an intelligent pollution cleaning method based on multi-modal deep learning, real-time monitoring of a pollution accumulation area is performed, and a real-time picture of a trash rack area is transmitted, a pollution detection data set is generated, and a pollution recognition algorithm based on deep learning is trained, the pollution recognition algorithm is used to recognize pollution in the real-time picture of the trash rack area and mark position information of the pollution, a multi-modal deep learning model is established to determine whether a pollution cleaning device is started, and factors such as an area of the pollution accumulation area, a water level difference of an inlet, weather conditions and rainfall are comprehensively considered, when an output result of the model reaches a preset value, it is determined that the pollution cleaning device starting requirement is met, through intelligent identification, automatic operation and informationized collaboration of the whole process of the hydropower station pollution cleaning system, the application improves the pollution cleaning efficiency of the trash rack of the hydropower station inlet and the reliability of the pollution cleaning device, and promotes the intelligent leap of the pollution cleaning system from passive response to active sensing.

Description

Technical Field

[0001] This invention relates to the field of hydropower station cleaning equipment technology, specifically an intelligent cleaning method and system based on multimodal deep learning. Background Technology

[0002] During the operation of a hydroelectric power station, the intake trash rack area is prone to accumulating large amounts of floating debris such as branches, tall crops, and wood. The main function of the trash rack is to intercept such large or long debris, preventing it from entering the turbine unit and causing equipment damage. At the same time, the continuous accumulation of debris in front of the rack can lead to blockage, resulting in a significant water level difference before and after the rack, which reduces the effective head of the unit and severely reduces the power generation efficiency of the hydroelectric power station.

[0003] To ensure the safety of the generating unit's head and power generation, hydropower stations must promptly remove debris from in front of the trash racks. Currently, the mainstream trash rack removal operation relies heavily on manual operation: operators must use trash racks at the unit's intake to retrieve debris and transport it to a designated waste disposal site. Furthermore, existing trash rack removal systems have limited intelligence and automation, often relying on mechanical timed triggering or pressure sensor threshold judgments for decision-making. These traditional methods have the following drawbacks: 1. Manual dredging and cleaning operations are time-consuming and labor-intensive, and the efficiency of cleaning is limited by the experience of personnel and environmental factors.

[0004] 2. A single decision-making mechanism based on mechanical timing or pressure sensors lacks real-time and accurate perception of the area of ​​dirt accumulation, resulting in unnecessary losses in the power generation efficiency of the unit.

[0005] 3. The level of intelligence is insufficient, and it is unable to make adaptive decisions based on complex factors such as the area of ​​sewage accumulation, the water level difference at the inlet, and the amount of rainfall.

[0006] In recent years, deep learning algorithms have made significant progress in fields such as smart industry, object detection, healthcare, and natural language processing. Applying deep learning algorithms to the field of hydropower station cleaning equipment to achieve automatic detection and identification of pollutants is of great significance for improving the operational efficiency and intelligence level of the cleaning system, and reducing the operation and maintenance costs and energy efficiency losses of hydropower stations.

[0007] Existing cleaning and decontamination systems still have shortcomings in real-time processing, detection accuracy, and intelligent decision-making, making it difficult to meet the actual needs of efficient and safe operation of hydropower stations.

[0008] Therefore, to address the aforementioned shortcomings, the development of an intelligent cleaning system and method based on multimodal deep learning has significant economic value and importance for improving power generation efficiency, ensuring operational safety, and reducing operation and maintenance costs in hydropower stations. Summary of the Invention

[0009] This invention provides an intelligent trash rack cleaning method and system based on multimodal deep learning. By realizing intelligent identification, automated operation and information-based collaboration of the entire process of the trash rack cleaning system in hydropower stations, it improves the cleaning efficiency of the trash rack at the water intake of hydropower stations and the reliability of the trash rack cleaning equipment, and promotes the intelligent leap of the trash rack cleaning system from passive response to active perception, so as to solve the problems in the background technology.

[0010] To achieve the above objectives, the technical solution of the present invention is as follows: A smart cleaning method based on multimodal deep learning includes the following steps: S1, Preliminary preparation: Install industrial cameras in the trash rack area at the intake of the hydropower station to monitor the area of ​​accumulated waste in real time and transmit real-time images of the trash rack area. S2, Model Training: Collect images of water surface debris from different perspectives in real-time footage of the debris barrier area, generate a debris detection dataset, and train a deep learning-based debris recognition algorithm. S3, dirt recognition, transmits real-time images of the trash rack area captured by industrial cameras to an IoT edge computer, and uses a dirt recognition algorithm to identify dirt in the real-time images of the trash rack area and mark its location information. S4, Multimodal Fusion Decision, establishes a multimodal deep learning model to determine whether the cleaning equipment should be started. It comprehensively considers factors such as the area of ​​the sewage accumulation area, the water level difference at the inlet, weather conditions and rainfall. When the model output reaches the preset value, it determines that the requirements for starting the cleaning equipment are met.

[0011] Preferably, the specific implementation steps of S2 are as follows: S21, Collect images of waste samples. When the intake of the hydropower station is open, take pictures of the trash rack area and images of different degrees of waste stacking. S22, Create a dirt dataset, using LabelImg as sample images, label the category and location information of dirt, and generate corresponding label files; S23, Dataset Augmentation, performs random scaling, offsetting, cropping, and combining on labeled images; S24, Training the model: Input the dirt detection dataset into the dirt recognition algorithm, train the algorithm network to detect different dirt, and output the dirt recognition algorithm after training is completed.

[0012] Preferably, the specific implementation steps of S4 are as follows: S41, after receiving information on the type and location of dirt in the image through a dirt recognition algorithm; Real-time calculation of the area of ​​waste accumulation ;like For areas with low-risk waste accumulation, the cleaning equipment does not need to be started; if For areas with high-risk waste accumulation, the cleaning equipment will immediately begin cleaning operations; among which... and This indicates the threshold range for low-risk waste accumulation areas. and A threshold representing the area of ​​low-risk waste accumulation; x represents the variable that affects the multimodal model. This indicates an accumulation of filth. This indicates the area of ​​accumulated waste; S42, the IoT edge computer collects water level sensor data to obtain the water level difference at the hydropower station's intake. ;like This corresponds to a low-risk water level difference range. The cleaning equipment does not need to be started. This corresponds to a high-risk water level difference range. The cleaning equipment immediately began cleaning operations; S43, taking into account weather conditions Rainfall The impact of high water season, normal water season and low water season, to establish the first Multimodal model of a trash rack inlet ; S44, when multimodal model When the corresponding warning signal is received, it indicates that the current multimodal model has reached the warning threshold. Within the specified range, the cleaning machine does not need to be started; if This corresponds to a warning signal, indicating that the current multimodal model has reached the warning threshold. Within the specified range, if the requirements for starting the cleaning equipment are met, the IoT edge computer sends an alarm signal to the cleaning intelligent control module and the programmable logic controller.

[0013] Preferably, establish the first Multimodal model of a trash rack inlet The process is as follows:

[0014] in, These represent the visual modality, hydrological modality, meteorological modality, and rainfall modality, respectively, and are thus the variables. The weighting coefficients must satisfy the following conditions: , and, through Set the hydrological mode to the highest priority. This is a seasonal bias term, derived from the effects of the wet season, normal season, and dry season, and calculated by weighting the rainfall amounts in different seasons.

[0015] Preferably, if the result of S4 indicates that the cleaning equipment can be started, then the following step S5 is executed: S51, after receiving a signal that the cleaning and pollution control module determines that the cleaning equipment can be started (i.e., an alarm signal), it immediately activates the audible and visual alarm lights and the intelligent positioning module, records the alarm time in real time, and provides the location information of the trash rack containing the dirt to the intelligent path algorithm. S52, the intelligent path algorithm combines the location of the cleaning equipment to generate the optimal cleaning operation sequence, and transmits the operation instruction to the programmable logic controller; S53, after receiving alarm signals and work instructions, activates the frequency converter anti-sway module; S54, the cleaning equipment moves to the orifice area to carry out cleaning operations, and the rake bucket moves to grab the dirt; S55, the cleaning equipment is moved to the sewage discharge area to perform sewage unloading operations; S56, the cleaning equipment returns to its initial position.

[0016] Preferably, the construction process of the intelligent path algorithm in S52 includes: Construct a linear layout space model, and set the unloading position as the origin coordinate of the cleaning machine. The trash racks are distributed equidistantly on one side of the origin along the vertical direction. The coordinates of each trash rack are obtained based on the trash rack location information. Then the first The actual physical distance from the trash rack to the origin is: :

[0017] A higher value indicates a more severe blockage and a greater urgency for cleaning. Constructing a weighted distance for multimodal perception This reflects the impact of pollution cleanup priority on route planning:

[0018] Among them, attenuation factor , which is a preset value, output by the multimodal model. Decision, and and Negative correlation The larger, The smaller, so when When it is larger, The smaller the value, the closer the algorithm perceives the trash rack in path planning, guiding the cleaning equipment to access it first; finally, based on the weighted distance... The Held-Karp dynamic programming algorithm is called to generate the optimal sequence of cleaning and decontamination operations.

[0019] Preferably, the water surface debris image in S1 is a typical water surface debris image, including lidded cans, plastic water bottles, and weeds.

[0020] A multimodal deep learning-based intelligent cleaning system, applicable to any of the above-mentioned intelligent cleaning methods, includes: a local control module, a frequency conversion anti-sway module, a data acquisition module, an intelligent positioning module, a dirt intelligent identification module, and a cleaning intelligent control module.

[0021] Preferably, the data acquisition module includes an industrial camera for capturing images of the trash rack area, a sensor for acquiring water level data, and an ultrasonic flow meter for acquiring real-time water flow data. The intelligent waste identification module includes a waterborne garbage dataset for training the waste identification model, a deep learning algorithm for real-time detection and data analysis, and an intelligent path algorithm for generating the optimal waste removal operation sequence. The intelligent pollution control module includes: a local platform for real-time edge computing and precise control, and a remote platform for global monitoring and strategy optimization; The intelligent positioning module includes: a lidar for positioning, an absolute encoder for precise positioning, and a wire displacement sensor for positioning the lifting device. The local control module includes: a programmable logic controller for providing hardware support and a local human-machine interface; an industrial network switch for data exchange and remote communication; and an IoT edge computer for deploying multimodal deep learning algorithms. The variable frequency anti-sway module includes: a motor for powering the cleaning equipment, a frequency converter for adjusting the operating status of the cleaning equipment, and an anti-sway algorithm for improving the stability of the hook or load.

[0022] Preferably, the connection relationships between the modules of the system are as follows: The data acquisition module outputs environmental perception data such as images of the trash rack area, water level, and water flow. The intelligent positioning module outputs real-time location data of the cleaning equipment. These two types of data are transmitted to the intelligent dirt identification module and the frequency converter anti-sway module, respectively. The intelligent dirt identification module generates dirt detection results and the optimal cleaning operation strategy, which are transmitted to the intelligent cleaning control module. The intelligent cleaning control module judges the operation status and issues control commands to the local control module. The local control module converts the control commands into executable signals and also undertakes data exchange and remote communication adaptation between modules. The signals are transmitted to the frequency converter anti-sway module. After the frequency converter anti-sway module performs the cleaning operation, it feeds back the real-time operating status data of the equipment to the intelligent cleaning control module.

[0023] As can be seen from the above technical solution compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs an intelligent closed loop covering the entire process of data acquisition, analysis and decision-making, scheduling and control, and execution feedback, thereby achieving automated operation and remote monitoring of cleaning and decontamination operations and significantly reducing the risk of accidents such as personnel falling into water, mechanical injuries, and falls from heights during cleaning and decontamination operations.

[0024] 2. This invention collects multi-dimensional environmental data such as images of the trash rack area, water level, and water flow, as well as real-time location data of the cleaning equipment. It combines multi-modal deep learning algorithms to detect contaminants and plan the optimal cleaning operation sequence, quickly responding to blockage problems and accurately removing blockages from the trash rack. This significantly reduces the time of blockage at the inlet, reduces water waste, and directly increases the power generation per unit flow of water. Especially during the peak of floating debris in the high-water season, it significantly improves power generation efficiency and economic benefits. 3. This invention reduces impact wear and accidental damage to key components of cleaning equipment through frequency conversion anti-sway control, and optimizes the operation process by combining intelligent path planning algorithm, making cleaning operations more energy-efficient, extending equipment service life, reducing the frequency of spare parts replacement and maintenance, and achieving a simultaneous reduction in operation and maintenance costs and energy consumption costs. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the method steps of the present invention; Figure 2 This is a flowchart of the training process for the deep learning-based dirt identification algorithm of this invention; Figure 3 This is a schematic diagram of the multimodal fusion decision-making process of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0027] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention, but should not be used to limit the scope of the present invention.

[0028] This invention provides an intelligent cleaning system based on multimodal deep learning, comprising: Local control module, frequency converter anti-sway module, data acquisition module, intelligent positioning module, intelligent dirt identification module, and intelligent cleaning and control module; The local control module provides programmable logic controllers with hardware support and local human-machine interface, industrial network switches for data exchange and remote communication, and IoT edge computers for deploying multimodal deep learning algorithms. The variable frequency anti-sway module includes: a motor for providing power to the cleaning equipment, a frequency converter for adjusting the operating status of the cleaning equipment, and an anti-sway algorithm for improving the stability of the hook or load; The data acquisition module includes: an industrial camera for capturing images of the trash rack area, a sensor for collecting water level data, and an ultrasonic flow meter for collecting real-time water flow data. Intelligent positioning module: LiDAR for coarse positioning of large or small vehicles, absolute encoder for precise positioning of large / small vehicles, and wire displacement sensor for positioning of lifting device; Intelligent waste identification module: A dataset of marine debris used to train the waste identification model, a deep learning algorithm for real-time detection and data analysis, and an intelligent path algorithm for generating the optimal waste removal operation sequence; Cleaning and pollution control module: a local platform for real-time edge computing and precise control, and a remote platform for global monitoring and strategy optimization.

[0029] Example: In this embodiment, when the cleaning equipment in the hydropower station is running normally, when the programmable logic controller is in fully automatic mode and receives the equipment start signal, the audible and visual alarm lights of the cleaning equipment sound, and the automatic cleaning operation is performed 30 seconds later.

[0030] Then the cleaning equipment runs to the orifice, the rake bucket opens and descends to the underload or lower limit stop, the rake bucket closes to grab the dirt, the rake bucket rises to the upper limit, runs to the designated dirt discharge position, the rake bucket opens to discharge the dirt, the cleaning machine runs to the orifice, cleans the orifice in sequence according to the above process, and returns to the parking position; To improve the intelligence and informatization of the aforementioned cleaning and decontamination equipment and reduce the operation and maintenance costs and safety risks of hydropower stations, an intelligent cleaning and decontamination system and method based on multimodal deep learning is proposed, such as... Figure 1 As shown, it includes the following steps: S1. Preliminary preparations: Based on the existing cleaning equipment of the hydropower station, industrial cameras will be installed in the trash rack area at the water intake of the hydropower station to monitor the area where the trash is piled up in real time and transmit real-time images of the trash rack area at regular intervals.

[0031] S2, Model Training, such as Figure 2 As shown: S21, Collect images of waste samples: When the intake of the hydropower station is open, take pictures of the trash rack area and images of different degrees of waste stacking. To meet actual needs, the number of images should be no less than 2,000.

[0032] S22, Create a dirt dataset: Use LabelImg to label the sample images with the category and location information of dirt, and generate the corresponding label files.

[0033] S23, Dataset Augmentation: Random scaling, offsetting, cropping, and combining of labeled images improves dataset diversity and model robustness.

[0034] S24, Training the model: Input the dirt detection dataset into the recognition algorithm and train the algorithm network to detect different types of dirt.

[0035] S3. Dirt Detection: Video data captured by industrial cameras is transmitted to an IoT edge computer, where a locally deployed dirt recognition algorithm annotates the images and outputs the results.

[0036] S4, Multimodal fusion decision-making, such as Figure 3 As shown: S41, after receiving information on the type and location of the contaminant, the IoT edge computer calculates the area of ​​the contaminant accumulation zone in real time. ;like For areas with low-risk waste accumulation, the cleaning equipment does not need to be started; if If the area corresponds to a high-risk area of ​​waste accumulation, the cleaning equipment will immediately begin cleaning operations. in, and This indicates the threshold range for low-risk waste accumulation areas. and This represents the threshold range for low-risk waste accumulation area, and x represents the variables affecting the multimodal model. This indicates an accumulation of filth. This indicates the area where dirt and filth accumulate.

[0037] S42, the IoT edge computer collects water level sensor data to obtain the water level difference at the hydropower station's intake. ;like This corresponds to a low-risk water level difference range. The cleaning equipment does not need to be started. This corresponds to a high-risk water level difference range. The cleaning equipment immediately began cleaning operations; S43, taking into account weather conditions Rainfall The impact of high water season, normal water season and low water season, to establish the first Multimodal model of a trash rack inlet ; Establish the first Multimodal model of a trash rack inlet The process is as follows:

[0038] in, These represent the visual modality, hydrological modality, meteorological modality, and rainfall modality, respectively, and are thus the variables. The weighting coefficients must satisfy the following conditions: , and, through Set the hydrological mode to the highest priority. This is a seasonal bias term, derived from the effects of the wet season, normal season, and dry season, and calculated by weighting the rainfall amounts in different seasons.

[0039] .

[0040] S44, when multimodal model When the corresponding warning signal is received, it indicates that the current multimodal model has reached the warning threshold. Within the specified range, the cleaning machine does not need to be started; if This corresponds to a warning signal, indicating that the current multimodal model has reached the warning threshold. Within the specified range, if the requirements for starting the cleaning equipment are met, the IoT edge computer sends an alarm signal to the cleaning intelligent control module and the programmable logic controller.

[0041] S5. Operation of cleaning equipment: Upon receiving an alarm signal, the S51 intelligent cleaning and pollution control module immediately activates the audible and visual alarm lights and the intelligent positioning module on the local platform. The remote platform records the alarm time in real time and pushes the alarm information and sensor data to the administrator terminal.

[0042] S52, the intelligent path algorithm generates the optimal cleaning operation sequence based on the location of the cleaning equipment, and transmits the operation instruction to the programmable logic controller.

[0043] The process of the intelligent path algorithm includes: The construction process of intelligent path algorithms includes: Construct a linear layout space model, and set the unloading position as the origin coordinate of the cleaning machine. The trash racks are distributed equidistantly on one side of the origin along the vertical direction. The coordinates of each trash rack are obtained based on the trash rack location information. Then the first The actual physical distance from the trash rack to the origin is: :

[0044] A higher value indicates a more severe blockage and a greater urgency for cleaning. Constructing a weighted distance for multimodal perception This reflects the impact of pollution cleanup priority on route planning:

[0045] Among them, attenuation factor , which is a preset value, output by the multimodal model. The value can be determined; for example, in this embodiment, it can be 0.8, and and Negative correlation The larger, The smaller, so when When it is larger, The smaller the value, the closer the algorithm perceives the trash rack in path planning, guiding the cleaning equipment to access it first; finally, based on the weighted distance... The Held-Karp dynamic programming algorithm is called to generate the optimal sequence of cleaning and decontamination operations.

[0046] The intelligent path algorithm uses dynamic programming to efficiently search all feasible sub-paths and combines multimodal perception with weighted distance to achieve an optimal balance between priority response and path length.

[0047] The S53 programmable logic controller activates the frequency converter anti-sway module after receiving alarm signals and operation instructions.

[0048] S54, the cleaning equipment moves to the orifice area to carry out cleaning operations, and the rake bucket moves to grab the dirt.

[0049] S55, the cleaning equipment is operated to the sewage discharge area to unload sewage.

[0050] S56, the cleaning equipment returns to its initial position.

[0051] The intelligent cleaning system and method based on multimodal deep learning achieves high accuracy, stability, and real-time performance in debris identification, while providing reliable decision-making basis for the operation of cleaning equipment at the intake of hydropower stations.

[0052] This intelligent cleaning system has low operation and maintenance costs and is easy to maintain. It breaks through the traditional cleaning bottlenecks with technological innovation and injects new momentum into the green and low-carbon development of the hydropower industry.

[0053] This method, through intelligent identification, automated execution, and information-based collaboration, meets the working requirements of the hydropower industry for "unmanned operation and minimal staffing."

[0054] The above descriptions of the embodiments are merely for the purpose of helping to understand the methods and core ideas of the embodiments of the present invention; at the same time, those skilled in the art will recognize that, based on the ideas of the embodiments of the present invention, there will be changes in specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation on the embodiments of the present invention.

[0055] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0056] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0057] For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media.

[0058] The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0059] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0060] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0061] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0062] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A smart cleaning method based on multimodal deep learning, characterized in that, Includes the following steps: S1, Preliminary preparation: Install industrial cameras in the trash rack area at the intake of the hydropower station to monitor the area of ​​accumulated waste in real time and transmit real-time images of the trash rack area. S2, Model Training: Collect images of water surface debris from different perspectives in real-time footage of the debris barrier area, generate a debris detection dataset, and train a deep learning-based debris recognition algorithm. S3, dirt recognition, transmits real-time images of the trash rack area captured by industrial cameras to an IoT edge computer, and uses a dirt recognition algorithm to identify dirt in the real-time images of the trash rack area and mark its location information. S4, Multimodal Fusion Decision, establishes a multimodal deep learning model to determine whether the cleaning equipment should be started. It comprehensively considers factors such as the area of ​​the sewage accumulation area, the water level difference at the inlet, weather conditions and rainfall. When the model output reaches the preset value, it determines that the requirements for starting the cleaning equipment are met.

2. The intelligent cleaning method based on multimodal deep learning as described in claim 1, characterized in that: The specific implementation steps of S2 are as follows: S21, Collect images of waste samples. When the intake of the hydropower station is open, take pictures of the trash rack area and images of different degrees of waste stacking. S22, Create a dirt dataset, using LabelImg as sample images, label the category and location information of dirt, and generate corresponding label files; S23, Dataset Augmentation, performs random scaling, offsetting, cropping, and combining on labeled images; S24, Training the model: Input the dirt detection dataset into the dirt recognition algorithm, train the algorithm network to detect different dirt, and output the dirt recognition algorithm after training is completed.

3. The intelligent cleaning method based on multimodal deep learning as described in claim 1, characterized in that: The specific implementation steps of S4 are as follows: S41, after receiving information on the type and location of dirt in the image through a dirt recognition algorithm; Real-time calculation of the area of ​​waste accumulation ;like For areas with low-risk waste accumulation, the cleaning equipment does not need to be started; if For areas with high-risk waste accumulation, the cleaning equipment will immediately begin cleaning operations; among which... and This indicates the threshold range for low-risk waste accumulation areas. and A threshold representing the area of ​​low-risk waste accumulation; x represents the variable that affects the multimodal model. This indicates an accumulation of filth. This indicates the area of ​​accumulated waste; S42, the IoT edge computer collects water level sensor data to obtain the water level difference at the hydropower station's intake. ;like This corresponds to a low-risk water level difference range. The cleaning equipment does not need to be started. This corresponds to a high-risk water level difference range. The cleaning equipment immediately began cleaning operations; S43, taking into account weather conditions Rainfall The impact of high water season, normal water season and low water season, to establish the first Multimodal model of a trash rack inlet ; S44, when multimodal model When the corresponding warning signal is received, it indicates that the current multimodal model has reached the warning threshold. Within the specified range, the cleaning machine does not need to be started; if This corresponds to a warning signal, indicating that the current multimodal model has reached the warning threshold. Within the specified range, if the requirements for starting the cleaning equipment are met, the IoT edge computer sends an alarm signal to the cleaning intelligent control module and the programmable logic controller.

4. The intelligent cleaning method based on multimodal deep learning as described in claim 3, characterized in that: The establishment of the first Multimodal model of a trash rack inlet The process is as follows: in, These represent the visual modality, hydrological modality, meteorological modality, and rainfall modality, respectively, and are thus the variables. The weighting coefficients must satisfy the following conditions: , and, through Set the hydrological mode to the highest priority. This is a seasonal bias term, derived from the effects of the wet season, normal season, and dry season, and calculated by weighting the rainfall amounts in different seasons.

5. The intelligent cleaning method based on multimodal deep learning as described in claim 4, characterized in that: If the result of step S4 indicates that the cleaning equipment can be started, then the following step S5 is executed: S51, after receiving a signal that the cleaning and pollution control module determines that the cleaning equipment can be started (i.e., an alarm signal), it immediately activates the audible and visual alarm lights and the intelligent positioning module, records the alarm time in real time, and provides the location information of the trash rack containing the dirt to the intelligent path algorithm. S52, the intelligent path algorithm combines the location of the cleaning equipment to generate the optimal cleaning operation sequence, and transmits the operation instruction to the programmable logic controller; S53, after receiving alarm signals and work instructions, activates the frequency converter anti-sway module; S54, the cleaning equipment moves to the orifice area to carry out cleaning operations, and the rake bucket moves to grab the dirt; S55, the cleaning equipment is moved to the sewage discharge area to perform sewage unloading operations; S56, the cleaning equipment returns to its initial position.

6. The intelligent cleaning method based on multimodal deep learning as described in claim 5, characterized in that: The construction process of the intelligent path algorithm in S52 includes: Construct a linear layout space model, and set the unloading position as the origin coordinate of the cleaning machine. The trash racks are distributed equidistantly on one side of the origin along the vertical direction. The coordinates of each trash rack are obtained based on the trash rack location information. Then the first The actual physical distance from the trash rack to the origin is: : A higher value indicates a more severe blockage and a greater urgency for cleaning. Constructing a weighted distance for multimodal perception This reflects the impact of pollution cleanup priority on route planning: Among them, attenuation factor , which is a preset value, output by the multimodal model. Decision, and and Negative correlation The larger, The smaller, so when When it is larger, The smaller the value, the closer the algorithm perceives the trash rack in path planning, guiding the cleaning equipment to access it first; finally, based on the weighted distance... The Held-Karp dynamic programming algorithm is called to generate the optimal sequence of cleaning and decontamination operations.

7. The intelligent cleaning method based on multimodal deep learning as described in claim 1, characterized in that: The water surface debris image in S1 is a typical water surface debris image, including lidded cans, plastic water bottles, and weeds.

8. A multimodal deep learning-based intelligent cleaning system applied to the intelligent cleaning method based on multimodal deep learning as described in any one of claims 1 to 7, characterized in that, include: The system includes a local control module, a frequency converter anti-sway module, a data acquisition module, an intelligent positioning module, a waste intelligent identification module, and a waste cleaning and control module.

9. The intelligent cleaning system based on multimodal deep learning as described in claim 8, characterized in that: The data acquisition module includes an industrial camera for capturing images of the trash rack area, a sensor for collecting water level data, and an ultrasonic flow meter for collecting real-time water flow data. The intelligent waste identification module includes a waterborne waste dataset for training the waste identification model, a deep learning algorithm for real-time detection and data analysis, and an intelligent path algorithm for generating the optimal waste removal operation sequence. The intelligent pollution control module includes: a local platform for real-time edge computing and precise control, and a remote platform for global monitoring and strategy optimization; The intelligent positioning module includes: a lidar for positioning, an absolute encoder for precise positioning, and a wire displacement sensor for positioning the lifting device. The local control module includes: a programmable logic controller for providing hardware support and a local human-machine interface, an industrial network switch for data exchange and remote communication, and an IoT edge computer for deploying multimodal deep learning algorithms. The variable frequency anti-sway module includes: a motor for providing power to the cleaning equipment, a frequency converter for adjusting the operating status of the cleaning equipment, and an anti-sway algorithm for improving the stability of the hook or load.

10. The intelligent cleaning system based on multimodal deep learning as described in claim 9, characterized in that: The connection relationships between the modules of the system are as follows: The data acquisition module outputs environmental perception data such as images of the trash rack area, water level, and water flow. The intelligent positioning module outputs real-time location data of the cleaning equipment. These two types of data are transmitted to the intelligent dirt identification module and the frequency converter anti-sway module, respectively. The intelligent dirt identification module generates dirt detection results and the optimal cleaning operation strategy, which are transmitted to the intelligent cleaning control module. The intelligent cleaning control module judges the operation status and issues control commands to the local control module. The local control module converts the control commands into executable signals and also undertakes data exchange and remote communication adaptation between modules. The signals are transmitted to the frequency converter anti-sway module. After the frequency converter anti-sway module performs the cleaning operation, it feeds back the real-time operating status data of the equipment to the intelligent cleaning control module.

Images