Water delivery trunk canal ice-free water delivery method and system based on machine vision and deep learning

Abstract

The invention discloses a water delivery trunk canal ice-free water delivery method based on machine vision and deep learning. The method comprises the following steps: step 1, installing a fixed machine vision optical system at a multi-state channel part which needs to be identified under a multi-ice-period water delivery condition, and enabling an image of each target to be within a calibration camera vision field range; 2, identifying the water flow velocity and the wind speed according to the motion mode scene through a machine vision optical system; meanwhile, the gate opening degree, the water level, the water temperature, the air temperature and the initial ice condition are recognized according to the edge and identification situations; and 3, analyzing the content identified by the machine vision optical system by adopting a deep learning unit, and finally realizing ice-free water delivery. The system has the advantages of no contact, high precision, real-time sensing, low cost, no influence on water quality and deep intelligence. The invention further discloses a water delivery trunk canal ice-free water delivery system based on machine vision and deep learning.

Description

Technical Field

[0001] The present invention relates to the fields of water conservancy, water services, and artificial intelligence. Specifically, it relates to a method for achieving ice-free water transfer in long-distance main canals during winter using multi-state variable machine vision and deep learning. More specifically, it relates to a method for achieving ice-free water transfer in main canals using machine vision and deep learning. The present invention also relates to an ice-free water transfer system for main canals employed in this method based on machine vision and deep learning. Background Art

[0002] Main water supply canals, especially artificial ones, are characterized by their wide channels and long distances. During ice age, water supply efficiency is approximately half of its normal level. In practice, due to factors such as flow rate, ice formation is difficult, making it prone to ice jams and ice dams, resulting in significantly lower water delivery rates. Furthermore, freezing of main water supply canals poses serious safety risks to control facilities such as gates, and can even cause permanent damage.

[0003] Therefore, it is necessary to develop a method to achieve ice-free water delivery in water supply canals. Summary of the Invention

[0004] The first purpose of the present invention is to provide a method for ice-free water delivery in water delivery main channels based on machine vision and deep learning. It is a method for achieving ice-free water delivery in long-distance main channels in winter through multi-state variable machine vision and deep learning. The method uses machine vision to identify the key state quantities of factors affecting the freezing of the main channels (such as wind speed, water temperature, air temperature and initial ice critical state, gate opening, water level, hydrodynamics, etc.), and uses machine deep learning to discover and master the freezing critical points of the main channels under various conditions. A new machine vision and AI algorithm is built to achieve self-decision-making, and ultimately achieve ice-free water delivery. The present invention can not only identify and learn the various state trends and influencing changes of water delivery in the main channels in winter through artificial intelligence (mainly identifying states (multi-variables) and obtaining state change trends through deep learning), but also has contactless It has the characteristics of touch, high precision, real-time perception, low cost (one map for multiple uses, video partitioning), no impact on water quality, and deep intelligence. It controls adjustable variables through input variables, discovers and masters the freezing critical points of the main channel under various conditions through machine deep learning, and realizes self-decision-making with artificial intelligence algorithms. With the help of mobile or permanent ice-free water transfer guarantee facilities and equipment dedicated to the main channel, it can perceive in time, respond quickly, make intelligent decisions, and deploy quickly to provide guarantees for ice-free water transfer, so that long-distance open water transfer main channels can adopt non-ice cover water transfer methods in winter, thereby reducing the factors and risks such as freezing, icicles, and ice damage that threaten the main channel facilities and water supply safety, which can greatly increase the water transfer safety, engineering safety, and water quality safety performance of long-distance main channels during the ice period, thereby greatly enhancing the winter water transfer capacity of the main channel.

[0005] The second purpose of the present invention is to provide an ice-free water delivery system adopted by an ice-free water delivery method for water delivery main channels based on machine vision and deep learning. It is a system for achieving ice-free water delivery in water delivery main channels through machine vision and deep learning. It is based on open water delivery and water diversion main channels, and utilizes the online recognition and reading of multi-state variables of the main channels by machine vision methods. Combined with machine deep learning, it accurately adjusts the controllable state variables on long-distance main channels in real time through algorithms and training, so that the long-distance main channels are always in a critical state before freezing in winter.

[0006] In order to achieve the first purpose of the present invention, the technical solution of the present invention is: a method for ice-free water delivery in a water delivery main channel based on machine vision and deep learning, comprising the following steps: Step 1: Install a fixed machine vision optical system at the location where multi-state channels (i.e., water delivery main channels, which are open water delivery channels, are generally artificial concrete lined channels with no cover on top. Both sides of the lined channels are generally provided with horse trails (concrete or asphalt pavement) for operation inspection and transportation and maintenance equipment. Buffer protection forest belts are generally provided on both sides of the horse trails for closed management and ensuring water quality safety) that need to be identified, and align the machine vision optical system with a front camera or a front side camera on the river surface including multiple targets such as gates, channels, water surfaces, temperature measurement, and wind measurement, so that the image of each target is within the field of view of the calibrated camera; Step 2: Use the machine vision optical system to calibrate the water flow according to the motion pattern scenario. The system can identify the speed and wind speed (rotating impeller speed), and for edge and marked scenarios, identify the gate opening, water level, water temperature (thermometer scale), air temperature (thermometer scale), initial ice conditions, etc.; including water flow rate (water flow texture recognition), gate opening (gate position recognition), instant wind speed (rotating anemometer speed recognition), water temperature (thermometer scale recognition), air temperature (thermometer scale recognition), and initial ice critical state (ice flower and ice floc recognition). Step three: Use deep learning units to analyze the content recognized by the machine vision optical system; through input variables, adjust variable control is performed, and through machine deep learning, the critical point of freezing in various situations of the main channel is discovered and mastered, and self-decision-making is achieved with artificial intelligence algorithms. With the help of mobile or permanent ice-free water supply guarantee facilities and equipment dedicated to the main channel, timely perception, rapid response, intelligent decision-making, and rapid deployment are achieved, ultimately achieving ice-free water supply.

[0007] In the above technical solution, the method for identifying water flow rate and wind speed (rotating impeller speed) in motion mode scenarios through a machine vision optical system is as follows: a high-dynamic, full-color camera is selected with a frame rate of 20fps to capture images of target parts in multi-state channels, and then a series of image preprocessing such as image filtering and image enhancement are performed. Then, the water flow rate or wind speed is detected in combination with the Meanshift target tracking method.

[0008] In the above technical solution, the water flow velocity or wind speed is detected by combining the Meanshift target tracking method. The specific method is as follows: first, the kernel function is applied to the target area; the kernel function is applied to the target area. Defined as any point in space To a certain center The monotonic function of the Euclidean distance between ; Set the size of the target area to the bandwidth of the kernel function At the same time, all pixel values ​​in the video frame are evenly divided into n intervals. At this time, the total number of eigenvalues ​​in the video frame is Each interval can correspond to one of the eigenvalues ​​according to the size of the range. The method for calculating the probability of each eigenvalue u is shown in formula (1): (1) Of which: represents the normalization constant of the target model; is the kronecker data function, which is used to determine whether the pixel value in the area is equal to the feature value. Secondly, in the next frame, a target box that may contain visual feature markers is found, and the centroid coordinates of the target box are used as the center coordinates of the kernel function. Based on this, the feature probability density distribution of the target box is calculated. If the center coordinate of the candidate target box area is y, the probability distribution of the candidate target model is calculated according to formula (2): (2) Of which: represents the normalization constant of the candidate target model; based on the target model and the candidate target model, the Bhattacharyya coefficient is selected as the similarity function to measure the similarity between the target and candidate target histograms, which is specifically expressed as: (3) According to the above formula, the similarity between the candidate region and the target region is determined. The larger the value of , the more similar the target model and the candidate model are. The meanshift vector of the target model is shown in formula (3): (4) Finally, the center position of the target frame in the previous frame is used as the center of the search window, and the Meanshift vector is continuously iterated to find the candidate area that maximizes the similarity function, which is the position of the target frame in the current frame. The pixel coordinates of the target are calculated and stored. After the center coordinates of the target marker are calculated and recorded, the water flow rate or wind speed (rotating impeller speed) is calculated by calculating the actual displacement of the target marker water flow or rotating impeller during the period of time (that is, the period between the shooting time of the two pictures).

[0009] In the above technical solution, in step 2, the method for identifying the gate opening, water level, water temperature (thermometer scale), air temperature (thermometer scale), initial ice conditions, etc. through the machine vision optical system for the edge and mark scenarios is: after selecting a 1fps low frame rate, high resolution, high precision camera to capture images of the target parts of the multi-state channel, first perform a series of image preprocessing such as image filtering and image enhancement, and then use the HOG feature-based method to identify the target markers in the captured image; the specific method of using the HOG feature-based method to identify the target markers in the captured image is: by calculating the gradient direction histogram of the captured image in the local area to form the target features, thereby extracting the target features and then describing the edges of the target.

[0010] In the above technical solution, the main process of target feature extraction is: first, the set image is divided into several pixel units, and the gradient direction is evenly divided into 9 intervals; then, in each unit, the gradient directions of all pixels in each direction interval are histogram-generated to obtain a 9-dimensional feature vector. Since each adjacent 4 units constitute a block, the feature vectors in a block are connected to obtain a 36-dimensional feature vector, thereby scanning the captured image with the block; finally, the features of all blocks are connected in series to obtain the target features. The specific steps are: 1) converting the input color image into a grayscale image; 2) using Gamma correction to obtain the target feature. The positive method standardizes the color space of the input image to adjust the contrast of the image, reduce the impact of local shadows and lighting changes in the image, and suppress the interference of noise; 3) calculates the gradient to capture contour information while further weakening the interference of lighting; 4) projects the gradient to the gradient direction of the unit to provide a code for the local image area; 5) normalizes all cells on the block to compress the lighting, shadows and edges; 6) collects the HOG features of all blocks in the detection space, and collects the HOG features of all overlapping blocks in the detection window, and combines them into a final feature vector for classification.

[0011] In the above technical solution, in step three, the specific method of using a deep learning unit to analyze the content recognized by the machine vision optical system is as follows: considering the key factors that affect whether the main canal is frozen, wind speed (m / s), water temperature (℃), air temperature (℃), and initial ice critical state (ice flower recognition, state 0 to 1) are used as input variables; gate opening, gate pump power (if any), water level, water power, etc. are used as controllable and adjustable variables to form a new time series data; CNN-LSTM network is used to learn the input variables to realize adjustable variable control. The specific method is: after the CNN model is used to learn and extract features of the input variables, the LSTM network The network is mainly responsible for predicting and controlling adjustable variables. The CNN model uses local connections and shared weights to extract data features. It directly obtains effective representations from the original data through alternating convolutional layers and pooling layers, automatically extracts local features of the data, and establishes dense and complete feature vectors. The LSTM network predicts adjustable variables. The specific prediction method is to set up a 4-layer LSTM network layer and use random deactivation to prevent overfitting of the model. The fully connected layer can output a vector in a specified format to achieve adjustable variable control. The LSTM network solves the problem of "gradient disappearance" in model training by adding an additional forget gate. Its calculation formula is as follows: (5) (6) (7) (8) (9) In formula (10), , , , , and Respectively represent the states of the forget gate, input gate, input node, output gate, state unit and intermediate output in the network, and Respectively represent the changes of sigmoid function and tanh function, , , , , , , and Represent the matrix weights multiplied by the input and intermediate output, , , , represent the bias terms, Represents the element-wise multiplication of each vector; through the above-mentioned machine deep learning method, the critical point of freezing in various situations of the main channel can be discovered and mastered, and self-decision-making can be achieved with artificial intelligence algorithms. With the help of mobile or permanent ice-free water supply guarantee facilities and equipment dedicated to the main channel, timely perception, rapid response, intelligent decision-making, and rapid deployment can be achieved, ultimately achieving ice-free water supply.

[0012] In order to achieve the second purpose of the present invention, the technical solution of the present invention is: an ice-free water supply system for a water supply canal based on machine vision and deep learning, including a multi-state machine vision perception system, an optical system, an image acquisition module, an image processing system, an input and output module, an early warning system and a state output interactive interface; the optical system is a necessary supplementary light source for the multi-state machine vision perception system to identify the water supply canal; the image acquisition module collects the image obtained by the multi-state machine vision perception system and transmits it to the image processing system for processing, and the processed image is input into the state output interactive interface through the input and output module; the early warning system is set on the state output interactive interface, and is used to generate an early warning signal when an error occurs that exceeds the minimum limit (early warning threshold) within the safety range. Changes in parameter values, such as excessive water level, excessive flow rate, dangerous wind speed, and large-scale icing, can send out warning signals; the multi-state machine vision perception system includes a gate opening machine vision system, a water flow rate machine vision system, a water level machine vision system, a wind speed machine vision system, a water temperature machine vision system, an air temperature machine vision system, and a critical state machine vision system for initial ice and drift ice; the gate opening machine vision system performs machine vision recognition and high-precision measurement of the gate opening (stroke) of flat doors or curved doors on the channel; low-frame-rate, high-definition industrial cameras can be used to share videos / images; the water flow rate machine vision system performs flow rate measurement based on water flow texture recognition near the gate or at locations where water flow rate needs to be measured; high-frame-rate, High-definition industrial cameras share videos / images; the water level machine vision system, corresponding to the water flow velocity measurement position, performs machine vision measurement based on the comparison and identification of the water flow bank and the marking scale; low-frame-rate, high-definition industrial cameras can be used to share videos / images; the wind speed machine vision system performs machine vision identification of the wind wheel speed at the same location measured by the water flow velocity machine vision system and the water level machine vision system, thereby obtaining the real-time wind speed at that location; similarly, the wind speed machine vision can obtain the required image data through high-frame-rate, high-definition industrial camera shared video / image processing; the water temperature machine vision system measures the water flow velocity machine vision system, the water level machine vision system and the wind speed machine vision system at the same location of the above multi-state measurement. The same part of the multi-state measurement is measured by the channel surface water temperature thermometer (scale) based on machine vision recognition; similarly, the shared video / image of the low frame rate, high-definition industrial camera can be used; the temperature machine vision system measures the temperature thermometer scale above the channel water surface based on machine vision recognition at the same part of the multi-state measurement measured by the water flow velocity machine vision system, water level machine vision system, wind speed machine vision system and water temperature machine vision system; the initial ice and drift critical state machine vision system performs machine vision recognition on the critical state or initial state of ice formation of the channel during the ice period, including initial ice state such as bank ice, water surface ice flowers, water surface drift ice, ice flocs, etc. based on color state and one-image multi-purpose and multi-target recognition;Water temperature, air temperature, wind speed, and initial ice conditions are input variables, while water level and water flow rate (flow) are controllable and adjustable output variables. Similarly, machine vision for initial ice conditions can utilize shared video / images from low-frame-rate, high-definition industrial cameras.

[0013] In the above technical solution, the image processing system includes an image acquisition card, ice-free water input and AI algorithm processing unit and deep learning unit; the image acquisition card is used to collect and pre-process the images obtained by the multi-state machine vision perception system, and the data interface determines the transmission bandwidth according to the resolution and frame rate. USB3.0, Camera Link or GigE interface; the ice-free water transfer parameter input and AI algorithm processing unit uses high-speed, high-dynamic machine vision imaging and edge recognition algorithms to accurately identify the various state parameter variables of water transfer during the main canal ice period; it can use a single image for multi-purpose and multi-target identification of water level, water flow rate, gate opening, and the first ice period of the main canal; among them, water temperature, air temperature, wind speed, and first ice conditions are input variables, and water level and water flow rate (flow) are controllable and adjustable output variables; the deep learning unit is used to perform time-domain comparative analysis and deep learning on the multi-state parameters obtained in real time. When a sudden change trend occurs, it can intelligently identify, forecast, and issue early warnings for extreme working conditions or meteorological phenomena such as excessive water level, excessive flow rate, dangerous wind speed, and large-scale icing.

[0014] In the above technical solution, the state output interactive interface uses a trend state output display device; the state output interactive interface is used to output the trend characteristics of natural condition input state variables, including water temperature, air temperature, wind speed, initial ice conditions, etc.

[0015] The long-distance trunk canal winter ice-free water supply system achieved through multi-state variable machine vision and deep learning also includes power and control cables, which are used to connect the power cable access and control cable connection of the above-mentioned machine vision system, special lighting devices, early warning and control systems.

[0016] In the above technical solution, the image acquisition module includes an industrial-grade camera. A high-dynamic, full-color camera can be used to identify motion patterns, such as water flow rate and wind speed (rotating impeller speed). A high-resolution, high-precision camera can be used to identify edges and landmarks, such as gate opening, water level, water temperature (thermometer scale), air temperature (thermometer scale), and initial ice conditions. The optical system includes a dedicated light source, which is used only as supplemental lighting for nighttime identification. The multi-state machine vision perception system of the present invention can identify all aspects of water level, gate opening, water flow rate, and initial ice conditions under natural light. The dedicated light source is necessary for nighttime identification.

[0017] The present invention has the following advantages:

[0018] (1) The present invention uses machine vision to identify and read multi-state variables of long-distance main canals, combined with machine deep learning, and adjusts multi-state controllable state variables under the same conditions in real time according to multi-state input variables under ice-period water delivery conditions, thereby ensuring that the main canal is always below the critical state of freezing, thereby realizing ice-free water delivery in open main canals in winter; (2) The method of the present invention uses machine vision to identify key factors affecting the freezing of main canals, including water flow rate (water flow texture recognition), gate opening (gate position recognition), instant wind speed (rotating anemometer speed recognition), water temperature (thermometer scale recognition), air temperature (thermometer scale recognition), initial ice critical state (ice flower, ice floc recognition), etc., and uses wind speed, water temperature, air temperature and initial ice critical state as input variables; gate opening, gate pump power (if any), water level, and water power as controllable and adjustable variables (controllable and adjustable variables are key factors for main canal freezing. Identifying and controlling these variables can keep the main canal below the critical point of freezing in winter, thereby realizing ice-free water delivery. Water); by controlling adjustable variables according to input variables, discovering and mastering the freezing critical points of the main canal under various conditions through machine deep learning, realizing self-decision-making with artificial intelligence algorithms, and making use of mobile or permanent ice-free water supply guarantee facilities and equipment dedicated to the main canal, timely perception, rapid response, intelligent decision-making, and rapid deployment are achieved, ultimately realizing ice-free water supply in open main canals in winter; (3) The present invention realizes the identification, acquisition, prediction, and early warning of multiple target state parameters under the conditions of water supply in the ice period of the main canal without contact, built-in sensors, or sampling through an optical system, an image acquisition module, an image processing system, and an image recognition algorithm based on machine vision; (4) The machine vision system in the present invention can be divided into two categories: high-dynamic, high-precision video images, and low-dynamic, high-precision video images, and the images / videos are shared according to the above two categories in a way of multiple uses and partitioned recognition. Based on phenomenon perception, new perception data is generated to realize contactless real-time perception without relying on sensors, improve recognition accuracy, and reduce costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a schematic diagram of the arrangement structure of the multi-state machine vision perception system in the present invention on a water supply canal; Figure 2 It is a principle flow chart of the present invention; Figure 3 This is a block diagram of the winter ice-free water delivery system for main canals achieved through multi-state variable machine vision and deep learning in the present invention; exist Figure 1 In the figure, A represents the main water channel, which is an open channel; B represents an industrial camera; C represents the direction of water flow; D represents the water surface line of the main channel; E represents the horseway of the open channel; F1 represents the water level machine vision; F2 represents the water temperature machine vision; F3 represents the water flow rate machine vision; F4 represents the water surface wind speed machine vision; and F5 represents the channel bank ice machine vision. DETAILED DESCRIPTION

[0020] The following detailed description of the embodiments of the present invention is given in conjunction with the accompanying drawings, which do not limit the present invention but are merely examples. The description makes the advantages of the present invention clearer and easier to understand.

[0021] This invention uses machine vision to identify and read various state variables related to the potential for freezing during winter water delivery in main canals, such as pre-sluice water level, post-sluice flow rate, interval water level, interval flow rate, air temperature (read by machine vision), water temperature (read by machine vision), wind speed (read by machine vision), and critical ice conditions (initial ice, ice bloom). By using the machine vision method of this invention to identify and read these multiple state variables in the main canal, combined with machine deep learning, algorithms, and training, it accurately and in real time adjusts controllable state variables in long-distance main canals, such as gate height opening, gate pump power, and water flow rate. Simultaneously, main canal insulation and heating facilities, as well as ice-disrupting and ice-blocking equipment, can be implemented to ensure that long-distance main canals remain in a critical pre-freezing state during winter, preventing freezing. Ultimately, this multi-state variable, machine vision-based, deep-learning-based artificial intelligence algorithm ensures ice-free water delivery in the main canals during winter. The method proposed in this paper not only uses artificial intelligence to identify and learn the various state trends and influencing changes in winter water delivery in main canals, but also features contactless, real-time monitoring, low system cost (multi-purpose use of a single image, video partitioning), no impact on water quality, and deep, global intelligence. This significantly improves the safety of winter water delivery, engineering, and water quality in long-distance main canals, while also significantly increasing the main canal's winter water delivery capacity. The method uses machine vision to identify water flow rate (water flow texture recognition), gate opening (gate position recognition), instantaneous wind speed (rotating anemometer speed recognition), water temperature (thermometer scale recognition), air temperature (thermometer scale recognition), and the critical state of initial freezing (ice flower and ice flake recognition). These variables are all key factors affecting whether the main canal freezes. Wind speed, water temperature, air temperature, and the critical state of initial freezing are input variables, while gate opening, gate pump power (if any), water level, and water dynamics are controllable variables. By controlling adjustable variables based on input variables and using deep machine learning, the critical freezing points of the main channel under various conditions are discovered and mastered, and self-decision-making is achieved using artificial intelligence algorithms. With the help of mobile or permanent ice-free water transfer guarantee facilities and equipment dedicated to the main channel, timely perception, rapid response, intelligent decision-making, and rapid deployment can be achieved, ultimately achieving ice-free water transfer. The method of the present invention can be used in any open water diversion main channel with ice-free water transfer needs, including various forms of open main channels such as deep excavation and high fill.

[0022] Example: The present invention will now be described in detail by taking the application of the present invention to a certain trunk canal section during the winter ice period as an example, which will also provide guidance for the application of the present invention to water delivery during the ice period in other areas.

[0023] In this embodiment, the water transfer in a certain trunk canal section during the winter ice period is a typical section north of a certain place in the middle line of the South-to-North Water Diversion Project, and the overall goal is to transfer water in this section of open trunk canal without ice. The system used in this embodiment includes: (2) industrial-grade camera (monocular or multi-lens); (3) special light source (supplementary lighting only for nighttime identification); (4) gate opening machine vision; (5) water flow rate machine vision; (6) water level machine vision; (7) wind speed machine vision; (8) water temperature machine vision; (9) air temperature machine vision; (10) first ice and drift critical state machine vision; (11) image acquisition card; (12) ice-free water transfer input and AI algorithm processing unit; (13) deep learning unit; (14) trend status output display device; (15) early warning system; (16) power supply and control cable. The main decomposition of the system and the equipment functions of this embodiment are as follows: (1) Water supply main channel (open channel) Open channel is generally an artificial concrete lined channel with no cover on top. There are usually horse paths (concrete or asphalt pavement) on both sides of the lined channel for operation inspection and transportation and maintenance equipment. There are also usually buffer protection forest belts on both sides of the horse paths for closed management and water quality safety; (2) Industrial-grade camera (monocular or multi-lens) can be selected as area array camera or line array camera. For the recognition of various states of water supply freezing in winter in this example, area array camera is often used. In the case of identifying motion patterns, such as water flow rate, wind speed (rotating impeller speed), high dynamic, full-color camera can be selected. In the case of identifying edges and signs, such as gate opening, water level, water temperature (thermometer scale), air temperature (thermometer scale), initial ice conditions, etc., high-resolution, high-precision camera can be selected. The camera resolution is calculated based on the actual image width and accuracy requirements. For high-precision and dynamic recognition required in this example, a higher resolution of 1920 pixels × 1080 pixels can be selected; all state quantities in this example are long-term state monitoring. For water level, gate opening, thermometer scale, first ice period recognition, etc., a low frame rate such as 1fps can be selected; for the recognition of water flow velocity and wind speed, a high frame rate such as 20fps can be selected.Since the positions of the machine vision elements in this example can be relatively fixed, the positions of the cameras are also relatively fixed, and the lenses can use fixed focus and fixed aperture; (3) Special light source (only for supplementary lighting for nighttime identification) In this example, all water levels, gate openings, water flow rates, and initial ice conditions can be identified under natural light. Special light source is a necessary supplementary light source for identification under nighttime conditions. In this example, there are many target objects to be identified, and either front or side light sources can be selected; (4) Gate opening machine vision can be used to perform machine vision identification and high-precision measurement of the gate opening (stroke) of the flat door or curved door on the channel for comparison and identification. Low-frame-rate, high-definition industrial cameras share videos / images; (5) Water velocity machine vision performs velocity measurement based on water texture recognition near the gate or at the location where water velocity needs to be measured, and uses high-frame-rate, high-definition industrial cameras to share videos / images; (6) Water level machine vision performs machine vision measurement based on the comparison and recognition of the water bank and the marking scale at the corresponding water velocity measurement location, and can use low-frame-rate, high-definition industrial cameras to share videos / images; (7) Wind speed machine vision performs machine vision recognition of the wind wheel speed at the same location as the above-mentioned water velocity and water level machine vision measurements, thereby obtaining the real-time wind speed at that location. Similarly, wind speed machine vision can obtain the required image data through high-frame-rate, high-definition industrial camera shared video / image processing; (8) Water temperature machine vision performs channel surface water temperature thermometer (scale) based on machine vision recognition measurement at the same location as the above-mentioned multi-state measurement. Similarly, the shared video / image of a low-frame, high-definition industrial camera can be used; (9) Air temperature machine vision can be used to measure the temperature thermometer scale above the channel water surface based on machine vision recognition for the same part of the above multi-state measurement. Similarly, the shared video / image of a low-frame, high-definition industrial camera can be used; (10) Initial ice and drift ice critical state machine vision can be used to perform machine vision recognition of the critical state or initial state of ice formation of the channel during the ice period, including initial ice state such as bank ice, surface ice flowers, surface drift ice, ice flakes, etc. Based on color state and one-image multi-purpose and multi-target recognition. Among them, water temperature, air temperature, wind speed, initial ice conditions are input variables, and water level and water flow rate (flow) are controllable and adjustable output variables. Identification of range changes. Similarly, machine vision for the initial ice state can utilize shared video / images from low-frame-rate, high-definition industrial cameras. (11) The image acquisition card is used for image acquisition and preprocessing of the industrial-grade camera lenses required for the above-mentioned scenarios. The data interface determines the transmission bandwidth based on the resolution and frame rate. Considering the transmission distance, USB3.0, Camera Link, or GigE interfaces can be selected. In this example, the USB3.0 interface is selected. (12) The ice-free water supply input and AI algorithm processing unit uses high-speed, high-dynamic machine vision imaging and edge recognition algorithms to accurately identify the various state parameters of the main canal water supply during the ice period.For water level, water flow rate, gate opening, first ice period of main channel, etc.; (13) The deep learning unit is used to perform time domain comparative analysis and deep learning on the multi-state parameters obtained in real time. When a sudden change trend occurs, it can intelligently identify, forecast and warn of extreme working conditions or meteorological phenomena such as excessive water level, excessive flow rate, dangerous wind speed, large-scale icing, etc.; (14) The trend state output display device is used to output the trend characteristics of the natural condition input state variables, including water temperature, air temperature, wind speed, first ice conditions, etc.; (15) The warning device is used to issue a warning signal when a parameter value change exceeds the minimum limit (warning threshold) within the safety range, such as excessive water level, excessive flow rate, dangerous wind speed, large-scale icing, etc.; (16) The power supply and control cable is used to connect the power cable access and control cable connection of the above-mentioned machine vision system, special lighting device, warning and control system.

[0024] The method of the present invention is only used during ice-period water delivery in main canals. This embodiment employs the method of the present invention for winter ice-free water delivery, including the following steps: First, a fixed machine vision optical system is installed at the multi-state channel portion where identification of water delivery under multiple ice-period conditions is required. A frontal camera or a frontal or side camera is positioned to align multiple river targets, including gates, channels, water surface, temperature measurement, and wind measurement, ensuring that the images of each target are within the calibrated camera's field of view. Two types of camera lenses are used for different scenarios: First, for identifying motion patterns, such as water flow velocity and wind speed (rotating impeller speed), a high-dynamic, full-color camera with a frame rate of 20 fps is selected. Second, for identifying edges and landmarks, such as gate opening, water level, water temperature (thermometer scale), air temperature (thermometer scale), and initial ice conditions, a low-frame-rate, high-resolution, high-precision camera with a frame rate of 1 fps is selected. The camera resolution is calculated based on the actual image width and accuracy requirements. For the high-precision, dynamic recognition required in this example, a higher resolution of 1920 pixels × 1080 pixels is selected. The positions of the multi-state elements in machine vision are relatively fixed, so the positions of each camera are also relatively fixed. The lens can adopt a fixed focus and fixed aperture. To control the image range, LED light sources are provided for nighttime illumination to meet the illumination requirements for high-definition on-site imaging. An image acquisition card is used for image acquisition and preprocessing of images captured by the aforementioned industrial-grade camera lens. The data interface uses a USB 3.0 interface for transmission bandwidth, determined by the aforementioned resolution and frame rate. Secondly, machine vision is used to identify water flow rate (water texture recognition), gate opening (gate position recognition), instantaneous wind speed (rotating anemometer speed recognition), water temperature (thermometer scale recognition), air temperature (thermometer scale recognition), and the critical state of initial ice (ice flowers and ice flakes recognition). The above-mentioned variables are all key factors affecting whether the main canal freezes. Among them, wind speed, water temperature, air temperature and the critical state of initial ice are input variables; gate opening, gate pump power (if any), water level, hydrodynamics, etc. are controllable variables; finally, by controlling the adjustable variables based on the input variables, through machine deep learning, the freezing critical point of the main canal under various conditions is discovered and mastered, and self-decision-making is achieved with artificial intelligence algorithms. With the help of mobile or permanent ice-free water transfer guarantee facilities and equipment dedicated to the main canal, timely perception, rapid response, intelligent decision-making, and rapid deployment can be achieved, ultimately achieving ice-free water transfer; in addition, when the parameter value changes within the monitoring target object range that exceed the minimum limit (warning threshold) within the safety range, such as excessive water level, excessive flow rate, dangerous wind speed, large-scale ice formation, etc., an early warning signal is issued. The warning target parameters can be analyzed by outputting the changing trend of the state parameters through a prediction method combining wavelet analysis and a long short-term memory (LSTM) network on two or more time series of △C (m) / day, △C / week or △C / month and the corresponding change rate △aC / day, △aC / week or △aC / month, and issuing a forecast warning; the present invention first identifies the water flow rate and wind speed (rotating impeller speed) for the motion mode scenario.A high-dynamic, full-color camera with a frame rate of 20fps is used to capture images of the target parts of the multi-state channel. After a series of image preprocessing such as image filtering and image enhancement, the water flow rate or wind speed is detected in combination with the Meanshift target tracking method. The specific method is as follows: first, the kernel function is applied to the target area; the kernel function is applied. Defined as any point in space To a certain center The monotonic function of the Euclidean distance between ; Set the size of the target area to the bandwidth of the kernel function At the same time, all pixel values ​​in the video frame are evenly divided into n intervals. At this time, the total number of eigenvalues ​​in the video frame is Each interval can correspond to one of the eigenvalues ​​according to the size of the range. The method for calculating the probability of each eigenvalue u is shown in formula (1): (1) In the formula, represents the normalization constant of the target model; is the kronecker data function, which is used to determine whether the pixel value in the area is equal to the feature value; secondly, in the next frame, the target box that may contain the visual feature marker is found, and the centroid coordinates of the target box are used as the center coordinates of the kernel function, based on which the feature probability density distribution of the target box is calculated; if the center coordinate of the candidate target box area is y, the probability distribution of the candidate target model is calculated according to (2): (2) In the formula, represents the normalization constant of the candidate target model; based on the target model and the candidate target model, the Bhattacharyya coefficient is selected as the similarity function to measure the similarity between the target and candidate target histograms, which is specifically expressed as: (3) According to the above formula, the similarity between the candidate region and the target region is determined. The larger the value of , the more similar the target model and the candidate model are. The meanshift vector of the target model is shown in formula (3): (4) Finally, the center position of the target frame in the previous frame is used as the center of the search window, and the Meanshift vector is continuously iterated to find the candidate area that maximizes the similarity function, which is the position of the target frame in the current frame. The pixel coordinates of the target are calculated and stored. After the center coordinates of the target marker are calculated and recorded, the water flow rate or wind speed (rotating impeller speed) is calculated by calculating the actual displacement of the target marker water flow or rotating impeller during the period (i.e., the period between the time when the two pictures were taken). Then, for the edge and mark scenes, the gate opening, water level, water temperature (thermometer scale), air temperature (thermometer scale), initial ice conditions, etc. are identified. A 1fps low frame rate, high resolution, and high precision camera is selected. The camera resolution is calculated based on the actual image width and accuracy requirements. After a series of image preprocessing such as image filtering and image enhancement, the target marker in the captured image is identified using a method based on HOG features. The specific method is as follows: by calculating the gradient direction histogram of the captured image in the local area to form the target feature, thereby extracting the target feature and then describing the edge of the target; the main process of target feature extraction is: first, the set image is divided into several pixel units, and the gradient direction is evenly divided into 9 intervals; then, in each unit, the gradient direction of all pixels in each direction interval is histogram-generated to obtain a 9-dimensional feature vector. Since each adjacent 4 units constitute a block, the feature vectors in a block are connected to obtain a 36-dimensional feature vector, so that the captured image is scanned with a block; finally, the features of all blocks are connected in series to obtain the target feature. The specific steps are: 1) Convert the input color image into a grayscale image; 2) Use the Gamma correction method to standardize the color space of the input image to adjust the contrast of the image, reduce the impact of local shadows and lighting changes in the image, and suppress noise interference; 3) Calculate Gradient to capture contour information while further weakening the interference of light; 4) Project the gradient to the gradient direction of the unit to provide a code for the local image area; 5) Normalize all cells on the block to compress light, shadow and edge; 6) Collect the HOG features of all blocks in the detection space, and collect the HOG features of all overlapping blocks in the detection window, and combine them into the final feature vector for classification; Based on machine vision recognition including water flow rate (water flow texture recognition), gate opening (gate position recognition), instant wind speed (rotating anemometer speed recognition), water temperature (thermometer scale recognition), air temperature (thermometer scale recognition), initial ice critical state (ice flower, ice floc recognition), deep learning unit is used to analyze it, realize the identification and parameterization (digitalization) of the critical state of ice formation and formation factors (variables) of the main channel, so as to realize ice-free water delivery, which can bring huge social and economic benefits of engineering safety and water delivery safety in the implementation of the main channel.This method considers the key factors that influence whether a main canal freezes, using wind speed, water temperature, air temperature, and the initial freezing critical state as input variables. It uses gate opening, gate pump power (if any), water level, and hydraulic dynamics as controllable variables to form a new time series data set. A CNN-LSTM network is then used to learn these input variables to achieve control of these adjustable variables. The CNN model extracts data features using local connections and shared weights. It directly extracts effective representations from the raw data through alternating convolutional and pooling layers, automatically extracting local features and building a dense, complete feature vector. The LSTM network addresses the "vanishing gradient" problem during model training by adding an additional forget gate. Its calculation formula is as follows: (5) (6) (7) (8) (9) In formula (10), , , , , and Respectively represent the states of the forget gate, input gate, input node, output gate, state unit and intermediate output in the network, and Respectively represent the changes of sigmoid function and tanh function, , , , , , , and Represent the matrix weights multiplied by the input and intermediate output, , , , represent the bias terms, Represents element-wise multiplication of each vector. After the CNN model learns and extracts features from the input variables, the LSTM network is primarily responsible for predicting and controlling the adjustable variables. The specific prediction method involves setting up a four-layer LSTM network and using random dropout to prevent overfitting. The fully connected layer can output a vector in a specified format to achieve control of the adjustable variables.

[0025] This embodiment uses the aforementioned machine deep learning method to discover and understand the critical freezing point of the main canal under various conditions (the manifestations of critical freezing in the main canal include the formation of ice flowers on the water surface and the decrease in surface water flow velocity (when ice flowers form, the water surface flow velocity is 0). It can achieve self-decision-making using artificial intelligence algorithms. With the help of mobile or permanent ice-free water supply guarantee facilities and equipment dedicated to the main canal, timely perception, rapid response, intelligent decision-making, and rapid deployment can be achieved, ultimately achieving ice-free water supply.

[0026] Other parts not described belong to the prior art.

Claims

1. A method for ice-free water delivery in a main water channel based on machine vision and deep learning, characterized by: The following steps are included: Step 1: Install a fixed machine vision optical system at the multi-state channel location where identification of water delivery conditions under multiple ice periods is required, and align the machine vision optical system with the river surface of multiple targets including gates, channels, water surface, temperature measurement, and wind measurement, so that the image of each target is within the field of view of the calibration camera; Step 2: Use the machine vision optical system to identify water flow rate and wind speed in motion pattern scenarios. At the same time, identify gate opening, water level, water temperature, air temperature, and initial ice conditions in edge and sign scenarios. Step 3: Use deep learning units to analyze the content recognized by the machine vision optical system; By inputting variables and controlling adjustable variables, the critical points of freezing in various conditions of the main channel can be discovered and mastered through deep machine learning. Self-decision-making can be achieved with artificial intelligence algorithms. With the help of mobile or permanent ice-free water supply guarantee facilities and equipment dedicated to the main channel, timely perception, rapid response, intelligent decision-making, and rapid deployment can be achieved, ultimately achieving ice-free water supply.

2. The method for ice-free water delivery in a main water channel based on machine vision and deep learning according to claim 1, characterized in that: The method for identifying water flow rate and wind speed based on motion pattern scenarios using a machine vision optical system is as follows: A high-dynamic, full-color camera with a frame rate of 20fps is used to capture images of the target parts of the multi-state channel. After image preprocessing such as image filtering and image enhancement, the water flow rate or wind speed is detected in combination with the Meanshift target tracking method.

3. The method for ice-free water delivery in a main water channel based on machine vision and deep learning according to claim 2, characterized in that: Combined with the Meanshift target tracking method to detect water flow rate or wind speed, the specific method is as follows: First, apply the kernel function to the target area; Defined as any point in space To a certain center The monotonic function of the Euclidean distance between ; Set the size of the target region to the bandwidth of the kernel function At the same time, all pixel values ​​in the video frame are evenly divided into n intervals. At this time, the total number of eigenvalues ​​in the video frame is Each interval can correspond to one of the eigenvalues ​​according to the size of the range. The method for calculating the probability of each eigenvalue u is shown in formula (1): （1） in: represents the normalization constant of the target model; It is the kronecker data function, which is used to determine whether the pixel value in the area is equal to the eigenvalue; Secondly, in the next frame, we search for a target frame that may contain a visual feature marker, and use the centroid coordinates of the target frame as the center coordinates of the kernel function. Based on this, we calculate the feature probability density distribution of the target frame. If the center coordinate of the candidate target frame area is y, the probability distribution of the candidate target model is calculated according to formula (2): （2） in: represents the normalization constant of the candidate target model; based on the target model and the candidate target model, the Bhattacharyya coefficient is selected as the similarity function to measure the similarity between the target and candidate target histograms, which is specifically expressed as: （3） According to the above formula, the similarity between the candidate region and the target region is determined. The larger the value of , the more similar the target model and the candidate model are. The meanshift vector of the target model is shown in formula (3): （4） Finally, the center position of the target frame in the previous frame is used as the center of the search window, and the Meanshift vector is continuously iterated to find the candidate area that maximizes the similarity function, which is the position of the target frame in the current frame. The pixel coordinates of the target are calculated and stored. After the center coordinates of the target marker are calculated and recorded, the water flow rate or wind speed is calculated by calculating the actual displacement of the target marker water flow or rotating impeller during this period.

4. The method for ice-free water delivery in a main water channel based on machine vision and deep learning according to claim 3, characterized in that: In step 2, the machine vision optical system is used to identify the gate opening, water level, water temperature, air temperature, and initial ice conditions based on the edge and marking scenarios. The method is as follows: After selecting a 1fps low-frame-rate, high-resolution, and high-precision camera to capture images of the target parts of the multi-state channel, image preprocessing such as image filtering and image enhancement is first performed, and then the target markers in the captured images are identified using a method based on HOG features; The specific method of using the HOG feature-based method to identify the target marker in the captured image is as follows: The target features are constructed by calculating the gradient direction histogram of the captured image in the local area, thereby extracting the target features and further describing the edge of the target.

5. The method for ice-free water delivery in a main water channel based on machine vision and deep learning according to claim 4, characterized in that: The method of target feature extraction is: First, the image is divided into several pixel units, and the gradient direction is evenly divided into 9 intervals; Then, in each unit, the gradient direction of all pixels in each direction interval is histogram-generated to obtain a 9-dimensional feature vector. Since each adjacent 4 units form a block, the feature vectors in a block are connected to obtain a 36-dimensional feature vector, thereby scanning the captured image with blocks. Finally, the features of all blocks are connected in series to obtain the features of the target. The specific steps are: 1) Convert the input color image to a grayscale image; 2) Gamma correction is used to standardize the color space of the input image to adjust the contrast of the image, reduce the impact of local shadows and lighting changes in the image, and suppress noise interference; 3) Calculate gradients to capture contour information while further weakening the interference of lighting; 4) Project the gradient to the gradient direction of the unit to provide an encoding for the local image region; 5) Normalize all cells on the block to compress lighting, shadows, and edges; 6) Collect the HOG features of all blocks in the detection space, collect the HOG features of all overlapping blocks in the detection window, and combine them into the final feature vector for classification.

6. The method for ice-free water delivery in a main water channel based on machine vision and deep learning according to claim 5, characterized in that: In step three, the specific method of using the deep learning unit to analyze the content recognized by the machine vision optical system is as follows: With wind speed, water temperature, air temperature and initial ice critical state as input variables; gate opening, gate pump power, water level and water dynamics as controllable variables, a new time series data is formed; The CNN-LSTM network is used to learn the input variables to achieve adjustable variable control. The specific method is as follows: After the CNN model is used to learn and extract features from the input variables, the LSTM network is mainly responsible for predicting and controlling the adjustable variables; The CNN model uses local connections and shared weights to extract data features. It directly obtains effective representations from the original data by alternating between convolutional layers and pooling layers, automatically extracts local features of the data, and establishes a dense and complete feature vector. The LSTM network predicts adjustable variables. The specific prediction method is as follows: set up a four-layer LSTM network layer and use the random deactivation method to prevent overfitting of the model. The fully connected layer can output a vector of a specified format to achieve adjustable variable control. The LSTM network solves the problem of "gradient disappearance" during model training by adding an additional forget gate. Its calculation formula is as follows: （5） (6) (7) (8) (9) (10) Where, , , , , and Respectively represent the states of the forget gate, input gate, input node, output gate, state unit and intermediate output in the network, and Respectively represent the changes of sigmoid function and tanh function, , , , , , , and Represent the matrix weights multiplied by the input and intermediate output, , , , represent the bias terms, Indicates element-wise multiplication of each vector; Through the above-mentioned machine deep learning method, the critical point of freezing in the main canal under various conditions can be discovered and mastered, and self-decision-making can be achieved with artificial intelligence algorithms. With the help of mobile or permanent ice-free water supply guarantee facilities and equipment dedicated to the main canal, timely perception, rapid response, intelligent decision-making, and rapid deployment can be achieved, ultimately achieving ice-free water supply.

7. The ice-free water delivery system used in the ice-free water delivery method for a main water delivery channel based on machine vision and deep learning according to any one of claims 1 to 6, characterized in that: It includes a multi-state machine vision perception system, an optical system, an image acquisition module, an image processing system, an input and output module, an early warning system, and a state output interactive interface; The optical system is a necessary supplementary light source for the multi-state machine vision perception system to identify the water supply canal. The image acquisition module collects images obtained by the multi-state machine vision perception system and transmits them to the image processing system for processing. The processed images are input into the state output interactive interface through the input and output module. The early warning system is set on the state output interactive interface to issue an early warning signal when a parameter value changes beyond the minimum limit within the safety range. The multi-state machine vision perception system includes a gate opening machine vision system, a water flow rate machine vision system, a water level machine vision system, a wind speed machine vision system, a water temperature machine vision system, an air temperature machine vision system, and a machine vision system for the critical state of initial ice and drift ice; Gate opening machine vision system, which can identify and measure the gate opening of flat doors or curved doors on the channel with machine vision and high precision; Water flow rate machine vision system, which measures flow rate based on water flow texture recognition near gates or other locations where water flow rate needs to be measured; The water level machine vision system corresponds to the water flow velocity measurement position and performs machine vision measurement based on the comparison and recognition of the water flow bank and the marking scale; The wind speed machine vision system performs machine vision recognition of the wind rotor speed at the same location measured by the water flow velocity machine vision system and the water level machine vision system, thereby obtaining the instantaneous wind speed at that location; The water temperature machine vision system measures the channel surface water temperature thermometer based on machine vision recognition at the same location measured by the water flow rate machine vision system, water level machine vision system and wind speed machine vision system for the same location measured by the above multi-state measurement; The air temperature machine vision system measures the air temperature thermometer scale above the channel water surface based on machine vision recognition at the same location measured by the water flow velocity machine vision system, water level machine vision system, wind speed machine vision system, and water temperature machine vision system in the same location as the multi-state measurement. The machine vision system for the critical state of first ice and drift ice performs machine vision identification on the critical state or initial state of ice formation of the channel during the ice period. The water temperature, air temperature, wind speed, and first ice conditions are input variables, and the water level and water flow rate are controllable and adjustable output variables.

8. The ice-free water delivery system for main water channels based on machine vision and deep learning according to claim 7, characterized in that: The image processing system includes an image acquisition card, ice-free water supply input, AI algorithm processing unit, and deep learning unit; The image acquisition card is used to collect and pre-process images acquired by the multi-state machine vision perception system; The ice-free water transfer input and AI algorithm processing unit uses high-speed, high-dynamic machine vision imaging and edge recognition algorithms to accurately identify various state parameter variables of main channel water transfer during the ice period. It also performs multi-purpose, multi-target recognition of water level, water flow rate, gate opening, and the first ice period of the main channel using a single image. Water temperature, air temperature, wind speed, and first ice conditions are input variables, while water level and water flow rate are controllable and adjustable output variables. The deep learning unit is used to perform comparative analysis and deep learning on the multi-state parameters obtained in real time in the time domain. When a sudden change trend occurs, it can intelligently identify, forecast and issue early warnings for extreme conditions or meteorological phenomena such as excessive water levels, excessive flow rates, dangerous wind speeds, and large-scale icing.

9. The ice-free water delivery system for main water delivery channels based on machine vision and deep learning according to claim 8, characterized in that: The state output interactive interface uses a trend state output display device; the state output interactive interface is used to output the trend characteristics of natural condition input state variables, including water temperature, air temperature, wind speed, and initial ice conditions.

10. The ice-free water delivery system for main water delivery channels based on machine vision and deep learning according to claim 9, characterized in that: The image acquisition module includes an industrial-grade camera; a high-dynamic, full-color camera is used in the motion pattern recognition scenario; Use high-resolution, high-precision cameras when identifying edges and logos; The optical system includes a dedicated light source for supplementary illumination for nighttime identification.

Images