Water flow video speed measurement method, spatiotemporal neural network model, electronic equipment and medium
Through the video speed measurement spatiotemporal neural network model of adjacent frame fusion, the existing water flow speed measurement method is solved, and the existing water flow speed measurement method is complex and resource consumption is achieved, efficient and accurate water flow speed measurement is achieved, and the model adaptability and deployment convenience are improved. .
Patent Information
- Application Number
- CN202411306336.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-19
AI Technical Summary
The existing water flow speed measurement method has complex calculations and high resource consumption, making it difficult to effectively deploy and operate in resource-constrained environments, and the parameter adjustment is cumbersome, which limits its universality and flexibility.
The video speed measurement spatiotemporal neural network model is adopted with adjacent frame fusion. By constructing data sets and labels, building spatiotemporal neural network model, decomposing flow field estimation into thermal maps and pixel displacement estimation, setting time dimensions, constructing loss functions, and performing the model Training and optimization, design efficient computing architectures to suit different scenarios and hardware conditions.
It realizes efficient and accurate water flow velocity measurement, reduces the computing resource requirements, improves the adaptability and deployment convenience of the model, and meets the speed measurement needs in diversified scenarios.
Smart Images

Figure CN118823056B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speed measurement, and in particular to a water flow video speed measurement method, a spatiotemporal neural network model, an electronic device and a medium. Background Art
[0002] The measurement of water flow velocity is crucial for water conservancy projects, environmental monitoring, and hydrological research. Traditional methods of measuring water flow velocity, such as direct measurement using a current meter or tracking water flow velocity using a buoy, usually face limitations such as cumbersome operation, low efficiency, and limited measurement range. In recent years, with the advancement of computer vision technology, water flow velocity measurement methods based on image processing have become a highly anticipated alternative.
[0003] Nevertheless, current image-based water velocity measurement technologies still face some challenges. First, these methods tend to have high computational complexity and require a lot of computing resources for image analysis and data processing. Second, in order to adapt to different application scenarios, existing algorithms usually require complex experimental parameter adjustments, which not only increases the difficulty of use but also limits their versatility and flexibility. In addition, due to the high computing requirements, these technologies are difficult to deploy and run effectively on resource-constrained devices, which further limits their widespread application in field operations and other practical application scenarios.
[0004] Therefore, it is particularly urgent to provide a water flow velocity measurement method that has high computational efficiency, easy parameter adjustment, and can operate in resource-constrained environments. Summary of the invention
[0005] In view of this, the embodiments of the present disclosure provide a water flow video speed measurement method, a spatiotemporal neural network model, an electronic device and a medium, which at least partially solve the problems existing in the prior art.
[0006] In a first aspect, an embodiment of the present disclosure provides a method for measuring water flow speed using video, which includes:
[0007] Build datasets and labels;
[0008] Construct a spatiotemporal neural network model for video speed measurement by fusing adjacent frames;
[0009] Decomposing the flow field estimation in the area to be measured into flow field thermal map estimation and pixel displacement estimation;
[0010] Set the time dimension;
[0011] Construct loss function;
[0012] Training and evaluating the spatiotemporal neural network model;
[0013] The trained spatiotemporal neural network model is configured as a standard model running in different scenarios and under different hardware conditions, and the spatiotemporal neural network model is monitored and optimized in real time.
[0014] According to a specific implementation of the embodiment of the present disclosure, the step of constructing a data set and a label includes:
[0015] Generate training labels using a flow velocity calculation method, wherein the labels include a water flow field distribution thermal map and a pixel displacement value;
[0016] The data in the dataset is cleaned and / or denoised and / or normalized.
[0017] According to a specific implementation method of an embodiment of the present disclosure, the spatiotemporal neural network model of video speed measurement with adjacent frame fusion is based on a two-dimensional convolutional neural network to fuse information between adjacent frames.
[0018] According to a specific implementation of the embodiment of the present disclosure, the value range of the flow field thermodynamic map is 0 to 1;
[0019] The value range of the pixel displacement is 0 to N, where N is a positive integer greater than or equal to 1.
[0020] According to a specific implementation of the embodiment of the present disclosure, the time dimension is 32 frames.
[0021] According to a specific implementation of the embodiment of the present disclosure, the constructing the loss function includes: using an activation function to map the flow field heat map value output by the spatiotemporal neural network model to between 0 and 1;
[0022] Use the loss function to regress the mapped flow field heat map value;
[0023] The loss function is used to regress the discrete values of the pixel displacement size.
[0024] According to a specific implementation of the embodiment of the present disclosure, the training and evaluation of the spatiotemporal neural network model includes: adjusting model parameters by stochastic gradient descent or the stochastic gradient descent variant algorithm;
[0025] Using a test data set to evaluate the performance of the trained spatiotemporal neural network model;
[0026] The spatiotemporal neural network model is optimized according to the performance evaluation results.
[0027] In a second aspect, the present disclosure provides a spatiotemporal neural network model for water flow video speed measurement, which includes:
[0028] An adjacent frame fusion module is used to receive continuous video frames as input, fuse the information between adjacent frames through adjacent frame fusion, and extract the preliminary spatiotemporal characteristics of water flow movement;
[0029] A feature extraction module is used to extract the temporal and spatial features of pixel displacement from the fused information;
[0030] The upsampling decoder module is used to generate high-resolution flow field heat maps through upsampling and feature fusion;
[0031] The prediction head module includes two prediction heads, which are used to predict the flow field thermal map value and the pixel displacement size respectively.
[0032] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising:
[0033] at least one processor; and,
[0034] a memory communicatively connected to the at least one processor; wherein,
[0035] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned water flow video speed measurement method.
[0036] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned water flow video speed measurement method.
[0037] In summary, this embodiment overcomes the limitations of traditional water flow velocity measurement methods in terms of accuracy, processing efficiency, and deployment flexibility. It provides a cutting-edge video spatiotemporal neural network model that can accurately capture and measure water flow velocity, while integrating efficient computing cores, lightweight architecture, and convenient deployment capabilities. In this way, a water flow velocity measurement system with excellent performance, high adaptability to different environments, and easy to be widely deployed and applied is created to fully meet the velocity measurement needs in diversified scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and drawings. Here, the exemplary implementation methods of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.
[0039] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0040] Figure 1 A schematic diagram of a flow chart of a water flow video velocity measurement method provided in the first embodiment of the present invention;
[0041] Figure 2 A schematic diagram of the fusion of two adjacent frames provided by the first embodiment of the present invention;
[0042] Figure 3 A schematic diagram of the decomposition of the flow field estimation in the area to be measured provided by the first embodiment of the present invention;
[0043] Figure 4 A schematic diagram of the structure of a video speed measurement neural model for adjacent frame fusion provided in the second embodiment of the present invention.
[0044] Figure 5 An exemplary structural diagram of a device capable of implementing the method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0045] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
[0046] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.
[0047] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.
[0048] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0049] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.
[0050] See also Figure 1 The first embodiment of the present invention provides a method for measuring water flow speed using video, which can be executed by an electronic device, and in particular, by one or more processors in the electronic device, to implement the following steps:
[0051] S101, build dataset and labels.
[0052] In this embodiment, the electronic device may be a device with computing capabilities such as a laptop computer, a desktop computer, a workstation, a server, a field programmable gate array (FPGA), etc.
[0053] Among them, in order to accurately measure the water flow velocity, traditional algorithms such as LSPIV (Large Scale Particle Image Velocimetry), OTV (Optical Flow Velocimetry), OFV (Optical Tracking Velocimetry), etc. have been widely used in practice. However, these methods generally have the problem of low operating efficiency and require specific parameter settings to be adjusted in different application scenarios, which limits their practical application.
[0054] The LSPIV algorithm estimates the flow velocity by tracking the movement of suspended particles in water. Its flow velocity calculation formula can be expressed as formula (1):
[0055] ;
[0056] in, Indicates flow rate, For particles at time Internal displacement.
[0057] The OTV algorithm uses optical flow technology to determine the movement direction and speed of objects in an image sequence. The flow rate calculation formula can be expressed as formula (2):
[0058] ;
[0059] in, Indicates flow rate, Indicates Particles in time The position at Indicates Particles in time The position at is the total number of particles.
[0060] The OFV algorithm infers the flow velocity by tracking the feature points in the image sequence. Its flow velocity calculation formula can be expressed as formula (3):
[0061] ;
[0062] Although the above algorithms have certain limitations, they are still widely accepted in the verification of flow velocity distribution. In order to improve the accuracy and solve the limitations of existing algorithms, the data generated by these algorithms can be used as labels for training neural networks to build more efficient and accurate water flow velocity measurement models. The labels include water flow field distribution thermal maps and pixel displacement values.
[0063] Furthermore, the data in the constructed dataset is preprocessed. In the preprocessing stage, the original data needs to be processed in a series of ways, including but not limited to data cleaning, denoising, and normalization, to ensure the quality and consistency of the data, thereby improving the training effect of the model.
[0064] S102, constructing a spatiotemporal neural network model for video speed measurement by fusing adjacent frames.
[0065] In this embodiment, the texture of the water flow has characteristics that can characterize the flow velocity in the time and space dimensions, especially the so-called "texture angle". However, in traditional methods, the artificial setting of the reference line has a significant impact on the final result, which makes the whole process more subjective and difficult to standardize.
[0066] To solve this problem, a deep learning algorithm based on a neural network can be used. This type of algorithm is widely used in many fields, including image analysis and pattern recognition, due to its powerful feature extraction capabilities. In this case, the task of selecting the reference line can be automatically assigned to the neural network. This means that it is no longer necessary to manually set the reference line, but only to prepare appropriate supervision information, that is, a labeled training data set, to guide the network to learn how to extract flow velocity-related features from the image. As shown in formula (4):
[0067] ;
[0068] in, For input data, is the convolution kernel.
[0069] This approach not only reduces the need for human intervention, but also improves the automation of the entire process, which may lead to more objective and consistent results. In addition, by using a large amount of diverse training data, the neural network can learn to recognize the water flow texture features in a variety of scenarios, further enhancing the generalization ability and adaptability of the algorithm.
[0070] Considering that the traditional 2D convolutional neural network can only extract features in the spatial dimension, but cannot directly process information in the temporal dimension, and although the 3D convolutional neural network can extract features in both the temporal and spatial dimensions, it is not suitable for existing computing resources due to its huge number of parameters. Therefore, this embodiment provides a video speed measurement neural network based on adjacent frame fusion of 2D (two-dimensional) convolutional neural network. Figure 2 .like Figure 2 As shown in , the network combines the efficient operation and spatial information extraction capabilities of 2D convolutional neural networks, and realizes feature extraction in the time dimension by fusing information between adjacent frames. Specifically, by fusing adjacent frames, the network can capture the change information between frames, thereby indirectly processing data in the time dimension. Figure 2 As shown, the first frame (Frame 1) is fused with the second frame (Frame 2), the third frame (Frame 3) is fused with the fourth frame (Frame 4), the fifth frame (Frame 1) is fused with the sixth frame (Frame 2), and the seventh frame (Frame 1) is fused with the eighth frame (Frame 2). Further, as Figure 2 As shown, the fused adjacent frames can be further fused. Figure 2The fusion of two adjacent frames is shown as an example, but it can also be a fusion of three or four adjacent frames. The embodiment of the present application does not specifically limit the number of adjacent frames to be fused.
[0071] In this embodiment, an attention mechanism can also be introduced to highlight the information in key areas, thereby increasing the network's sensitivity to important features. By adding residual connections, the network can better propagate gradients, alleviate the gradient vanishing problem in deep networks, and improve the training efficiency and generalization ability of the model.
[0072] Through these technical means, the video speed measurement neural network for adjacent frame fusion provided in this embodiment can not only efficiently extract spatial features, but also perform effective information exchange between adjacent frames, thereby improving the performance and generalization ability of the model.
[0073] S103, decomposing the flow field estimation of the area to be measured into flow field thermal map estimation and pixel displacement estimation.
[0074] In this embodiment, the water flow velocity can be reflected by the flow field distribution, that is, the flow velocity values measured at different positions and different times (expressed as pixel displacement). These values are usually discrete (ranging from 0 to N), which may bring challenges to the calculation of the loss function of the neural network. Therefore, the flow field estimation problem can be decomposed into two subtasks: flow field heat map estimation (value range is 0 to 1) and pixel displacement estimation (value range is 0 to N). Among them, the interval from 0 to 1 is more suitable for neural network learning. See Figure 3 , Figure 3 Schematic diagram of the decomposition of flow field estimation in the area to be measured.
[0075] It should be noted that a decoder (based on upsampling convolutional neural network structure) can be used to generate a flow field heat map. The value range of this heat map is limited to between 0 and 1, which is more suitable for neural network learning. An encoder similar to the VGG structure can be used to estimate the size of the pixel displacement. This type of encoder can effectively extract spatial features, and the displacement size of its output ranges from 0 to N. Among them, N is a positive integer greater than or equal to 1.
[0076] S104, setting the time dimension.
[0077] In this embodiment, in view of the fact that the information of the time dimension is taken into account from the perspective of the spatiotemporal texture, the video speed measurement neural network of the adjacent frame fusion in the embodiment of the present application also fuses the information between adjacent frames. Therefore, in each reasoning process, 32 continuous frames of data can be used. The selection of this number of frames not only avoids the problem of unclear information in the time dimension due to too few frames, but also prevents excessive changes in flow velocity values in the same time period due to too many frames. Assuming that the time interval of each frame is Δt, the total time span when 32 frames are used is 32Δt. Through a large number of experiments and data analysis, it can be determined that such a frame number range is optimal to ensure that the model can accurately capture the spatiotemporal characteristics of the water flow.
[0078] Of course, other frame numbers of time dimensions may also be set, and the embodiment of the present application does not impose any specific restrictions on the set time dimensions.
[0079] S105, construct a loss function.
[0080] In this embodiment, for the flow field thermal map, an activation function may be used, for example, a sigmoid function maps the output result of the video speed measurement neural network fused with adjacent frames to the interval between 0 and 1. The mathematical expression of the sigmoid function is shown in formula (5):
[0081] ;
[0082] in, It is the flow field heat map value output by the video speed measurement neural network of adjacent frame fusion. By using the sigmoid function, it can be ensured that each value of the flow field heat map is between 0 and 1, which is helpful for network learning and optimization.
[0083] In order to perform regression tasks, a loss function can be used, such as the L1-loss (absolute error loss) function, to regress the flow field heat map values fused by the mapped adjacent frames. The mathematical expression of the L1-loss function is shown in formula (6):
[0084] ;
[0085] in, is the true value, is the predicted value, and the L1-loss function calculates the average absolute error between the predicted value and the true value.
[0086] For the discrete value regression of pixel displacement, the loss function of the regression problem can be used, for example, the Huber loss function can be used. Huber loss is a loss function that combines the advantages of L1 and L2 loss and is suitable for processing data containing outliers. The mathematical expression of the Huber loss function is shown in formula (7):
[0087] ;
[0088] in, is the parameter of Huber loss function. By properly setting the weight and hyperparameters of the loss function, the model training process can be optimized and the accuracy and stability of the model can be improved.
[0089] S106, training and evaluating the spatiotemporal neural network model.
[0090] In this embodiment, the video speed measurement spatiotemporal neural network model of adjacent frame fusion constructed in the above embodiment is trained using training data, and the prediction results output by the spatiotemporal neural network model are made as close to the label results as possible by adjusting the model parameters.
[0091] During the training process, Stochastic Gradient Descent (SGD) and its variant algorithms are used to adjust the model parameters. For example, a commonly used variant is the Adam optimization algorithm, which combines the ideas of momentum and adaptive learning rate. Specifically, the update rule of the Adam algorithm is as follows:
[0092] First, calculate the gradient. The gradient calculation is shown in formula (8):
[0093] ;
[0094] in, is at the time step The gradient of is the parameter of the previous time step, is the loss function.
[0095] Then, update the first-order moment estimate and the second moment estimate , as shown in formula (9) and formula (10):
[0096]
[0097] in, and is the decay rate, usually close to 1.
[0098] Next, the first-order moment estimate and the second-order moment estimate are bias-corrected, as shown in formula (11) and formula (12):
[0099] ;
[0100] Finally, update the parameters as shown in formula (13):
[0101] ;
[0102] in, is the learning rate, is a small constant used to prevent the denominator from being zero.
[0103] Furthermore, the performance of the trained model can be verified using an independent test data set. During the evaluation process, mean absolute error (MAE), mean square error (MSE), and root mean square error (RMSE) can be used. These indicators quantify the difference between the model's predicted value and the actual value from different dimensions, providing a solid foundation for a comprehensive evaluation of the model's performance.
[0104] Subsequently, based on the above evaluation results, the spatiotemporal neural network model optimization phase is entered. The optimization strategy is formulated based on the specific findings of the error analysis, including but not limited to: adjusting the model structure and parameters. According to the error distribution characteristics, it may be necessary to fine-tune the model's complexity, learning rate, regularization strength and other parameters to balance the overfitting and underfitting problems of the model and improve the generalization ability.
[0105] Increase or optimize training data, identify potential biases or deficiencies in the data, and enrich the representativeness of the training data set by collecting more diverse samples, data cleaning and enhancement, thereby improving the accuracy of the model.
[0106] S107, configuring the trained spatiotemporal neural network model as a standard model that runs in different scenarios and under different hardware conditions, and performing real-time monitoring and optimization on the model.
[0107] In this embodiment, in order to ensure that the trained model can run efficiently in a variety of environments, the trained model is converted into a unified standard format, such as the ONNX (Open Neural Network Exchange) model, to facilitate cross-platform deployment. The FastAPI framework is used to encapsulate the ONNX model as an HTTP interface service to facilitate external calls. The encapsulated service is containerized using Docker to ensure that the service can run consistently in different environments.
[0108] When deploying services on cloud servers, you can enable GPU (Graphics Processing Unit) acceleration mode to fully utilize the powerful computing power of the GPU and significantly improve the reasoning speed. When deploying services on edge boxes or other resource-constrained devices, you can choose to use CPU or NPU (Neural Processing Unit) mode for reasoning to adapt to different computing resource conditions. When deploying services on devices with a certain computing power, you can also choose CPU or NPU mode for reasoning, and flexibly configure according to different hardware conditions. Continuously monitor the performance of the deployed model to ensure its reliability and efficiency in actual applications. Based on the monitoring results, perform necessary optimizations in real time to maintain the optimal performance of the model.
[0109] In summary, compared with the prior art, this embodiment has the following advantages:
[0110] 1. Improve the accuracy of speed measurement: Use the video speed measurement neural network that fuses adjacent frames to deeply mine and learn the spatiotemporal features in the water flow video, accurately capture the complex movement patterns of the water flow, and thus significantly improve the accuracy of water flow speed measurement.
[0111] 2. Improve computing efficiency: By optimizing the neural network architecture and algorithm flow, the amount of calculation and operation time can be reduced to achieve efficient water flow velocity measurement and meet the needs of real-time or quasi-real-time velocity measurement.
[0112] 3. Make the model lightweight: By designing an efficient and streamlined neural network structure, it can run easily on resource-constrained devices.
[0113] 4. Enhance deployment convenience: Design simple interfaces and deployment processes to ensure that the algorithm can be quickly deployed on various hardware platforms and operating systems, lowering the application threshold.
[0114] 5. Adapt to diverse scenarios: By building a rich water flow video dataset, the algorithm can adapt to scenarios with different water flow environments, video quality, and measurement requirements.
[0115] 6. Promote the development of water flow velocity measurement technology: By providing an innovative and efficient water flow velocity measurement algorithm, promote the widespread application of water flow velocity measurement technology in water conservancy projects, hydrological research, environmental monitoring and other fields, and provide strong support for related decision-making and research.
[0116] To sum up, this embodiment solves the problems existing in traditional water flow velocity measurement methods, and provides an accurate, efficient, lightweight and easy-to-deploy water flow video velocity measurement algorithm to meet complex and diverse water flow velocity measurement needs, and promote technological progress and application expansion in related fields.
[0117] The second embodiment of the present invention provides a video speed measurement neural model (such as Figure 4 shown), which includes:
[0118] The adjacent frame fusion module 410 is used to receive continuous video frames as input, fuse the information between adjacent frames through the adjacent frame fusion video speed measurement neural network, and extract the preliminary spatiotemporal characteristics of water flow movement. This step is crucial for capturing the dynamic changes of water flow.
[0119] The feature extraction module 420 is used to extract the spatiotemporal features of pixel displacement from the fused information. For example, the VGG (Visual Geometry Group) network structure can be used to perform deep feature extraction on the fused spatiotemporal features. The VGG network is well-known for its powerful feature representation ability, which can further mine the subtle features in the water flow video and provide a rich information basis for subsequent velocity field prediction.
[0120] The upsampling decoder module 430 is used to generate a high-resolution flow field thermogram by upsampling and feature fusion. In order to recover a high-resolution flow field thermogram from a low-resolution feature map, the present embodiment introduces an upsampling decoder module. The module gradually recovers the spatial resolution of the flow field thermogram by gradually upsampling and feature fusion, and optimizes the detailed information of the flow field thermogram in the process.
[0121] The prediction head module 440 includes two prediction heads in this embodiment, one prediction head is used to predict the flow field thermal map value, and the other prediction head is used to predict the pixel displacement size, so that the network can be optimized specifically for different velocity components, thereby improving the accuracy and robustness of the overall prediction.
[0122] The spatiotemporal neural network model for video speed measurement by fusing adjacent frames provided in this embodiment pays special attention to parameters such as network depth, width, and number of channels to ensure that the network can estimate the flow field efficiently and accurately.
[0123] A third embodiment of the present invention further provides an electronic device, the electronic device comprising:
[0124] at least one processor; and,
[0125] a memory communicatively connected to the at least one processor; wherein,
[0126] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the water flow video speed measurement method of any of the aforementioned embodiments.
[0127] The fourth embodiment of the present invention further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the water flow video speed measurement method described in any of the aforementioned embodiments.
[0128] The fifth embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the water flow video speed measurement method of any of the aforementioned embodiments.
[0129] The sixth embodiment of the present invention further provides a computer program, which includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the water flow video speed measurement method of any of the above embodiments.
[0130] Figure 5 A schematic diagram of a method or device 1000 that can implement an embodiment of the present invention is shown, which may include more or fewer devices than shown in the figure in some embodiments. In some embodiments, it can be implemented using a single or multiple devices. In some embodiments, it can be implemented using cloud or distributed devices.
[0131] like Figure 5 As shown, the device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) 1002 or the program and / or data loaded from the storage part 1008 to the random access memory (RAM) 1003. Processor 1001 can be a multi-core processor or include multiple processors. In some embodiments, processor 1001 can include a general main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1003, various programs and data required for the operation of device 1000 are also stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.
[0132] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.
[0133] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed. Figure 5 Only some components are schematically shown in the figure, which does not mean that the device 1000 only includes Figure 5 Components shown.
[0134] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.
[0135] Although not shown, in an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the water flow video speed measurement method described in Example 1 is implemented.
[0136] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
[0137] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which implements the water flow video speed measurement method described in Example 1 when executed by a processor.
[0138] The methods, programs, systems, devices, etc. of the embodiments of the present invention may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of this specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.
[0139] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.
[0140] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present invention do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.
[0141] In this article, multiple embodiments of the present invention are described, but for the sake of simplicity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to be applicable to at least one embodiment or example of the present invention, but not all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.
[0142] The exemplary systems and methods of the present invention have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present invention as defined in the appended claims.
Claims
1. A method for measuring water flow speed using video. It is characterized in that include: Build datasets and labels; Construct a spatiotemporal neural network model for video speed measurement by fusing adjacent frames; Decomposing the flow field estimation in the area to be measured into flow field thermal map estimation and pixel displacement estimation; Set the time dimension; Construct loss function; Training and evaluating the spatiotemporal neural network model; Configuring the trained spatiotemporal neural network model as a standard model running in different scenarios and under different hardware conditions, and performing real-time monitoring and optimization on the spatiotemporal neural network model; The step of constructing a data set and labels includes: Generate training labels using a flow velocity calculation method, wherein the labels include a water flow field distribution thermal map and a pixel displacement; Cleaning and / or denoising and / or normalizing the data in the dataset; The constructing loss function includes: using an activation function to map the flow field heat map value output by the spatiotemporal neural network model to between 0 and 1; Use the loss function to regress the mapped flow field heat map value; The loss function is used to regress the discrete values of the pixel displacement size.
2. The water flow video velocity measurement method according to claim 1, It is characterized in that The spatiotemporal neural network model of video speed measurement by fusing adjacent frames is based on a two-dimensional convolutional neural network and fuses information between adjacent frames.
3. The water flow video velocity measurement method according to claim 1, It is characterized in that The value range of the flow field thermal map is 0 to 1; The value range of the pixel displacement is 0 to N, where N is a positive integer greater than or equal to 1.
4. The water flow video velocity measurement method according to claim 1, It is characterized in that The time dimension is 32 frames.
5. The water flow video velocity measurement method according to claim 2, It is characterized in that The training and evaluation of the spatiotemporal neural network model includes: adjusting model parameters by stochastic gradient descent or the stochastic gradient descent variant algorithm; Using a test data set to evaluate the performance of the trained spatiotemporal neural network model; The spatiotemporal neural network model is optimized according to the performance evaluation results.
6. A spatiotemporal neural network model for water flow video velocity measurement, It is characterized in that include: An adjacent frame fusion module is used to receive continuous video frames as input, fuse the information between adjacent frames through adjacent frame fusion, and extract the preliminary spatiotemporal characteristics of water flow movement; A feature extraction module is used to extract the temporal and spatial features of pixel displacement from the fused information; The upsampling decoder module is used to generate high-resolution flow field heat maps through upsampling and feature fusion; A prediction head module, wherein the prediction head module includes two prediction heads, which are used to predict the flow field thermal map value and the pixel displacement size respectively; It also includes building datasets and labels, including: Generate training labels using a flow velocity calculation method, wherein the labels include a water flow field distribution thermal map and a pixel displacement; Cleaning and / or denoising and / or normalizing the data in the dataset; It also includes constructing a loss function, including: using an activation function to map the flow field heat map value output by the spatiotemporal neural network model to between 0 and 1; Use the loss function to regress the mapped flow field heat map value; The loss function is used to regress the discrete values of the pixel displacement size.
7. An electronic device, It is characterized in that The electronic device includes: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the water flow video speed measurement method described in any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium, It is characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the water flow video speed measurement method described in any one of claims 1 to 5.
