River surface flow velocity identification method and system based on directional constraint deformable convolution

Abstract

The application provides a river surface flow velocity identification method and system based on direction constraint deformable convolution, which is different from the mode of blindly learning spatial features of a general convolutional neural network, and the application creatively proposes a space-time image flow velocity calculation method based on a direction constraint deformable convolutional neural network for a river surface flow velocity measurement scene. The method adaptively captures water flow texture features through orthogonal and linear sampling shapes, breaks through the limitations of fixed geometry of a standard convolution kernel and square growth of a parameter quantity of a traditional deformable convolution and no physical constraint of an offset, and realizes reduction of a calculation cost while enhancing the geometric modeling capability of the network for textures in space-time images and noise resistance robustness.

Description

Technical Field

[0001] This invention relates to the fields of hydrological element monitoring and computer vision technology, specifically to a method and system for identifying river surface velocity based on directionally constrained deformable convolution, applicable to non-contact hydrological monitoring, smart water conservancy, flood early warning and other application scenarios. Background Technology

[0002] Surface velocity measurement of water bodies such as rivers and open channels is a crucial aspect of hydrological monitoring, water conservancy projects, and disaster early warning. Space-Time Image Velocimetry (STIV) analyzes the movement of floating objects or textures on a fixed velocity measurement line over time to generate a space-time image of the velocity line, such as... Figure 1 As shown in the image, there is a clear linear relationship between the tilt angle of the texture and the water flow velocity. Therefore, accurately extracting the texture angle is a core step in realizing video-based non-contact flow velocity measurement.

[0003] Traditional methods such as Hough transform, Radon transform, or Fourier spectrum analysis perform well in scenes with clear textures and minimal interference. However, in complex real-world environments, factors such as lighting changes, water surface fluctuations, raindrop interference, and shadows often cause texture features to become blurred, fragmented, or subject to severe noise interference, leading to decreased accuracy and insufficient robustness of these methods. In recent years, deep learning technology has made significant progress in image processing, but directly applying general neural networks to STIV texture angle recognition presents the following problems: (1) It relies heavily on a large amount of labeled data, but the cost and difficulty of obtaining labeled data in the STIV field are high; (2) Poor generalization ability; the model has difficulty maintaining good adaptability between different rivers and different shooting conditions. (3) Robustness under complex lighting conditions still needs further research; (4) The network has poor interpretability.

[0004] Therefore, there is an urgent need for a texture angle recognition method that has strong generalization ability, is robust to noise, and can still maintain high accuracy under small sample conditions. Summary of the Invention

[0005] To overcome the above shortcomings, this invention proposes a method for river surface velocity recognition based on orientation-constrained deformable convolution. Unlike the approach of general convolutional neural networks that blindly learn spatial features, this invention innovatively proposes a spatiotemporal image velocity calculation method based on orientation-constrained deformable convolutional neural networks, specifically for river surface velocity measurement scenarios. This method adaptively captures water flow texture features through orthogonal and linear sampling shapes, overcoming the limitations of fixed geometry in standard convolutional kernels and the squared growth of parameters and lack of physical constraints on offset in traditional deformable convolutions. This achieves improved geometric modeling ability and noise robustness of the network for textures in spatiotemporal images while reducing computational overhead. Specifically, it includes the following steps: Step S1: Obtain the video sequence of the river surface, extract the spatiotemporal image, and perform preprocessing to obtain the preprocessed image; Step S2: Based on the structural tensor method, roughly estimate the principal direction of the texture of the spatiotemporal image to obtain the prior angle; Step S3, construct orthogonal convolutional blocks and texture convolutional blocks: First, initialize the sampling shapes of orthogonal convolutional blocks and texture convolutional blocks according to prior angles, and then design an offset prediction mechanism with orientation constraints to constrain the sampling offsets of orthogonal convolutional blocks and texture convolutional blocks respectively. Step S4: Construct a three-stage network architecture: First, perform initial feature extraction, then extract multi-scale features through orthogonal convolution, texture convolution, and standard deformable convolution stages, and finally map the high-dimensional feature map to a 180-dimensional angular classification space through the classification head module to obtain the predicted texture angle; where the orthogonal convolution stage contains several orthogonal convolution blocks, the texture convolution stage contains several texture convolution blocks, and the standard deformable convolution stage contains several standard deformable convolution blocks; Step S5: Combine the video frame time interval with the spatial resolution of the velocity measurement line to convert the predicted texture angle into the river surface velocity.

[0006] Furthermore, the specific implementation of step S2 is as follows: The preprocessed image is divided into Overlapping or non-adjacent sub-blocks , Indicates the first Sub-blocks; using the Sobel operator and Solve for the gray-level gradient:

[0007] For each sub-block Calculate the grayscale gradient tensor elements:

[0008]

[0009]

[0010] in, Representing an image In spatial location grayscale gradient at that location Representing an image In time gradient of direction, Indicates the sub-block Integral or summation of the squared directional gradient. Indicates the sub-block The integral of the squared directional gradient. Indicates the sub-block gradient and The integral of the product of gradients; then the first... The main direction of the texture of each block It is given by the following formula:

[0011] At the same time, a correlation parameter is introduced. To characterize the clarity of the texture of this sub-block:

[0012] ; Finally, a weighted average of the angles of all sub-blocks is taken to obtain the global coarse angle. :

[0013] The range of values ​​is .

[0014] Furthermore, initializing the sampling shape of the orthogonal convolution block includes: For a given number of sampling points Let the number of sampling points in the principal direction, i.e., the prior angle direction, be... The number of sampling points in the vertical direction is ,satisfy ; Main direction coordinates ( , The generation method is as follows: ; Vertical coordinates ( During generation, to avoid duplication with the center point of the main direction, the origin position is excluded. Coordinates:

[0015] ; like If the number is odd, skip duplicate center coordinates during merging; After concatenating the coordinates of the principal direction and the vertical direction, based on the prior angle... Apply a two-dimensional rotation transformation, with a rotation angle of . The orthogonal initial sampling shape is obtained. , among which the former Each point belongs to the main direction, and then... Each point belongs to the vertical direction; Concatenate all sampled coordinates into a tensor form , among which the former Storage in one channel Coordinates, after Storage in one channel Coordinates, serving as a buffer; This refers to the initial set of sampling coordinates for a directionally constrained linear deformable convolution module, used to define the basic sampling position distribution of the convolution kernel before an offset is applied.

[0016] Furthermore, initializing the texture convolution block sampling shape includes: For a given number of sampling points First, generate the data along the horizontal axis. shaft Equally spaced coordinates, and then centered: ; in, and They refer to the first The horizontal direction of each sampling point in the spatial coordinate system is... Axis and vertical direction The initial coordinate components of the axis, This indicates a floor operation; centering ensures the sample shape is about the origin. symmetry; Subsequently, based on the prior angle Apply a two-dimensional rotation transformation to rotate an axis-aligned straight line shape to the target orientation: ; in, The coordinates are in radians and after rotation. This refers to the initial sampling shape as a straight line, which is the sampling shape of the texture convolution block.

[0017] 5. The river surface velocity identification method based on orientation-constrained deformable convolution as described in claim 1, characterized in that: a scalar offset prediction strategy is used in step S4, which only predicts... The channel offset tensor is directly mapped to a two-dimensional offset using the direction basis vectors, significantly reducing the number of parameters. The specific implementation is as follows: (1) Offset prediction: through Convolutional layer Predicting scalar offset tensor : ; in, For the input feature map, and The number of output channels is a learnable parameter. , Indicates the first The sample, the first Position, number scalar offset value of each sampling point; (2) Direction vector calculation: based on the current texture angle Calculate the direction unit vector, where For learnable small-angle correction parameters, As a priori perspective; The principal direction unit vector is: ; The vertical unit vector is: ; (3) Linear mapping strategy: For linear convolutional blocks, i.e. texture convolutional blocks, all Each sampling point is along Directional distribution directly maps scalar offsets to two-dimensional offset components: ; in, and The first Each sampling point is at and The final offset of the direction; (4) Orthogonal mapping strategy: For orthogonal convolutional blocks, the first Each sampling point belongs to the main direction, and then... Each sampling point belongs to the vertical direction; The main direction offset mapping is the same as the linear strategy; Vertical offset Apply amplitude constraints to prevent disruption of the orthogonal structure: ; in, Let be the hyperbolic tangent function, mapping any real number to... interval, The constraint coefficient hyperparameter controls the maximum allowable offset in the vertical direction; The constrained scalar is then mapped to two-dimensional offset components: ; in, and They represent the first Each sampling point is at and Offset in direction; (5) Final sampling coordinate calculation: All The two-dimensional offset components of each sampling point are concatenated into a tensor. , and the initial sampling coordinates Basic grid coordinates Add them together to get the final sampling position: ; in, The basic grid coordinates corresponding to each position in the output feature map are calculated using a grid generation algorithm.

[0018] Furthermore, step S4 also includes learning a weight coefficient for each sampling point to characterize its importance; through independent Convolutional layer prediction Channel modulation tensor ,pass Activation function mapping to Interval: ; in, express function, Indicates the first The sample, the first Position, number Modulation weights for each sampling point; Based on the final sampling location The input feature map is obtained through bilinear interpolation. The mid-sampled feature values ​​specifically include: Neighborhood coordinate calculation: for each sampling point Calculate the integer coordinates of its four neighborhoods: ; in, Indicates rounding down, subscript These represent the four neighboring points: top left, bottom right, bottom left, and top right, respectively. The superscripts x and y represent the x-axis and y-axis directions, respectively. Calculate the weights of four neighboring points using bilinear interpolation , , , ; Weighted Sampling and Modulation: Obtaining Feature Values ​​of Four Neighborhoods via Tensor Indexing Operations , Weighted summation and application of modulation weights: ; in, The output features are obtained by convolution sampling and weight modulation; Feature rearrangement and fusion: all The features of each sampling point are stacked along the spatial dimension, and then rearranged. Change the shape of the tensor, and finally through Column convolution completes feature fusion and channel transformation: ; The kernel size of column convolution is... Step size is This enables feature aggregation along the sampling point direction.

[0019] Furthermore, in the orthogonal convolution stage, let the first... The input to each orthogonal convolutional block is The output is Each orthogonal convolutional block contains two cascaded, direction-constrained, linearly deformable convolutional layers. The calculation process is as follows: ; ; ; in, This represents the orthogonal convolution operation. This represents a residual join mapping. It is an identity mapping when the input and output dimensions are the same, otherwise it is not. convolution; Wherein, the orthogonal convolution operation The specific calculation process is as follows: Firstly, through Convolutional layer predict Scalar offset tensor of the channel and Channel modulation weight tensor : ; in, The number of sampling points. for function, Input feature map; Based on the current texture angle Calculate the unit vector of the principal direction and the unit vector in the vertical direction Mapping scalar offsets to two-dimensional constraint offsets ; For the sampling points in the main direction : ; For vertical sampling points : ; in, This is the vertical constraint coefficient; Finally, the output is obtained by bilinear interpolation sampling and weighted summation: ; in, For column convolution operations, For feature rearrangement operations, This is a bilinear interpolation operation; This outputs the base grid coordinates for each location in the feature map. These are the initial sampling coordinates.

[0020] Furthermore, in the texture convolution stage, the first texture convolution block is downsampled by a factor of 2, and let the... The input to each texture convolution block is The output is The calculation process is as follows: ; ; ; in, Stride size; Texture convolution operation Similar to orthogonal convolution, the difference is that the sampling shape is linear, and all sampling points are along... Direction mapping offset.

[0021] Furthermore, in the standard deformable convolution stage, the first standard deformable convolution block... Perform a 2x downsampling; let the first... indivual The input is The output is The calculation process is as follows: ; ; ; in, The stride is specified; standard deformable convolution operation. The specific calculation process is as follows: Firstly, through two independent Convolutional layer Predict separately Two-dimensional offset tensor of each channel and Modulation weight tensor of each channel :

[0022] in, Each channel corresponds to convolution kernel Each sampling point 3D offset, without directional constraints The input feature map is used; the initial sampling shape is fixed. Grid The final sampling coordinates are: ; in, These are the initial sampling coordinates. Based on the base grid coordinates; sampled using bilinear interpolation and applied with modulation weights, finally multiplied and summed with the convolution kernel weights:

[0023] in, For deformable convolution kernel weights, This is a bias term.

[0024] The present invention also provides a river surface velocity recognition system based on orientation-constrained deformable convolution, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute the river surface velocity recognition method based on orientation-constrained deformable convolution as described in the above technical solution.

[0025] The present invention provides a method for identifying river surface velocity based on orientation-constrained deformable convolution, which has the following advantages: (1) High recognition accuracy: The deformable convolution with directional constraints is adopted to ensure that the sampling points are strictly distributed along the texture direction or orthogonal direction, avoiding feature aliasing caused by the mismatch between the standard convolution sampling grid and the water flow texture direction. Combined with the multi-stage feature fusion architecture, the angle resolution capability and classification accuracy are effectively improved.

[0026] (2) Strong robustness: The sampling points of the convolution kernel strictly maintain collinear or orthogonal distribution to match the physical characteristics of water flow texture, thereby enhancing the network's ability to geometrically model textures in spatiotemporal images and its noise resistance robustness.

[0027] (3) Good adaptability to small samples: The prior angle estimated by the gray gradient tensor method guides the network initialization, which reduces the learning burden of data-driven learning. Combined with the scalar offset prediction mechanism, the number of parameters of the offset prediction layer is halved, which effectively prevents overfitting in small sample training scenarios.

[0028] (4) Good generalization: The network focuses on the physical nature of water flow texture rather than background noise through the directional constraint mechanism, and is suitable for flow velocity measurement tasks under different river cross sections, camera angles and shooting conditions. Attached Figure Description

[0029] Figure 1 This is a spatiotemporal image of the STIV method velocity measurement line; Figure 2 This is a schematic diagram of the overall process of the method of the present invention; Figure 3 This is a schematic diagram of the orientation-constrained deformable convolutional neural network architecture of the present invention; Figure 4 This is an example of texture corner feature recognition in complex environments according to the present invention; Figure 5 These are the accuracy, recall, and F1 score on the validation set during model iteration. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings.

[0031] This invention provides a method for identifying river surface velocity based on orientation-constrained deformable convolution, the overall flowchart of which is shown below. Figure 2 As shown, the specific steps include: Step 1: Image preprocessing; The input is a batch of spatiotemporal imagery of speed measurement lines, and its tensor shape is... ,in 1 represents the batch size, and 1 represents the number of channels (grayscale image). Image height (spatial orientation) The image width (time direction) is denoted as . Necessary preprocessing is performed on the input image: first, it is converted to grayscale (skipped if the input is already grayscale), then median filtering is used to remove noise, and finally histogram equalization is used to enhance contrast. The preprocessed image is denoted as . ,in Represents spatial coordinates, Represents the time coordinate.

[0032] Step 2: Estimate the global angle based on the gray-level gradient tensor method; This step utilizes the classic gray-level gradient tensor method (structure tensor method) to perform a coarse global angle estimation on the preprocessed spatiotemporal image, obtaining the prior angle. This method fully utilizes the physical prior that texture angles in spatiotemporal imagery of velocity lines exhibit a diagonal (striped) distribution. To improve noise resistance and obtain the globally dominant direction, the image is divided into... Overlapping or non-adjacent sub-blocks (This invention uses sub-blocks of 16×16 pixels with a step size of 16 pixels.) Indicates the first Sub-blocks. Using the Sobel operator. and Solve for the gray-level gradient:

[0033] For each sub-block Calculate the grayscale gradient tensor elements:

[0034]

[0035]

[0036] in, Representing an image In spatial location The grayscale gradient at that location (by...) (obtained by convolution) Representing an image In time gradient of direction (by) (obtained by convolution) Indicates the sub-block Integral (or summation) of the squared directional gradient. Indicates the sub-block The integral of the squared directional gradient. Indicates the sub-block (Spatial) gradient and The integral of the product of (time) gradients (mixed second moment).

[0037] Then the first The main direction of the texture of each block It is given by the following formula:

[0038] At the same time, a correlation parameter is introduced. To characterize the clarity of the texture of this sub-block:

[0039] The larger the value, the clearer the texture and the more reliable the angle estimation.

[0040] Finally, a weighted average of the angles of all sub-blocks is taken to obtain the global coarse angle. The range of values ​​is :

[0041] Step 3: Design of the core module of the orientation-constrained deformable convolutional network; This step constructs the core computing unit of this invention—direction-constrained deformable convolution (…). This computational unit aims to address the problems of standard convolution's inability to adapt to changes in water flow texture direction and the large number of parameters and scattered sampling points in standard deformable convolution.

[0042] 3.1 Initial sampling shape generation algorithm (orthogonal and linear types); For a given number of sampling points (That is, the number of convolution kernel parameters, which supports any positive integer and achieves linear parameter growth), and generates the corresponding initial sampling coordinate set according to the convolution block type:

[0043] in, This refers to the initial set of sampling coordinates for a direction-constrained linear deformable convolution module, used to define the basic sampling position distribution of the convolution kernel before an offset is applied; and They refer to the first The horizontal direction of each sampling point in the spatial coordinate system ( (axis) and vertical direction ( (Axis) Initial coordinate components.

[0044] (1) Linear sampling shape generation (for texture convolution blocks) When the convolution block type is linear At that time, the sampling points are linearly arranged along a single direction. First, samples are generated along the horizontal axis ( (axis) Equally spaced coordinates, and then centered:

[0045] in, This indicates a floor operation; centering ensures the sample shape is about the origin. symmetry.

[0046] Subsequently, based on the prior angle (Unit: degrees) Apply a 2D rotation transformation to rotate an axis-aligned straight line shape to the target direction:

[0047] in, The coordinates are in radians and after rotation. This is the initial sampling shape of a straight line.

[0048] (2) Orthogonal sampling shape generation (for orthogonal convolution blocks) When the convolution block type is orthogonal ( When sampling points are distributed along two mutually perpendicular directions, they form a cross structure.

[0049] Let the main direction be ( The number of sampling points (direction) is Vertical direction ( The number of sampling points (direction) is ,satisfy .

[0050] The method for generating the principal direction coordinates is as follows:

[0051] When generating vertical coordinates, to avoid duplication with the center point of the main direction, the origin position is excluded, and the generated coordinates are... Coordinates:

[0052]

[0053] like If the number is odd, duplicate center coordinates must be skipped during merging.

[0054] After concatenating the coordinates of the principal direction and the vertical direction, the same two-dimensional rotation transformation matrix as described above is applied, with a rotation angle of . The orthogonal initial sampling shape is obtained. Among them, the former Each point belongs to the main direction, and then... Each point belongs to the vertical direction.

[0055] Concatenate all sampled coordinates into a tensor form , among which the former Storage in one channel Coordinates, after Storage in one channel Coordinates, used as a buffer in the network (not updated during backpropagation).

[0056] 3.2 Scalar Offset Prediction and Direction Mapping Mechanism: In standard deformable convolution, the offset prediction layer output The channel offset tensor. This invention innovatively employs a scalar offset prediction strategy, where the network only predicts the channel offset tensor. The channel offset tensor is directly mapped to a two-dimensional offset through the direction basis vector, which significantly reduces the number of parameters.

[0057] (1) Offset prediction: through Convolutional layer prediction scalar offset tensor :

[0058] in, For the input feature map, and The number of output channels is a learnable parameter. , Indicates the first The sample, the first Position, number Scalar offset value for each sampling point.

[0059] (2) Direction vector calculation: based on the current texture angle in Calculate the direction unit vector (for learnable small angle correction parameters).

[0060] The principal direction unit vector is:

[0061] The vertical unit vector is:

[0062] (3) Linear mapping strategy: For linear convolutional blocks, all Each sampling point is along Directional distribution. Scalar offset is directly mapped to two-dimensional offset components:

[0063] in, and The first Each sampling point is at and The final offset in direction. This mapping ensures that the offset direction of all sampling points is strictly parallel to... .

[0064] (4) Orthogonal mapping strategy: For orthogonal convolutional blocks, the first Each sampling point belongs to the main direction, and then... Each sampling point belongs to the vertical direction.

[0065] The main direction offset mapping is the same as the linear strategy.

[0066] Vertical offsets require amplitude constraints to prevent disruption of the orthogonal structure.

[0067] in, Let be the hyperbolic tangent function, mapping any real number to... interval, As a constraint coefficient hyperparameter, controlling the maximum allowable offset in the vertical direction, this invention takes... =0.5.

[0068] The constrained scalar is then mapped to two-dimensional offset components:

[0069] in, and They represent the first Each sampling point at and Offset in direction; (5) Final sampling coordinate calculation: All The two-dimensional offset components of each sampling point are concatenated into a tensor. , and the initial sampling coordinates Basic grid coordinates Add them together to get the final sampling position:

[0070] in, The basic grid coordinates corresponding to each position in the output feature map are calculated using a grid generation algorithm.

[0071] 3.3, Deformable convolutional modulation weight mechanism; To enhance the adaptive capability of feature selection, a weight coefficient is learned for each sampling point to represent its importance. This is achieved through independent... Convolutional layer prediction Channel modulation tensor ,pass Activation function mapping to Interval:

[0072] in, express function, Indicates the first The sample, the first Position, number The modulation weights of each sampling point. The closer the weight value is to 1, the more important the feature of that sampling point is; the closer it is to 0, the more suppressed the sampling point is.

[0073] 3.4, Bilinear Interpolation Sampling and Feature Rearrangement; Based on the final sampling location The input feature map is obtained through bilinear interpolation. Sampling feature values.

[0074] (1) Neighborhood coordinate calculation: For each sampling point Calculate the integer coordinates of its four neighborhoods:

[0075] in, Indicates rounding down. These represent the four neighboring points: top left, bottom right, bottom left, and top right, respectively.

[0076] (2) Bilinear interpolation weight calculation:

[0077]

[0078]

[0079]

[0080] (3) Weighted sampling and modulation: The feature values ​​of the four neighborhoods are obtained through tensor indexing operations. , Weighted summation and application of modulation weights:

[0081] (4) Feature rearrangement and fusion: all The features of each sampling point are stacked along the spatial dimension, and then rearranged. Transform the tensor shape from Convert to Finally passed Column convolution completes feature fusion and channel transformation:

[0082] The kernel size of column convolution is... Step size is This enables feature aggregation along the sampling point direction.

[0083] Step 4: Specific implementation of orthogonal convolution blocks and texture convolution blocks; Based on the above The core module constructs two types of specialized convolutional blocks: 4.1 Orthogonal Convolutional Layer: Inherited from Module, set during initialization Number of sampling points (This invention takes) ,Right now Vertical constraint coefficient Take 0.5.

[0084] Used in the early stages of the network, it captures the local directionality and cross-structure features of texture through cross-sampling. It receives the input feature map. ,predict Channel scalar offset, based on point affiliation (front) One or after Each feature map is mapped to either the principal direction or the vertical direction to generate orthogonal sampling coordinates. Bilinear interpolation sampling and feature rearrangement are then performed, and the fused feature map is output. .

[0085] 4.2 Texture Convolutional Layer: Inherited from Module, set during initialization Number of sampling points odd number (take) The vertical constraint coefficient is fixed at 1. (That is, vertical offset is completely prohibited).

[0086] Used in the mid-stage of the network, it performs linear sampling along the main texture direction to finely extract continuous texture features along the water flow direction. Similar to orthogonal convolution blocks, the difference is that the sampling shape is linear and all scalar offsets are mapped to... direction.

[0087] Both types of convolutional layers include batch normalization layers. )and Activation functions are used to enhance training stability and nonlinear expressive power.

[0088] Step 5: Overall network architecture design; This invention employs a method similar to A multi-stage architecture is used to construct a direction-constrained deformable convolutional neural network (DDCNet) for multi-stage progressive feature extraction, such as... Figure 3 As shown, this network employs a residual connection architecture, fusing directionally constrained linear deformable convolutions with standard deformable convolutions. Through a strategy of capturing local texture structures early, extracting directional continuous features in the middle stage, and fusing global contextual information in the later stage, it achieves high-precision texture corner recognition.

[0089] 5.1 Initial Feature Extraction: The preprocessed spatiotemporal image is denoted as... ,in For batch size, Image height, The image width is [value]. The initial feature extraction process is as follows: First, the preprocessed image I goes through a... A two-dimensional convolutional layer with a stride of 2 and padding of 2 is used to increase the number of channels from 1 to 64. This is followed by a batch normalization layer, a non-linear activation function, and a max-pooling layer. The stride is 2. Let the convolution output be... The calculation process is as follows:

[0090] in, express Convolution operation, This indicates a batch normalization operation. This represents the linear rectification activation function.

[0091] 5.2, Orthogonal Convolution Stage: Contains two orthogonal convolution blocks, employing a residual structure to maintain spatial resolution. Let the... The input to each orthogonal convolutional block is The output is ,in Each block contains two cascaded, orientation-constrained, linearly deformable convolutional layers. The calculation process is as follows:

[0092]

[0093]

[0094] in, This represents the orthogonal convolution operation. This represents a residual join mapping (an identity mapping when the input and output dimensions are the same, otherwise it is an identity mapping). convolution).

[0095] Wherein, the orthogonal convolution operation The specific calculation process is as follows: Firstly, through Convolutional layer prediction Scalar offset tensor of the channel and Channel modulation weight tensor :

[0096] in, The number of sampling points. for function.

[0097] Based on the current texture angle Calculate the unit vector of the principal direction and the unit vector in the vertical direction Mapping scalar offsets to two-dimensional constraint offsets : For the sampling points in the main direction :

[0098] For vertical sampling points :

[0099] in, This is the vertical constraint coefficient.

[0100] Finally, the output is obtained by bilinear interpolation sampling and weighted summation:

[0101] in, For column convolution operations, For feature rearrangement operations, This is a bilinear interpolation operation.

[0102] The final output of the orthogonal convolution stage is denoted as .

[0103] 5.3, Texture Convolution Stage: Contains two texture convolution blocks, using a residual structure. The first block undergoes a 2x downsampling. Let the... The input to each texture convolution block is The output is ,in The calculation process is as follows:

[0104]

[0105]

[0106] Among them, when convolution kernel stride The spatial resolution is reduced to half of its original value; when hour, .

[0107] The texture convolution operation Similar to orthogonal convolution, the difference is that the sampling shape is linear, and all sampling points are along... Direction mapping offset, vertical direction constraint coefficient Number of sampling points .

[0108] The final output of the texture convolution stage is denoted as .

[0109] 5.4, ​​Standard Deformable Convolution Stage: Contains 2 standard deformable convolution blocks ( The residual structure is used, and the first block is downsampled by a factor of 2. This stage uses standard... This module supports adaptive sampling in any direction. Let the first... indivual The input of the block is The output is ,in The calculation process is as follows:

[0110]

[0111]

[0112] Among them, when hour, The spatial resolution is reduced to half of its original value; when hour, .

[0113] The standard operate The specific calculation process is as follows: Firstly, through two independent Convolutional layers predict respectively Two-dimensional offset tensor of the channel and Channel modulation weight tensor :

[0114] in, Corresponding channel convolution kernel Each sampling point 3D offset, with no directional constraints. The initial sampling shape is fixed. Grid The final sampled coordinates are:

[0115] The sample is obtained by bilinear interpolation and modulation weights are applied. Finally, the result is multiplied by the convolution kernel weights and summed.

[0116] in, For deformable convolution kernel weights, This is a bias term.

[0117] The final output of the standard deformable convolution stage is denoted as .

[0118] 5.5 Classification Head Module: The classification head module maps the high-dimensional feature map to a 180-dimensional angular classification space. The calculation process is as follows: After a globally adaptive average pooling layer This reduces the spatial resolution to 2 in both the height and width dimensions. Let the pooling output be... The calculation process is as follows:

[0119] The feature map is flattened into a one-dimensional vector and mapped to it through a fully connected layer. dimension

[0120]

[0121] in, This indicates a fully connected layer operation. For flattening operation, This is the original score vector (i.e., logits) output by the network. The Dimensional value The corresponding angle category is The unnormalized score, For angle category index.

[0122] During reasoning, through The function will Convert to a probability distribution, and take the angle category with the highest probability as the predicted texture angle. :

[0123] Step 6: Loss function design; Using the cross-entropy loss function:

[0124] in, Indicates the first Each sample is predicted as an angle category. of value, This is a label for the actual angle.

[0125] Step 7: Training and Reasoning Process; 7.1 Training Strategy: A two-stage training strategy is adopted, with a batch size of 32. Weight decay .

[0126] Phase 1: Fixing Angle Correction Parameters Only optimize the main network parameters, learning rate 50 rounds of training ( .

[0127] Phase Two: Thawing By jointly optimizing the perspective and network, the learning rate was reduced to Fine-tuning was then performed.

[0128] 7.2, Reasoning Flow: Input after preprocessing Image, network forward propagation output 180 dimensions ,through Convert to a probability distribution, and take the angle category with the highest probability as the prediction result. .

[0129] Step 8: Calculate the surface velocity of the river; Based on the identified texture angle (Unit: degrees), combined with the spatial resolution of the speed measurement line (Unit: meters / pixel) and video frame time interval (Unit: seconds / frame) Calculate the surface velocity of the river. :

[0130] The following specific implementation case illustrates the implementation effect of the present invention.

[0131] Figure 4 This illustration demonstrates the application of the invention in different video environments, showing that the invention maintains good performance even under conditions of blur, obstacles, and ripples. For objective evaluation, the invention uses precision, recall, and F1 score to evaluate model performance. Precision is the percentage of correctly predicted results out of the total samples; recall is the probability of a sample that is actually positive being predicted as positive; the F1 score considers both precision and recall, aiming to maximize both while achieving a balance. The expressions for these three are as follows:

[0132]

[0133]

[0134] Where TP represents a correct prediction that actually occurred; FP represents a correct prediction that actually occurred; FN represents a wrong prediction that actually occurred; and TN represents a wrong prediction that actually occurred. The accuracy rate is TP / (TP+FP).

[0135] Figure 5The results for accuracy, loss, F1 score, and recall during the training and validation periods with 100 model iterations show that the model begins to converge around 50 iterations. During the validation period, the accuracy, F1 score, and recall are all around 0.9, and the loss is around 0.2.

[0136] To verify the generalization and reliability of the model, the constructed network was applied to data from a certain hydrological station. The camera, model DS-2DF8C842IXS, was installed approximately 20 m above the water surface, with a focal length range of 7.5 to 300 mm. Water surface information was captured every 15 minutes for 30 seconds, with an image size of 3840×2160.

[0137] Table 1 compares the results of different STIV calculation methods with those of the rotor current meter. Both the gradient tensor method (GT) and the two-dimensional Fourier method (FFT) tend to underestimate the cross-sectional velocity of the river. The average cross-sectional velocity calculated by the GT method is 1.59 m / s, and by the FFT method it is 1.39 m / s, both lower than the 1.67 m / s calculated by the rotor current meter. The GT method shows relatively stable results, with average relative error and RMSE of 12.17% and 0.19, respectively. The FFT method shows greater fluctuations, with average relative error and RMSE reaching 20.37% and 0.30, respectively. The results of the method of this invention are very close to those of the manual labeling method, and are more stable than those of the GT and FFT methods. The errors of the average cross-sectional velocity, average relative error, and RMSE of the method of this invention and the manual labeling method compared with the measured results are within 0.1 m / s, 10%, and 0.20, respectively.

[0138] Table 1. Comparison of errors from different methods

[0139] On the other hand, embodiments of the present invention also provide a river surface velocity recognition system based on orientation-constrained deformable convolution, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute the river surface velocity recognition method based on orientation-constrained deformable convolution as described in the above technical solution.

[0140] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0141] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for identifying river surface velocity based on orientation-constrained deformable convolution, characterized in that, Includes the following steps: Step S1: Obtain the video sequence of the river surface, extract the spatiotemporal image, and perform preprocessing to obtain the preprocessed image; Step S2: Based on the structural tensor method, roughly estimate the principal direction of the texture of the spatiotemporal image to obtain the prior angle; Step S3, construct orthogonal convolutional blocks and texture convolutional blocks: First, initialize the sampling shapes of orthogonal convolutional blocks and texture convolutional blocks according to prior angles, and then design an offset prediction mechanism with orientation constraints to constrain the sampling offsets of orthogonal convolutional blocks and texture convolutional blocks respectively. Step S3 employs a scalar offset prediction strategy, predicting only... The channel offset tensor is directly mapped to a two-dimensional offset using the direction basis vectors, significantly reducing the number of parameters. The specific implementation is as follows: (1) Offset prediction: through Convolutional layer Predicting scalar offset tensor : ； in, For the input feature map, and The number of output channels is a learnable parameter. , Indicates the first The sample, the first Position, number scalar offset value of each sampling point; (2) Direction vector calculation: based on the current texture angle Calculate the direction unit vector, where For learnable small-angle correction parameters, As a priori perspective; The principal direction unit vector is: ; The vertical unit vector is: ; (3) Linear mapping strategy: For linear convolutional blocks, i.e. texture convolutional blocks, all Each sampling point is along Directional distribution directly maps scalar offsets to two-dimensional offset components: ； in, and The first Each sampling point is and The final offset of the direction; (4) Orthogonal mapping strategy: For orthogonal convolutional blocks, the first Each sampling point belongs to the main direction, and then... Each sampling point belongs to the vertical direction; The main direction offset mapping is the same as the linear strategy; Vertical offset Apply amplitude constraints to prevent disruption of the orthogonal structure: ； in, Let be the hyperbolic tangent function, mapping any real number to... interval, The constraint coefficient hyperparameter controls the maximum allowable offset in the vertical direction; The constrained scalar is then mapped to two-dimensional offset components: ； in, and They represent the first Each sampling point is and Offset in direction; (5) Final sampling coordinate calculation: All The two-dimensional offset components of each sampling point are concatenated into a tensor. , and the initial sampling coordinates Basic grid coordinates Add them together to get the final sampling position: ； in, The basic grid coordinates corresponding to each position in the output feature map are calculated using a grid generation algorithm. Step S4: Construct a three-stage network architecture: First, perform initial feature extraction, then extract multi-scale features through orthogonal convolution, texture convolution, and standard deformable convolution stages, and finally map the high-dimensional feature map to a 180-dimensional angular classification space through the classification head module to obtain the predicted texture angle; where the orthogonal convolution stage contains several orthogonal convolution blocks, the texture convolution stage contains several texture convolution blocks, and the standard deformable convolution stage contains several standard deformable convolution blocks; Step S5: Combining the spatial resolution of the velocity measurement line with the video frame time interval, the predicted texture angle is converted into the river surface velocity.

2. The river surface velocity identification method based on orientation-constrained deformable convolution as described in claim 1, characterized in that: The specific implementation method of step S2 is as follows: The preprocessed image is divided into Overlapping or non-adjacent sub-blocks , Indicates the first Sub-blocks; using the Sobel operator and Solve for the gray-level gradient: For each sub-block Calculate the grayscale gradient tensor elements: in, Representing an image In spatial location grayscale gradient at that location Representing an image In time gradient of direction, Indicates the sub-block Integral or summation of the squared directional gradient. Indicates the sub-block The integral of the squared directional gradient. Indicates the sub-block gradient and The integral of the product of gradients; then the first... The main direction of the texture of each block It is given by the following formula: At the same time, a correlation parameter is introduced. To characterize the clarity of the texture of this sub-block: ； Finally, a weighted average of the angles of all sub-blocks is taken to obtain the prior angle. : The range of values ​​is .

3. The river surface velocity identification method based on orientation-constrained deformable convolution as described in claim 1, characterized in that: Initializing the sampling shape of orthogonal convolution blocks includes: For a given number of sampling points Let the number of sampling points in the principal direction, i.e., the prior angle direction, be... The number of sampling points in the vertical direction is ,satisfy ; Main direction coordinates ( , The generation method is as follows: ； Vertical coordinates ( During generation, to avoid duplication with the center point of the main direction, the origin position is excluded. Coordinates: ； like If the number is odd, skip duplicate center coordinates during merging; After concatenating the coordinates of the principal direction and the vertical direction, based on the prior angle... Apply a two-dimensional rotation transformation, with a rotation angle of . The orthogonal initial sampling shape is obtained. , among which the former Each point belongs to the main direction, and then... Each point belongs to the vertical direction; Concatenate all sampled coordinates into a tensor form , among which the former Storage in one channel Coordinates, after Storage in one channel Coordinates, serving as a buffer; This refers to the initial set of sampling coordinates for a directionally constrained linear deformable convolution module, used to define the basic sampling position distribution of the convolution kernel before an offset is applied.

4. The river surface velocity identification method based on orientation-constrained deformable convolution as described in claim 1, characterized in that: Initializing the texture convolution block sampling shape includes: For a given number of sampling points First, generate the data along the horizontal axis. shaft Equally spaced coordinates, and then centered: ； in, and They refer to the first The horizontal direction of each sampling point in the spatial coordinate system is... Axis and vertical direction The initial coordinate components of the axis, This indicates a floor function, and centering ensures that the sampled shape is about the origin. symmetry; Subsequently, based on the prior angle Apply a two-dimensional rotation transformation to rotate an axis-aligned straight line shape to the target orientation: ； in, The coordinates are in radians and after rotation. This refers to the initial sampling shape as a straight line, which is the sampling shape of the texture convolution block.

5. The river surface velocity identification method based on orientation-constrained deformable convolution as described in claim 1, characterized in that: Step S4 also includes learning a weight coefficient for each sampling point to characterize its importance; Through independent Convolutional layer prediction Channel modulation tensor ,pass Activation function mapping to Interval: ； in, express function, Indicates the first The sample, the first Position, number Modulation weights for each sampling point; Based on the final sampling location The input feature map is obtained through bilinear interpolation. The mid-sampled feature values ​​specifically include: Neighborhood coordinate calculation: for each sampling point Calculate the integer coordinates of its four neighborhoods: ； in, Indicates rounding down, subscript These represent the four neighboring points: top left, bottom right, bottom left, and top right, respectively. The superscripts x and y represent the x-axis and y-axis directions, respectively. Calculate the weights of four neighboring points using bilinear interpolation , , , ; Weighted Sampling and Modulation: Obtaining Feature Values ​​of Four Neighborhoods via Tensor Indexing Operations , Weighted summation and application of modulation weights: ； in, The output features are obtained by convolution sampling and weight modulation; Feature rearrangement and fusion: All The features of each sampling point are stacked along the spatial dimension, and then rearranged. Change the shape of the tensor, and finally through Column convolution completes feature fusion and channel transformation: ； The kernel size of column convolution is... Step size is This enables feature aggregation along the sampling point direction.

6. The river surface velocity identification method based on orientation-constrained deformable convolution as described in claim 1, characterized in that: In the orthogonal convolution stage, let the first... The input to each orthogonal convolutional block is The output is Each orthogonal convolutional block contains two cascaded, direction-constrained, linearly deformable convolutional layers. The calculation process is as follows: ； ； ； in, This represents the orthogonal convolution operation. This represents a residual join mapping. It is an identity mapping when the input and output dimensions are the same, otherwise it is not. convolution; Wherein, the orthogonal convolution operation The specific calculation process is as follows: Firstly, through Convolutional layer predict Scalar offset tensor of the channel and Channel modulation weight tensor : ； in, The number of sampling points. for function, Input feature map; Based on the current texture angle Calculate the unit vector of the principal direction and the vertical unit vector Mapping scalar offsets to two-dimensional constraint offsets ; For the sampling points in the main direction : ； For vertical sampling points : ； in, This is the vertical constraint coefficient; Finally, the output is obtained by bilinear interpolation sampling and weighted summation: ； in, For column convolution operations, For feature rearrangement operations, This is a bilinear interpolation operation; This outputs the base grid coordinates for each location in the feature map. These are the initial sampling coordinates.

7. The river surface velocity identification method based on orientation-constrained deformable convolution as described in claim 6, characterized in that: In the texture convolution stage, the first texture convolution block is downsampled by a factor of 2. Let the... The input to each texture convolution block is The output is The calculation process is as follows: ； ； ； in, Stride size; Texture convolution operation Similar to orthogonal convolution, the difference is that the sampling shape is linear, and all sampling points are along... Direction mapping offset.

8. The river surface velocity identification method based on orientation-constrained deformable convolution as described in claim 1, characterized in that: Standard deformable convolution stage, the first standard deformable convolution block Perform a 2x downsampling; let the first... indivual The input is The output is The calculation process is as follows: ； ； ； in, The stride is specified; standard deformable convolution operation. The specific calculation process is as follows: Firstly, through two independent Convolutional layer Predict separately Two-dimensional offset tensor of each channel and Modulation weight tensor of each channel : in, Each channel corresponds to convolution kernel Each sampling point 3D offset, without directional constraints The input feature map is used; the initial sampling shape is fixed. Grid The final sampling coordinates are: ； in, These are the initial sampling coordinates. Based on the base grid coordinates; sampled using bilinear interpolation and applied with modulation weights, finally multiplied and summed with the convolution kernel weights: in, For deformable convolution kernel weights, This is a bias term.

9. A river surface velocity recognition system based on orientation-constrained deformable convolution, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the program instructions in the memory to execute the river surface velocity identification method based on orientation-constrained deformable convolution as described in any one of claims 1-8.

Images