A method for producing a real water scene optical flow dataset

Abstract

The present invention provides a method for producing an optical flow dataset of a real water surface scene, which is specifically as follows: 1: acquiring a dynamic image of the flow on the water surface to obtain a sequence of continuous frame images; 2: performing orthorectification on the image according to the position and posture information when the image is captured and the orientation elements in the camera; 3: forming an image pair from a current frame image and a next frame image, and adopting an improved multi-level pyramid grid iterative interpolation flow field estimation method to obtain the optical flow field of the image pair; 4: performing flow field motion error elimination on the obtained optical flow field; 5: using the optical flow field after error elimination as an optical flow label, and using the optical flow label to perform forward warping mapping on the current frame image using a feature warping mapping method to obtain a new image; 6: adopting a bilinear interpolation method to interpolate and fill the new image; 7: constructing a dataset sample; 8: repeating steps 1 to 7 until the characteristics and quantity of the obtained data samples meet the requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for producing an optical flow data set of a real water surface scene. Background Art

[0002] River velocity is an important indicator of hydrodynamic characteristics and plays a key role in flow estimation, ecological and environmental protection, and flood warning. Traditional physical measurement methods, such as those using acoustic Doppler current profilers (ADCPs) or rotating velocimeters, have limited detection ranges, cannot be deployed over large areas, and cannot meet the current intelligent requirements of water conservancy monitoring. Therefore, image-based velocity measurement methods have gradually become mainstream. These methods can adapt to different water surface texture characteristics and improve the accuracy of velocity measurement, which is crucial for water resource management and flood monitoring. Due to its wide monitoring range, rapid response, low cost, flexibility and efficiency, image velocimetry technology has been widely used in river velocity detection. However, these methods often perform poorly when dealing with fast-moving or complex scenes, especially in weakly textured water surface environments, where their measurement accuracy is often unsatisfactory.

[0003] With advances in computer vision technology, deep optical flow estimation has been widely used in common scenarios such as autonomous driving and trajectory tracking, enabling precise measurement of object motion characteristics. In the field of river velocity measurement, the accuracy of deep learning optical flow estimation methods is highly dependent on high-quality, real-world optical flow datasets. Due to the complex texture of river water surfaces and the influence of factors such as topography, wind speed, and water depth, existing deep learning models, lacking training data tailored to specific surface flow characteristics, can lead to significant errors in velocity estimation. Furthermore, unlike rigid body motion, surface flow patterns exhibit random, non-rigid characteristics. Current common optical flow datasets such as Flychairs, Sintel, and KITTI suffer from the following significant limitations: First, motion feature mismatches. The FlyingChairs dataset generates optical flow data by moving synthetic foreground objects against a static background. However, this simple rigid motion fails to capture the non-rigid deformations and complex motion patterns of water flow. The Sintel dataset uses 3D rendering technology to generate synthetic scenes, where object motion is governed by animation rules. This lacks the realistic physical characteristics of water flow scenes and is therefore expensive to produce. The KITTI dataset is primarily targeted at autonomous driving scenarios. Its optical flow labels rely on the simultaneous calculation of stereo cameras and GPS / IMU sensors, and its motion model is primarily rigid-body motion, making it unsuitable for fluid environments. Secondly, the dataset lacks realism. The optical flow in existing datasets primarily targets solid objects or artificially synthesized environments, making it difficult to realistically simulate the texture variations and optical properties (such as waves, refraction, and transmission) of water in natural scenes. The dynamics of water flow are difficult to accurately reproduce in current datasets, making it difficult to describe these fluid motion variations. Therefore, effectively applying deep learning methods to river velocity measurement requires constructing a high-quality, realistic optical flow dataset encompassing diverse flow velocities, flow regimes, and environmental conditions to improve model generalization and estimation accuracy.

[0004] In recent years, progress has been made in constructing optical flow datasets for fluid scenarios. For example, the visual effects of water surface flow fields are simulated by camera motion, and optical flow is calculated. However, this method mainly relies on camera motion to generate optical flow labels and fails to accurately describe the dynamic characteristics of the fluid itself. By numerically simulating fluid motion and adding synthetic particles to the flow field, the particle motion trajectory is tracked using a fluid model to generate high-precision optical flow labels. However, this method is mainly applicable to laboratory environments and is limited to its single flow scenario, making it difficult to apply to the complex flow characteristics of natural water bodies. There are also methods for generating optical flow datasets by deep pre-training optical flow models. However, such methods themselves rely on existing optical flow datasets for training. Therefore, the characteristic motion of optical flow labels in weakly textured water surface areas is difficult to obtain, which affects the accuracy of the velocity field. The construction of optical flow datasets for river velocity measurement still faces many challenges. How to accurately obtain high-quality optical flow data that conforms to the dynamic characteristics of river water remains an urgent problem to be solved. Summary of the Invention

[0005] Purpose of the invention: In order to solve the problems existing in the above-mentioned prior art, the present invention provides a method for producing an optical flow dataset of a real water surface scene.

[0006] Technical solution: The present invention discloses a method for producing an optical flow dataset of a real water surface scene, which specifically includes the following steps:

[0007] Step 1: Acquire dynamic images of the water surface flow to obtain a continuous frame image sequence;

[0008] Step 2: Based on the position and posture information when the image was taken and the orientation elements within the camera , orthorectify the image; represents the offset of the principal point, and f represents the focal length of the camera;

[0009] Step 3: Convert the current frame image and the next frame Composed image pairs, using an improved multi-level pyramid grid iterative interpolation flow field estimation method to obtain the optical flow field of the image pairs;

[0010] Step 4: Eliminate the error of flow field motion for all optical flow fields;

[0011] Step 5: Use the optical flow field after error elimination as the optical flow label. Through the optical flow label, use the feature warping mapping method to perform forward warping mapping on the current frame image to obtain a new image frame. ;Will , ,as well as The corresponding optical flow labels are used as optical flow image pairs;

[0012] Step 6: Right Perform interpolation filling;

[0013] Step 7: , after filling And the corresponding optical flow labels as dataset samples;

[0014] Step 8: Repeat steps 1 to 7 until the number of data samples obtained meets the requirements.

[0015] Furthermore, the step 2 is specifically as follows:

[0016] Step 2.1: Convert the current frame image from the scanning coordinate system to the ground coordinate system; obtain the coordinates of the four vertex corners of the current frame image in the ground coordinate system;

[0017] Step 2.2: Get the maximum and minimum values of the horizontal coordinates of the four vertex coordinates , , the maximum and minimum values of the vertical axis , ;based on , , and Construct the projection range of the current frame image;

[0018] Step 2.3: Calculate the width and height of the corrected image, project the current frame image to the ground coordinate system using the collinearity condition equation, and delete the pixels outside the projection range;

[0019] Step 2.4: Use bilinear interpolation method to interpolate and correct the projected image.

[0020] Furthermore, the step 2.1 is specifically as follows:

[0021] Step 2.1.1: Establish the transformation relationship between the scanning coordinate system and the image space coordinate system:

[0022] ;

[0023] in, is the coordinate of the pixel point in the scanning coordinate system, (x, y, -f) is Convert to the coordinates in the image space coordinate system; is the physical size of the pixel, Respectively represent the width and height of the image;

[0024] Step 2.1.2: Establish the transformation relationship from the image space coordinate system to the image space auxiliary coordinate system:

[0025] ;

[0026] in, Represents the coordinate point in the auxiliary coordinate system of the image space corresponding to the (x, y, -f) in the image space coordinate system, , as well as Both represent the direction cosine value between two coordinate axes. , the expression is:

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] in, Indicates the pitch angle of the drone, Indicates the yaw angle of the drone, Indicates the roll angle of the drone;

[0037] Step 2.1.3: Establish the transformation relationship from the image space auxiliary coordinate system to the ground coordinate system:

[0038] ;

[0039] in, for point Coordinates in the ground coordinate system; is the coordinate of the photography center S in the ground coordinate system, is the scale factor;

[0040] Step 2.1.4: Based on the transformation relationship from the image space auxiliary coordinate system to the ground coordinate system in step 2.1.3 and the transformation relationship from the image space coordinate system to the image space auxiliary coordinate system in step 2.1.2, construct the transformation relationship from the image space coordinate system to the ground coordinate system:

[0041] ;

[0042] .

[0043] Furthermore, the width of the corrected image in step 2.3 is and height The expression is as follows:

[0044] ;

[0045] ;

[0046] in, is the ground spatial resolution corresponding to a single pixel of the image, , H is the altitude when the image is captured, and s represents the physical size of the pixel.

[0047] Furthermore, step 3 is specifically as follows:

[0048] Step 3.1: Set the first to third windows. The size of the first window is larger than the size of the second window, and the size of the second window is larger than the size of the third window.

[0049] Step 3.2: Using the first window, obtain the initial sparse optical flow field Flow1 between the image pairs based on the cross-correlation matching method;

[0050] Step 3.3: Use the interpolation function to interpolate the sparse optical flow field Flow1 into a dense optical flow field through the neighborhood features in the first window ;

[0051] Step 3.4: Based on dense optical flow field , the image feature distortion mapping method is used to transform the current frame image Map to the new position and get the intermediate image img1;

[0052] Step 3.5: Use the cross-correlation matching method to calculate the current frame image under the second window The sparse optical flow field Flow2 between the intermediate image img1 and Flow2 is interpolated to obtain the dense optical flow field corresponding to Flow2 ;

[0053] Step 3.6: Based on dense optical flow field , the image feature distortion mapping method is used to transform the current frame image Map to the new position and get the intermediate image img2;

[0054] Step 3.7: Use the cross-correlation matching method to calculate the current frame image under the third window The sparse optical flow field Flow3 between the intermediate image img2;

[0055] Step 3.8: Extract the sampling point nodes according to the third window and Optical flow value , ; Then calculate the sparse optical flow field by the following formula :

[0056] ;

[0057] Step 3.9: Use Gaussian radial interpolation basis function pair Perform interpolation filling.

[0058] Furthermore, step 4 is specifically as follows:

[0059] Step 4.1: Set the maximum optical flow threshold , and remove the optical flow greater than Pixels:

[0060] ;

[0061] in, is the mean value of the optical flow field, is the standard deviation of the optical flow field, and n is the proportional multiple of the standard deviation;

[0062] Step 4.2: Set the minimum optical flow threshold :

[0063] ;

[0064] in, is the empirical scaling factor;

[0065] Step 4.3: Count the optical flow fields with optical flow less than The total number of pixels, the optical flow is less than The proportion of pixels ,when Exceeding the preset threshold When the optical flow field is less than Pixels.

[0066] Furthermore, the method also includes performing statistical analysis on the constructed optical flow data set, specifically: calculating the maximum displacement value and the minimum displacement value, setting the displacement interval, and counting the pixel proportions of different displacement intervals. If the pixel proportion of the required displacement interval is less than a preset threshold, the interval frame number between image frames is adjusted or the optical flow field amplitude is amplified.

[0067] A computer device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the steps of the method for producing a real water surface scene optical flow dataset are implemented.

[0068] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for producing a real water surface scene optical flow data set.

[0069] Beneficial effects:

[0070] 1) This invention overcomes the inconsistency between river surface texture and fluid motion characteristics, maintaining their consistency. It can directly utilize real river image data for optical flow annotation, providing high-quality training data for deep learning models.

[0071] 2) Without a pre-trained model, this method calculates the local feature optical flow displacement through a layer-by-layer window. Using accumulated optical flow data, it ensures the consistency of flow field feature motion, making the velocity estimate more consistent with the actual river motion characteristics. Furthermore, using a feature warping mapping method, it achieves a precise correspondence between optical flow labels and actual feature motion, avoiding error accumulation and improving the reliability of optical flow data.

[0072] 3) The water optical flow dataset construction method provided by the present invention is highly adaptable and can generate fluid data that conforms to specific motion characteristics for different river velocity measurement application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is an overall flow chart of the method of the present invention;

[0074] Figure 2 This is the orthorectification process flow chart;

[0075] Figure 3 This is a flowchart of the improved multi-level pyramid grid iterative interpolation flow field estimation method;

[0076] Figure 4 is the original image, where (a) is the original image at time T, and (b) is the original image at time T+1;

[0077] Figure 5 is the generated optical flow label visualization result map;

[0078] Figure 6 It is the distribution histogram of pixel displacement after eliminating the error;

[0079] Figure 7 1 is a comparison diagram between the method of the present invention and the Realfow method, wherein (a) is the original image at time t, (b) is the flow field result diagram after the original image is processed by the method of the present invention, and (c) is the flow field result diagram after the original image is processed by the Realfow method. DETAILED DESCRIPTION

[0080] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0081] like Figure 1 As shown, the present invention provides a method for producing a real water surface scene optical flow dataset, which specifically includes the following steps:

[0082] Step S1, acquiring dynamic images of the flow on the water surface, and obtaining continuous frame image sequence data of natural fluid characteristics in different flow scenes.

[0083] Step S2: performing orthorectification and image segmentation on the image according to the position and posture information of the image when it was captured and the camera's internal orientation elements.

[0084] Step S3: using the previous and next frame images and the improved multi-level pyramid grid iterative interpolation flow field estimation method to obtain the approximate optical flow field of the image pair.

[0085] Step S4: Eliminate flow field motion errors from the acquired optical flow field.

[0086] In step S5, the optical flow field after the error optical flow vector elimination process is used as an optical flow label. Based on the optical flow label, a feature warping mapping method is used to construct a second frame image corresponding to the optical flow label using the first frame image.

[0087] Step S6: For the null value areas that may appear in the reconstructed second frame image, a bilinear interpolation method is used to fill the holes.

[0088] Step S7: Integrate the first frame image, the forward reconstructed second frame image, and the corresponding real optical flow labels to form a complete data set sample.

[0089] Step S8, repeating steps 1 to 7, and using the above steps one by one to create data set samples for the acquired image data until a sufficient number of data set samples are obtained.

[0090] In step S9, statistical analysis is performed on the constructed optical flow dataset to evaluate the distribution of pixel displacements in the optical flow labels to ensure that the dataset covers the optical flow displacement characteristics within the natural river flow velocity range. If the pixel displacement distribution requirements are not met, the interval between the sequence image frames or the optical flow amplitude is adjusted, and the optical flow field calculation is performed again.

[0091] The river scene image video data can be any fixed-point video data obtained by any shooting equipment and production means to describe the fluid motion characteristics. In this embodiment, the river image video obtained by the camera mounted on the drone is selected. The image resolution is 3840*2160, the original frame rate of the image is 25 frames / second, and it is sampled at 12.5Hz (half the original 25Hz rate), which is a full-color image. The ground sampling distance (GSD) of the image is 0.021m / px, the altitude H is 41m, the physical size of a single pixel is 6.12μm, the focal length f is 12mm, and the coordinates of the principal point in the ground coordinate system are (40371692.99547118,3538499.811265133,41), like the principal point offset (0.0000002, 0.0000002), like the principal point offset It is used to describe the coordinate deviation between the principal point of the image and the coordinate origin in the image plane coordinate system. The unit is meter, that is, the deviation between the principal point of the image in the actual image scanning coordinate system and the coordinate origin of the image plane coordinate system.

[0092] As shown in Figure 2, the attitude information in step S2 is the rotation angle of the drone equipped with the camera during shooting, usually expressed in Euler angles. express, is the yaw angle, is the pitch angle, is the roll angle, in this embodiment , the interior orientation element is expressed as , orthorectification specifically includes:

[0093] Establish the transformation between the scanning coordinate system and the image space coordinate system. The image plane coordinates usually set the upper left corner of the image as the coordinate origin, while the image space coordinate system sets the center of the image as the origin. The transformation relationship between the scanning coordinate system and the image space coordinate system is:

[0094]

[0095] in, is the coordinate of the pixel point in the scanning coordinate system, (x, y, -f) is Convert to the coordinates in the image space coordinate system; is the physical size of the pixel, Represents the width and height of the image respectively.

[0096] Establish the image space coordinate system to the image space auxiliary coordinate system transformation, knowing any point in the image space coordinate system , its coordinates in the auxiliary coordinate system of the image space can be expressed as , the coordinates in the image space coordinate system are , the coordinate relationship between the two coordinate systems is:

[0097]

[0098] in, , Represents the direction cosine value between two coordinate axes. , pitch angle , roll angle ) is calculated. R is the rotation matrix, which can be obtained by the following rotation matrix formula:

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112] Establish the conversion from the image space auxiliary coordinate system to the ground coordinate system. The coordinates of the photography center S and point A in the ground coordinate system are and , point A is the image point The corresponding point in the ground coordinate system is, The three points are collinear, and using the relationship between similar triangles we can get:

[0113]

[0114] in is the scale factor, which is determined by the altitude Camera focal length The ratio of is determined, and the transformation from the space auxiliary coordinate system to the ground coordinate system is written in matrix form as follows:

[0115]

[0116] Using the relationship between the image space coordinates of the image point and the image space auxiliary coordinates, the transformation formula from the image space coordinate system to the image space auxiliary coordinate system can be obtained as follows:

[0117] =

[0118] Establish the conversion between the image space coordinate system and the ground coordinate system. Utilize the relationship between the image space coordinate system, the image space auxiliary coordinate system, and the ground coordinate system, and realize the conversion from image space coordinates to ground coordinates through the collinearity condition equation:

[0119]

[0120]

[0121] In the formula, The altitude H can be replaced by the relationship between similar triangles, that is, H= ).

[0122] The determination of the orthorectified image range can determine the coordinates of the four corner points of the image in the image space coordinate system:

[0123] Top left corner:

[0124] Top right corner:

[0125] Bottom right corner:

[0126] Lower left corner:

[0127] in, is the width of the original image; is the height of the original image. Calculate the coordinates of the four vertices in the ground coordinate system. The maximum and minimum coordinate values of each corner point are calculated from the four corner points of the four ground coordinate systems. The width and height of the corrected image can be calculated using the spatial resolution of the captured image. The calculation formulas for the corrected height and width are as follows:

[0128]

[0129]

[0130]

[0131] in, is the ground spatial resolution corresponding to a single pixel of the image, H is the flight altitude, The coordinates of the four corner points of the image are calculated to correspond to the ground coordinates, and the correction coordinate range of the pixels inside the image is specified (if a pixel is projected out of the coordinate range when it is projected to the ground coordinate system, the pixel is discarded). Through the indirect method of image resampling, the coordinates of all image points corresponding to the ground points are calculated using the collinearity condition equation. The corrected image is interpolated using the bilinear interpolation method to achieve the image orthorectification without control points.

[0132] As shown in FIG3 , step S3 specifically includes:

[0133] Input two consecutive frames of images and Different sizes of local windows are used to obtain feature blocks. Different local window sizes are set to large grid, medium grid, and small grid. t represents the time. The large grid size used in this embodiment is 64, the medium grid size is 32, and the small grid size is 16. Two consecutive frames of images and like Figure 4 shown.

[0134] The node flow field of the large grid is calculated for the image pair composed of two frames, and the cross-correlation matching method is used to obtain the initial sparse optical flow field between the image pairs. Through the neighborhood features in the feature window, the interpolation function is used to interpolate the sparse optical flow field Flow1 into a dense optical flow field to obtain the dense optical flow field corresponding to the large grid feature window. .

[0135] For the input image frame and , constructed by and dense flow fields The intermediate frame image img1 of the coordinate mapping is recorded and saved at the same time as the sparse optical flow field Flow1. The intermediate frame image img1 is constructed by the feature warping mapping method and is used to perform spatial mapping of images or features under a given transformation field (such as the optical flow field). The position of pixels or feature points is adjusted using the homogeneous coordinate transformation vector. The expression formula is as follows:

[0136]

[0137] in: is the coordinate point after transformation, are the original coordinates, is the displacement vector given by the optical flow field.

[0138] Using the cross-correlation matching method, the grid window is calculated The sparse optical flow field Flow2 between the intermediate image img1 is obtained by bilinear interpolation to obtain a dense optical flow field .

[0139] use The dense optical flow obtained under the grid window , the intermediate image img2 is mapped through the image feature distortion mapping method.

[0140] Using the cross-correlation matching method, calculate the The sparse optical flow field Flow3 between the image img2 and the intermediate image img2.

[0141] The dense optical flow field obtained under the large grid and medium grid windows is extracted according to the sampling nodes of the small grid to extract the optical flow displacement value. , , and accumulate it with the sparse optical flow field results calculated under the small grid. The formula is as follows:

[0142]

[0143] in 、 Dense optical flow fields in large and medium grids respectively and The optical flow displacement value extracted by the small grid node, Flow3 is the sparse optical flow field calculated under the small grid, is the total displacement field.

[0144] Gaussian radial interpolation basis function is used to interpolate and fill the obtained total displacement field. Gaussian radial interpolation basis function can interpolate discrete and non-uniformly distributed data points. The obtained velocity field is smoother and more in line with the characteristics of flow field motion, realizing the transition from sparse optical flow field to dense optical flow field. In order to obtain a dense optical flow field and realize the production of intermediate images, in view of the computational efficiency of the algorithm and the coordination of the characteristic effects of the optical flow field, the interpolation method used in the large grid and medium grid calculation stages is the bilinear interpolation method.

[0145] The error elimination of the flow field data in step S4 refers to eliminating certain abnormal value points in the data by analyzing the distribution of pixel displacement in the optical flow field. The threshold setting strategies for the maximum and minimum values are different.

[0146] Maximum value elimination: When there are local drastic changes in flow rate, the simple maximum / minimum threshold elimination method will misjudge normal data in high-flow area. The n-times standard deviation method can dynamically adapt to the overall change trend of the data (n=3 in this example), improving the generalization of the elimination strategy. The calculation formula is as follows:

[0147] The horizontal component of the optical flow data is , the vertical component is , then the optical flow field is:

[0148]

[0149] in, is the total number of optical flow data points.

[0150] The displacement of the optical flow field Defined as:

[0151]

[0152] Calculate the mean of the optical flow field and standard deviation :

[0153]

[0154]

[0155] Set the maximum optical flow threshold:

[0156]

[0157] Abnormal flow rates greater than this threshold are eliminated.

[0158] Minimum value elimination: The characteristics of natural fluid motion and the influence of weak texture areas on optical flow estimation will cause small displacement data of varying proportions to appear in the optical flow field. Based on the proportion of small displacement points in the entire flow field, different thresholds are set for error elimination. The process is as follows:

[0159] Set a minimum optical flow threshold ,in is the empirical scaling factor;

[0160] Calculate all optical flows less than The percentage of points:

[0161]

[0162] in Indicates the proportion of optical flow less than the threshold, is an indicator function that takes the value 1 if the condition is met and 0 otherwise.

[0163] like Exceeding a preset threshold , then remove the optical flow data points whose optical flow threshold is less than the minimum optical flow threshold. The optical flow field after eliminating the error is used as the optical flow label. The optical flow label is as follows: Figure 5 shown.

[0164] In step S5, the image feature distortion mapping method uses forward distortion mapping to synthesize the previous frame image, that is, the given previous frame image , the next frame image , and the corresponding optical flow label, using the previous frame image After the forward mapping, a new image corresponding to the dense optical flow field is obtained ,Depend on and and the optical flow labels together form a set of optical flow image pairs.

[0165] The statistical analysis in step S9 is specifically as follows: Figure 6The displacement distribution in the optical flow field data is shown in the figure. The maximum and minimum displacement values are calculated, and the pixel proportions of different displacement intervals and the proportion of different displacement regions to all pixels are counted to evaluate the representativeness and rationality of the optical flow data. If the analysis results indicate that the dataset does not fully cover the typical flow velocity distribution in a natural water flow environment, it is necessary to adjust the interval between image frames or amplify the optical flow field amplitude to change the time difference between adjacent frames, thereby affecting the calculated optical flow displacement amplitude.

[0166] After the above process, select Figure 5 The frame images before and after the flow field are used as an example to illustrate. Within the corresponding complete monitoring flow field range, the average optical flow displacement calculated after eliminating the error is 8.40 pixels, the maximum displacement pixel is 20.0, the minimum displacement pixel is 2.0, and the effective pixel ratio is 71.81%. The specific pixel displacement ratio is Figure 6 shown.

[0167] In order to verify the effectiveness and reliability of the method for constructing a river optical flow dataset proposed in the present invention, in this embodiment, the present method is compared with the existing pre-trained Realflow model. Specifically, for river images with different characteristics, the evaluation focuses on two key indicators: one is the effective pixel ratio in various river images, and the other is the optical flow coverage effect of the optical flow field in the weak texture area. Through in-depth analysis of these two indicators, the pros and cons of the present method in producing optical flow datasets can be scientifically and objectively judged. The experimental results are shown in Table 1. The quantitative evaluation indicator effective pixel ratio is the ratio of the pixels that meet a specific pixel range in the calculated optical flow label to the total pixels. This indicator can well reflect whether the pixel displacement distribution of the flow field of the generated optical flow label meets the real flow characteristics. The higher the value, to a certain extent, it can represent that the generated optical flow label can better simulate the motion characteristics of the weak texture area. It can be seen from Table 1 that the production effect of the optical flow dataset adopted by the present invention is better than Realflow in the effective flow pixel ratio of the flow field of the water body, and it is Figure 7 It can be seen that the pixel coverage and overall distribution of this method are better than those of the Realflow method, and can better achieve the marking of optical flow in weak texture areas. The flow field effect is more consistent with the motion characteristics of the actual flow field.

[0168] Table 1

[0169]

[0170] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction. To avoid unnecessary repetition, the present invention will not further describe various possible combinations.

Claims

1. A method for producing a real water scene optical flow dataset, characterized in that: The specific steps include: Step 1: Acquire dynamic images of the water surface flow to obtain a continuous frame image sequence; Step 2: Based on the position and posture information when the image was taken and the orientation elements within the camera , orthorectify the image; represents the offset of the principal point, and f represents the focal length of the camera; Step 3: Convert the current frame image and the next frame Composed image pairs, using an improved multi-level pyramid grid iterative interpolation flow field estimation method to obtain the optical flow field of the image pairs; Step 4: Eliminate the error of flow field motion for all optical flow fields; Step 5: Use the optical flow field after error elimination as the optical flow label. Through the optical flow label, use the feature warping mapping method to perform forward warping mapping on the current frame image to obtain a new image frame. ;Will , ,as well as The corresponding optical flow labels are used as optical flow image pairs; Step 6: Right Perform interpolation filling; Step 7: , after filling And the corresponding optical flow labels as dataset samples; Step 8: Repeat steps 1 to 7 until the number of data samples obtained meets the requirements; The step 2 is specifically as follows: Step 2.1: Convert the current frame image from the scanning coordinate system to the ground coordinate system; obtain the coordinates of the four vertex corners of the current frame image in the ground coordinate system; Step 2.2: Get the maximum and minimum values of the horizontal coordinates of the four vertex coordinates , , the maximum and minimum values of the vertical axis , ;based on , , and Construct the projection range of the current frame image; Step 2.3: Calculate the width and height of the corrected image, project the current frame image to the ground coordinate system using the collinearity condition equation, and delete the pixels outside the projection range; Step 2.4: Use bilinear interpolation method to interpolate and correct the projected image.

2. The method for producing a real water surface scene optical flow dataset according to claim 1, characterized in that: The step 2.1 is specifically as follows: Step 2.1.1: Establish the transformation relationship between the scanning coordinate system and the image space coordinate system: ； in, is the coordinate of the pixel point in the scanning coordinate system, for Convert to the coordinates in the image space coordinate system; is the physical size of the pixel, Respectively represent the width and height of the image; Step 2.1.2: Establish the transformation relationship from the image space coordinate system to the image space auxiliary coordinate system: ； in, Indicates the coordinate point corresponding to (x, y, -f) in the image space auxiliary coordinate system and the image space coordinate system, , as well as Both represent the direction cosine value between two coordinate axes. , the expression is: ； ； ； ； ； ； ； ； ； in, Indicates the pitch angle of the drone, Indicates the yaw angle of the drone, Indicates the roll angle of the drone; Step 2.1.3: Establish the transformation relationship from the image space auxiliary coordinate system to the ground coordinate system: ； in, for point Coordinates in the ground coordinate system; is the coordinate of the photography center S in the ground coordinate system, is the scale factor; Step 2.1.4: Based on the transformation relationship from the image space auxiliary coordinate system to the ground coordinate system in step 2.1.3 and the transformation relationship from the image space coordinate system to the image space auxiliary coordinate system in step 2.1.2, construct the transformation relationship from the image space coordinate system to the ground coordinate system: ； 。 3. The method for producing a real water surface scene optical flow dataset according to claim 1, characterized in that: The width of the corrected image in step 2.3 and height The expression is as follows: ； ； in, is the ground spatial resolution corresponding to a single pixel of the image, , H is the altitude when the image is captured, and s represents the physical size of the pixel.

4. The method for producing a real water surface scene optical flow dataset according to claim 1, characterized in that: Step 3 is as follows: Step 3.1: Set the first to third windows. The size of the first window is larger than the size of the second window, and the size of the second window is larger than the size of the third window. Step 3.2: Using the first window, obtain the initial sparse optical flow field Flow1 between the image pairs based on the cross-correlation matching method; Step 3.3: Use the interpolation function to interpolate the sparse optical flow field Flow1 into a dense optical flow field through the neighborhood features in the first window ; Step 3.4: Based on dense optical flow field , the image feature distortion mapping method is used to transform the current frame image Map to the new position and get the intermediate image img1; Step 3.5: Use the cross-correlation matching method to calculate the current frame image under the second window The sparse optical flow field Flow2 between the intermediate image img1 and Flow2 is interpolated to obtain the dense optical flow field corresponding to Flow2 ; Step 3.6: Based on dense optical flow field , the image feature distortion mapping method is used to transform the current frame image Map to the new position and get the intermediate image img2; Step 3.7: Use the cross-correlation matching method to calculate the current frame image under the third window The sparse optical flow field Flow3 between the intermediate image img2; Step 3.8: Extract the sampling point nodes according to the third window and Optical flow value , ; Then calculate the sparse optical flow field by the following formula : ； Step 3.9: Use Gaussian radial interpolation basis function pair Perform interpolation filling.

5. The method for producing a real water surface scene optical flow dataset according to claim 1, characterized in that: Step 4 is as follows: Step 4.1: Set the maximum optical flow threshold , and remove the optical flow greater than Pixels: ； in, is the mean value of the optical flow field, is the standard deviation of the optical flow field, and n is the proportional multiple of the standard deviation; Step 4.2: Set the minimum optical flow threshold : ； in, is the empirical scaling factor; Step 4.3: Count the optical flow fields with optical flow less than The total number of pixels, the optical flow is less than The proportion of pixels ,when Exceeding the preset threshold When the optical flow field is less than Pixels.

6. The method for producing a real water surface scene optical flow dataset according to claim 1, characterized in that: The method also includes performing statistical analysis on the constructed optical flow dataset, specifically: calculating the maximum and minimum displacement values, setting the displacement interval, and counting the pixel proportions of different displacement intervals. If the pixel proportion of the required displacement interval is less than a preset threshold, the interval frame number between image frames is adjusted or the optical flow field amplitude is amplified.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the processor implements the steps of the method for producing a real water surface scene optical flow dataset as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for producing a real water surface scene optical flow dataset are implemented as described in any one of claims 1 to 6.

Images