Image water level recognition method and device based on scene simulation

Abstract

The present invention relates to the field of image or video recognition or understanding technology, and specifically to an image water level recognition method and device based on scene simulation, wherein the method comprises: collecting multi-view images of a target river section that meets preset observation conditions, and estimating corresponding camera parameters, constructing and adjusting a neural radiation field model for reconstructing a river scene, generating corresponding two-dimensional images based on river scenes at different water levels, and performing data enhancement on the two-dimensional images to obtain corresponding image data sets, training a deep convolutional neural network to obtain a final deep convolutional neural network, and using the final deep convolutional neural network to output a corresponding water level value or water level grade according to an actual river scene image of a river to be identified. Thus, the technical problems in the related art that the cost of acquiring training data is high, making it difficult for the collected data to cover enough scenes, it is difficult to extract water surface features in a complex environment, the algorithm has poor robustness, and is heavily dependent on the stability of the camera are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image or video recognition or understanding, and in particular to an image water level recognition method and device based on scene simulation. Background Art

[0002] Water level is one of the important observation elements in hydrological testing, and can provide data support for hydrological forecasting, water resources management, etc. At present, the telemetry water level gauges equipped in hydrological stations are usually contact monitoring devices such as float type, pressure type, and bubble type, which have limitations such as high construction and maintenance costs and susceptibility to damage by water impact. With the development of artificial intelligence and computer vision technology, water level monitoring methods based on image recognition have been studied and applied. As a non-contact monitoring method, river image water level recognition technology can effectively avoid the interference of water impact on equipment operation, and transmit and process river scene images in real time, quickly monitor water levels, and have a high level of automation and intelligence.

[0003] However, in the related art, the image water level recognition technology still has certain shortcomings. The cost of obtaining training data is high, and it often takes a long time to collect data to cover enough scenes. It also requires tedious data labeling. And when data acquisition is insufficient, it may be difficult to extract water surface features in complex environments such as lighting changes, water surface fluctuations, and obstructions, resulting in poor algorithm robustness. In addition, the image water level monitoring equipment in the related art is usually a camera fixed on the river bank. Since the calculation of the water level value is based on the camera parameters, the image water level recognition algorithm is heavily dependent on the stability of the camera. When the camera is disturbed, it is often necessary to recalibrate the parameters, which is in urgent need of improvement. Summary of the invention

[0004] The present invention provides an image water level recognition method and device based on scene simulation to solve technical problems in related technologies, such as high cost of acquiring training data, which makes it difficult for the collected data to cover enough scenes, and it is difficult to extract water surface features in complex environments, the algorithm has poor robustness and is heavily dependent on the stability of the camera, the preparation work is cumbersome, and the labor cost is high.

[0005] The first aspect of the present invention provides an image water level recognition method based on scene simulation, comprising the following steps: acquiring multi-perspective images of a target river section that meets preset observation conditions, and estimating camera parameters corresponding to each image of the multi-perspective images; constructing a neural radiation field model for reconstructing a river scene; adjusting the neural radiation field model using at least one preset water body feature constraint, so as to simulate river scenes under different water levels using the adjusted neural radiation field model; generating corresponding two-dimensional images based on the river scenes under the different water levels, and performing data enhancement on the two-dimensional images based on actual environmental factors to obtain a corresponding image data set; training a deep convolutional neural network using the image data set to obtain a final deep convolutional neural network, so as to use the final deep convolutional neural network to output corresponding water level values ​​or water level levels according to the actual river scene image of the river to be identified.

[0006] Optionally, in one embodiment of the present invention, the acquiring of multi-perspective images of a target river section that meets preset observation conditions and the estimation of camera parameters corresponding to each image of the multi-perspective images include: sparsely reconstructing the multi-perspective images using a structure-from-motion algorithm to obtain a sparse representation of the camera parameters and the three-dimensional scene corresponding to each image.

[0007] Optionally, in one embodiment of the present invention, before constructing a neural radiation field model for reconstructing a river scene, it also includes: constructing a plurality of rays starting from the center of the camera and passing through points on the imaging plane, and sampling three-dimensional space points on the rays to obtain corresponding sampling points; calculating the ray direction in the world coordinate system according to the camera parameters and the pixel coordinates on the imaging plane with the upper left corner of the image as the origin, so as to use the ray direction and the three-dimensional coordinates of the sampling point as input of the neural radiation field model.

[0008] Optionally, in one embodiment of the present invention, constructing a neural radiation field model for reconstructing a river scene includes: inputting the three-dimensional coordinates into a first fully connected neural network, outputting the density of the sampling points and intermediate features reflecting the spatial position information; splicing and fusing the ray directions with the intermediate features to obtain a fusion result; inputting the fusion result into a second fully connected neural network to obtain the color of the sampling points; obtaining a two-dimensional image of the river scene under the corresponding perspective based on the density and color, and calculating the loss function value between the two-dimensional image of the river scene and the corresponding multi-perspective image; judging whether the loss function value meets a preset convergence condition, if the preset convergence condition is met, the neural radiation field model is composed of the first fully connected neural network and the second fully connected neural network, otherwise, updating the parameters of the first fully connected neural network and the second fully connected neural network until the loss function value meets the preset convergence condition.

[0009] Optionally, in one embodiment of the present invention, the water body feature constraint includes at least one of a water body surface ripple and wave constraint, a water surface transparency and turbidity constraint, a water surface gloss constraint, and a water body color constraint.

[0010] Optionally, in one embodiment of the present invention, the corresponding two-dimensional images are generated based on the river scenes under the different water levels, and the two-dimensional images are data enhanced based on actual environmental factors to obtain corresponding image data sets, including: sphericalizing the camera distribution, and synthesizing the multi-view images to obtain river images under different viewpoints; inputting the river images under different viewpoints into the adjusted neural radiation field model to obtain the two-dimensional images; generating corresponding editing instructions based on the actual environmental factors; inputting the editing instructions and the two-dimensional images into a pre-trained graph-to-graph network to obtain new images of the river scenes under the corresponding style, and obtaining the image data set based on the new images.

[0011] The second aspect of the present invention provides an image water level recognition device based on scene simulation, including: an acquisition module, used to acquire multi-perspective images of a target river section that meets preset observation conditions, and estimate the camera parameters corresponding to each image of the multi-perspective images; a construction module, used to construct a neural radiation field model for reconstructing a river scene; an adjustment module, used to adjust the neural radiation field model using at least one preset water body feature constraint, so as to simulate river scenes under different water levels using the adjusted neural radiation field model; an enhancement module, used to generate corresponding two-dimensional images based on the river scenes under the different water levels, and perform data enhancement on the two-dimensional images based on actual environmental factors to obtain corresponding image data sets; a training module, used to train a deep convolutional neural network using the image data set to obtain a final deep convolutional neural network, so as to use the final deep convolutional neural network to output corresponding water level values ​​or water level levels according to the actual river scene image of the river to be identified.

[0012] Optionally, in one embodiment of the present invention, the acquisition module includes: a reconstruction unit, configured to perform sparse reconstruction on the multi-view images using a structure-from-motion algorithm to obtain a sparse representation of the camera parameters and the three-dimensional scene corresponding to each image.

[0013] Optionally, in one embodiment of the present invention, it also includes: a sampling module, used to construct a plurality of rays that pass through imaging plane points with the camera center as the starting point, and sample three-dimensional space points on the rays to obtain corresponding sampling points; a calculation module, used to calculate the ray direction in the world coordinate system according to the camera parameters and the pixel coordinates on the imaging plane with the upper left corner of the image as the origin, so as to use the ray direction and the three-dimensional coordinates of the sampling point as input to the neural radiation field model.

[0014] Optionally, in one embodiment of the present invention, the construction module includes: a first input unit, used to input the three-dimensional coordinates into a first fully connected neural network, and output the density of the sampling points and the intermediate features reflecting the spatial position information; a fusion unit, used to splice and fuse the ray direction with the intermediate features to obtain a fusion result; a second input unit, used to input the fusion result into a second fully connected neural network to obtain the color of the sampling point; a calculation unit, used to obtain a two-dimensional image of the river scene under the corresponding perspective based on the density and color, and calculate the loss function value between the two-dimensional image of the river scene and the corresponding multi-perspective image; a judgment unit, used to judge whether the loss function value meets a preset convergence condition. If the preset convergence condition is met, the neural radiation field model is composed of the first fully connected neural network and the second fully connected neural network. Otherwise, the parameters of the first fully connected neural network and the second fully connected neural network are updated until the loss function value meets the preset convergence condition.

[0015] Optionally, in one embodiment of the present invention, the water body feature constraint includes at least one of a water body surface ripple and wave constraint, a water surface transparency and turbidity constraint, a water surface gloss constraint, and a water body color constraint.

[0016] Optionally, in one embodiment of the present invention, the enhancement module includes: a synthesis unit, used to sphericalize the camera distribution and synthesize the multi-view images to obtain river images at different viewpoints; a third input unit, used to input the river images at different viewpoints into the adjusted neural radiation field model to obtain the two-dimensional image; a generation unit, used to generate corresponding editing instructions based on the actual environmental factors; and a fourth input unit, used to input the editing instructions and the two-dimensional image into a pre-trained graph-to-graph network to obtain a new image of the river scene in the corresponding style, and obtain the image data set based on the new image.

[0017] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image water level recognition method based on scene simulation as described in the above embodiment.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the image water level recognition method based on scene simulation as described in the above embodiment.

[0019] A fifth aspect of the present invention provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned image water level recognition method based on scene simulation.

[0020] The embodiment of the present invention can use the neural radiation field model and the reference object of known size to construct the three-dimensional river scene from the two-dimensional image at low cost, and simulate the images taken under different perspectives and different water levels, and then combine the image generation model to perform data enhancement, so that the obtained synthetic data greatly reduces the time cost and labor cost of data collection on the basis of ensuring the authenticity and diversity of the scene, and is based on the data of multiple perspectives, multiple water levels, and multiple environments, and realizes end-to-end calculation from image to water level through a deep neural network, avoiding the influence of camera position disturbance on the monitoring results, breaking through the limitations of camera position and camera model, and using any shooting equipment to flexibly and quickly obtain real-time water level, expanding the implementation scope of image water level monitoring. Therefore, the technical problems in the related technology that the cost of obtaining training data is high, making it difficult for the collected data to cover enough scenes, it is difficult to extract water surface features in complex environments, the algorithm has poor robustness, and is heavily dependent on the stability of the camera, the preparation work is cumbersome, and the labor cost is high are solved.

[0021] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flow chart of an image water level recognition method based on scene simulation according to an embodiment of the present invention;

[0024] Figure 2 is a flow chart of an image water level recognition method based on scene simulation according to an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of the structure and principle of a neural radiation field according to an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of a convolutional neural network structure for image water level recognition based on scene simulation according to an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of the structure of an image water level recognition device based on scene simulation according to an embodiment of the present invention;

[0028] Figure 6It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0030] The following describes the scene simulation-based water level recognition method and device of the embodiment of the present invention with reference to the accompanying drawings. In view of the related technologies mentioned in the above background technology, the high cost of training data acquisition makes it difficult for the collected data to cover enough scenes, it is difficult to extract water surface features in complex environments, the algorithm has poor robustness, and it is seriously dependent on the stability of the camera, the preparation work is cumbersome, and the labor cost is high. The present invention provides a scene simulation-based water level recognition method, in which a neural radiation field model and a reference object of known size can be used to construct a three-dimensional river scene from a two-dimensional image at a low cost, and simulate images taken at different perspectives and different water levels, and then combine the image generation model to perform data enhancement, so that the obtained synthetic data greatly reduces the time cost and labor cost of data collection on the basis of ensuring the authenticity and diversity of the scene, and is based on multi-perspective, multi-water level, and multi-environment data, and realizes end-to-end calculation from image to water level through a deep neural network, avoiding the influence of camera position disturbance on the monitoring results, breaking through the limitations of camera position and camera model, and using any shooting device to flexibly and quickly obtain real-time water level, expanding the implementation scope of image water level monitoring. As a result, technical problems in related technologies such as the high cost of acquiring training data making it difficult for the collected data to cover enough scenes, difficulty in extracting water surface features in complex environments, poor algorithm robustness, and heavy reliance on camera stability, cumbersome preparation work, and high labor costs were solved.

[0031] Specifically, Figure 1 A schematic flow chart of an image water level recognition method based on scene simulation provided in an embodiment of the present invention.

[0032] like Figure 1 As shown, the image water level recognition method based on scene simulation includes the following steps:

[0033] In step S101, multi-view images of a target river section that meet preset observation conditions are collected, and camera parameters corresponding to each image of the multi-view images are estimated.

[0034] In the actual implementation process, the embodiment of the present invention can collect multi-view images of the target river section and estimate the camera parameters corresponding to each image of the multi-view images. Among them, the multi-view images need to meet the following observation conditions:

[0035] Collect multi-view images of the river section to be observed in the same time period to avoid changes in lighting conditions. Shooting should be carried out during periods when the lighting is uniform, the water flow is gentle, there are no floating objects on the water surface, and the water level is at a low level. When shooting, you can use smartphones, cameras, drones and other equipment to shoot videos or pictures from multiple angles around the river section to be observed in the form of a hemisphere, partial sphere or cylinder, and try to ensure the integrity of the collection area and the continuity and overlap of the collected images. The captured river section scene should include a water ruler or other reference objects of known size to determine the true scale of the reconstructed scene, reflect the actual water surface elevation, and provide necessary information for the implementation of subsequent steps.

[0036] Optionally, in one embodiment of the present invention, multi-perspective images of a target river section that meets preset observation conditions are collected, and camera parameters corresponding to each image of the multi-perspective images are estimated, including: sparsely reconstructing the multi-perspective images using a motion recovery structure algorithm to obtain a sparse representation of the camera parameters corresponding to each image and the three-dimensional scene.

[0037] For example, an embodiment of the present invention can use SfM (Structure from Motion) to sparsely reconstruct the collected multi-view images after removing out-of-focus, blurred and abnormally exposed images, so as to obtain the camera parameters corresponding to each image and a sparse representation of the three-dimensional scene.

[0038] Among them, SfM mainly includes: feature point detection, feature point matching, bundle adjustment iterative optimization and other steps. Since the structure from motion algorithm has been relatively maturely applied, the above analysis process can be completed using open source software frameworks such as OpenSfM and Colmap, and the embodiment of the present invention will not be elaborated in detail.

[0039] For example, there are 17 camera parameters corresponding to each image, 15 of which can be expressed as a 3×5 matrix, which is used to convert the pixel coordinates of the image into the input form of the neural radiation field:

[0040] ,

[0041] Among them, the 3×3 matrix on the left is the rotation matrix, which represents the direction of the X, Y, and Z axes of the camera coordinate system in the world coordinate system; the fourth column is the translation vector, which represents the corresponding position of the camera origin in the world coordinate system; H ,W are the height and width of the image respectively; f is the focal length of the camera. The other two parameters , They represent the closest and farthest distances between the sparse 3D point cloud and the camera, respectively, which limits the scene range under the camera pose and is used to determine the sampling interval.

[0042] In step S102, a neural radiation field model is constructed for reconstructing the river scene.

[0043] The embodiments of the present invention can construct a neural radiation field model to reconstruct a river scene, so as to construct a three-dimensional river scene from a two-dimensional image at low cost by using the neural radiation field model and a reference object of known size.

[0044] Among them, the neural radiation field is an implicit expression of the three-dimensional scene. After the model training is completed, the position of the spatial point X With observation direction D You can query the corresponding color ( c ) and density ( ) information, which can be expressed as:

[0045] .

[0046] Optionally, in one embodiment of the present invention, before constructing a neural radiation field model for reconstructing a river scene, it also includes: constructing multiple rays that pass through imaging plane points with the camera center as the starting point, and sampling three-dimensional space points on the rays to obtain corresponding sampling points; calculating the ray direction in the world coordinate system according to the camera parameters and the pixel coordinates on the imaging plane with the upper left corner of the image as the origin, so as to use the ray direction and the three-dimensional coordinates of the sampling point as input of the neural radiation field model.

[0047] In some embodiments, before training the neural radiation field, the embodiments of the present invention can construct a number of rays starting from the center of the camera and passing through the imaging plane point, and sample the three-dimensional space points on the rays, and use the ray direction and the three-dimensional coordinates of the sampling points as the input of the model. According to the camera parameters and the pixel coordinates on the imaging plane with the upper left corner of the picture as the origin ( i , u ), calculate the ray direction in the world coordinate system.

[0048] The embodiment of the present invention can first calculate the ray direction in the camera coordinate system, with the camera coordinate system as the origin, that is, the coordinates are (0,0,0); the pixel point coordinates on the imaging plane are , is the coordinate of the optical center with the upper left corner of the image as the origin, and its value is Therefore, the direction vector of the ray in the camera coordinate system is , and normalize it:

[0049] .

[0050] Using the rotation matrix and translation vector, we can Convert to the direction vector in the world coordinate system :

[0051] .

[0052] The depth in the ray direction is Discrete sampling is performed within the range to obtain the coordinates of several points The direction vector and the three-dimensional coordinates are encoded and then input into the model. By mapping the information into a high-dimensional space, the contrast between different positions and directions is expanded. The encoding formula is as follows:

[0053] ,

[0054] .

[0055] Optionally, in one embodiment of the present invention, a neural radiation field model for reconstructing a river scene is constructed, including: inputting three-dimensional coordinates into a first fully connected neural network, outputting the density of the sampling points and intermediate features reflecting the spatial position information; splicing and fusing the ray directions with the intermediate features to obtain a fusion result; inputting the fusion result into a second fully connected neural network to obtain the color of the sampling points; obtaining a two-dimensional image of the river scene under the corresponding perspective based on the density and color, and calculating the loss function value between the two-dimensional image of the river scene and the corresponding multi-perspective image; judging whether the loss function value meets a preset convergence condition, if the preset convergence condition is met, the neural radiation field model is composed of the first fully connected neural network and the second fully connected neural network, otherwise, updating the parameters of the first fully connected neural network and the second fully connected neural network until the loss function value meets the preset convergence condition.

[0056] In some other embodiments, the neural radiation field model body may include two fully connected neural networks, numbered A (first fully connected neural network) and B (second fully connected neural network). The input of network A is the coordinate information of the three-dimensional point, and the output is the density ( ) and intermediate features, which can reflect spatial position information. The ray direction information is spliced ​​and fused with the intermediate features output by network A and then input into network B, and the output is color ( r , g , b ). The parameters to be optimized for network A and network B are the weight matrix and bias vector of each layer.

[0057] The embodiment of the present invention can perform rendering after obtaining the density and color of the sampling points to obtain a two-dimensional image at a corresponding viewing angle. Rendering is performed according to the following formula, where: r Represents rays, is the color of the pixel point on the imaging plane corresponding to the ray, t represents the sampling point on the ray, for t The coordinates of the point, d represents the direction of the ray, Represents the rays transmitted to t Light ratio of a point:

[0058] ,

[0059] .

[0060] The embodiment of the present invention can calculate the following loss function value based on the rendered two-dimensional image and the actual image taken: is the color value of the rendered pixel. The color value of the corresponding pixel in the captured image:

[0061] .

[0062] The embodiment of the present invention can use back propagation and gradient descent method to update parameters and gradually iterate the training until the model converges or the loss function reaches a set threshold.

[0063] In step S103, the neural radiation field model is adjusted using at least one preset water feature constraint to simulate river scenes at different water levels using the adjusted neural radiation field model. The water feature constraint includes at least one of a water surface ripple and wave constraint, a water surface transparency and turbidity constraint, a water surface gloss constraint, and a water color constraint.

[0064] Based on the constructed neural radiation field model of the river section and the water gauge in the scene, the scaling ratio between the reconstructed scene and the real scene can be calculated. , the embodiment of the present invention can set a number of actual water surface elevations, multiplied by Convert it to a collection of water surface elevations within the reconstruction scene H .

[0065] Determine the water surface elevation in the scene h Then, combined with the gravity direction line , the water surface can be approximated as a horizontal plane, and the following formula is the representation of the horizontal plane in the world coordinate system:

[0066] .

[0067] In order to meet the water flow state in actual situations, it is necessary to add waves on the water surface. The embodiment of the present invention can use sine waves, Gerstner waves or fast Fourier transforms to simulate the water surface wave field and obtain the three-dimensional coordinates of the water surface point and the water surface normal vector. N The embodiment of the present invention can be described by taking the Gerstner wave as an example, and its basic idea is to simulate the wave by applying disturbance to each point on the plane:

[0068] ,

[0069] ,

[0070] ,

[0071] in,( x , y , z ) is the coordinate of the initial water surface point, is the coordinate corresponding to the fluctuation, N is the number of superimposed Gerstner waves, For the i The amplitude of a wave, For the i The wave number vector of the wave, For the i The angular frequency of a wave, For the i The initial phase of the wave, t By adjusting the time t The value of can simulate the dynamic changes of waves.

[0072] Furthermore, the embodiment of the present invention can set the water surface density after the fluctuation according to the following rules: is the normal at the water surface:

[0073] .

[0074] Considering the gloss effect of the water surface, the bidirectional reflectance distribution function is introduced , representing a point X In the outgoing direction Light intensity and incident direction The embodiment of the present invention can adopt a spherical Gaussian model to construct a bidirectional reflectance distribution function, d is the viewing direction vector, N For point X The normal vector at for d About Normal Vectors N The reflection vector of is the sharpness factor:

[0075] ,

[0076] .

[0077] In addition, considering the Fresnel effect of rendering water, reflectivity is introduced , represents the ratio of the light intensity observed in the reflection direction to the light intensity at that point. The reflectivity can be calculated according to the following formula: is the angle between the normal and the viewing direction, is the angle between the normal and the refracted light, is the refractive index of air, is the refractive index of water:

[0078] ,

[0079] ,

[0080] ,

[0081] .

[0082] The color of the water surface point can be calculated according to the following formula: For the color of water, Represents the incident angle exist X The color formed by the point can be calculated according to the above rendering formula. X 10 rays are sampled to simulate the color reflected from the water surface.

[0083] .

[0084] The color of underwater points needs to take into account the scattering and absorption of light by water, and is calculated according to the following formula: is the transmission coefficient of light in water, is the refraction direction angle:

[0085] .

[0086] In step S104, corresponding two-dimensional images are generated based on river scenes at different water levels, and data enhancement is performed on the two-dimensional images based on actual environmental factors to obtain corresponding image data sets.

[0087] During the actual execution process, the embodiments of the present invention can adjust different camera viewing angles, render and generate corresponding two-dimensional images, and construct text instructions to edit images for data enhancement based on actual environmental factors, such as lighting conditions and water surface conditions, to obtain an image data set.

[0088] Optionally, in one embodiment of the present invention, corresponding two-dimensional images are generated based on river scenes under different water levels, and data enhancement is performed on the two-dimensional images based on actual environmental factors to obtain corresponding image data sets, including: sphericalizing the camera distribution and synthesizing multi-view images to obtain river images under different viewpoints; inputting the river images under different viewpoints into the adjusted neural radiation field model to obtain two-dimensional images; generating corresponding editing instructions based on actual environmental factors; inputting the editing instructions and the two-dimensional images into a pre-trained graph-to-graph network to obtain new images of the river scene under the corresponding style, and obtaining an image data set based on the new images.

[0089] The embodiment of the present invention can first sphericalize the camera distribution and synthesize river images under different viewing angles for river scenes at different water levels. The position of the camera can be determined by the radius from the origin of the world coordinate system. r , world coordinate system z Axis xz Angle of plane rotation , world coordinate system x Axis xy Angle of plane rotation Once the three parameters are determined, the translation vector of the camera in the world coordinate system can be expressed as .right r , , Random sampling can obtain a series of camera positions and viewing angles. Then, rendering based on the constructed neural radiation field model can obtain a series of two-dimensional images of the river at different angles and water levels.

[0090] Next, the embodiment of the present invention can construct editing instructions according to complex actual environments, such as: out-of-focus and blurred images; images with raindrops blocking the lens; images with insufficient lighting and dim images; images shot by near-infrared cameras at night, etc. The editing instructions and the original images are input into the pre-trained graph-generated graph network to obtain new images of the river scene in the corresponding style. This method can be used to expand the data set to enhance the environmental adaptability and robustness of the water level recognition algorithm. Each image in the final data set has a corresponding water level value label.

[0091] In step S105, a deep convolutional neural network is trained using the image data set to obtain a final deep convolutional neural network, so that the final deep convolutional neural network can be used to output a corresponding water level value or water level grade according to the actual river scene image of the river to be identified.

[0092] As a possible implementation method, an embodiment of the present invention can train an image water level depth convolutional network based on a synthetic data set.

[0093] For example, an embodiment of the present invention can train a deep convolutional neural network based on an image data set, wherein the deep convolutional neural network is a type of feedforward neural network that includes convolution calculations and has a deep structure, and is composed of a cascade of several convolutional layers, pooling layers, and fully connected layers. The deep convolutional neural network loss function is combined to perform parameter optimization to obtain an optimized deep convolutional neural network.

[0094] The embodiments of the present invention can be based on the downstream tasks oriented to the monitoring data, wherein the deep convolutional network can select the following regression or classification functions for training, that is, the corresponding precise water level value or water level grade (water level range) can be output by inputting the captured river scene image.

[0095] Combination Figures 2 to 4 As shown, the working principle of the image water level recognition method based on scene simulation according to an embodiment of the present invention is described in detail by taking an embodiment as an example.

[0096] like Figure 2 As shown, the embodiment of the present invention may include the following steps:

[0097] Step S201: Collect multi-perspective images of the scene to be monitored. The embodiment of the present invention can collect multi-perspective images of the river section to be observed in the same time period to avoid changes in lighting conditions. The shooting should be carried out as much as possible during the period when the lighting is uniform, the water flow is gentle, there are no floating objects on the water surface, and the water level is at a low level. When shooting, you can use smart phones, cameras, drones and other equipment to shoot videos or pictures from multiple angles around the river section to be observed in the form of a hemisphere, a partial sphere or a cylinder, and try to ensure the integrity of the collection area and the continuity and overlap of the collected images. The captured river section scene should include a water ruler or other reference objects of known size to determine the true scale of the reconstructed scene, reflect the actual water surface elevation, and provide necessary information for the implementation of subsequent steps.

[0098] The embodiment of the present invention can use SfM (Structure from Motion) to perform sparse reconstruction on the collected multi-view images after removing out-of-focus, blurred and abnormally exposed images, so as to obtain the camera parameters corresponding to each image and the sparse representation of the three-dimensional scene.

[0099] Step S202: Construct a neural radiation field model of the scene to be monitored. Figure 3 The figure shows the structure and principle of the neural radiation field. The embodiment of the present invention can construct a neural radiation field model to reconstruct a river scene, so as to construct a three-dimensional river scene from a two-dimensional image at low cost by using the neural radiation field model and a reference object of known size.

[0100] Among them, the neural radiation field is an implicit expression of the three-dimensional scene. After the model training is completed, the position of the spatial point XWith observation direction D You can query the corresponding color ( c ) and density ( ) information, which can be expressed as:

[0101] .

[0102] Step S203: Simulate the monitored scenes under different water levels. Under the given river water level conditions, adjust the neural radiation field model to simulate the river scenes under different water levels.

[0103] Step S204: synthesize and enhance the image water level data set. The embodiment of the present invention can adjust different camera viewing angles, render and generate corresponding two-dimensional images, and construct text instructions to edit images for data enhancement based on actual environmental factors such as lighting conditions and water surface conditions to obtain an image data set.

[0104] Step S205: training the image water level recognition network based on scene simulation. Figure 4 The figure shows a schematic diagram of the structure of a convolutional neural network for image water level recognition based on scene simulation. The embodiment of the present invention can train a deep convolutional neural network based on an image data set, wherein the deep convolutional neural network is a type of feedforward neural network that includes convolution calculations and has a deep structure, and is composed of a cascade of several convolutional layers, pooling layers, and fully connected layers. The deep convolutional neural network loss function is combined to perform parameter optimization to obtain an optimized deep convolutional neural network.

[0105] The embodiments of the present invention can be based on the downstream tasks oriented to the monitoring data, wherein the deep convolutional network can select the following regression or classification functions for training, that is, inputting the captured river scene image to output the corresponding accurate water level value or water level range.

[0106] According to the image water level recognition method based on scene simulation proposed in the embodiment of the present invention, the neural radiation field model and the reference object of known size can be used to construct a three-dimensional river scene from a two-dimensional image at a low cost, and simulate images taken at different perspectives and different water levels, and then combine the image generation model for data enhancement, so that the obtained synthetic data greatly reduces the time cost and labor cost of data collection on the basis of ensuring the authenticity and diversity of the scene, and is based on multi-perspective, multi-water level, and multi-environment data, and realizes end-to-end calculation from image to water level through a deep neural network, avoiding the influence of camera position disturbance on the monitoring results, breaking through the limitations of camera position and camera model, and using any shooting equipment to flexibly and quickly obtain real-time water level, expanding the implementation scope of image water level monitoring. Therefore, the technical problems in the related technology that the high cost of training data acquisition makes it difficult for the collected data to cover enough scenes, it is difficult to extract water surface features in complex environments, the algorithm has poor robustness, and is heavily dependent on the stability of the camera, the preparation work is cumbersome, and the labor cost is high are solved.

[0107] Next, the image water level recognition device based on scene simulation proposed in an embodiment of the present invention is described with reference to the accompanying drawings.

[0108] Figure 5 4 is a block diagram of an image water level recognition device based on scene simulation according to an embodiment of the present invention.

[0109] like Figure 5 As shown, the image water level recognition device 10 based on scene simulation includes: a collection module 100, a construction module 200, an adjustment module 300, an enhancement module 400 and a training module 500.

[0110] Specifically, the acquisition module 100 is used to acquire multi-view images of a target river section that meets preset observation conditions, and estimate camera parameters corresponding to each image of the multi-view images.

[0111] The construction module 200 is used to construct a neural radiation field model for reconstructing a river scene.

[0112] The adjustment module 300 is used to adjust the neural radiation field model using at least one preset water body characteristic constraint, so as to simulate river scenes under different water levels using the adjusted neural radiation field model.

[0113] The enhancement module 400 is used to generate corresponding two-dimensional images based on river scenes at different water levels, and perform data enhancement on the two-dimensional images based on actual environmental factors to obtain corresponding image data sets.

[0114] The training module 500 is used to train a deep convolutional neural network using an image data set to obtain a final deep convolutional neural network, so as to use the final deep convolutional neural network to output a corresponding water level value or water level grade according to an actual river scene image of a river to be identified.

[0115] Optionally, in one embodiment of the present invention, the acquisition module 100 includes: a reconstruction unit.

[0116] The reconstruction unit is used to perform sparse reconstruction on multi-view images using a structure-from-motion algorithm to obtain a sparse representation of the camera parameters and three-dimensional scene corresponding to each image.

[0117] Optionally, in one embodiment of the present invention, the image water level recognition device 10 based on scene simulation further includes: a sampling module and a calculation module.

[0118] The sampling module is used to construct multiple rays starting from the camera center and passing through the imaging plane points, and to sample the three-dimensional space points on the rays to obtain corresponding sampling points.

[0119] The calculation module is used to calculate the ray direction in the world coordinate system according to the camera parameters and the pixel coordinates on the imaging plane with the upper left corner of the image as the origin, so as to use the ray direction and the three-dimensional coordinates of the sampling point as the input of the neural radiation field model.

[0120] Optionally, in one embodiment of the present invention, the construction module 200 includes: a first input unit, a fusion unit, a second input unit, a calculation unit and a judgment unit.

[0121] Among them, the first input unit is used to input the three-dimensional coordinates into the first fully connected neural network, and output the density of the sampling points and the intermediate features reflecting the spatial position information.

[0122] The fusion unit is used to merge the ray direction with the intermediate features to obtain a fusion result.

[0123] The second input unit is used to input the fusion result into the second fully connected neural network to obtain the color of the sampling point.

[0124] The calculation unit is used to obtain a two-dimensional image of the river scene under the corresponding viewing angle based on density and color, and calculate the loss function value between the two-dimensional image of the river scene and the corresponding multi-view image.

[0125] A judgment unit is used to judge whether the loss function value meets the preset convergence condition. If the preset convergence condition is met, the neural radiation field model is composed of the first fully connected neural network and the second fully connected neural network. Otherwise, the parameters of the first fully connected neural network and the second fully connected neural network are updated until the loss function value meets the preset convergence condition.

[0126] Optionally, in one embodiment of the present invention, the water body feature constraint includes at least one of a water body surface ripple and wave constraint, a water surface transparency and turbidity constraint, a water surface gloss constraint, and a water body color constraint.

[0127] Optionally, in one embodiment of the present invention, the enhancement module 400 includes: a synthesis unit, a third input unit, a generation unit and a fourth input unit.

[0128] The synthesis unit is used to sphericalize the camera distribution and synthesize the multi-view images to obtain river images from different perspectives.

[0129] The third input unit is used to input the river channel images at different viewing angles into the adjusted neural radiation field model to obtain a two-dimensional image.

[0130] The generating unit is used to generate corresponding editing instructions based on actual environmental factors.

[0131] The fourth input unit is used to input the editing instruction and the two-dimensional image into the pre-trained image-generating network to obtain a new image of the river scene in the corresponding style, and obtain an image data set based on the new image.

[0132] It should be noted that the aforementioned explanation of the embodiment of the image water level recognition method based on scene simulation is also applicable to the image water level recognition device based on scene simulation of this embodiment, and will not be repeated here.

[0133] According to the image water level recognition device based on scene simulation proposed in the embodiment of the present invention, the neural radiation field model and the reference object of known size can be used to construct a three-dimensional river scene from a two-dimensional image at a low cost, and simulate images taken at different perspectives and different water levels, and then combine the image generation model for data enhancement, so that the obtained synthetic data greatly reduces the time cost and labor cost of data collection on the basis of ensuring the authenticity and diversity of the scene, and is based on the data of multiple perspectives, multiple water levels, and multiple environments, and realizes end-to-end calculation from image to water level through a deep neural network, avoiding the influence of camera position disturbance on the monitoring results, breaking through the limitations of camera position and camera model, and using any shooting equipment to flexibly and quickly obtain real-time water level, expanding the implementation scope of image water level monitoring. Therefore, the technical problems in the related technology that the high cost of training data acquisition makes it difficult for the collected data to cover enough scenes, it is difficult to extract water surface features in complex environments, the algorithm has poor robustness, and is heavily dependent on the stability of the camera, the preparation work is cumbersome, and the labor cost is high are solved.

[0134] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0135] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0136] When the processor 602 executes the program, the image water level recognition method based on scene simulation provided in the above embodiment is implemented.

[0137] Furthermore, the electronic device further comprises:

[0138] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0139] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0140] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0141] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0142] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0143] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0144] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned image water level recognition method based on scene simulation is implemented.

[0145] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by a processor, the image water level recognition method based on scene simulation provided by an embodiment of the present invention is implemented.

[0146] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0147] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0148] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0149] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0150] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0151] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0152] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0153] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. An image water level recognition method based on scene simulation, characterized in that: The following steps are involved: Collecting multi-view images of a target river section that meets preset observation conditions, and estimating camera parameters corresponding to each image of the multi-view images; Constructing a neural radiation field model for reconstructing river scenes; The neural radiation field model is adjusted using at least one preset water body characteristic constraint, so as to simulate river scenes under different water levels using the adjusted neural radiation field model; Generate corresponding two-dimensional images based on the river scenes at different water levels, and perform data enhancement on the two-dimensional images based on actual environmental factors to obtain corresponding image data sets; Using the image data set to train a deep convolutional neural network to obtain a final deep convolutional neural network, so as to use the final deep convolutional neural network to output a corresponding water level value or water level grade according to an actual river scene image of a river to be identified; Among them, the corresponding two-dimensional images are generated based on the river scenes under the different water levels, and the two-dimensional images are data enhanced based on the actual environmental factors to obtain the corresponding image data set, including: sphericalizing the camera distribution, and synthesizing the multi-view images to obtain river images under different perspectives; inputting the river images under different perspectives into the adjusted neural radiation field model to obtain the two-dimensional images; generating corresponding editing instructions based on the actual environmental factors; inputting the editing instructions and the two-dimensional image into a pre-trained graph-to-graph network to obtain a new image of the river scene under the corresponding style, and obtaining the image data set based on the new image.

2. The image water level recognition method based on scene simulation according to claim 1 is characterized in that: The collecting of multi-view images of the target river section that meets the preset observation conditions and estimating the camera parameters corresponding to each image of the multi-view images includes: The multi-view images are sparsely reconstructed using a structure-from-motion algorithm to obtain a sparse representation of the camera parameters and the three-dimensional scene corresponding to each image.

3. The image water level recognition method based on scene simulation according to claim 1 is characterized in that: Before building the neural radiation field model for reconstructing the river scene, it also includes: Constructing multiple rays starting from the camera center and passing through the imaging plane points, and sampling three-dimensional space points on the rays to obtain corresponding sampling points; The ray direction in the world coordinate system is calculated according to the camera parameters and the pixel coordinates on the imaging plane with the upper left corner of the image as the origin, so that the ray direction and the three-dimensional coordinates of the sampling point are used as inputs of the neural radiation field model.

4. The image water level recognition method based on scene simulation according to claim 3 is characterized in that: The neural radiation field model for reconstructing the river scene is constructed, including: Inputting the three-dimensional coordinates into a first fully connected neural network, and outputting the density of the sampling points and intermediate features reflecting spatial position information; The ray direction is spliced ​​and fused with the intermediate feature to obtain a fusion result; Inputting the fusion result into a second fully connected neural network to obtain the color of the sampling point; Obtaining a two-dimensional image of the river scene at a corresponding viewing angle based on the density and color, and calculating a loss function value between the two-dimensional image of the river scene and the corresponding multi-view image; Determine whether the loss function value satisfies a preset convergence condition; if so, the neural radiation field model is composed of the first fully connected neural network and the second fully connected neural network; otherwise, update the parameters of the first fully connected neural network and the second fully connected neural network until the loss function value satisfies the preset convergence condition.

5. The image water level recognition method based on scene simulation according to claim 1 is characterized in that: The water body feature constraint includes at least one of a water body surface ripple and wave constraint, a water surface transparency and turbidity constraint, a water surface gloss constraint, and a water body color constraint.

6. An image water level recognition device based on scene simulation, characterized in that: include: An acquisition module, used to acquire multi-view images of a target river section that meet preset observation conditions, and estimate camera parameters corresponding to each image of the multi-view images; A construction module for building a neural radiation field model for reconstructing river scenes; An adjustment module, used to adjust the neural radiation field model using at least one preset water body characteristic constraint, so as to simulate river scenes under different water levels using the adjusted neural radiation field model; An enhancement module, used to generate corresponding two-dimensional images based on the river scenes at different water levels, and perform data enhancement on the two-dimensional images based on actual environmental factors to obtain corresponding image data sets; A training module, used to train a deep convolutional neural network using the image data set to obtain a final deep convolutional neural network, so as to use the final deep convolutional neural network to output a corresponding water level value or water level according to an actual river scene image of a river to be identified; Among them, the enhancement module includes: a synthesis unit, which is used to sphericalize the camera distribution and synthesize the multi-view images to obtain river images at different perspectives; a third input unit, which is used to input the river images at different perspectives into the adjusted neural radiation field model to obtain the two-dimensional image; a generation unit, which is used to generate corresponding editing instructions based on the actual environmental factors; and a fourth input unit, which is used to input the editing instructions and the two-dimensional image into a pre-trained graph-to-graph network to obtain a new image of the river scene in the corresponding style, and obtain the image data set based on the new image.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image water level recognition method based on scene simulation as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the image water level recognition method based on scene simulation as described in any one of claims 1 to 5.

9. A computer program product, characterized in that It comprises a computer program, which, when executed, is used to implement the image water level recognition method based on scene simulation as described in any one of claims 1 to 5.

Images