Monitoring video stream distortion correction and two-dimensional-to-three-dimensional fusion method and system

Abstract

The invention belongs to the technical field of computer videos, and particularly relates to a monitoring video stream distortion correction and two-dimensional-to-three-dimensional fusion method and a monitoring video stream distortion correction and two-dimensional-to-three-dimensional fusion system. The non-uniform mapping relation is established by calculating the distortion offset of the pixel coordinates, and the interpolation weight is dynamically calculated based on the pixel distance and the local edge intensity, so that the non-uniform distortion of the lens can be accurately corrected, the edge detail features of the image are effectively reserved, image blurring or detail loss in the correction process is avoided, and the correction efficiency is improved. And the definition and the accuracy of the corrected image are greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of computer video technology, specifically relating to a method and system for distortion correction and 2D-to-3D conversion and fusion of surveillance video streams. Background Technology

[0002] In practical applications of computer video technology, the need for processing surveillance video streams and reconstructing 3D scenes is becoming increasingly urgent, and it is widely used in many fields such as security monitoring, intelligent transportation, and virtual reality.

[0003] Because the optical characteristics of surveillance camera lenses inevitably cause distortion in video images, including radial and tangential distortion, this distortion will cause deviations in the shape and position of objects in the image. This not only affects the visual effect, but also seriously interferes with subsequent processing procedures such as feature extraction and depth calculation. As a result, the accuracy of the 3D reconstruction results based on the distorted image will be biased and cannot accurately reflect the spatial relationship of the real scene. To address this, an improved method and system for distortion correction and 2D-to-3D fusion of surveillance video streams was designed. Summary of the Invention

[0004] To address the aforementioned shortcomings in the existing technology, this invention provides a method and system for distortion correction and 2D-to-3D fusion of surveillance video streams to solve the problems mentioned in the background technology.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method for distortion correction and 2D-to-3D conversion fusion of surveillance video streams includes the following steps: S1: Obtain the monitoring video stream, and extract video stream image frames from the video stream as the original image; S2: The original image is corrected for distortion using a non-uniform grid interpolation algorithm; S3: Use a CNN convolutional neural network to extract image features from the corrected image and generate an image feature map. Based on the image features, generate an initial depth map through a depth prediction network. S4: Based on the initial depth map and the pre-constructed sparse scene point cloud, the initial depth map is optimized using a bundle adjustment algorithm to obtain an optimized depth map; S5: The optimized depth map and the image feature map of the corrected image are fused through a cross-modal attention mechanism to generate an enhanced 3D scene representation containing depth perception information.

[0006] Preferably, the distortion correction of the original image based on a non-uniform grid interpolation algorithm includes: calculating the distortion offset of pixel coordinates in the original image according to predetermined lens distortion model parameters; determining a non-uniform mapping relationship from the corrected image coordinates to the original image coordinates based on the distortion offset; calculating interpolation weights based on the non-uniform mapping relationship, according to the distance between the pixel and the target coordinates and local image features; and performing weighted calculations on the original image pixels according to the interpolation weights to obtain the corrected image.

[0007] Preferably, in step S2, the lens distortion model parameters include coefficients for correcting radial and tangential distortion, wherein the distortion coordinates with normalized ideal coordinates The relationship is defined by the following model: in , The radial distortion coefficient is... denoted as the tangential distortion coefficient.

[0008] Preferably, the process of calculating the interpolation weights includes: For the target pixel in the corrected image, its corresponding target sampling coordinates in the original image, obtained through the non-uniform mapping relationship, are: ; Obtain image features of the original image near the target sampling coordinates, the image features including edge intensity; Select with Centered The nearest neighbor pixel is used as the sampling point; for the nth nearest neighbor pixel, the sampling point is... Each sampling point, its weight Based on the Euclidean distance from the target sampling point coordinates and the edge strength of its location The calculation formula is as follows: in As a regulating factor, To prevent extremely small positive numbers from being divided by zero, all weights are normalized to obtain normalized weights for pixel value calculation.

[0009] Preferably, in step S3, the training method for the deep prediction network is as follows: S501: Obtain the training dataset by collecting images from different perspectives of the same scene and performing the distortion correction processing on the images; S502: Construct a deep prediction network with an encoder-decoder structure; S503: Define the total loss function of the deep prediction network. for: in, The depth prediction loss is calculated based on the difference between the depth map predicted by the network and the actual depth. For the multi-view geometric consistency loss, the photometric reprojection error and structural similarity error under the predicted depth map are calculated by utilizing the correspondence between images from different viewpoints in the training dataset. The depth smoothing loss is used to constrain the local smoothness of the predicted depth map in regions of small image gradients. , and These are preset weighting coefficients; S504: Input the training data into the depth prediction network using forward propagation to obtain the predicted depth map, and calculate the loss value according to the total loss function; optimize the parameters of the depth prediction network using the backpropagation algorithm; S505: During training, the model converges when the relative decrease rate of the loss value is less than 0.1% over 10 epochs.

[0010] Preferably, in step S503, the multi-view geometric consistency loss The calculation process is as follows: For training images And the depth map predicted from the training image. and images from adjacent viewpoints ,use Will The pixels in the image are synthesized through 3D transformation and reprojection. Virtual images from different perspectives The consistency loss of the multi-view set Due to breadth loss and structural similarity loss Weighted composition, i.e. ,in: in This is a function for calculating structural similarity. As weight.

[0011] Preferably, in step S4, the pre-constructed sparse scene point cloud is obtained by: acquiring multiple frames of scene images during a static period when there are no dynamic targets in the monitored scene, processing the multiple frames of images using the structure-in-motion algorithm, and reconstructing the sparse 3D point cloud of the current monitored scene.

[0012] Preferably, the optimized depth map includes the following steps: S801: Based on the corrected image and the initial depth map, the initial depth map is back-projected into three-dimensional space to generate a dense initial three-dimensional point cloud. ; S802: Construct a joint optimization problem whose optimization variables include the dense initial 3D point cloud. The three-dimensional coordinates of the midpoint, and the pre-constructed sparse scene point cloud. The three-dimensional coordinates of the midpoint; S803: Define the objective function as the dense initial 3D point cloud. With the pre-constructed sparse scene point cloud The sum of reprojection errors generated when reprojecting back to the corrected graphics platform is minimized by an iterative optimization algorithm to simultaneously optimize the three-dimensional coordinates of the dense initial three-dimensional point cloud and the sparse point cloud. S804: From the optimized dense 3D point cloud, based on the correspondence between each 3D point and the corrected image, recalculate its depth value on the image plane to generate the optimized depth map.

[0013] Preferably, in step S4, the cross-modal attention mechanism is specifically implemented as follows: the image feature map is used as the query vector, the depth features obtained after feature extraction from the optimized depth map are used as the key vector, the similarity matrix between the query vector and the key vector is calculated and normalized into an attention weight matrix, the image features and the depth features are combined as the value vector, the attention weight matrix is ​​used to perform a linear transformation on the value vector, and the enhanced 3D scene representation containing depth perception information is output.

[0014] A system for distortion correction and 2D-to-3D conversion and fusion of surveillance video streams includes: Image acquisition module: used to acquire the monitoring video stream and extract video stream image frames from the video stream as the original image; Distortion correction module: used to correct the distortion of the original image using a non-uniform grid interpolation algorithm to obtain the corrected image; Depth prediction module: used to extract image features of the corrected image using a CNN convolutional neural network and generate an image feature map, and generate an initial depth map based on the image features through a depth prediction network; Depth optimization module: Based on the initial depth map and the pre-constructed sparse scene point cloud, it optimizes the initial depth map using a bundle adjustment algorithm to obtain an optimized depth map; Feature fusion module: used to fuse the optimized depth map and the image feature map of the corrected image through a cross-modal attention mechanism to generate an enhanced 3D scene representation containing depth perception information.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention employs a distortion correction method based on a non-uniform grid interpolation algorithm. Combined with pre-calibrated lens distortion model parameters, it establishes a non-uniform mapping relationship by calculating the distortion offset of pixel coordinates and dynamically calculates interpolation weights based on pixel distance and local edge intensity. This method can accurately correct the non-uniform distortion of the lens, effectively preserve the edge detail features of the image, avoid image blurring or loss of details during the correction process, and significantly improve the clarity and accuracy of the corrected image. 2. The depth prediction network adopts an encoder-decoder structure and designs a total loss function that includes depth prediction loss, multi-view geometric consistency loss, and depth smoothing loss. Through multi-dimensional constraints, the accuracy and consistency of depth prediction are improved. At the same time, the bundle adjustment algorithm is used to jointly optimize the initial depth map with the pre-constructed sparse scene point cloud, minimizing the reprojection error between dense 3D point cloud and sparse point cloud. This effectively corrects the local error of single-frame depth prediction, improves the global consistency and stability of the depth map, and provides reliable depth data support for subsequent 3D scene representation. 3. This invention employs a cross-modal attention mechanism to fuse image features and depth features. By using image features as query vectors and depth features as key vectors to calculate the attention weight matrix, it can adaptively mine the correlation information between the two modal features, highlighting the contribution of key features. This mechanism can more fully integrate depth perception information and image semantic features, and the generated enhanced 3D scene representation not only has accurate spatial depth relationships but also retains rich image details, making the 3D scene more in line with the real environment and meeting the needs of high-precision scene reconstruction, intelligent analysis and other high-end applications. Attached Figure Description

[0016] Figure 1 A framework diagram for a method of distortion correction and 2D-to-3D conversion fusion of monitoring video streams. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0018] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual images. They should not be construed as limiting the scope of this patent. To better illustrate the embodiments of the present invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0019] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0020] In the description of this invention, unless otherwise explicitly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] Example 1: like Figure 1 The method and system for distortion correction and 2D-to-3D fusion of surveillance video streams, as shown, employs a surveillance camera to capture images of the monitored scene. Frames of the continuous video stream are parsed according to a preset frame extraction frequency to extract single-frame images. These extracted single-frame images are then processed and uniformly converted to a preset image format and resolution to ensure consistency in subsequent processing. The processed single-frame image is the original image. Furthermore, the lens distortion model parameters of the current surveillance camera are obtained beforehand through camera calibration experiments. These parameters include coefficients used to correct radial distortion. And the coefficients used to correct tangential distortion. ; By calling the lens distortion model parameters, the distortion offset of each pixel coordinate in the original image is calculated. The calculation logic is as follows: Using the bottom left corner of the original image as the origin, the normalized ideal coordinates of any pixel in the original image are... Through formula Calculate the distance from the normalized ideal coordinates to the origin, and then substitute the obtained distance into the formula for calculating distorted coordinates. Obtain the distorted coordinates Then, based on the obtained distorted coordinates and ideal coordinates, the distortion offset of the pixel is calculated. Based on the obtained distortion offset, a non-uniform mapping relationship from the corrected image coordinates to the original image coordinates is derived in reverse. That is, for any target pixel coordinate in the corrected image... The target sampling coordinates in the original image are calculated in reverse using the offset. This non-uniform mapping relationship changes dynamically with the pixel position to adapt to the non-uniform distortion characteristics of the lens; Based on the non-uniform mapping relationship, the interpolation weight is calculated by combining the distance between the pixel and the target coordinates and the local features of the image. The calculation process is as follows: obtain the original image at the target sampling coordinates. Nearby image features are used to calculate the gradient value of the region using the Sobel operator edge detection algorithm. The magnitude of the gradient value is taken as the edge strength, and a selection is made based on the gradient magnitude. Centered The nearest neighbor pixel is used as the sampling point for the nth... Each sampling point is used to calculate its coordinates relative to the target. European distance , And based on the edge strength of that point Then substitute it into the interpolation weight calculation formula. in The preset adjustment factor, the To prevent extremely small positive numbers from being divided by zero, the original image is weighted according to normalized values. The pixel values ​​of each sampling point are weighted and calculated, that is... in To correct the pixel values ​​of the target pixels in the image, For the first The pixel values ​​of each sampling point are used to traverse all target pixels in the corrected image, complete the distortion correction of the entire image, and output the corrected image. Initial depth map generation based on depth prediction network: A CNN convolutional neural network is used to extract multi-scale features from the corrected image, including low-level edge and texture features and high-level semantic features, to form a multi-dimensional image feature map and image features. The multi-dimensional image feature map is input into the trained and converged depth prediction network. Through the forward inference operation of encoder-decoder, the initial depth map corresponding to the corrected image is output. Furthermore, the CNN convolutional neural network specifically comprises: Input layer: Receives the image after it has been corrected by a non-uniform grid interpolation algorithm, providing high-quality input for CNN feature extraction; Convolutional layer group: It consists of 4 convolutional layers. The first and second layers are low-level convolutional layers, using 3×3 small-sized convolutional kernels to focus on extracting detailed features such as edges and textures, and to preserve local information of the image. The third and fourth layers are high-level convolutional layers, using 5×5 large-sized convolutional kernels, combined with receptive field expansion, to extract semantic features such as target contours and scene layout. Pooling layer: A 2×2 pooling window is inserted between every two convolutional layers, and a max pooling layer with a stride of 2 is used to reduce the resolution of the feature map, reduce the amount of computation, and retain key features. Activation layer: The entire network uses the ReLU activation function, which enhances the feature representation ability through nonlinear transformation and avoids the gradient vanishing problem; Output layer: Outputs multi-dimensional image feature maps and multi-dimensional image features. The extracted multi-dimensional image feature maps are directly input into the encoder of the deep prediction network.

[0022] Furthermore, the training method for deep prediction networks is as follows: Images from different perspectives of the same scene are collected as training data. All training images are first processed by distortion correction to ensure the consistency between the training data and the actual input data. A depth prediction network with an encoder-decoder structure is constructed. The encoder is responsible for downsampling and abstracting the features of the input image to extract the core features. The decoder is responsible for upsampling and resolution restoration of the core features and outputting an initial depth map with the same size as the input image. Definition of total loss function: The formula for calculating the total loss function is: in, The depth prediction loss is calculated using the average error as the basis for determining the difference between the depth map predicted by the network and the actual depth. For the multi-view geometric consistency loss, the photometric reprojection error and structural similarity error under the predicted depth map are calculated by utilizing the correspondence between images from different viewpoints in the training dataset. The depth smoothing loss is used to constrain the local smoothness of the predicted depth map in regions of small image gradients. , and These are preset weighting coefficients used to balance the proportion of the three losses in the total loss; Multi-view geometric consistency loss The calculation process is as follows: For training images And the depth map predicted from the training image. and images from adjacent viewpoints ,use Will The pixels in the image are synthesized through 3D transformation and reprojection. Virtual images from different perspectives The consistency loss of the multi-view set Due to breadth loss and structural similarity loss Weighted composition, i.e. ,in: in This is a function for calculating structural similarity. As weight; The The calculation process for deep smoothing loss is as follows: in For pixels The predicted depth value, For pixels Image intensity, Tables are respectively in The gradient in the direction is used to input the training data into the depth prediction network through forward propagation to obtain the predicted depth map and calculate the loss value according to the total loss function; the network parameters are adjusted through the backpropagation algorithm; during the training process, the change of the loss value is monitored in real time, and when the relative decrease rate of the loss value within 10 epochs is less than 0.1%, the model is determined to have converged, training is stopped and the network parameters after training are saved.

[0023] Depth map optimization based on beam adjustment algorithm: The method for pre-constructing sparse scene point clouds of the monitoring scene is as follows: During a static period in the monitored scene where there are no dynamic targets, 20 scene images were acquired. Feature matching, camera pose estimation, and triangulation were performed on the acquired multi-frame images. Then, the sparse 3D point cloud of the current monitored scene was reconstructed using a structure-of-motion motion (SOM) algorithm. The generated initial depth map is then stored in the scene database. Based on the parameters of the monitoring camera within the corrected image, the generated initial depth map is back-projected into 3D space to generate a dense initial 3D point cloud. The calculation logic is as follows: For any pixel in the corrected image Its corresponding depth value The depth value is taken from the initial depth map output by the depth prediction network. Therefore, the 3D spatial coordinates of this pixel are: The depth remains constant, where To monitor the coordinates of the main point of the camera, Each of the surveillance cameras is located at The focal length of the direction is used to construct a dense initial 3D point cloud. Point clouds in sparse scenes The problem is a joint optimization problem with the three-dimensional coordinates as the optimization variables, and the objective function is defined as follows: and The sum of reprojection errors generated when reprojecting back to the corrected image plane, i.e., for any point in the middle Its three-dimensional coordinates are The reprojected pixel coordinates are The actual corresponding pixel coordinates are Then the reprojection error at that point is: Similarly, calculation The reprojection error at each point is defined as the objective function, which is the sum of the reprojection errors of all stores. Gradient descent is used to minimize the objective function, and simultaneous optimization is performed. and The 3D coordinates of all points are calculated until the objective function value converges to 5 to the power of negative 4. From the optimized dense 3D point cloud, based on the correspondence between each 3D point and the pixels of the corrected image, the depth value of each pixel on the image plane is recalculated. Generate an optimized depth map with the same dimensions as the corrected image.

[0024] Feature fusion based on cross-modal attention mechanism: Feature extraction is performed on the optimized depth map. The same CNC convolutional neural network is used to convert the depth map into a depth feature map consistent with the one obtained by "using a CNN convolutional neural network to perform multi-scale feature extraction on the corrected image, forming a multi-dimensional image feature map." This ensures the fusion of the two modalities. The phrase "using a CNN convolutional neural network to perform multi-scale feature extraction on the corrected image, forming a multi-dimensional image feature map" is used as the query vector. The image feature map extracted from the optimized depth map is used as the key vector. The similarity matrix between the query vector and the key vector is calculated using the dot product similarity method. This similarity matrix is ​​then normalized using the Softmax function to obtain the attention weight matrix. The attention weight matrix is ​​used to characterize the correlation strength between image features and depth features. A multi-scale feature map generated by using a CNN convolutional neural network to perform multi-scale feature extraction on the corrected image is combined with the depth feature map obtained by feature extraction from the optimized depth map in the previous step to obtain a value vector. Using attention weight matrix value vector Perform a linear transformation, that is: in This is the weight matrix. As the bias term, the output after linear transformation is the enhanced 3D scene representation containing depth-sensing information.

[0025] The above are merely embodiments of the present invention. The circuits, electronic components, and modules involved are all prior art, fully achievable by those skilled in the art, and require no further explanation. The scope of protection in this application does not involve improvements to the software and methods. Commonly known structures and characteristics in the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all prior art in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A method for distortion correction and 2D-to-3D conversion fusion of surveillance video streams, characterized in that, Includes the following steps: S1: Obtain the monitoring video stream, and extract video stream image frames from the video stream as the original image; S2: The original image is subjected to distortion correction using a non-uniform grid interpolation algorithm to obtain the corrected image; S3: Use a CNN convolutional neural network to extract image features from the corrected image and generate an image feature map. Based on the image features, generate an initial depth map through a depth prediction network. S4: Based on the initial depth map and the pre-constructed sparse scene point cloud, the initial depth map is optimized using a bundle adjustment algorithm to obtain an optimized depth map; S5: The optimized depth map and the image feature map of the corrected image are fused through a cross-modal attention mechanism to generate an enhanced 3D scene representation containing depth perception information.

2. The method for distortion correction and 2D-to-3D conversion fusion of a surveillance video stream according to claim 1, characterized in that, The distortion correction of the original image based on the non-uniform grid interpolation algorithm includes: calculating the distortion offset of pixel coordinates in the original image according to the predetermined lens distortion model parameters; determining the non-uniform mapping relationship from the corrected image coordinates to the original image coordinates according to the distortion offset; calculating the interpolation weights based on the non-uniform mapping relationship, according to the distance between the pixel and the target coordinates and the local features of the image; and performing weighted calculation on the original image pixels according to the interpolation weights to obtain the corrected image.

3. The method for distortion correction and 2D-to-3D conversion fusion of a surveillance video stream according to claim 2, characterized in that, In step S2, the lens distortion model parameters include coefficients for correcting radial and tangential distortion, wherein the distortion coordinates... with normalized ideal coordinates The relationship is defined by the following model: in , The radial distortion coefficient is... denoted as the tangential distortion coefficient.

4. The method for distortion correction and 2D-to-3D conversion fusion of a surveillance video stream according to claim 3, characterized in that, The process of calculating the interpolation weights includes: For the target pixel in the corrected image, its corresponding target sampling coordinates in the original image, obtained through the non-uniform mapping relationship, are: ; Obtain image features of the original image near the target sampling coordinates, the image features including edge intensity; Select with Centered The nearest neighbor pixel is used as the sampling point; for the nth nearest neighbor pixel, the sampling point is... Each sampling point, its weight Based on the Euclidean distance from the target sampling point coordinates and the edge strength of its location The calculation formula is as follows: in As a regulating factor, To prevent extremely small positive numbers from being divided by zero, all weights are normalized to obtain normalized weights for pixel value calculation.

5. The method for distortion correction and 2D-to-3D conversion fusion of a surveillance video stream according to claim 1, characterized in that, In step S3, the training method for the deep prediction network is as follows: S501: Obtain the training dataset by collecting images from different perspectives of the same scene and performing the distortion correction processing on the images; S502: Construct a deep prediction network with an encoder-decoder structure; S503: Define the total loss function of the deep prediction network. for: in, The depth prediction loss is calculated based on the difference between the depth map predicted by the network and the actual depth. For the multi-view geometric consistency loss, the photometric reprojection error and structural similarity error under the predicted depth map are calculated by utilizing the correspondence between images from different viewpoints in the training dataset. The depth smoothing loss is used to constrain the local smoothness of the predicted depth map in regions of small image gradients. , and These are preset weighting coefficients; S504: Input the training data into the depth prediction network using forward propagation to obtain the predicted depth map, and calculate the loss value according to the total loss function; optimize the parameters of the depth prediction network using the backpropagation algorithm; S505: During training, the model converges when the relative decrease rate of the loss value is less than 0.1% over 10 epochs.

6. The method for distortion correction and 2D-to-3D conversion fusion of a surveillance video stream according to claim 5, characterized in that, In step S503, the multi-view geometric consistency loss The calculation process is as follows: For training images And the depth map predicted from the training image. and images from adjacent viewpoints ,use Will The pixels in the image are synthesized through 3D transformation and reprojection. Virtual images from different perspectives The consistency loss of the multi-view set Due to breadth loss and structural similarity loss Weighted composition, i.e. ,in: in This is a function for calculating structural similarity. As weight.

7. The method for distortion correction and 2D-to-3D conversion fusion of a surveillance video stream according to claim 1, characterized in that, In step S4, the pre-constructed sparse scene point cloud is obtained in the following way: during a static period when there are no dynamic targets in the monitored scene, multiple frames of scene images are acquired, and the multiple frames of images are processed by the motion recovery structure algorithm to reconstruct the sparse three-dimensional point cloud of the current monitored scene.

8. The method for distortion correction and 2D-to-3D conversion fusion of a surveillance video stream according to claim 7, characterized in that, The optimized depth map includes the following steps: S801: Based on the corrected image and the initial depth map, the initial depth map is back-projected into three-dimensional space to generate a dense initial three-dimensional point cloud. ; S802: Construct a joint optimization problem whose optimization variables include the dense initial 3D point cloud. The three-dimensional coordinates of the midpoint, and the pre-constructed sparse scene point cloud. The three-dimensional coordinates of the midpoint; S803: Define the objective function as the dense initial 3D point cloud. With the pre-constructed sparse scene point cloud The sum of reprojection errors generated when reprojecting back to the corrected graphics platform is minimized by an iterative optimization algorithm to simultaneously optimize the three-dimensional coordinates of the dense initial three-dimensional point cloud and the sparse point cloud. S804: From the optimized dense 3D point cloud, based on the correspondence between each 3D point and the corrected image, recalculate its depth value on the image plane to generate the optimized depth map.

9. The method for distortion correction and 2D-to-3D conversion fusion of a surveillance video stream according to claim 1, characterized in that, In step S4, the cross-modal attention mechanism is specifically implemented as follows: the image feature map is used as the query vector, the depth features obtained after feature extraction of the optimized depth map are used as the key vector, the similarity matrix between the query vector and the key vector is calculated and normalized into an attention weight matrix, the image features and the depth features are combined as the value vector, the attention weight matrix is ​​used to perform a linear transformation on the value vector, and the enhanced 3D scene representation containing depth perception information is output.

10. A system for distortion correction and 2D-to-3D conversion and fusion of surveillance video streams, characterized in that, include: Image acquisition module: used to acquire the monitoring video stream and extract video stream image frames from the video stream as the original image; Distortion correction module: used to correct the distortion of the original image using a non-uniform grid interpolation algorithm to obtain the corrected image; Depth prediction module: used to extract image features of the corrected image using a CNN convolutional neural network and generate an image feature map, and generate an initial depth map based on the image features through a depth prediction network; Depth optimization module: Based on the initial depth map and the pre-constructed sparse scene point cloud, it optimizes the initial depth map using a bundle adjustment algorithm to obtain an optimized depth map; Feature fusion module: used to fuse the optimized depth map and the image feature map of the corrected image through a cross-modal attention mechanism to generate an enhanced 3D scene representation containing depth perception information.

Images