A method and system for fusion of virtuality and reality in ultra-large space based on metaverse

Abstract

The present invention discloses a method and system for fusion of virtual and real in a very large space based on a metaverse, and relates to the technical field of fusion of virtual and real, including: obtaining continuous video frame images as scene training data, using a key frame detection method to obtain global features in real time for the scene training data, setting similarity thresholds for adjacent video frame images according to the global features, and obtaining local parameters of the real-time scene; using a multi-scale space detection algorithm to obtain key feature points for the local parameters, calculating the scene depth value of the video frame image through the key feature points, and obtaining scene fusion data; performing feature extraction on the scene fusion data, obtaining overlapping texture data, and using a texture weighting method to fuse adjacent overlapping areas of the images, and obtaining virtual and real fusion scene data, which can effectively reduce texture distortion, ensure that the monitoring video after the camera parameters change is automatically fused with the three-dimensional virtual scene in real time, and avoid tedious manual operations and lags.

Description

Technical Field

[0001] The present invention relates to the field of virtual-reality fusion technology, and in particular to a super-large space virtual-reality fusion method and system based on a metaverse. Background Art

[0002] With the development of economy and the advancement of science and technology, more and more surveillance cameras are deployed in the same park and scene. The traditional split-screen multi-image surveillance system has become increasingly unable to meet people's needs: the multi-channel surveillance images that are separated from each other are not conducive to users' analysis and tracking of moving targets; faced with a large number of complex two-dimensional surveillance images, it is also difficult for users to establish an intuitive three-dimensional spatial perception and global vision of the surveillance scene. Therefore, an intelligent monitoring system based on an enhanced virtual environment is adopted to superimpose the surveillance video as real information into a virtual three-dimensional environment, realizing the mapping and conversion of video content from two-dimensional to three-dimensional.

[0003] However, the existing methods and systems for ultra-large space virtual-real fusion still have the following problems:

[0004] (1) The premise of fusing surveillance video images into a 3D virtual scene is to know the pose parameters of the real camera in the 3D virtual scene. This requires manually annotating the camera pose to solve the problem. However, this method is not only time-consuming and laborious, but also difficult to apply when the camera in the real scene needs to adjust its pose.

[0005] (2) Existing virtual-reality fusion systems use a stitching method to process overlapping areas of multiple videos, and then superimpose them with the three-dimensional scene. The disadvantage of this method is that it can only obtain a good observation effect from the video shooting perspective, and serious distortion will occur when observed from other perspectives. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for ultra-large space virtual-real fusion based on the metaverse, so as to solve the technical problems in the prior art of difficulty in calibrating camera pose parameters, poor adaptability, and easy impact on scene effects during video overlapping area processing.

[0007] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0008] The first aspect of the present invention provides a method for integrating virtuality and reality in a very large space based on a metaverse, comprising the following steps:

[0009] Acquire continuous video frame images as scene training data, use key frame detection method to acquire global features in real time for the scene training data, set similarity thresholds for adjacent video frame images according to the global features, and update video frame image parameters in real time by judging whether the similarity between adjacent video frame images corresponds to the similarity threshold value, so as to acquire local parameters of the real-time scene;

[0010] A multi-scale spatial detection algorithm is used to obtain key feature points for the local parameters, a scene depth value of the video frame image is calculated through the key feature points, and the image is mapped into a three-dimensional virtual scene in units of video frames according to the scene depth value to obtain scene fusion data;

[0011] Feature extraction is performed on the scene fusion data to obtain overlapping texture data, the image overlapping areas are divided according to the overlapping texture data, and the adjacent image overlapping areas are fused using a texture weighting method to obtain virtual-real fusion scene data.

[0012] As a preferred solution of the present invention, a key frame detection method is used to obtain global features in real time for the scene training data, including:

[0013] The continuously acquired video frame images are taken as M I Unordered scene training data For feature point extraction, each image I m Get one by M m A set of local feature points , Represents image pixel coordinates The feature description operator at ;

[0014] Extract the feature description operator frame by frame The color of the video image frame at the position, calculate the probability of each color in the continuous video frame image, and construct an image color probability waveform corresponding to the video image frame with the video frame timing as the horizontal axis;

[0015] Divide different areas in the video image frame according to the image color probability waveform diagram, and calculate the image pixel coordinates in the video image frame The pixel value at the position is used to divide the number of pixels in each frame of the video by the pixel value, the color of the video frame image is calibrated by the pixel number, and the global feature is obtained.

[0016] As a preferred solution of the present invention, a similarity threshold is set for adjacent video frame images according to the global feature, and the similarity between adjacent video frame images is determined to correspond to the similarity threshold, including:

[0017] The image probability of two adjacent frames of video images is calculated by the global feature, the image frame histogram of the video stream is obtained at the image pixel coordinates, and the video frame image difference value is calculated according to the image frame histogram, and its expression is:

[0018]

[0019] in, represents the histogram of the video stream image frame at the image pixel coordinate x, represents the histogram of the video stream image frame at the image pixel coordinate x+1, n represents the image color level, j represents the image color difference, Represents the pixel value of the video stream image frame at the image pixel coordinate x;

[0020] A similarity threshold is set according to the video frame image difference value, and when the difference value of two adjacent video frames is greater than the similarity threshold, the video frame image parameters are updated to obtain the local parameters of the real-time scene.

[0021] As a preferred solution of the present invention, a multi-scale spatial detection algorithm is used to obtain key feature points for the local parameters, including:

[0022] The local parameters are filtered using a Gaussian operator, and the local parameters of the scene in the multi-scale space are smoothed using a spatial mapping function to obtain a multi-scale spatial graph. , whose expression is:

[0023]

[0024] in, represents the video frame image data at coordinates x and y, and z represents the spatial scale factor;

[0025] For the multi-scale spatial graph Calculate the image pixel values ​​between adjacent video frames, and transform the multi-scale spatial graph into Perform differential processing to obtain spatial extreme points;

[0026] The image data within the spatial extreme point is differentiated at a scale k, and a corner detection algorithm is used to extract a feature vector from the differential image at each scale;

[0027] The feature vector is convolved with the global image to obtain local key feature points.

[0028] As a preferred solution of the present invention, calculating the scene depth value of the video frame image by using the key feature points includes:

[0029] Constructing homogeneous vertex coordinates of the key feature points in the form of a set, and mapping the homogeneous vertex coordinates to an imaging plane to establish image space coordinates;

[0030] Dividing the horizontal coordinate of the image space coordinate by the width and height of the corresponding image to obtain the texture normalized coordinate, and transforming the texture normalized coordinate in sequence through a matrix to obtain the texture coordinate in the texture space;

[0031] The scene image depth value of any of the key feature points is calculated using the texture coordinates to obtain a real-time scene depth value.

[0032] As a preferred solution of the present invention, according to the scene depth value, the image is mapped into a three-dimensional virtual scene in units of video frames to obtain scene fusion data, including:

[0033] Creating a depth texture region of a virtual image using the scene depth value, and mapping the depth value to a texture via a buffer within the depth texture region;

[0034] By repeatedly calculating the pixel color value of the video image, determining whether each pixel of the video image is in a shadow area, the video image is divided into texture areas;

[0035] The pixel value of the texture area video image is fused with the texture area of ​​each frame image to obtain scene fusion data.

[0036] As a preferred solution of the present invention, feature extraction is performed on the scene fusion data to obtain overlapping texture data, including:

[0037] The scene fusion data is constructed into a set T in the form of primitives, valid primitives are extracted from the set T, and a unit direction vector of a pixel point is calculated with the valid primitive as the center point;

[0038] A monitoring area is constructed by connecting the unit direction vectors between different pixel points, and a primitive weight is calculated for the monitoring area in the form of texture sampling weights;

[0039] Dynamically fused overlapping texture data is obtained in a virtual three-dimensional environment according to the primitive weights.

[0040] As a preferred solution of the present invention, the overlapping image regions are divided by the overlapping texture data, and the adjacent overlapping image regions are fused by the texture weighting method to obtain a virtual-real fusion scene image, including:

[0041] splicing the overlapping texture data in the direction of the unit direction vector to construct an overlapping area, and fusing the virtual image in the overlapping area by a texture weighting method;

[0042] The stitching lines of the fused virtual image are eliminated in the adjacent image overlapping areas to obtain a virtual-real fusion scene image.

[0043] As a preferred solution of the present invention, a texture weighting method is used to fuse the adjacent overlapping areas of the images, wherein the texture weighting method steps are:

[0044] Calculate the horizontal distance and vertical distance of the texture overlapping area according to the three-dimensional coordinate points of adjacent overlapping texture data;

[0045] Calculate the distance between the embedding point coordinates corresponding to the current overlapping texture data and the diagonal line of the overlapping area, and obtain the weights of different diagonal lines of the overlapping area;

[0046] The color value of the texture at the overlapping area is divided according to the diagonal weight, and the color value is embedded in the corresponding overlapping area as the optimal texture value.

[0047] The second aspect of the present invention provides a super-large space virtual-real fusion system based on the metaverse, comprising:

[0048] The video image acquisition module is used to acquire a wide range of video data, including real-time video data and historical data frame images acquired from various video surveillance cameras and various storage devices, and generate a real-time video image of each frame;

[0049] The virtual-real fusion control module calculates the image data and parameters of the video image acquisition module, converts them into three-dimensional virtual coordinates in real time, and realizes the registration and fusion of virtual and real scenes by constructing a matching function;

[0050] The scene correction module monitors the scene posture and video image in real time, extracts and matches local features of the video image data, and ensures that the data can be updated in real time after the scene changes;

[0051] The virtual scene interaction module integrates the video image with the three-dimensional virtual scene through the three-dimensional virtual scene visualization interface and displays it in real time.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The present invention adopts a multi-video and virtual environment fusion strategy based on dynamic textures, which is suitable for videos shot in ultra-large scenes and at high camera positions. It selects the optimal video texture for mixing in real time according to the camera posture, which can effectively reduce texture distortion and enable users to obtain reasonable observation results from any viewing angle.

[0054] The multi-scale space detection algorithm is used to detect the global features of the scene in real time, and the camera parameters are obtained in real time for accurate local feature extraction and matching. The feature points of different scale spaces are extracted in the multi-scale space through the texture weighting method. It has fast computing speed and image rotation invariance, improves the number of feature points and detection accuracy, and can ensure that the surveillance video after the camera parameters change is automatically integrated with the three-dimensional virtual scene in real time, avoiding tedious manual operations and lags. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0056] Figure 1 A flow chart of a method for fusion of virtuality and reality in a very large space based on the metaverse provided in an embodiment of the present invention;

[0057] Figure 2 A block diagram of a super-large space virtual-reality fusion system based on the metaverse provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] like Figure 1 As shown, the present invention provides a method for integrating virtuality and reality in a very large space based on a metaverse, comprising the following steps:

[0060] Acquire continuous video frame images as scene training data, use key frame detection method to acquire global features in real time for the scene training data, set similarity thresholds for adjacent video frame images according to the global features, and update video frame image parameters in real time by judging whether the similarity between adjacent video frame images corresponds to the similarity threshold value, so as to acquire local parameters of the real-time scene;

[0061] In this embodiment, a key frame detection method is used to monitor the global features of the video frame image in real time, extract the color of the features of the video image frame, calculate the probability of each color in the video space, and then form a color histogram corresponding to the video image frame. Then, based on the distance between the color histogram and the color, the color similarity between the color histogram and the video image frame is determined. This can effectively express the color features and spatial distribution of each frame of the video, and accurately reflect the occurrence frequency, color range and color distribution of different colors in the image.

[0062] A multi-scale spatial detection algorithm is used to obtain key feature points for the local parameters, a scene depth value of the video frame image is calculated through the key feature points, and the image is mapped into a three-dimensional virtual scene in units of video frames according to the scene depth value to obtain scene fusion data;

[0063] In this embodiment, a multi-scale spatial detection algorithm is used to perform grayscale registration, template registration, transform domain registration and feature registration on the image. Feature registration is used as the basis to retain the original features of the image. Through local feature extraction and matching, the positional relationship of the feature points in the two frames before and after the video image can be obtained to make the camera parameters before and after the change related. At the same time, the matching speed can also be optimized to meet the requirements of feature point detection accuracy and real-time performance in the application scenario.

[0064] Feature extraction is performed on the scene fusion data to obtain overlapping texture data, the image overlapping areas are divided according to the overlapping texture data, and the adjacent image overlapping areas are fused using a texture weighting method to obtain virtual-real fusion scene data.

[0065] In this embodiment, the fusion of video images in the three-dimensional virtual scene can rely on a separate three-dimensional engine to achieve the fusion application of a small-scale or partial three-dimensional virtual scene with video images and other resources, or it can rely on GIS to achieve the fusion application of three-dimensional geographic information visualization on a global scale.

[0066] The key frame detection method is used to obtain global features in real time for the scene training data, including:

[0067] The continuously acquired video frame images are taken as M I Unordered scene training data For feature point extraction, each image I m Get one by M m A set of local feature points , Represents image pixel coordinates The feature description operator at ;

[0068] In this embodiment, the image pixel coordinates Describing feature points has radiation invariance and geometric invariance, thus ensuring that the same feature points can be extracted from images taken under different lighting and rotation conditions.

[0069] Extract the feature description operator frame by frame The color of the video image frame at the position, calculate the probability of each color in the continuous video frame image, and construct an image color probability waveform corresponding to the video image frame with the video frame timing as the horizontal axis;

[0070] Divide different areas in the video image frame according to the image color probability waveform diagram, and calculate the image pixel coordinates in the video image frame The pixel value at the position is used to divide the number of pixels in each frame of the video by the pixel value, the color of the video frame image is calibrated by the pixel number, and the global feature is obtained.

[0071] In this embodiment, after the feature points of all images are extracted, preliminary matching needs to be performed based on the feature points to ensure that the two images in the potential matching set can achieve matching feature mapping through a transformation matrix.

[0072] In this embodiment, two images are selected from a region where the scene matching graph nodes are densely populated as the initial state, and the pose calibration of all images is completed by calculating the shooting pose and camera intrinsic parameters of the images, thereby obtaining global features.

[0073] Setting a similarity threshold for adjacent video frame images according to the global feature, and determining whether the similarity between adjacent video frame images corresponds to the similarity threshold, including:

[0074] The image probability of two adjacent frames of video images is calculated by the global feature, the image frame histogram of the video stream is obtained at the image pixel coordinates, and the video frame image difference value is calculated according to the image frame histogram, and its expression is:

[0075]

[0076] in, represents the histogram of the video stream image frame at the image pixel coordinate x, represents the histogram of the video stream image frame at the image pixel coordinate x+1, n represents the image color level, j represents the image color difference, Represents the pixel value of the video stream image frame at the image pixel coordinate x;

[0077] In this embodiment, the color difference values ​​between adjacent image frames are calculated to calibrate the color characteristics and spatial distribution of each frame of the video. The color space is usually divided into different intervals, and the number of pixels whose pixel values ​​fall within each frame of the video is calculated as the color histogram of the video image frame, which can effectively reduce the computational complexity.

[0078] A similarity threshold is set according to the video frame image difference value, and when the difference value of two adjacent video frames is greater than the similarity threshold, the video frame image parameters are updated to obtain the local parameters of the real-time scene.

[0079] In this embodiment, by setting a similarity threshold, the color histogram method can be used to quickly filter video image frames in large-scale monitoring visualization, thereby improving data calculation efficiency.

[0080] A multi-scale spatial detection algorithm is used to obtain key feature points for the local parameters, including:

[0081] The local parameters are filtered using a Gaussian operator, and the local parameters of the scene in the multi-scale space are smoothed using a spatial mapping function to obtain a multi-scale spatial graph. , whose expression is:

[0082]

[0083] in, represents the video frame image data at coordinates x and y, and z represents the spatial scale factor;

[0084] For the multi-scale spatial graph Calculate the image pixel values ​​between adjacent video frames, and transform the multi-scale spatial graph into Perform differential processing to obtain spatial extreme points;

[0085] The image data within the spatial extreme point is differentiated at a scale k, and a corner detection algorithm is used to extract a feature vector from the differential image at each scale;

[0086] In this embodiment, after two adjacent scales are separated by a constant k to obtain a differential image, FAST corner point detection is used on the differential image at each scale to extract feature points and the BRIEF algorithm is used to convert these feature points into feature vectors. After detecting the feature points in all scale spaces, non-maximum suppression is used to output feature points whose values ​​obtained by FAST corner point detection are greater than a similarity threshold. The similarity threshold can be set according to different scene data, and the present invention sets it to 27.

[0087] The feature vector is convolved with the global image to obtain local key feature points.

[0088] In this embodiment, a spatial mapping function is used to generate a scale space for local parameters, which can effectively preserve severely blurred feature points and increase the number of feature points. Different orders of spatial mapping functions can be selected according to the signal-to-noise ratio of the image, which has a better filtering effect.

[0089] Calculating the scene depth value of the video frame image by using the key feature points includes:

[0090] Constructing homogeneous vertex coordinates of the key feature points in the form of a set, and mapping the homogeneous vertex coordinates to an imaging plane to establish image space coordinates;

[0091] Dividing the horizontal coordinate of the image space coordinate by the width and height of the corresponding image to obtain the texture normalized coordinate, and transforming the texture normalized coordinate in sequence through a matrix to obtain the texture coordinate in the texture space;

[0092] The scene image depth value of any of the key feature points is calculated using the texture coordinates to obtain a real-time scene depth value.

[0093] In this embodiment, when the video image is fused into the three-dimensional virtual scene, the depth value of the scene is stored in a buffer. The depth information of the scene can be obtained without calculating the color buffer and related to the light source and texture. The obtained depth information can be stored in a newly created depth texture. By mapping the depth value of the buffer to the texture, a depth texture is created for each camera. As long as the objects and light sources in the three-dimensional virtual scene do not move relative to each other, the depth texture does not need to be updated.

[0094] Mapping the image to a three-dimensional virtual scene in units of video frames according to the scene depth value to obtain scene fusion data includes:

[0095] Creating a depth texture region of a virtual image using the scene depth value, and mapping the depth value to a texture via a buffer within the depth texture region;

[0096] By repeatedly calculating the pixel color value of the video image, determining whether each pixel of the video image is in a shadow area, the video image is divided into texture areas;

[0097] The pixel value of the texture area video image is fused with the texture area of ​​each frame image to obtain scene fusion data.

[0098] In this embodiment, the perspective in the real scene is observed by the scene depth value. When the video image is integrated into the three-dimensional virtual scene, the original two adjacent static textures are replaced with the video image texture, and the extended real-time shadow algorithm is used for processing to determine the mapping area corresponding to the surveillance video image, so as to obtain the correct surveillance video image texture.

[0099] Extracting features from the scene fusion data to obtain overlapping texture data includes:

[0100] The scene fusion data is constructed into a set T in the form of primitives, valid primitives are extracted from the set T, and a unit direction vector of a pixel point is calculated with the valid primitive as the center point;

[0101] A monitoring area is constructed by connecting the unit direction vectors between different pixel points, and a primitive weight is calculated for the monitoring area in the form of texture sampling weights;

[0102] Dynamically fused overlapping texture data is obtained in a virtual three-dimensional environment according to the primitive weights.

[0103] In this embodiment, when the three-dimensional virtual scene is fused from the perspective of the real scene, it is determined whether each pixel of the video image is in the occluded area, and the pixel coordinates of the video image are converted into coordinates in the clipping space using a matrix, and overlapping texture data is generated by recalculating the pixel color value.

[0104] Dividing the image overlapping area by the overlapping texture data, fusing the adjacent image overlapping areas by a texture weighting method, and obtaining a virtual-real fusion scene image, including:

[0105] splicing the overlapping texture data in the direction of the unit direction vector to construct an overlapping area, and fusing the virtual image in the overlapping area by a texture weighting method;

[0106] The stitching lines of the fused virtual image are eliminated in the adjacent image overlapping areas to obtain a virtual-real fusion scene image.

[0107] Adopting texture weighting method to fuse the adjacent overlapping areas of the images, wherein the steps of texture weighting method are:

[0108] Calculate the horizontal distance and vertical distance of the texture overlapping area according to the three-dimensional coordinate points of adjacent overlapping texture data;

[0109] Calculate the distance between the embedding point coordinates corresponding to the current overlapping texture data and the diagonal line of the overlapping area, and obtain the weights of different diagonal lines of the overlapping area;

[0110] In this embodiment, the weights of different diagonals in the overlapping area are calculated, which can effectively improve the operating efficiency and reduce high-consumption redundant local feature extraction operations. At the same time, image detection uses global color features to meet real-time requirements, making the fusion effect more acceptable and the visual effect better.

[0111] The color value of the texture at the overlapping area is divided according to the diagonal weight, and the color value is embedded in the corresponding overlapping area as the optimal texture value.

[0112] In this embodiment, for the stitching lines in the overlapping areas, the texture weighting method is used to fuse adjacent images, so as to achieve the image gradient effect of eliminating the stitching lines. In the overlapping part of the texture, the weighted fusion operation is performed along the diagonal direction of the stitching overlapping area. As the diagonal distance increases, the weight of one video image gradually decreases, while the weight of the other video image gradually increases. The weight is determined by the distance from the current point to the diagonal of the overlapping area.

[0113] like Figure 2 As shown, the second embodiment of the present invention is a super-large space virtual-real fusion system based on the metaverse, comprising:

[0114] The video image acquisition module is used to acquire a wide range of video data, including real-time video data and historical data frame images acquired from various video surveillance cameras and various storage devices, and generate a real-time video image of each frame;

[0115] The virtual-real fusion control module calculates the image data and parameters of the video image acquisition module, converts them into three-dimensional virtual coordinates in real time, and realizes the registration and fusion of virtual and real scenes by constructing a matching function;

[0116] The scene correction module monitors the scene posture and video image in real time, extracts and matches local features of the video image data, and ensures that the data can be updated in real time after the scene changes;

[0117] The virtual scene interaction module integrates the video image with the three-dimensional virtual scene through the three-dimensional virtual scene visualization interface and displays it in real time.

[0118] In this embodiment, by collecting images of the monitoring scene and inputting them into the system, the position and posture of the image camera, the dense point cloud and the precise model of the monitoring scene can be obtained. In order to meet the real-time requirements, the precise model can be simplified and cropped according to the hardware device configuration and needs. After the simplified model is imported into the system, the static virtual environment can be obtained through texture mapping. For newly shot monitoring videos, the position and posture of the monitoring camera can be calibrated by extracting and matching feature points of the video screen, thereby realizing the dynamic fusion of multiple monitoring videos and the virtual environment.

[0119] The present invention adopts a multi-video and virtual environment fusion strategy based on dynamic textures, which is suitable for videos shot in ultra-large scenes and at high camera positions. It selects the optimal video texture for mixing in real time according to the camera posture, which can effectively reduce texture distortion and enable users to obtain reasonable observation results from any viewing angle.

[0120] The multi-scale space detection algorithm is used to detect the global features of the scene in real time, and the camera parameters are obtained in real time for accurate local feature extraction and matching. The feature points of different scale spaces are extracted in the multi-scale space through the texture weighting method. It has fast computing speed and image rotation invariance, improves the number of feature points and detection accuracy, and can ensure that the surveillance video after the camera parameters change is automatically integrated with the three-dimensional virtual scene in real time, avoiding tedious manual operations and lags.

[0121] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A method for fusion of virtuality and reality in a very large space based on the metaverse, characterized in that: The following steps are involved: Acquire continuous video frame images as scene training data, use key frame detection method to acquire global features in real time for the scene training data, set similarity thresholds for adjacent video frame images according to the global features, and update video frame image parameters in real time by judging whether the similarity between adjacent video frame images corresponds to the similarity threshold value, so as to acquire local parameters of the real-time scene; The local parameters are filtered using a Gaussian operator, and the local parameters of the scene in the multi-scale space are smoothed using a spatial mapping function to obtain a multi-scale spatial graph. , whose expression is: in, represents the video frame image data at coordinates x and y, and z represents the spatial scale factor; For the multi-scale spatial graph Calculate the image pixel values ​​between adjacent video frames, and transform the multi-scale spatial graph into Perform differential processing to obtain spatial extreme points; The image data within the spatial extreme point is differentiated at a scale k, and a corner detection algorithm is used to extract a feature vector from the differential image at each scale; Convolving the feature vector with the global image to obtain local key feature points; Calculating the scene depth value of the video frame image through the key feature points, mapping the image to a three-dimensional virtual scene in units of video frames according to the scene depth value, and acquiring scene fusion data; Performing feature extraction on the scene fusion data to obtain overlapping texture data; splicing the overlapping texture data in the direction of the unit direction vector to construct an overlapping area, and fusing the virtual image in the overlapping area by a texture weighting method; Eliminate the stitching lines in the overlapping areas of adjacent images of the fused virtual image to obtain a virtual-real fused scene image; The steps of the texture weighting method are: Calculate the horizontal distance and vertical distance of the texture overlapping area according to the three-dimensional coordinate points of adjacent overlapping texture data; Calculate the distance between the embedding point coordinates corresponding to the current overlapping texture data and the diagonal line of the overlapping area, and obtain the weights of different diagonal lines of the overlapping area; The color value of the texture at the overlapping area is divided according to the diagonal weight, and the color value is embedded in the corresponding overlapping area as the optimal texture value.

2. According to the method for fusion of virtuality and reality in a large space based on the metaverse according to claim 1, it is characterized in that: The key frame detection method is used to obtain global features in real time for the scene training data, including: The continuously acquired video frame images are taken as M I Unordered scene training data For feature point extraction, each image I m Get one by M m A set of local feature points , Represents image pixel coordinates The feature description operator at ; Extract the feature description operator frame by frame The color of the video image frame at the position, calculate the probability of each color in the continuous video frame image, and construct an image color probability waveform corresponding to the video image frame with the video frame timing as the horizontal axis; Divide different areas in the video image frame according to the image color probability waveform diagram, and calculate the image pixel coordinates in the video image frame The pixel value at the position is used to divide the number of pixels in each frame of the video by the pixel value, the color of the video frame image is calibrated by the pixel number, and the global feature is obtained.

3. According to the method of ultra-large space virtual-real fusion based on metaverse in claim 2, it is characterized in that: Setting a similarity threshold for adjacent video frame images according to the global feature, and determining whether the similarity between adjacent video frame images corresponds to the similarity threshold, including: The image probability of two adjacent frames of video images is calculated by the global feature, the image frame histogram of the video stream is obtained at the image pixel coordinates, and the video frame image difference value is calculated according to the image frame histogram, and its expression is: in, represents the histogram of the video stream image frame at the image pixel coordinate x, represents the histogram of the video stream image frame at the image pixel coordinate x+1, n represents the image color level, j represents the image color difference, Represents the pixel value of the video stream image frame at the image pixel coordinate x; A similarity threshold is set according to the video frame image difference value, and when the difference value of two adjacent video frames is greater than the similarity threshold, the video frame image parameters are updated to obtain the local parameters of the real-time scene.

4. According to claim 3, a method for fusion of virtuality and reality in a large space based on the metaverse is characterized in that: Calculating the scene depth value of the video frame image by using the key feature points includes: Constructing homogeneous vertex coordinates of the key feature points in the form of a set, and mapping the homogeneous vertex coordinates to an imaging plane to establish image space coordinates; Dividing the horizontal coordinate of the image space coordinate by the width and height of the corresponding image to obtain the texture normalized coordinate, and transforming the texture normalized coordinate in sequence through a matrix to obtain the texture coordinate in the texture space; The scene image depth value of any of the key feature points is calculated using the texture coordinates to obtain a real-time scene depth value.

5. According to claim 4, a method for fusion of virtuality and reality in a very large space based on the metaverse is characterized in that: Mapping the image to a three-dimensional virtual scene in units of video frames according to the scene depth value to obtain scene fusion data includes: Creating a depth texture region of a virtual image using the scene depth value, and mapping the depth value to a texture via a buffer within the depth texture region; By repeatedly calculating the pixel color value of the video image, determining whether each pixel of the video image is in a shadow area, the video image is divided into texture areas; The pixel value of the texture area video image is fused with the texture area of ​​each frame image to obtain scene fusion data.

6. The method for fusion of virtuality and reality in a large space based on the metaverse according to claim 5, characterized in that: Extracting features from the scene fusion data to obtain overlapping texture data includes: The scene fusion data is constructed into a set T in the form of primitives, valid primitives are extracted from the set T, and a unit direction vector of a pixel point is calculated with the valid primitive as the center point; A monitoring area is constructed by connecting the unit direction vectors between different pixel points, and a primitive weight is calculated for the monitoring area in the form of texture sampling weights; Dynamically fused overlapping texture data is obtained in a virtual three-dimensional environment according to the primitive weights.

7. A metaverse-based ultra-large space virtual-real fusion system based on the metaverse-based ultra-large space virtual-real fusion method according to any one of claims 1 to 6, characterized in that: include: The video image acquisition module is used to acquire a wide range of video data, including real-time video data and historical data frame images acquired from various video surveillance cameras and various storage devices, and generate a real-time video image of each frame; The virtual-real fusion control module calculates the image data and parameters of the video image acquisition module, converts them into three-dimensional virtual coordinates in real time, and realizes the registration and fusion of virtual and real scenes by constructing a matching function; The scene correction module monitors the scene posture and video image in real time, extracts and matches local features of the video image data, and ensures that the data can be updated in real time after the scene changes; The virtual scene interaction module integrates the video image with the three-dimensional virtual scene through the three-dimensional virtual scene visualization interface and displays it in real time.

Images