Method and device for integrating large-field-of-view video and three-dimensional scene

Abstract

The present disclosure provides a method, device, equipment and medium for fusing a large-field-of-view video with a three-dimensional scene, including receiving multiple video data, processing the received video data, fusing videos with different perspectives in overlapping areas into a large-field-of-view video; constructing a corresponding three-dimensional scene by means of three-dimensional modeling and rendering; overlaying the large-field-of-view video and the three-dimensional scene image into the three-dimensional scene by means of projected texture mapping, and outputting a three-dimensional video fusion scene. The present disclosure fuses the video image into the three-dimensional scene by means of projected texture mapping, and gives the video spatial position information, thereby solving the problem of limited field of view of ordinary videos, and at the same time combining the spatial intuitiveness of the three-dimensional scene with the temporal continuity of the video, truly achieving time-space integration and four-dimensional linkage.

Description

Technical Field

[0001] The present disclosure relates to the technical field of coordinated optimization of carbon emission reduction, and in particular to a method, device, equipment and medium for fusing large-field-of-view video with three-dimensional scenes. Background Art

[0002] The current use of video visualization can be roughly divided into two categories. One is based on a large screen to display video images of multiple different shooting points. Due to the lack of information such as spatial location, operators need to search for content they are interested in in multiple video images. This method is very inefficient and has a very limited field of view.

[0003] Another method is to mark the camera positions at different points in the 3D scene. If the operator wants to retrieve the video footage of a certain location, he only needs to open the annotation of the corresponding location. However, this method simply superimposes the 2D video footage as a pop-up window on the front-end page with the 3D scene as the background. It is just a "big screen" with 3D location information, and does not fully utilize the intuitiveness of the 3D scene.

[0004] With the introduction of concepts such as digital twins and metaverse, multidimensional visualization has become more integrated into people's lives. As a branch of digital twin technology, three-dimensional video fusion technology has attracted widespread attention. Among them, Cui proposed an enhanced virtual environment that integrates video and three-dimensional scenes. By quickly reconstructing static three-dimensional virtual scenes from images taken by mobile phones or drones, multiple surveillance videos are integrated in real time, and the entire scene is observed from a bird's-eye view to obtain a more intuitive and global experience. Liu et al. proposed a method for quickly determining objects that integrate three-dimensional GIS scenes with multi-channel videos. A topological network model is dynamically established for the rendering objects of the video object set in the three-dimensional scene cone. The topological relationship between the video and the rendering object is established based on the topological network model, and the rapid determination and fusion rendering of the fused object are completed. However, there are problems in the existing video visualization technology that the fusion of three-dimensional scenes and videos is not in-depth and the three-dimensional intuitiveness is not fully utilized. Summary of the invention

[0005] The purpose of the present disclosure is to provide a method, device, equipment and medium for fusing large-field-of-view video with three-dimensional scene, so as to solve the problem that the fusion of three-dimensional scene and video is not deep and the three-dimensional intuitiveness is not fully utilized.

[0006] According to one aspect of the present disclosure, a method for fusing a wide-field-of-view video with a three-dimensional scene is provided, comprising: receiving multiple channels of video data, and processing the received video data to fuse videos with different perspectives having overlapping areas into a video with a large field of view; constructing a corresponding three-dimensional scene by means of three-dimensional modeling and rendering; overlaying the wide-field-of-view video and the three-dimensional scene image into the three-dimensional scene by means of projected texture mapping, and outputting a three-dimensional video fusion scene.

[0007] According to some embodiments of the present disclosure, receiving multiple channels of video data and processing the received video data to merge videos with different viewing angles having overlapping areas into a large field of view video specifically includes:

[0008] Acquire multi-channel video data; decode the acquired multi-channel videos, and the set of video frames corresponding to each channel of video is f; perform projection transformation on the video frames to obtain video frames f' after projection transformation; use SIFT algorithm to detect and match feature points of video frames with the same timestamp to obtain feature set F; use RANSAC algorithm to remove mismatches of feature set F, and the processed feature set is F; find the best stitching, fuse video frames, and output the fused video frames; synthesize a large field of view video V according to the video frames.

[0009] According to some embodiments of the present disclosure, constructing a corresponding three-dimensional scene by means of three-dimensional modeling and rendering specifically includes: inputting source data, the source data including point cloud, oblique photography, and physical photos; performing point cloud visualization, enhancing the point cloud by spatial displacement overlapping, and performing three-dimensional reconstruction, giving texture, and obtaining a basic model and terrain; constructing a plain model, and giving color, texture, and material to obtain a refined physical model, and establishing a model library; using Three.js and Cesium three-dimensional engine tools to render the three-dimensional scene and establish a scene library.

[0010] According to some embodiments of the present disclosure, a large-field-of-view video and a three-dimensional scene image are overlaid onto a three-dimensional scene by projecting texture mapping, and a three-dimensional video fusion scene is output, including: inputting a large-field-of-view video and a three-dimensional scene; fusion point calibration: calibrating points corresponding to multiple groups of video images and three-dimensional scenes; correction matrix calculation: serializing the corresponding coordinate points obtained, and filtering and calculating the corresponding homography matrix through the RANSAC algorithm. The optimal solution of the homography matrix is ​​the video fusion correction transformation matrix, which is denoted as H; projection matrix calculation: calculating the model view matrix M of the camera in the three-dimensional space. mv and the projection matrix M p ; Depth map generation: Get the depth d of the current field of view; Video fusion: Through the view matrix M mv , projection matrix M p The three-dimensional coordinates are transformed into screen coordinates and depth d, and then converted into world coordinates for screen post-processing. The final video stream image is converted into pixels seen at the specified position on the screen after rasterization.

[0011] According to some embodiments of the present disclosure, the video fusion further includes: transforming the three-dimensional space coordinates into screen space coordinates: ft = M p ·M mvv; where v is the coordinate of the vertex on the model surface in the 3D scene, and ft is the corresponding texture coordinate of the fragment; the screen space coordinate is calculated back to the world coordinate w corresponding to the 3D space: w = (M mv ·M p ) -1 ·f(ft,d), calculate the texture coordinates vt of the corresponding video stream image: vt=w·H, and output the 3D video fusion scene.

[0012] According to some embodiments of the present disclosure, in the projection matrix calculation, the camera position and the actual camera direction vector are used to calculate the model view matrix M of the camera in the three-dimensional space. mv and the projection matrix M p .

[0013] According to the second aspect of the present disclosure, a virtual coordination optimization device based on carbon emission reduction targets is provided, including: a multi-channel video fusion module, used to receive multi-channel video data, and process the received video data, and fuse videos with different perspectives with overlapping areas into a large field of view video; a three-dimensional modeling module, used to construct a corresponding three-dimensional scene through three-dimensional modeling and rendering methods; a three-dimensional video fusion module, used to overlay the large field of view video and the three-dimensional scene image into the three-dimensional scene through projection texture mapping, and output the three-dimensional video fusion scene.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, comprising a processor, and a memory and a network interface connected to the processor; the network interface is connected to a non-volatile memory in a server; when the processor is running, it retrieves a computer program from the non-volatile memory through the network interface, and runs the computer program through the memory to execute the method of fusing a large field of view video with a three-dimensional scene as described above.

[0015] According to a fourth aspect of the present disclosure, a storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method for fusing a large field of view video with a three-dimensional scene as described above is implemented.

[0016] It can be seen from the above technical solutions that the present disclosure proposes a method, device, equipment and medium for fusing large-field-of-view video with three-dimensional scenes, which fuses video images into three-dimensional scenes by means of projection texture mapping and gives the video spatial position information. On the one hand, it fuses multiple videos with different perspectives and overlapping areas to solve the problem of limited field of view of ordinary videos; on the other hand, it gives the video spatial position information, so that the spatial intuitiveness of the three-dimensional scene and the temporal continuity of the video can be combined, truly achieving time-space integration and four-dimensional linkage. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A flow chart of a method for fusing a large field of view video with a three-dimensional scene according to the present disclosure is presented;

[0018] Figure 2 A schematic diagram of a multi-channel video fusion process according to an embodiment of the present disclosure is presented;

[0019] Figure 3 A schematic diagram of the three-dimensional model construction and scene rendering process according to an embodiment of the present disclosure is presented;

[0020] Figure 4 A schematic diagram of a three-dimensional video fusion process according to an embodiment of the present disclosure is presented;

[0021] Figure 5 A schematic diagram of a computer device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0023] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0024] Exemplary Methods

[0025] like Figure 1 As shown, the present disclosure provides a method for fusing a large field of view video with a three-dimensional scene, comprising the following steps:

[0026] S1: receiving multiple channels of video data, and processing the received video data to fuse videos of different viewing angles with overlapping areas into a large-field-of-view video.

[0027] S11: Input multiple video channels V1, V2, ..., V n ;

[0028] S12: Video decoding, the set of video frames corresponding to each video channel is f;

[0029] f={f1,f2,…,f t},t∈[0,s V T V ]

[0030] Among them, t is the timestamp, s V is the frame rate of the video, T V is the video duration;

[0031] S13: Projection transformation. If the video frames are directly spliced, the consistency of the field of view will be destroyed, making the spliced ​​panorama look incoherent. Therefore, all video frames need to be projected onto the cylinder through projection transformation. The present invention uses bilinear interpolation to solve the projection error. The video frame after projection transformation is represented by f'.

[0032] f'=BI(Pt(f))

[0033] Among them, Pt represents projection transformation, BI represents bilinear interpolation;

[0034] S14: Use SIFT algorithm to detect and match feature points of video frames with the same timestamp to obtain feature set F.

[0035] F = SIFT (f 1t ',f 2t ',…,f nt ')

[0036] Among them, t is the timestamp.

[0037] S15: Use the RANSAC algorithm to remove mismatches from the feature set F. The processed feature set is

[0038]

[0039] S16: Find the best seam. The seam refers to the most similar line in the overlapping area between images, usually in the position where the color difference is small or the gradient change is large. There are several commonly used methods for finding the seam line. 1) Point-by-point method: the simplest, and the stitching effect is relatively poor; 2) Graph cut method: high computational complexity, and very good stitching effect; 3) Dynamic programming method: equivalent to the neutralization of the first two, with both computational complexity and stitching effect in the middle. The present invention uses dynamic programming to find the best seam, and the best seam is represented by L.

[0040]

[0041] Where DP stands for dynamic programming algorithm.

[0042] S17: Seam fusion. The video frames are fused according to the best seam found. The fused video frame is denoted by f”;

[0043] f"=L(f 1t ',f 2t ',…,f nt ')

[0044] S18: output the fused video frame;

[0045] S19: synthesizing a large field of view video V according to the video frames;

[0046]

[0047] S2: Construct corresponding 3D scenes through 3D modeling and rendering;

[0048] S21: Input source data: point cloud, oblique photography, entity photos;

[0049] S22: Automatic modeling: Use tools such as Context Capture and PhotoScan to visualize point clouds, enhance point clouds through spatial displacement and overlap, and then perform 3D reconstruction, and finally give textures to obtain basic models and terrain features;

[0050] S23: Manual modeling: Use 3D Studio Max, Blender and other tools to build a model in proportion to the real size, then give it color, texture and material to obtain a refined solid model and establish a model library;

[0051] S24: Model fusion: Use tools such as MeshMixer and 3D Studio Max to stitch and fuse the refined entity model with the basic model and terrain;

[0052] S25: Scene rendering: Use 3D engine tools such as Three.js and Cesium to render 3D scenes and build a scene library;

[0053] S26: Output the three-dimensional scene.

[0054] S3: Overlay the large field of view video and the 3D scene image onto the 3D scene by projecting texture mapping, and output the 3D video fusion scene;

[0055] S31: input large field of view video and three-dimensional scene;

[0056] S32: fusion point calibration: calibrate the points corresponding to the multiple sets of video images and the three-dimensional scene. For the convenience of calculation, it is assumed that the captured plane is located at Z=0 in the world coordinate system;

[0057] S33: Correction matrix calculation: the corresponding coordinate points obtained in S32 are serialized, and the corresponding homography matrix is ​​screened and calculated by the RANSAC algorithm. The optimal solution of the homography matrix is ​​the video fusion correction transformation matrix, which is denoted as H.

[0058]

[0059] S34: Projection matrix calculation: Use the camera position, actual camera direction vector and other camera pose information to calculate the camera's model view matrix M in three-dimensional space. mv and the projection matrix M p ;

[0060] S35: Depth map generation: obtaining the depth d of the current field of view;

[0061] S36: Video fusion: Use the above matrix and depth information to transform the three-dimensional coordinates into screen coordinates, and then convert them into world coordinates for screen post-processing. The final video stream image is converted into pixels seen at the specified position on the screen after rasterization operation.

[0062] First, transform the 3D space coordinates to screen space coordinates:

[0063] ft=M p ·M mv ·v

[0064] Among them, v is the vertex coordinate of the model surface in the three-dimensional scene, and ft is the corresponding fragment texture coordinate.

[0065] Then calculate the screen space coordinates back to the world coordinates w corresponding to the three-dimensional space:

[0066] w=(M mv ·M p ) -1 f(ft,d)

[0067] Finally, the texture coordinates vt of the corresponding video stream image are calculated:

[0068] vt=w·H

[0069] S37: Output the three-dimensional video fusion scene.

[0070] Exemplary Devices

[0071] According to an embodiment of the present application, a device for fusing a large-field-of-view video and a three-dimensional scene includes:

[0072] The multi-channel video fusion module is used to receive multi-channel video data and process the received video data to fuse videos with different perspectives in overlapping areas into a large-field-of-view video; the 3D modeling module is used to construct the corresponding 3D scene through 3D modeling and rendering; the 3D video fusion module is used to overlay the large-field-of-view video and 3D scene image into the 3D scene through projection texture mapping, and output the 3D video fusion scene.

[0073] Exemplary Electronic Devices

[0074] Below, reference Figure 5 The electronic device according to the embodiment of the present application is described. The electronic device may be an electronic device integrated with a first imaging device, or a stand-alone device independent of the first imaging device, and the stand-alone device may communicate with the first imaging device to receive the collected input signal therefrom.

[0075] Figure 5 A block diagram of an electronic device according to an embodiment of the present application is shown.

[0076] like Figure 5 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0077] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0078] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the moving object tracking method of each embodiment of the present application described above and / or other desired functions. Various contents such as the current frame image, the previous frame image, the result image of the differential processing, etc. may also be stored in the computer-readable storage medium.

[0079] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0080] For example, when the electronic device is integrated with the first imaging device, the input device 13 may be the first imaging device, such as a camera, for capturing each frame of an image of a moving object. When the electronic device is a stand-alone device, the input device 13 may be a communication network connector for receiving the collected input signal from the first imaging device.

[0081] In addition, the input device 13 may also include, for example, a keyboard, a mouse, and the like.

[0082] The output device 14 can output various information to the outside, including the determined deflection angle information, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0083] Of course, to simplify, Figure 5 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may also include any other appropriate components.

[0084] Exemplary computer program products and computer-readable storage media

[0085] An embodiment of the present invention also provides a computer-readable storage medium, on which computer program instructions are stored; when the computer program instructions are executed by a processor, the training method of the photovoltaic power generation prediction model provided by the embodiment of the present invention is implemented, or the photovoltaic power generation power prediction method provided by the embodiment of the present invention is implemented.

[0086] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0087] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present disclosure are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (Read-Only Memory, ROM), flash memory, erasable read-only memory (ErasableRead Only Memory, EROM), floppy disks, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0088] It should also be noted that the exemplary embodiments mentioned in the present disclosure describe some methods or systems based on a series of steps or devices. However, the present disclosure is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0089] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0090] The above is only a specific implementation of the present disclosure. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present disclosure is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure.

Claims

1. A method for fusing a large field of view video with a three-dimensional scene, characterized in that: include: Receive multiple video data channels, process the received video data, and fuse videos with different viewing angles and overlapping areas into a large-viewing-field video; Construct corresponding 3D scenes through 3D modeling and rendering; Overlay the large field of view video and 3D scene image onto the 3D scene by projecting texture mapping, and output the 3D video fusion scene; The method of overlaying the large-field-of-view video and the three-dimensional scene image onto the three-dimensional scene by projecting texture mapping to output the three-dimensional video fusion scene includes: Input large field of view videos and 3D scenes; Fusion point calibration: calibrate the points corresponding to multiple sets of video images and 3D scenes; Correction matrix calculation: The corresponding coordinate points are serialized, and the corresponding homography matrix is ​​calculated by screening through the RANSAC algorithm. The optimal solution of the homography matrix is ​​the video fusion correction transformation matrix, denoted as H; Projection matrix calculation: Calculate the model view matrix M of the camera in three-dimensional space mv and the projection matrix M p ; Depth map generation: Get the depth d of the current field of view; Video fusion: through the view matrix M mv , projection matrix M p The three-dimensional coordinates are transformed into screen coordinates and then into world coordinates for screen post-processing. The final video stream image is converted into pixels seen at the specified position on the screen after rasterization operation; Transform 3D space coordinates to screen space coordinates: Among them, v is the coordinate of the vertex on the model surface in the three-dimensional scene, is the corresponding fragment texture coordinate; Calculate the screen space coordinates back to the corresponding world coordinates w in three-dimensional space: Calculate the texture coordinates of the corresponding video stream image : Output 3D video fusion scene.

2. The method according to claim 1, characterized in that Receive multiple video data channels and process the received video data to merge videos with different viewing angles and overlapping areas into a large field of view video, including: Obtain multiple channels of video data; Decode the acquired multiple videos, and the set of video frames corresponding to each video is f; Performing a projection transformation on the video frame to obtain a video frame after projection transformation ; Use SIFT algorithm to detect and match feature points of video frames with the same timestamp to obtain feature set F; The RANSAC algorithm is used to remove mismatches from the feature set F. The processed feature set is: ; Find the best stitching, fuse the video frames, and output the fused video frames; A large field of view video V is synthesized according to the video frames.

3. The method according to claim 1, characterized in that The construction of the corresponding three-dimensional scene by three-dimensional modeling and rendering means specifically includes: Input source data, the source data including point cloud, oblique photography, and physical photos; Perform point cloud visualization, enhance point clouds through spatial displacement and overlap, and perform 3D reconstruction and texture to obtain basic models and terrain features; Construct a plain model, and give it color, texture, and material to obtain a refined solid model and establish a model library; Use Three.js and Cesium 3D engine tools to render 3D scenes and build a scene library.

4. The method according to claim 3, characterized in that In the projection matrix calculation, the camera position and the actual camera direction vector are used to calculate the camera's model view matrix M in three-dimensional space. mv and the projection matrix M p .

5. The device used in the method for fusing a large field of view video with a three-dimensional scene according to claim 1, characterized in that: include: A multi-channel video fusion module is used to receive multi-channel video data, process the received video data, and fuse videos with different viewing angles in overlapping areas into a large-viewing-field video; A 3D modeling module is used to construct corresponding 3D scenes through 3D modeling and rendering; The 3D video fusion module is used to overlay the large-field-of-view video and 3D scene image into the 3D scene by projecting texture mapping, and output the 3D video fusion scene.

Images