Video fusion method, device, system, equipment and medium for three-dimensional scene

By obtaining video stream and real-time parameter information, preprocessing and UV mapping, the video stream is integrated with three-dimensional scenes, and the problem of insufficient fusion of video streams in traditional methods is solved, and efficient and accurate video fusion and user experience in three-dimensional scenes is achieved promote.

CN118694910BActive Publication Date: 2024-11-05中电信数字城市科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411187268.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-05
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

Traditional methods are difficult to integrate the surveillance video stream in motion into a three-dimensional scene, cannot fully utilize the intuitiveness and three-dimensional sense of the three-dimensional scene, and it is difficult to adapt to the real-time and dynamic video streaming needs, and cannot achieve efficient and accurate video fusion.

Method used

By obtaining the image data of each frame of the video stream and the real-time parameter information of the acquisition device, preprocessing is performed to obtain the texture image, setting the camera viewport of the video stream in a three-dimensional scene, and determining the camera viewport based on the latitude, longitude, height and rotation angle Scaling value, fusing texture images with 3D scenes based on UV mapping.

Benefits of technology

Real-time interaction and integration between dynamic video surveillance streams and three-dimensional models in three-dimensional scenes is realized, visualization effects and user experience are improved, and the problem of insufficient integration of video streams in traditional methods is solved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118694910B_ABST
    Figure CN118694910B_ABST
Patent Text Reader

Abstract

The present invention provides a video fusion method, device, system, equipment and medium for three-dimensional scenes, and relates to the technical field of video processing. The method comprises: obtaining image data of each frame of a video stream and real-time parameter information of an acquisition device; preprocessing the image data to obtain a texture image; setting a camera viewport corresponding to the video stream in the three-dimensional scene displayed on a UE client, and determining a scaling value of the camera viewport based on the longitude and latitude, the altitude and the rotation angle; determining UV mapping based on the scaling value of the camera viewport; and fusing the texture image with the three-dimensional scene based on the UV mapping, thereby realizing real-time interaction and fusion of a dynamic video surveillance stream with a three-dimensional model in the three-dimensional scene, and improving visualization effects and user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a video fusion method, device, system, equipment and medium for three-dimensional scenes. Background Art

[0002] With the rapid development of cutting-edge concepts such as smart cities and the metaverse, urban management and service methods are undergoing profound changes. The construction of smart cities relies on the integrated application of advanced technologies such as big data, the Internet of Things, cloud computing, and artificial intelligence. As an emerging digital space, the metaverse provides new perspectives and tools for urban governance. In this context, combining urban surveillance video streams with three-dimensional twin scenes has become an important means to improve urban management efficiency and enhance urban perception capabilities.

[0003] In traditional methods, urban surveillance video streams are mainly embedded into three-dimensional scenes through simple coordinate transformation to achieve basic video display functions. However, this method has obvious limitations: on the one hand, the simple superposition of two-dimensional surveillance video and three-dimensional scene cannot fully utilize the intuitiveness and stereoscopic sense of three-dimensional scene, resulting in users lacking depth perception when observing surveillance images; on the other hand, with the widespread application of new surveillance equipment such as drones, traditional methods are difficult to adapt to the real-time and dynamic video stream transmission needs, and cannot achieve efficient and accurate video fusion. Summary of the invention

[0004] The purpose of the present invention is to provide a video fusion method, device, system, equipment and medium for three-dimensional scenes, so as to solve the technical problem existing in the prior art that a moving surveillance video stream cannot be fused into a three-dimensional scene for flow field demonstration.

[0005] In a first aspect, an embodiment of the present invention provides a video fusion method for a three-dimensional scene, comprising:

[0006] Obtain image data of each frame of the video stream and real-time parameter information of the acquisition device, the acquisition device is used to acquire the video stream, and the real-time parameter information includes longitude and latitude, height of the acquisition device from the ground, and rotation angle of the acquisition device;

[0007] Preprocess the image data to obtain a texture image;

[0008] Set the camera viewport corresponding to the video stream in the 3D scene displayed on the UE client, and determine the zoom value of the camera viewport based on the latitude and longitude, altitude and rotation angle;

[0009] Determine UV mapping based on the zoom value of the camera viewport;

[0010] Based on UV mapping, the texture image is integrated with the 3D scene.

[0011] In an optional embodiment, the image data is preprocessed, including:

[0012] Decode the image data format into RGB format;

[0013] Sort the decoded image data.

[0014] In an optional embodiment, setting a camera viewport corresponding to a video stream in a three-dimensional scene includes:

[0015] Determine the location of the acquisition device in the three-dimensional scene based on latitude, longitude and altitude;

[0016] Set the camera viewport according to the position of the acquisition device in the 3D scene;

[0017] Adjusts the orientation of the camera viewport based on the rotation angle.

[0018] In an optional embodiment, the zoom value of the camera viewport is determined based on the latitude and longitude, altitude and rotation angle:

[0019] The zoom value of the camera viewport is determined by the following zoom formula:

[0020]

[0021] Where FOV is the field of view of the camera, FPD is the far focal length, and AR is the resolution aspect ratio.

[0022] In an optional embodiment, based on the zoom value of the camera viewport, determining the UV mapping includes:

[0023] Get scene depth value and pixel depth;

[0024] Get the model depth information based on the scene depth value, pixel depth value and camera viewport scaling value;

[0025] Based on the model depth information and the relative direction vector, the UV mapping is obtained. The relative direction is the direction vector of each pixel point between the camera viewport position and the ground position.

[0026] In an optional embodiment, the texture image is merged with the three-dimensional scene based on UV mapping, including:

[0027] Based on the model depth information and UV mapping, a depth map corresponding to the texture image is generated, and the depth map is fused with the three-dimensional scene.

[0028] In a second aspect, an embodiment of the present invention provides a video fusion device for a three-dimensional scene, comprising:

[0029] An acquisition module is used to acquire image data of each frame of the video stream and real-time parameter information of an acquisition device, the acquisition device is used to acquire the video stream, and the real-time parameter information includes longitude and latitude, height of the acquisition device from the ground, and rotation angle of the acquisition device;

[0030] A preprocessing module, used for preprocessing image data to obtain a texture image;

[0031] A setting module is used to set the camera viewport corresponding to the video stream in the three-dimensional scene displayed on the UE client, and determine the zoom value of the camera viewport based on the latitude and longitude, altitude and rotation angle;

[0032] UV module, used to determine UV mapping based on the zoom value of the camera viewport;

[0033] The fusion module is used to fuse the texture image with the three-dimensional scene based on UV mapping.

[0034] In a third aspect, an embodiment of the present invention provides a video fusion system for a three-dimensional scene, including: a UE client and a collection device, wherein the collection device is connected to the UE client;

[0035] The acquisition device is used to acquire the monitored video stream;

[0036] The UE client is used to obtain the image data of each frame of the video stream and the real-time parameter information of the acquisition device, the real-time parameter information includes the longitude and latitude, the height of the acquisition device from the ground and the rotation angle of the acquisition device; pre-process the image data to obtain a texture image; set the camera viewport corresponding to the video stream in the three-dimensional scene displayed on the UE client, and determine the zoom value of the camera viewport based on the longitude and latitude, height and rotation angle; determine the UV mapping based on the zoom value of the camera viewport; based on the UV mapping, fuse the texture image with the three-dimensional scene.

[0037] In a fourth aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods in the first aspect when executing the computer program.

[0038] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any method in the first aspect are implemented.

[0039] The present invention provides a video fusion method, device, system, equipment and medium for three-dimensional scenes, which obtain image data of each frame of a video stream and real-time parameter information of an acquisition device; pre-process the image data to obtain a texture image; set a camera viewport corresponding to the video stream in the three-dimensional scene, and determine the zoom value of the camera viewport based on the longitude and latitude, altitude and rotation angle; determine UV mapping based on the zoom value of the camera viewport; and fuse the texture image with the three-dimensional scene based on the UV mapping, thereby realizing real-time interaction and fusion of a dynamic video surveillance stream and a three-dimensional model in the three-dimensional scene, thereby improving visualization effects and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of a flow chart of a video fusion method for a three-dimensional scene provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of video fusion provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the structure of a video fusion device for a three-dimensional scene provided by an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of the structure of a video fusion system for a three-dimensional scene provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0047] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In order to facilitate the understanding of the embodiments of the present application, the names appearing in the present application are explained below:

[0049] For three-dimensional scenes, UE (Unreal Engine) provides a comprehensive development environment, including a powerful rendering engine, physics engine, particle system, animation tools, audio system, etc., as well as an intuitive editor that allows developers to create high-quality three-dimensional scenes, characters, animations, and interactive content.

[0050] RGB is a format for recording color images. The RGB color model is a color standard in the industry. It obtains various colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue. This standard covers almost all colors that can be perceived by human vision and is one of the most widely used color systems.

[0051] YUV is a color encoding method. In the YUV format, color information is separated into two parameters: brightness (Y) and chrominance (U, V). This separation method gives YUV an advantage in video compression and transmission.

[0052] RTSP (Real Time Streaming Protocol) is a control protocol for real-time streaming media transmission and playback. It was jointly proposed by Real Network and Netscape and complies with the IETF RFC 2326 specification.

[0053] UV: UV coordinates mean that all image files are a two-dimensional plane, U in the horizontal direction and V in the vertical direction. Through the two-dimensional UV coordinate system of this plane, any pixel on the image can be located. The UV coordinates are a coordinate system used in three-dimensional computer graphics to map two-dimensional images (textures) to the surface of three-dimensional models. In the texture mapping process, the UV coordinates define how each point on the surface of the three-dimensional model corresponds to the pixels in the texture image.

[0054] Shader is an editable program used to implement image rendering and replace the fixed rendering pipeline. Vertex Shader is mainly responsible for the calculation of the geometric relationship of vertices, and Pixel Shader is mainly responsible for the calculation of the color of the fragment. Shader replaces the traditional fixed rendering pipeline and can realize the relevant calculations in 3D graphics. Due to its editability, it can achieve a variety of image effects without being limited by the fixed rendering pipeline of the graphics card.

[0055] Now the technology has improved the fusion method of monitoring video and three-dimensional scene. For example, through UV expansion and Decal projection, the intuitiveness and stereoscopic sense of monitoring video in three-dimensional scene can be improved. However, this solution still faces the problem of insufficient adjustment of dynamic video stream, which cannot meet the seamless fusion requirements of real-time video stream transmitted by mobile monitoring equipment such as drones and UE three-dimensional scene; or it mainly relies on laser mapping instruments to measure relative position information. Although this method is accurate, it is costly and difficult to adapt to the scene changes of real-time longitude and latitude transmission, which limits its flexibility and extensiveness in practical applications. Based on this, the present application provides a video fusion method, device, system, equipment and medium for three-dimensional scene, by obtaining the image data of each frame of the video stream and the real-time parameter information of the acquisition device; pre-processing the image data to obtain a texture image; setting the camera viewport corresponding to the video stream in the three-dimensional scene, and determining the zoom value of the camera viewport based on the longitude and latitude, height and rotation angle; determining the UV mapping based on the zoom value of the camera viewport; based on the UV mapping, the texture image is fused with the three-dimensional scene, which reflects the intuitiveness and stereoscopic sense of the three-dimensional scene, and solves the limitations of the actual application scene.

[0056] Figure 1 The present invention provides a flow chart of a video fusion method for a three-dimensional scene, wherein the execution subject of the method is a UE client with a UE engine function, and the UE client is connected to a collection device, and the collection device is used to collect video surveillance streams. Figure 1 As shown, the method includes:

[0057] Step S110, obtaining image data of each frame of the video stream and real-time parameter information of the acquisition device, the acquisition device is used to acquire the video stream, and the real-time parameter information includes longitude and latitude, height of the acquisition device from the ground, and rotation angle of the acquisition device.

[0058] Among them, data is read frame by frame from the video stream in a loop or event-driven manner, and the read data is parsed to extract image data of each frame.

[0059] The real-time parameter information is transmitted to the UE client through the acquisition device, wherein the real-time parameter information includes the latitude and longitude of the acquisition device, the height of the acquisition device from the ground, the rotation angle corresponding to the video stream, and the field of view, far focal length and near focal length of the acquisition device.

[0060] Step S120: pre-process the image data to obtain a texture image.

[0061] In a possible embodiment, step S120 includes:

[0062] Decode the image data format into RGB format;

[0063] Sort the decoded image data.

[0064] The format of the image data obtained from the video stream is YUV format, where Y in YUV represents brightness, U and V represent chrominance, which is used to specify pixel color and describe image saturation and color. In order to achieve a more realistic and rich visual experience and presentation effect, the YUV format of the image data is decoded into RGB format, and the decoded image is a texture image.

[0065] In this application, the YUV format data is converted to RGB format using the following conversion formula:

[0066]

[0067] According to the transformation formula between YUV and RGB, the YUV image can be converted into an RGB image by accessing each point in the image data pixel by pixel. The process of converting the YUV format of the image data into the RGB format is to decode the format of the image data.

[0068] Furthermore, the decoded image data is sorted according to the serial numbers, and the image data is mapped to the corresponding video stream according to the serial numbers, and the texture of the image data is set, so as to obtain the texture image corresponding to each frame of the video stream.

[0069] By converting image data in YUV format into texture images in RGB format, the image color reproduction can be accurately improved, giving users a more realistic and rich visual experience, and optimizing the image presentation on display screens or other display devices. The transformation formula can efficiently complete the format conversion, reducing the time in the decoding process, thereby improving the speed and efficiency of data processing.

[0070] Step S130: Set the camera viewport corresponding to the video stream in the three-dimensional scene displayed on the UE client, and determine the zoom value of the camera viewport based on the longitude and latitude, altitude and rotation angle.

[0071] In a possible embodiment, in step S130, setting a camera viewport corresponding to the video stream in the three-dimensional scene includes:

[0072] Based on the longitude, latitude and altitude, determine the position of the acquisition device in the three-dimensional scene displayed by the UE client;

[0073] Set the camera viewport according to the position of the acquisition device in the 3D scene displayed by the UE client;

[0074] Adjusts the orientation of the camera viewport based on the rotation angle.

[0075] Specifically, the longitude, latitude and altitude of the acquisition device are converted into three-dimensional coordinates in the three-dimensional scene, that is, the geographic coordinates are converted into the actual coordinates of the scene, so as to ensure that the content in the video stream is consistent with the virtual environment in the three-dimensional scene in terms of geographical location. Usually, the geographic coordinate point is mapped to a point in three-dimensional space, which can be achieved by converting the geographic coordinate system to the three-dimensional Cartesian coordinate system.

[0076] Furthermore, after converting the geographic coordinates to the actual scene coordinates, since the 3D scene usually uses its own coordinate system, the definitions of the origin, X-axis, Y-axis, and Z-axis may be different from the geographic coordinate system. Therefore, after converting the latitude, longitude, and altitude to 3D coordinates, they need to be adjusted according to the coordinate system of the 3D scene. Generally, the coordinate system of the 3D scene is the same as the geographic coordinate system.

[0077] A camera viewport is set in front of the actual coordinates of the scene. The camera viewport is used to simulate the viewing angle of the acquisition device. The content in the video stream can be observed through the camera viewport.

[0078] Based on the rotation angle of the acquisition device, the camera viewport is adjusted to the correct direction to simulate the viewing angle of the acquisition device, thereby ensuring that when the video stream is displayed in the three-dimensional scene, its viewing angle is consistent with the viewing angle of the acquisition device when it is actually shot.

[0079] In a possible embodiment, in step S130, determining the zoom value of the camera viewport based on the latitude and longitude, altitude and rotation angle includes:

[0080] The zoom value of the camera viewport is determined by the following zoom formula:

[0081]

[0082] Where FOV is the field of view of the camera, FPD is the far focal length, and AR is the resolution aspect ratio.

[0083] Furthermore, in the scaling formula The value is the scaling along the X axis in the scene; The value is the scaling along the Y axis in the scene; The value is the scaling along the Z axis in the scene, where the X axis is parallel to the X axis of the two-dimensional image, i.e. the horizontal direction, the Y axis is parallel to the Y axis of the two-dimensional image, i.e. the vertical direction, and the Z axis is perpendicular to the plane of the two-dimensional image.

[0084] In the present applicant, taking drone flight photography as an example, the drone first transmits the longitude and latitude, the height and rotation angle of the drone from the ground, and the field of view, far focal length and near focal length of the projection camera on the drone to the UE client in real time, and the UE engine of the UE client converts the longitude and latitude into the actual coordinates of the scene. In the three-dimensional scene of the UE client, the position of the drone in the three-dimensional scene is determined according to the longitude and latitude and the altitude, and a camera viewport is set in front of the position. The camera viewport is a regular tetrahedron with a length, width and height of 2, 2 and 1 respectively as the scene perspective (camera viewport), and the vertex position is the drone position. Since the regular tetrahedron conforms to the shape of the camera wide angle, a regular tetrahedron is selected and scaled, and its scaling size is:

[0085]

[0086] The part where the regular tetrahedron overlaps with the ground in the three-dimensional scene is the part where the video transmitted by the drone is projected into the three-dimensional scene.

[0087] Then, through the scene capture component in the three-dimensional scene, the corresponding camera viewport parameters are set based on the field of view, far focal length and near focal length of the projection camera on the drone, such as the field of view, near focal length, far focal length and dynamically created depth map.

[0088] Step S140, determining UV mapping based on the zoom value of the camera viewport;

[0089] In a possible embodiment, step S140, determining UV mapping based on the zoom value of the camera viewport, includes:

[0090] Get scene depth value and pixel depth;

[0091] Get the model depth information based on the scene depth value, pixel depth value and camera viewport scaling value;

[0092] Based on the model depth information and the relative direction vector, the UV mapping is obtained. The relative direction is the direction vector of each pixel point between the camera viewport position and the ground position.

[0093] Among them, the scene depth value is the actual distance from the acquisition device to a point in the scene. This distance can be obtained through the depth buffer of graphics rendering or through geometric calculation before rendering. The scene depth is the sampling depth at any position.

[0094] Pixel depth is the scene depth information represented by each pixel in the rendered image. Pixel depth is the sampling depth of the currently drawn pixel.

[0095] The model depth information is:

[0096]

[0097] Where Va is the normal that converts the y vector value from the relative coordinate system to the world coordinate system multiplied by the height of the drone; Vb is the scene depth divided by the pixel depth, multiplied by the absolute world coordinate minus the camera world coordinate, and the result plus the position of the camera in the three-dimensional scene; Vc is the vector that converts the x vector value from the relative coordinate system to the world coordinate system using the vertex iterator output vector; Vd is the normal that converts the z vector value from the relative coordinate system to the world coordinate system using the vertex iterator output vector, where the y vector value is the vector value in the Y-axis direction in the scaling formula, the x vector value is the vector value in the X-axis direction in the scaling formula, and the z vector value is the vector value in the Z-axis direction in the scaling formula.

[0098] In a possible embodiment, in this application, The vector is converted to relative coordinates to get the UV of the model, where 100 can be adjusted according to the projection size in the 3D scene.

[0099] The relative coordinate system is the relative coordinate of the object, and the world coordinate system is the coordinate of the object in the scene. For example, there are 100 points in the object, and the coordinate of one of the points is (0,0,100). This coordinate is the relative coordinate. Place the object in the scene, and the scene coordinate is (100,0,0). Then the world coordinate of this point is (100,0,100).

[0100] Step S150: Based on UV mapping, the texture image is merged with the three-dimensional scene.

[0101] In a possible embodiment, in step S150, the texture image is fused with the three-dimensional scene based on UV mapping, including:

[0102] Based on the model depth information and UV mapping, a depth map corresponding to the texture image is generated, and the depth map is fused with the three-dimensional scene.

[0103] Specifically, first, the model depth information and UV mapping are used as input, where the model depth information is the distance from the camera to each point on the surface of the 3D model, and the UV mapping is the corresponding position of these points on the texture image. According to the model depth information and UV mapping, the depth map corresponding to the texture image is obtained, and the depth map is configured with the corresponding depth value, which is usually a scalar, indicating the distance from the point to the camera.

[0104] Furthermore, by configuring the corresponding depth value in the depth map, the self-luminous color of each pixel can be calculated by the shader according to the UV mapping and the depth information.

[0105] Then, the depth map is applied to the self-luminous material of the 3D model to complete the fusion of the texture image and the 3D scene, such as Figure 2 shown.

[0106] In this application, by obtaining image data of each frame of the video stream and real-time parameter information of the acquisition device, the zoom value of the camera viewport is determined based on the longitude and latitude, altitude and rotation angle; based on the zoom value of the camera viewport, the UV mapping is determined; based on the UV mapping, the texture image is fused with the three-dimensional scene, thereby realizing real-time interaction and fusion of the dynamic video surveillance stream and the three-dimensional model in the three-dimensional scene, thereby improving the visualization effect and user experience.

[0107] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0108] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0109] Figure 3 A schematic diagram of the structure of a video fusion device for a three-dimensional scene provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the device comprises:

[0110] An acquisition module 210 is used to acquire image data of each frame of the video stream and real-time parameter information of an acquisition device, the acquisition device is used to acquire the video stream, and the real-time parameter information includes longitude and latitude, height of the acquisition device from the ground, and rotation angle of the acquisition device;

[0111] A preprocessing module 220 is used to preprocess the image data to obtain a texture image;

[0112] A setting module 230, used to set the camera viewport corresponding to the video stream in the three-dimensional scene, and determine the zoom value of the camera viewport based on the latitude and longitude, altitude and rotation angle;

[0113] UV module 240, for determining UV mapping based on the zoom value of the camera viewport;

[0114] The fusion module 250 is used to fuse the texture image with the three-dimensional scene based on UV mapping.

[0115] In a possible embodiment, the preprocessing module 220 is further configured to:

[0116] Decode the image data format into RGB format;

[0117] Sort the decoded image data.

[0118] In a possible embodiment, the setting module 230 is further configured to:

[0119] Determine the location of the acquisition device in the three-dimensional scene based on latitude, longitude and altitude;

[0120] Set the camera viewport according to the position of the acquisition device in the 3D scene;

[0121] Adjusts the orientation of the camera viewport based on the rotation angle.

[0122] In a possible embodiment, the setting module 230 is further configured to:

[0123] The zoom value of the camera viewport is determined by the following zoom formula:

[0124]

[0125] Where FOV is the field of view of the camera, FPD is the far focal length, and AR is the resolution aspect ratio.

[0126] In a possible embodiment, the UV module 240 is further used for:

[0127] Get scene depth value and pixel depth;

[0128] Get the model depth information based on the scene depth value, pixel depth value and camera viewport scaling value;

[0129] Based on the model depth information and the relative direction vector, the UV mapping is obtained. The relative direction is the direction vector of each pixel point between the camera viewport position and the ground position.

[0130] In a possible embodiment, the fusion module 250 is further configured to:

[0131] Based on the model depth information and UV mapping, a depth map corresponding to the texture image is generated, and the depth map is fused with the three-dimensional scene.

[0132] Figure 4 A schematic diagram of the structure of a video fusion system for a three-dimensional scene provided by an embodiment of the present invention is shown in FIG. Figure 4As shown, the device includes a UE client 300 and a collection device 310, and the collection device is connected to the UE client; the collection device is used to collect the monitored video stream; the UE client is used to obtain the image data of each frame of the video stream and the real-time parameter information of the collection device, and the real-time parameter information includes longitude and latitude, the height of the collection device from the ground, and the rotation angle of the collection device; the image data is preprocessed to obtain a texture image; the camera viewport corresponding to the video stream in the three-dimensional scene is set, and the zoom value of the camera viewport is determined based on the longitude and latitude, height and rotation angle; UV mapping is determined based on the zoom value of the camera viewport; based on the UV mapping, the texture image is merged with the three-dimensional scene.

[0133] The above is an embodiment of the system of the present disclosure, which can be used to execute the embodiment of the method of the present disclosure. For details not disclosed in the embodiment of the system of the present disclosure, please refer to the embodiment of the method of the present disclosure.

[0134] Reference below Figure 5 , which shows a structural schematic diagram of an electronic device 400 suitable for implementing some embodiments of the present disclosure. Figure 5 The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0135] like Figure 5 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0136] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0137] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0138] It should be noted that the computer-readable medium mentioned above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0139] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0140] The computer-readable medium may be included in the device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the image data of each frame of the video stream and the real-time parameter information of the acquisition device; pre-processes the image data to obtain a texture image; sets the camera viewport corresponding to the video stream in the UE scene, and determines the zoom value of the camera viewport based on the longitude and latitude, altitude and rotation angle; determines the UV mapping based on the zoom value of the camera viewport; and fuses the texture image with the three-dimensional scene based on the UV mapping.

[0141] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, may be described as: a processor includes an acquisition module, a preprocessing module, a setting module, a UV module and a fusion module. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the acquisition module may also be described as "acquiring image data of each frame of the video stream and real-time parameter information of the acquisition device".

[0144] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate array (FPGA), application specific integrated circuit (ASIC), application specific standard product (ASSP), system on chip (SOC), complex programmable logic device (CPLD), etc. It should be noted that similar reference numerals and letters represent similar items in the following figures, so once an item is defined in one figure, it is not necessary to further define and explain it in the subsequent figures. In the description of the present invention, it should be noted that the terms "horizontal", "vertical", "overhanging" and the like do not mean that the components are required to be absolutely horizontal or overhanging, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video fusion method for a three-dimensional scene, characterized in that: include: Acquire image data of each frame of the video stream and real-time parameter information of a collection device, the collection device being used to collect the video stream, the real-time parameter information including longitude and latitude, the height of the collection device from the ground, and the rotation angle of the collection device; Preprocessing the image data to obtain a texture image; The camera viewport corresponding to the video stream in the three-dimensional scene displayed on the UE client is set, and the scaling value of the camera viewport is determined based on the longitude and latitude, the altitude and the rotation angle; wherein, the camera viewport corresponding to the video stream in the three-dimensional scene displayed on the UE client is set, including: determining the position of the acquisition device in the three-dimensional scene based on the longitude and latitude and the altitude; setting the camera viewport according to the position of the acquisition device in the three-dimensional scene; adjusting the orientation of the camera viewport based on the rotation angle; and determining the scaling value of the camera viewport by the following scaling formula: Where FOV is the field of view of the camera, FPD is the far focal length, and AR is the resolution aspect ratio. The value is the X-axis scaling in the 3D scene. The value is the scaling along the Y axis in the 3D scene and the FPD value is the scaling along the Z axis in the 3D scene; Based on the zoom value of the camera viewport, UV mapping is determined, wherein the scene depth value and the pixel depth value are obtained; based on the scene depth value, the pixel depth value and the zoom value of the camera viewport, the model depth information is obtained; based on the model depth information and the relative direction vector, UV mapping is obtained, wherein the relative direction is the direction vector of each pixel point between the position of the camera viewport and the ground position; the scene depth value is the actual distance from the acquisition device to a certain point in the scene; the pixel depth value is the scene depth information represented by each pixel in the rendered image; the model depth information is: Wherein, Va is the normal for converting the y vector value from the relative coordinate system to the world coordinate system multiplied by the height of the drone; Vb is the scene depth divided by the pixel depth, multiplied by the absolute world coordinate minus the world coordinate of the camera, and the result plus the position of the camera in the three-dimensional scene; Vc is the vector vertex iterator output vector for converting the x vector value from the relative coordinate system to the world coordinate system; Vd is the normal vertex iterator output vector for converting the z vector value from the relative coordinate system to the world coordinate system, where the y vector value is the vector value in the Y-axis direction in the scaling formula, the x vector value is the vector value in the X-axis direction in the scaling formula, and the z vector value is the vector value in the Z-axis direction in the scaling formula; the relative coordinate system is the relative coordinate of the object, and the world coordinate system is the coordinate of the object in the three-dimensional scene; Based on the UV mapping, the texture image is fused with the three-dimensional scene.

2. The method according to claim 1, characterized in that Preprocessing the image data includes: Decode the image data format into RGB format; Sort the decoded image data.

3. The method according to claim 1, characterized in that Based on the UV mapping, the texture image is merged with the three-dimensional scene, including: Based on the model depth information and UV mapping, a depth map corresponding to the texture image is generated, and the depth map is fused with the three-dimensional scene.

4. A video fusion device for three-dimensional scenes, characterized in that: include: An acquisition module, used to acquire image data of each frame of the video stream and real-time parameter information of an acquisition device, the acquisition device being used to acquire the video stream, the real-time parameter information including longitude and latitude, the height of the acquisition device from the ground, and the rotation angle of the acquisition device; A preprocessing module, used for preprocessing the image data to obtain a texture image; A setting module is used to set the camera viewport corresponding to the video stream in the three-dimensional scene displayed by the UE client, and determine the scaling value of the camera viewport based on the longitude and latitude, the altitude and the rotation angle; wherein, setting the camera viewport corresponding to the video stream in the three-dimensional scene displayed by the UE client includes: determining the position of the acquisition device in the three-dimensional scene based on the longitude and latitude and the altitude; setting the camera viewport according to the position of the acquisition device in the three-dimensional scene; adjusting the orientation of the camera viewport based on the rotation angle; and determining the scaling value of the camera viewport by the following scaling formula: Where FOV is the field of view of the camera, FPD is the far focal length, and AR is the resolution aspect ratio. The value is the X-axis scaling in the 3D scene. The value is the scaling along the Y axis in the 3D scene and the FPD value is the scaling along the Z axis in the 3D scene; A UV module is used to determine a UV mapping based on the zoom value of the camera viewport; wherein, by obtaining a scene depth value and a pixel depth value; obtaining model depth information based on the scene depth value, the pixel depth value and the zoom value of the camera viewport; obtaining a UV mapping based on the model depth information and a relative direction vector, wherein the relative direction is a direction vector of each pixel point between the position of the camera viewport and the ground position; the scene depth value is an actual distance from a capture device to a certain point in the scene; the pixel depth value is the scene depth information represented by each pixel in the rendered image; the model depth information is: Wherein, Va is the normal for converting the y vector value from the relative coordinate system to the world coordinate system multiplied by the height of the drone; Vb is the scene depth divided by the pixel depth, multiplied by the absolute world coordinate minus the world coordinate of the camera, and the result plus the position of the camera in the three-dimensional scene; Vc is the vector vertex iterator output vector for converting the x vector value from the relative coordinate system to the world coordinate system; Vd is the normal vertex iterator output vector for converting the z vector value from the relative coordinate system to the world coordinate system, where the y vector value is the vector value in the Y-axis direction in the scaling formula, the x vector value is the vector value in the X-axis direction in the scaling formula, and the z vector value is the vector value in the Z-axis direction in the scaling formula; the relative coordinate system is the relative coordinate of the object, and the world coordinate system is the coordinate of the object in the three-dimensional scene; A fusion module is used to fuse the texture image with the three-dimensional scene based on the UV mapping.

5. A video fusion system for three-dimensional scenes, characterized in that: include: A collection device and a UE client, wherein the collection device is connected to the UE client; The acquisition device is used to acquire the monitored video stream; The UE client is used to obtain image data of each frame of the video stream and real-time parameter information of the acquisition device, wherein the real-time parameter information includes longitude and latitude, the height of the acquisition device from the ground, and the rotation angle of the acquisition device; Preprocess the image data to obtain a texture image; set a camera viewport corresponding to the video stream in the three-dimensional scene displayed by the UE client, and determine the scaling value of the camera viewport based on the longitude and latitude, the altitude and the rotation angle; wherein setting the camera viewport corresponding to the video stream in the three-dimensional scene displayed by the UE client includes: determining the position of the acquisition device in the three-dimensional scene based on the longitude and latitude and the altitude; setting the camera viewport according to the position of the acquisition device in the three-dimensional scene; adjusting the orientation of the camera viewport based on the rotation angle; and determining the scaling value of the camera viewport by the following scaling formula: Where FOV is the field of view of the camera, FPD is the far focal length, and AR is the resolution aspect ratio. The value is the X-axis scaling in the 3D scene. The value is the scaling of the Y-axis direction in the three-dimensional scene and the FPD value is the scaling of the Z-axis direction in the three-dimensional scene; based on the scaling value of the camera viewport, the UV mapping is determined; wherein, by obtaining the scene depth value and the pixel depth value; based on the scene depth value, the pixel depth value and the scaling value of the camera viewport, the model depth information is obtained; based on the model depth information and the relative direction vector, the UV mapping is obtained, wherein the relative direction is the direction vector of each pixel point between the position of the camera viewport and the ground position; the scene depth value is the actual distance from the acquisition device to a certain point in the scene; the pixel depth value is the scene depth information represented by each pixel in the rendered image; the model depth information is: Wherein, Va is the normal for converting the y vector value from the relative coordinate system to the world coordinate system multiplied by the height of the drone; Vb is the scene depth divided by the pixel depth, multiplied by the absolute world coordinate minus the world coordinate of the camera, and the result plus the position of the camera in the three-dimensional scene; Vc is the vector vertex iterator output vector for converting the x vector value from the relative coordinate system to the world coordinate system; Vd is the normal vertex iterator output vector for converting the z vector value from the relative coordinate system to the world coordinate system, wherein the y vector value is the vector value in the Y-axis direction in the scaling formula, the x vector value is the vector value in the X-axis direction in the scaling formula, and the z vector value is the vector value in the Z-axis direction in the scaling formula; the relative coordinate system is the relative coordinate of the object, and the world coordinate system is the coordinate of the object in the three-dimensional scene; based on the UV mapping, the texture image is fused with the three-dimensional scene.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.