A dynamic texture fusion method and device with adaptive observation viewpoint

Abstract

The present invention relates to a method and device for dynamic texture fusion of adaptive observation viewpoints. The method and device automatically select two optimal surveillance videos according to the posture of the observation camera controlled by the user, and fuse the two optimal surveillance videos as textures, thereby realizing fully automatic calculation of video fusion textures, without the need for manual operation or human intervention, and improving efficiency. When the posture of the observation camera and the surveillance camera differ greatly, severe distortion will occur. The present invention can match the video of the optimal surveillance camera as texture according to the posture of the observation camera and fuse them, that is, obtain the optimal real-time fusion result according to the posture of the observation camera, and reduce distortion.

Description

Technical Field

[0001] The invention relates to a dynamic texture fusion method and device for adaptive observation viewpoint. Background Art

[0002] Pictures of surveillance scenes taken with the help of drones or digital cameras can be used to reconstruct three-dimensional scenes. Before reconstruction, there are certain standards for the shooting of pictures. First, the pictures taken should completely cover the scene to be reconstructed; second, each picture should overlap with at least one other picture by more than 30%; finally, each picture should not contain close-ups. Once all the pictures are taken, three-dimensional reconstruction is performed using the SfM and MVS algorithms to obtain the camera pose of the pictures, the sparse point cloud of the scene, and the dense point cloud. With the dense point cloud, the three-dimensional model of the surveillance scene can be obtained through Poisson reconstruction and mesh simplification algorithms.

[0003] After completing the 3D reconstruction of the surveillance scene with the image, the surveillance video image is mapped as a texture onto the model of the 3D scene. Therefore, it is necessary to obtain the intrinsic parameters of the surveillance camera and calculate the camera pose. Since the surveillance camera is fixed most of the time, it is only necessary to calculate the camera pose through a certain frame of the surveillance camera. When the surveillance camera pose changes, take a frame and recalculate. First, extract the SIFT feature points of the video frame to be calibrated, and then match them with the pictures taken for the 3D reconstruction of the surveillance scene. The pose of the video camera can be calculated with the help of the SfM method. For a single surveillance video, after obtaining the camera pose and intrinsic parameters, the texture coordinates of each vertex in the 3D model can be calculated. Finally, the texture can be updated according to the video image to achieve video texture mapping.

[0004] In the prior art, there is manual adjustment to splice videos to achieve multi-video fusion, and the effect is as follows Figure 1 shown. Figure 1 This is a video fusion effect diagram obtained by manually stitching video textures and 3D scenes. In the prior art, there is also a method of manually selecting a mixed area from the overlapping parts of multiple videos for mixing. The schematic diagram and effect diagram are as follows Figure 2 As shown. Figure 2 The left side is a schematic diagram of the manual selection and mixing part, and the right side is a diagram of the effect before and after manual mixing. However, the existing technology has the following disadvantages:

[0005] 1. Manual operation is required;

[0006] 2. Video texture fusion is a static method, and it is impossible to select the most reasonable video as texture according to the observation camera posture. When the position of the monitoring video camera and the observation camera is very different, serious distortion will occur. Summary of the invention

[0007] The embodiment of the present invention provides a dynamic texture fusion method and device with adaptive observation viewpoint, so as to at least solve the technical problems that the prior art requires manual intervention and cannot adapt to changes in observation angles, resulting in distortion.

[0008] According to an embodiment of the present invention, a dynamic texture fusion method with adaptive observation viewpoint is provided, comprising the following steps:

[0009] Automatically select two optimal surveillance videos based on the position of the observation camera controlled by the user;

[0010] The two best surveillance videos are fused as textures.

[0011] Furthermore, automatically selecting two optimal surveillance videos according to the position of the observation camera controlled by the user includes:

[0012] For each triangle patch T in the virtual 3D monitoring scene i , first calculate from T i Center to surveillance camera V j The unit direction vector v ij When the observation camera O moves and rotates in the virtual scene, the real-time calculation from T i Unit direction vector u from the center to the viewing camera O io , then for the triangle patch T i , the surveillance camera V corresponding to the optimal texture j* Calculated by the following formula:

[0013]

[0014] Next, find another surveillance camera that meets the conditions and traverse all the cameras except V. j* All surveillance video cameras except V k , connect V j* and V k Get line segment L jk , connect V j* and observe the camera O to obtain the line segment L oj , the line segment L oj Projection onto line segment L jk If the projection point is on the line segment L jk In addition, V k Not meeting the requirements;

[0015] Among the remaining surveillance cameras, the second optimal surveillance camera V that meets the conditions is obtained by using the above formula. k* .

[0016] Furthermore, the second optimal surveillance camera V that meets the conditions is obtained by using the above formula among the remaining surveillance cameras. k* When Vk* If V j* It is the only surveillance camera.

[0017] Furthermore, fusing the two best surveillance videos as textures includes:

[0018] The textures of the two surveillance cameras are blended according to the following weights:

[0019]

[0020] Among them, α and β are the monitoring camera V j* and V k* The corresponding texture is mixed with weight, θ oj* is the direction vector u io and v ij* The angle, θ ok* is the direction vector u io and v ik* Angle.

[0021] Furthermore, after the optimal cameras and weights of all triangles are calculated, the entire scene is rendered.

[0022] Furthermore, the method further comprises:

[0023] The fused video texture is displayed in a three-dimensional virtual scene.

[0024] According to another embodiment of the present invention, a dynamic texture fusion device with adaptive observation viewpoint is provided, comprising:

[0025] An optimal monitoring video selection unit, used to automatically select two optimal monitoring videos according to the posture of the observation camera controlled by the user;

[0026] The virtual three-dimensional monitoring scene fusion unit is used to fuse two optimal monitoring videos as textures.

[0027] Furthermore, the device also includes:

[0028] The scene display unit is used to display the fused video texture in a three-dimensional virtual scene.

[0029] A storage medium stores a program file capable of implementing any one of the above-mentioned adaptive observation viewpoint dynamic texture fusion methods.

[0030] A processor is used to run a program, wherein when the program is run, any one of the above-mentioned dynamic texture fusion methods of adaptive observation viewpoint is executed.

[0031] The method and device for dynamic texture fusion of adaptive observation viewpoints in the embodiment of the present invention automatically selects two optimal surveillance videos according to the posture of the observation camera controlled by the user, and fuses the two optimal surveillance videos as textures, thereby realizing fully automatic calculation of video fusion textures, without the need for manual operation or human intervention, and improving efficiency. Moreover, when the posture of the observation camera and the surveillance camera differ greatly, severe distortion will occur. The present invention can match the video of the optimal surveillance camera as texture and fuse it according to the posture of the observation camera, that is, obtain the optimal real-time fusion result according to the posture of the observation camera, and reduce distortion.

[0032] The beneficial effects of the present invention are at least:

[0033] 1. The traditional static video texture fusion method requires manual operation of the splicing or fusion process. The present invention realizes the fully automatic calculation of video fusion texture, which does not require manual operation or human intervention, thereby improving efficiency.

[0034] 2. After obtaining the fusion result, the traditional static video texture fusion method will not change the fusion result no matter how the observation camera changes. When the posture of the observation camera and the monitoring camera is greatly different, serious distortion will occur. The present invention can match the optimal monitoring camera video as texture according to the posture of the observation camera and fuse it, that is, obtain the optimal real-time fusion result according to the posture of the observation camera, reducing the distortion phenomenon. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0036] Figure 1 This is the effect of manually stitching video textures to merge the video and the 3D scene.

[0037] Figure 2 A schematic diagram and effect diagram of manually selecting a mixing area from overlapping parts of multiple videos for mixing;

[0038] Figure 3 This is the entire scene graph rendered by the dynamic texture fusion method of the adaptive observation viewpoint of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] Example 1

[0042] According to an embodiment of the present invention, a dynamic texture fusion method with adaptive observation viewpoint is provided, comprising the following steps:

[0043] Automatically select two optimal surveillance videos based on the position of the observation camera controlled by the user;

[0044] The two best surveillance videos are fused as textures.

[0045] The dynamic texture fusion method of adaptive observation viewpoint in the embodiment of the present invention automatically selects two optimal monitoring videos according to the posture of the observation camera controlled by the user, and fuses the two optimal monitoring videos as textures, thereby realizing fully automatic calculation of video fusion texture, without manual operation or human intervention, and improving efficiency. Moreover, when the posture of the observation camera and the monitoring camera are greatly different, serious distortion will occur. The present invention can match the video of the optimal monitoring camera as texture according to the posture of the observation camera and fuse them, that is, obtain the optimal real-time fusion result according to the posture of the observation camera, reducing the distortion phenomenon.

[0046] Among them, automatically selecting two optimal surveillance videos according to the posture of the observation camera controlled by the user includes:

[0047] For each triangle patch T in the virtual 3D monitoring scene i , first calculate from T i Center to surveillance camera Vj The unit direction vector v ij When the observation camera O moves and rotates in the virtual scene, the real-time calculation from T i Unit direction vector u from the center to the viewing camera O io , then for the triangle patch T i , the surveillance camera V corresponding to the optimal texture j* Calculated by the following formula:

[0048]

[0049] Next, find another surveillance camera that meets the conditions and traverse all the cameras except V. j* All surveillance video cameras except V k , connect V j* and V k Get line segment L jk , connect V j* and observe the camera O to obtain the line segment L oj , the line segment L oj Projection onto line segment L jk If the projection point is on the line segment L jk In addition, V k Not meeting the requirements;

[0050] Among the remaining surveillance cameras, the second optimal surveillance camera V that meets the conditions is obtained by using the above formula. k* .

[0051] Among them, the second optimal surveillance camera V that meets the conditions is obtained by using the above formula among the remaining surveillance cameras k* When V k* If V j* It is the only surveillance camera.

[0052] Among them, fusing two optimal surveillance videos as textures includes:

[0053] The textures of the two surveillance cameras are blended according to the following weights:

[0054]

[0055] Among them, α and β are the monitoring camera V j* and V k* The corresponding texture is mixed with weight, θ oj* is the direction vector u io and v ij* The angle, θ ok* is the direction vector u io and v ik* Angle.

[0056] Among them, after the optimal cameras and weights of all triangle faces are calculated, the entire scene is rendered.

[0057] The method further includes:

[0058] The fused video texture is displayed in a three-dimensional virtual scene.

[0059] The following is a detailed description of the adaptive viewpoint dynamic texture fusion method of the present invention with specific embodiments:

[0060] The present invention aims to dynamically and real-time calculate the multi-video fusion texture according to the position and posture of the observation camera in the three-dimensional virtual scene, so as to solve the problems of picture fragmentation and weak sense of space in the traditional multi-camera monitoring system, and to solve the problems of the traditional method requiring manual intervention and being unable to adapt to the changes in observation angle and causing distortion.

[0061] The present invention can automatically select two optimal monitoring videos as textures for fusion according to the posture of the observation camera controlled by the user, and display them in a three-dimensional virtual scene.

[0062] After the three-dimensional model of the monitoring scene is reconstructed and the posture of the monitoring camera is calibrated, video texture mapping can be performed and the texture content can be updated in real time in each frame. Because the monitoring video may capture part of the same scene, multiple textures may overlap. The present invention proposes a dynamic texture fusion method for adaptive observation viewpoints. Since the posture of the monitoring video camera is fixed, and the observation camera controlled by the user can move and rotate freely in the virtual scene, the present invention aims to achieve the following two goals: ① When the observation camera moves to the position of a monitoring video camera, the scene should show the entire content of the monitoring video camera; ② When the monitoring observation camera moves to the middle of two monitoring video cameras, the scene should show the result of the weighted fusion of the two videos to reduce distortion and deformation.

[0063] In order to achieve the first goal, for each triangle patch T in the virtual 3D monitoring scene i The present invention first calculates from T i Center to surveillance camera V j The unit direction vector v ij When the observation camera O moves and rotates in the virtual scene, the real-time calculation from T i Unit direction vector u from the center to the viewing camera O io , then for the triangle patch T i , the surveillance camera V corresponding to the optimal texture j* It can be calculated by the following formula:

[0064]

[0065] In order to achieve the second goal, the present invention needs to find two optimal surveillance cameras, so in addition to V j* In addition, we need to find another surveillance camera that meets the conditions. First, we traverse all the cameras except V j* All surveillance video cameras except V k , connect V j* and V k Get line segment L jk , connect V j* and observe the camera O to obtain the line segment L oj , the line segment L oj Projection onto line segment L jk If the projection point is on the line segment L jk In addition, V k It does not meet the requirements. The above formula can be used to obtain the second optimal surveillance camera V that meets the conditions in the remaining surveillance cameras. k* If V k* If V j* As the only surveillance camera, the textures of the two surveillance cameras are mixed according to the following weights:

[0066]

[0067] Where α and β are the surveillance camera V j* and V k* The corresponding texture is mixed with weight, θ oj* is the direction vector u io and v ij* The angle, θ ok* is the direction vector u io and v ik* When the optimal cameras and weights of all triangles are calculated, the entire scene can be rendered. The entire scene after rendering is as follows Figure 3 As shown. Figure 3 The two pictures on the left and right at the bottom are the photos that need to be fused. Figure 3 The upper part located between the two lower pictures is the fusion result picture.

[0068] Example 2

[0069] According to another embodiment of the present invention, a dynamic texture fusion device with adaptive observation viewpoint is provided, comprising:

[0070] An optimal monitoring video selection unit, used to automatically select two optimal monitoring videos according to the posture of the observation camera controlled by the user;

[0071] The virtual three-dimensional monitoring scene fusion unit is used to fuse two optimal monitoring videos as textures.

[0072] The device further comprises:

[0073] The scene display unit is used to display the fused video texture in a three-dimensional virtual scene.

[0074] The dynamic texture fusion device with adaptive observation viewpoint in the embodiment of the present invention automatically selects two optimal monitoring videos according to the posture of the observation camera controlled by the user, and fuses the two optimal monitoring videos as textures, thereby realizing fully automatic calculation of video fusion texture, without manual operation or human intervention, and improving efficiency. Moreover, when the posture of the observation camera and the monitoring camera are greatly different, serious distortion will occur. The present invention can match the video of the optimal monitoring camera as texture according to the posture of the observation camera and fuse them, that is, obtain the optimal real-time fusion result according to the posture of the observation camera, reducing the distortion phenomenon.

[0075] The following is a detailed description of the dynamic texture fusion device for adaptive observation viewpoints of the present invention with reference to a specific embodiment:

[0076] The present invention aims to dynamically and real-time calculate the multi-video fusion texture according to the position and posture of the observation camera in the three-dimensional virtual scene, so as to solve the problems of picture fragmentation and weak sense of space in the traditional multi-camera monitoring system, and to solve the problems of the traditional method requiring manual intervention and being unable to adapt to the changes in observation angle and causing distortion.

[0077] The present invention can automatically select two optimal monitoring videos as textures for fusion according to the posture of the observation camera controlled by the user, and display them in a three-dimensional virtual scene by a scene display unit.

[0078] After the three-dimensional model of the monitoring scene is reconstructed and the posture of the monitoring camera is calibrated, video texture mapping can be performed and the texture content can be updated in real time in each frame. Because the monitoring video may capture part of the same scene, multiple textures may overlap. The present invention proposes a dynamic texture fusion device with adaptive observation viewpoints. Since the posture of the monitoring video camera is fixed, and the observation camera controlled by the user can move and rotate freely in the virtual scene, the present invention aims to achieve the following two goals: ① When the observation camera moves to the position of a monitoring video camera, the scene should show the entire content of the monitoring video camera; ② When the monitoring observation camera moves to the middle of two monitoring video cameras, the scene should show the result of the weighted fusion of the two videos to reduce distortion and deformation.

[0079] The optimal surveillance video selection unit, for each triangular face T in the virtual 3D surveillance scene i The present invention first calculates from T i Center to surveillance camera V jThe unit direction vector v ij When the observation camera O moves and rotates in the virtual scene, the real-time calculation from T i Unit direction vector u from the center to the viewing camera O io , then for the triangle patch T i , the surveillance camera V corresponding to the optimal texture j* It can be calculated by the following formula:

[0080]

[0081] The present invention needs to find two optimal surveillance cameras, so in addition to V j* In addition, we need to find another surveillance camera that meets the conditions. First, we traverse all the cameras except V j* All surveillance video cameras except V k , connect V j* and V k Get line segment L jk , connect V j* and observe the camera O to obtain the line segment L oj , the line segment L oj Projection onto line segment L jk If the projection point is on the line segment L jk In addition, V k It does not meet the requirements. The above formula can be used to obtain the second optimal surveillance camera V that meets the conditions in the remaining surveillance cameras. k* If V k* If V j* It is the only surveillance camera.

[0082] Virtual 3D monitoring scene fusion unit: For the two monitoring cameras obtained by the optimal monitoring video selection unit, the two monitoring images are mixed according to the following weights:

[0083]

[0084] Where α and β are the surveillance camera V j* and V k* The corresponding texture is mixed with weight, θ oj* is the direction vector u io and v ij* The angle, θ ok* is the direction vector u io and v ik* When the optimal cameras and weights of all triangles are calculated, the entire scene can be rendered. The entire scene after rendering is as follows Figure 3 As shown. Figure 3 The two pictures on the left and right at the bottom are the photos that need to be fused. Figure 3The upper part located between the two lower pictures is the fusion result picture.

[0085] Example 3

[0086] A storage medium stores a program file capable of implementing any one of the above-mentioned adaptive observation viewpoint dynamic texture fusion methods.

[0087] Example 4

[0088] A processor is used to run a program, wherein when the program is run, any one of the above-mentioned dynamic texture fusion methods of adaptive observation viewpoint is executed.

[0089] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0090] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the system embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0093] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0095] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A dynamic texture fusion method with adaptive observation viewpoint, characterized in that: The following steps are involved: Automatically select two optimal surveillance videos based on the position of the observation camera controlled by the user; Fusion of two optimal surveillance videos as textures; The automatic selection of two optimal surveillance videos according to the posture of the observation camera controlled by the user includes: For each triangle patch T in the virtual 3D monitoring scene i , first calculate from T i Center to surveillance camera V j The unit direction vector v ij When the observation camera O moves and rotates in the virtual scene, the real-time calculation from T i Unit direction vector u from the center to the viewing camera O io , then for the triangle patch T i , the surveillance camera V corresponding to the optimal texture j* Calculated by the following formula: Next, find another surveillance camera that meets the conditions and traverse all the cameras except V. j* All surveillance video cameras except V k , connect V j* and V k Get line segment L jk , connect V j* and observe the camera O to obtain the line segment L oj , the line segment L oj Projection onto line segment L jk If the projection point is on the line segment L jk In addition, at this time V k Not meeting the requirements; Among the remaining surveillance cameras, the second optimal surveillance camera V that meets the conditions is obtained by using the above formula. k* .

2. The method for dynamic texture fusion based on adaptive observation viewpoint according to claim 1, characterized in that: Among the remaining surveillance cameras, the second optimal surveillance camera V that meets the conditions is obtained by using the above formula. k* When V k* If V j* It is the only surveillance camera.

3. The method for dynamic texture fusion based on adaptive observation viewpoint according to claim 1, characterized in that: The fusing of the two optimal surveillance videos as textures includes: The textures of the two surveillance cameras are blended according to the following weights: Where α and β are the surveillance camera V j* and V k* The corresponding texture is mixed with weight, θ oj* is the direction vector u io and v ij* The angle, θ ok* is the direction vector u io and v ik* Angle.

4. The method for dynamic texture fusion based on adaptive observation viewpoint according to claim 3, characterized in that: Once the optimal cameras and weights for all triangles are calculated, the entire scene is rendered.

5. The method for dynamic texture fusion based on adaptive observation viewpoint according to claim 1, characterized in that: The method further comprises: The fused video texture is displayed in a three-dimensional virtual scene.

6. A dynamic texture fusion device for adaptive observation viewpoints using the dynamic texture fusion method for adaptive observation viewpoints according to claim 1, characterized in that: include: An optimal monitoring video selection unit, used to automatically select two optimal monitoring videos according to the posture of the observation camera controlled by the user; The virtual three-dimensional monitoring scene fusion unit is used to fuse two optimal monitoring videos as textures.

7. The dynamic texture fusion device with adaptive observation viewpoint according to claim 6, characterized in that: The device also includes: The scene display unit is used to display the fused video texture in a three-dimensional virtual scene.

8. A storage medium, characterized in that: The storage medium stores a program file capable of implementing the dynamic texture fusion method for adaptive observation viewpoints as described in any one of claims 1 to 5.

9. A processor, characterized in that: The processor is used to run a program, wherein the program executes the dynamic texture fusion method of adaptive observation viewpoint described in any one of claims 1 to 5 when running.

Images