A Real-Time Image Analysis Method Based on Multi-Video Stream Fusion

Abstract

This invention provides a real-time image analysis method for multi-video stream fusion. The method involves real-time tracking of video image parameters; then, calculating the view frustum corresponding to the video image in 3D space based on these parameters; calculating the visible video images from the main video viewpoint; scheduling these visible video images; then, calculating the association between each visible video image and virtual objects in the 3D scene; and fusing the video images with the virtual objects based on these associations; finally, visualizing the fusion result in a virtual environment to obtain the fused video. This method offers advantages such as high real-time performance and fast fusion speed.

Description

Technical Field

[0001] This invention relates to the field of multi-video stream fusion technology, and in particular to a real-time image analysis method for multi-video stream fusion. Background Technology

[0002] With the advancement of science and technology, people have increasingly higher requirements for video images, especially in fields such as medicine, remote sensing, surveillance, and the military, where the richer the information contained in the video image, the better. In this context, traditional single sensors cannot meet these requirements. For example, visible light imaging sensors can obtain images with rich color and detail information during the day, but they cannot produce ideal images at night. Infrared imaging sensors have good imaging effects at night, but this type of sensor has low resolution and can only produce grayscale images, lacking rich color information. It is clear that a single imaging sensor cannot provide scene images with rich information. Therefore, we need image fusion technology to combine multiple images with complementary information into a single image, facilitating subsequent processing.

[0003] Meanwhile, with the development of big data and artificial intelligence algorithms, image recognition is being applied in more and more fields. Image recognition is a technology that uses computers or embedded devices to fuse, analyze, and understand images acquired by sensors in order to recognize targets and objects of various patterns. Image fusion is a technology that combines multiple images obtained from multiple imaging sensors into a single image. Because video image fusion involves a large amount of data, it places very high demands on hardware. Furthermore, current image fusion systems need to process several video streams, resulting in a very large data volume. Moreover, before fusing multiple images, the input video images need to be registered, making the entire video image fusion algorithm quite complex.

[0004] The common method currently used is to store multiple high-definition real-time images separately in a buffer or memory as images. After a single image is filled, the data in the buffer is read and fused. When there are many data sources, this storage and transmission mechanism severely reduces the fusion speed of the embedded system. In particular, due to the parallel fusion characteristics of FPGAs, this storage mechanism causes the parallel processor to be idle for a long time, waiting for all the data to be stored in a single image before fusion is performed. This severely limits the fusion speed and real-time performance of multiple video images. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a real-time image analysis method for multi-video stream fusion, which solves the problems of poor speed and real-time performance in multi-video image fusion in existing technologies.

[0006] According to an embodiment of the present invention, a real-time image analysis method for multi-video stream fusion includes the following steps:

[0007] S1. Acquire multiple original video images, select the video with the most covered image parameters as the main video, calculate the parameter information of the video images, and use the parameter information to schedule the fusion of the video images and the main video.

[0008] S2. Calculate the view frustum corresponding to the video in three-dimensional space based on the parameter information of the video image. The view frustum is an approximate or precise expression of the spatial range of the real environment when the video was captured in the virtual environment. Calculate the set of videos currently visible to the main video viewpoint based on the viewpoint position and direction of the main video image in the virtual environment, and schedule the video images visible to the main video viewpoint.

[0009] S3. For video images in the visible set, calculate the association between the video images and virtual objects in the 3D scene based on the parameter information, and use the association to fuse the video images with the main video images.

[0010] S4. Visualize the fusion result of all video images and the main video image in a virtual environment.

[0011] Furthermore, for each video in step S2, the view frustum corresponding to the video image in three-dimensional space is calculated based on the parameter information at the time of acquisition. The view frustum defines the observation range of the video image in the virtual environment, which corresponds to the real geographic space when the video image was captured. Then, based on the position and direction of the current viewpoint of the video image, the intersection of each video view frustum with the main video observation area is calculated, and the visibility of the video image is determined according to the following rules:

[0012] If the position of the video image is more than a certain distance from the viewpoint of the main video, the camera is considered invisible; if the position of the video image is within a certain distance from the viewpoint of the main video, and the video image's view frustum is not within the viewpoint area of ​​the main video, or the angle between the optical axis of the video image and the viewpoint direction exceeds a certain angle, the video image is considered invisible.

[0013] If the video image is within a certain distance from the main video viewpoint and meets the following conditions: the video image's view frustum intersects with or is within the main video region, and the angle between the video image's optical axis direction and the main video viewpoint direction is no greater than a certain angle, then the video image is considered visible.

[0014] The above process for determining the visibility of video images is accelerated by grouping video images according to their location distribution and searching for nearby groups based on the main video viewpoint during calculation, thereby reducing the amount of computation. After calculating the visibility of the video images, the list of visible video images in use, U, is updated.

[0015] Furthermore, the visible video image U is updated using the following steps:

[0016] S2.1 Save the video image visibility result obtained in the last calculation, clear the exit list Q and the list to be added J, and set the video image flag bit in the visibility list V to -1; where a video image flag bit of 1 indicates that the video image is visible, a flag bit of 0 indicates that the video image is not visible, and -1 indicates that it is unknown; proceed to step S2.2.

[0017] S2.2 For each video image in list V, if its position is more than a certain distance away from the current viewpoint, the video image is considered invisible and its flag is set to 0; otherwise, proceed to step S2.3.

[0018] S2.3 If the distance between the video image and the main video viewpoint is within a certain range: calculate the bounding box of the video image's view frustum, calculate the observation area of ​​the main video image at the current viewpoint, if the bounding box does not intersect with the observation area, then the video image is considered invisible, and the flag position of the video image is set to 0; calculate the angle between the optical axis direction of the video image and the direction of the main video viewpoint, if the angle is greater than a certain angle, set the flag position of the video image to 0; otherwise, proceed to step S2.4;

[0019] S2.4 If the bounding box of the video image's view frustum intersects with or is within the main video viewing area, and the angle between the video image's optical axis direction and the main video viewpoint direction is not greater than a certain angle, then the video image is considered visible, and the flag position of the video image is set to 1; compare the visibility list obtained in this calculation with the visibility list obtained in the previous calculation. If the video image is visible in this calculation but not in the previous calculation, then add it to the list to be added J; if the video image is not visible in this calculation but was visible in the previous calculation, then add it to the exit list Q; proceed to step S2.5;

[0020] S2.5. Extract the video images to be added to list J and send them to candidate list C; select an appropriate number of video images from C and send them to the list of visible video images in use U as required; remove the same video images from candidate list C and list in use U as those in exit list Q and release the related parameters; perform synchronization processing on each video image in list U using timestamp parameters as required by the application.

[0021] Furthermore, step S1 uses one or more video images, the parameters of which are continuous, including position, orientation, focal length, and timestamp; the parameter information is encoded and transmitted or saved in real time along with the video images.

[0022] Furthermore, in step S3, for each visible video image in the scene, the association relationship between its content and the virtual objects in the 3D scene is calculated based on the parameter information of the video image. This calculation only needs to be performed once if the video image parameters do not change. Then, based on the association relationship, multiple video images are fused with the virtual objects in the 3D scene through video projection. The objects include points, lines, surfaces, and combinations of the three. The content of multiple video images can be fused with the same object. The intersection of the video images is processed using feature matching, alpha channels, and weighted averaging methods. During fusion, the occlusion relationship between objects needs to be considered. The occlusion relationship between objects is quickly determined using the depth map method. The occluded parts use the original texture or are specified according to specific requirements, while the unoccluded parts are fused with the video images.

[0023] Furthermore, step S4 visualizes the fusion result of the video image and the 3D scene in a virtual environment, and additionally displays the original 3D scene, the original video image, and the fusion effect under the main video viewpoint as needed.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The fusion calculation can be dynamically updated. After the parameters of the acquired video images change, the existing fusion results can be updated simply by recalculating the parameters once.

[0026] 2. It has good scalability and can adapt to different ranges of scenarios. It can merge videos with various objects in the scene and takes into account the occlusion relationship between objects.

[0027] 3. By integrating multiple dispersed videos into a unified virtual environment, the spatiotemporal correlation between videos is enhanced, which helps users understand the static and dynamic content in multiple videos. Detailed Implementation

[0028] The technical solutions of the present invention will be further described below with reference to the embodiments.

[0029] S1. Acquire multiple original video images, select the video with the most covered image parameters as the main video, calculate the parameter information of the video images, and use the parameter information to schedule the fusion of the video images and the main video.

[0030] S2. Calculate the view frustum corresponding to the video in three-dimensional space based on the parameter information of the video image. The view frustum is an approximate or precise expression of the spatial range of the real environment when the video was captured in the virtual environment. Calculate the set of videos currently visible to the main video viewpoint based on the viewpoint position and direction of the main video image in the virtual environment, and schedule the video images visible to the main video viewpoint.

[0031] Furthermore, for each video in step S2, the view frustum corresponding to the video image in three-dimensional space is calculated based on the parameter information at the time of acquisition. The view frustum defines the observation range of the video image in the virtual environment, which corresponds to the real geographic space when the video image was captured. Then, based on the position and direction of the current viewpoint of the video image, the intersection of each video view frustum with the main video observation area is calculated, and the visibility of the video image is determined according to the following rules:

[0032] If the position of the video image is more than a certain distance from the viewpoint of the main video, the camera is considered invisible; if the position of the video image is within a certain distance from the viewpoint of the main video, and the video image's view frustum is not within the viewpoint area of ​​the main video, or the angle between the optical axis of the video image and the viewpoint direction exceeds a certain angle, the video image is considered invisible.

[0033] If the video image is within a certain distance from the main video viewpoint and meets the following conditions: the video image's view frustum intersects with or is within the main video region, and the angle between the video image's optical axis direction and the main video viewpoint direction is no greater than a certain angle, then the video image is considered visible.

[0034] The above process for determining the visibility of video images is accelerated by grouping video images according to their location distribution and searching for nearby groups based on the main video viewpoint during calculation, thereby reducing the amount of computation. After calculating the visibility of the video images, the list of visible video images in use, U, is updated.

[0035] Furthermore, the visible video image U is updated using the following steps:

[0036] S2.1 Save the video image visibility result obtained in the last calculation, clear the exit list Q and the list to be added J, and set the video image flag bit in the visibility list V to -1; where a video image flag bit of 1 indicates that the video image is visible, a flag bit of 0 indicates that the video image is not visible, and -1 indicates that it is unknown; proceed to step S2.2.

[0037] S2.2 For each video image in list V, if its position is more than a certain distance away from the current viewpoint, the video image is considered invisible and its flag is set to 0; otherwise, proceed to step S2.3.

[0038] S2.3 If the distance between the video image and the main video viewpoint is within a certain range: calculate the bounding box of the video image's view frustum, calculate the observation area of ​​the main video image at the current viewpoint, if the bounding box does not intersect with the observation area, then the video image is considered invisible, and the flag position of the video image is set to 0; calculate the angle between the optical axis direction of the video image and the direction of the main video viewpoint, if the angle is greater than a certain angle, set the flag position of the video image to 0; otherwise, proceed to step S2.4;

[0039] S2.4 If the bounding box of the video image's view frustum intersects with or is within the main video viewing area, and the angle between the video image's optical axis direction and the main video viewpoint direction is not greater than a certain angle, then the video image is considered visible, and the flag position of the video image is set to 1; compare the visibility list obtained in this calculation with the visibility list obtained in the previous calculation. If the video image is visible in this calculation but not in the previous calculation, then add it to the list to be added J; if the video image is not visible in this calculation but was visible in the previous calculation, then add it to the exit list Q; proceed to step S2.5;

[0040] S2.5. Extract the video images to be added to list J and send them to candidate list C; select an appropriate number of video images from C and send them to the list of visible video images in use U as required; remove the same video images from candidate list C and list in use U as those in exit list Q and release the related parameters; perform synchronization processing on each video image in list U using timestamp parameters as required by the application.

[0041] S3. For video images in the visible set, calculate the association between the video image and virtual objects in the 3D scene based on parameter information, and use this association to fuse the video image with the main video image. Specifically, for each camera in list U, analyze the association R between the content of the video image and virtual objects such as points, lines, surfaces, and volumes in the virtual environment. Render the depth map D of the scene seen by the camera from its viewpoint. Based on R and D, convert the pixel Cp of the video image to the texture color Ct corresponding to the scene object. The calculation formula is as follows:

[0042]

[0043] Among them, I t d represents the texture coordinates corresponding to the object. t D represents the depth value corresponding to the object. t Let f be the depth value in the depth map D, and let f be the depth value calculated based on the correlation R, image pixel Cp, and texture coordinates. t A function for calculating texture color. When multiple images are merged into the same object, the contribution of each pixel in the image to the object is calculated, and this contribution is used as the weight to calculate the object's texture. The calculation formula is:

[0044] C=∑ω i C ti

[0045] Where i is the index of the image, C ti ω is the texture color of the object calculated from image i. i Let be the contribution value of image i to the object. Through the above calculation process, the fusion process of multiple video streams and various objects in the 3D scene is realized.

[0046] S4. Visualize the fusion result of all video images and the main video image in a virtual environment.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method of real-time image analysis of multi-video stream fusion, characterized in that, Includes the following steps: S1. Acquire multiple original video images, select the video with the most covered image parameters as the main video, calculate the parameter information of the video images, and use the parameter information to schedule the fusion of the video images and the main video. S2. Calculate the view frustum corresponding to the video in three-dimensional space based on the parameter information of the video image. The view frustum is an approximate or precise expression of the spatial range of the real environment when the video was captured in the virtual environment. Calculate the set of videos currently visible to the main video viewpoint based on the viewpoint position and direction of the main video image in the virtual environment, and schedule the video images visible to the main video viewpoint. S3. For video images in the visible set, calculate the association between the video images and virtual objects in the 3D scene based on the parameter information, and use the association to fuse the video images with the main video images. S4. Visualize the fusion result of all video images and the main video image in a virtual environment.

2. The method of real-time image analysis of multiple video stream fusion of claim 1, wherein: For each video in step S2, the view frustum corresponding to the video image in three-dimensional space is calculated based on the parameter information at the time of acquisition. The view frustum defines the observation range of the video image in the virtual environment, which corresponds to the real geographic space when the video image was captured. Then, based on the position and direction of the current viewpoint of the video image, the intersection of each video view frustum with the main video observation area is calculated, and the visibility of the video image is determined according to the following rules: If the position of the video image is more than a set distance from the viewpoint of the main video, the camera is considered invisible; if the position of the video image is within a set distance from the viewpoint of the main video, and the video image's view frustum is not within the viewpoint area of ​​the main video, or the angle between the optical axis of the video image and the viewpoint direction exceeds a set angle, the video image is considered invisible. If the video image position is within a set distance from the main video viewpoint position, and the following conditions are met: the video image cone intersects with the main video area or is within the main video area, and the angle between the optical axis direction of the video image and the direction of the main video viewpoint is not greater than a set angle, then the video image is considered visible. To accelerate the process of determining the visibility of video images, the video images are grouped according to their location distribution. During the calculation, the nearby groups are searched based on the main video viewpoint, thereby reducing the amount of computation. After calculating the visibility of the video images, update the list U of visible video images in use.

3. The method of real-time image analysis of a fusion of multiple video streams of claim 2, wherein, Update the visible video image U using the following steps: S2.1 Save the video image visibility result obtained in the last calculation, clear the exit list Q and the list to be added J, and set the video image flag bit in the visibility list V to -1; where a video image flag bit of 1 indicates that the video image is visible, a flag bit of 0 indicates that the video image is not visible, and -1 indicates that it is unknown; proceed to step S2.

2. S2.2 For each video image in list V, if its position is more than a set distance away from the current viewpoint, the video image is considered invisible and its flag is set to 0; otherwise, proceed to step S2.

3. S2.3 If the distance between the video image and the main video viewpoint is within a set distance: calculate the bounding box of the video image's frustum, calculate the observation area of ​​the main video image at the current viewpoint, and if the bounding box does not intersect with the observation area, the video image is considered invisible, and the flag position of the video image is set to 0; calculate the angle between the optical axis direction of the video image and the direction of the main video viewpoint, and if the angle is greater than a set angle, set the flag position of the video image to 0; otherwise, proceed to step S2.4; S2.4 If the bounding box of the video image's view frustum intersects with or is within the main video viewing area, and the angle between the video image's optical axis direction and the main video viewpoint direction is not greater than a set angle, then the video image is considered visible, and the video image's flag position is set to 1; compare the visibility list obtained in this calculation with the visibility list obtained in the previous calculation. If the video image is visible in this calculation but not in the previous calculation, then add it to the add list J; if the video image is not visible in this calculation but was visible in the previous calculation, then add it to the exit list Q; proceed to step S2.5; S2.

5. Extract the video images to be added to list J and send them to candidate list C; select several video images from C and send them to list of visible video images in use U as required; remove the same video images from candidate list C and list in use U as those from exit list Q and release the related parameters; perform synchronization processing on each video image in list U using timestamp parameters as required by the application.

4. The real-time image analysis method for multi-video stream fusion as described in claim 1, characterized in that: Step S1 uses one or more video images. The parameters of the video images are continuous, including position, orientation, focal length, and timestamp. The parameter information is encoded and transmitted or saved in real time along with the video images.

5. The real-time image analysis method for multi-video stream fusion as described in claim 1, characterized in that: Step S3 calculates the association between the content of each visible video image in the scene and the virtual objects in the 3D scene based on the parameter information of the video image. This calculation only needs to be performed once if the video image parameters do not change. Then, based on the association, multiple video images are fused with virtual objects in the 3D scene using video projection. The objects include points, lines, surfaces, and combinations of the three. The content of multiple video images can be fused with the same object. The intersection of video images is processed using feature matching, alpha channels, and weighted averaging methods. During fusion, the occlusion relationship between objects needs to be considered. The occlusion relationship between objects is quickly determined using the depth map method. The occluded parts use the original texture or are specified according to specific requirements, while the unoccluded parts are fused with the video images.

6. The real-time image analysis method for multi-video stream fusion as described in claim 1, characterized in that: Step S4 visualizes the fusion result of the video image and the 3D scene in the virtual environment, and adds the original 3D scene, the original video image, and the fusion effect under the main video viewpoint as needed.

Images