A hierarchical fusion method for comprehensive visual images

Abstract

The present application provides a method for layered fusion of a comprehensive visual picture, the method comprising: drawing a virtual map picture and a symbol picture in non-overlapping blocks to obtain a block-drawing picture; transmitting the block-drawing picture to a fusion module; the fusion module divides the block-drawing picture according to a fixed resolution to extract a virtual map picture and a symbol picture; the extracted virtual map picture is fused with a video picture collected at a unified time to obtain a fused picture; the extracted symbol picture is scaled according to a fixed resolution to obtain a transformed symbol picture, the transformed symbol picture has the same size as the fused picture; pixel-segmenting the transformed symbol picture according to a preset picture background pixel range to remove its background area to obtain an effective symbol area; superimposing the effective symbol area on the fused picture to obtain a final layered fused comprehensive visual picture.

Description

Technical Field

[0001] The invention belongs to the technical field of computer graphics and image processing, and in particular relates to a method for layered fusion of comprehensive visual scenes. Background Art

[0002] The continuous development of airborne flight vision system technology has more effectively improved the pilot's situational awareness of the extra-cabin environment and enhanced flight safety. The integrated vision system combines the external three-dimensional terrain with the sensor video data by fusing the terrain image with the sensor video. The map vision generated by the three-dimensional terrain database is relatively clear, has a high resolution, and can present any field of view angle and distance. The sensor video data has higher real-time performance, which effectively improves the lag of pure terrain images and the disadvantages of sensor video in imaging, resolution and viewing angle. In the process of virtual and real vision fusion, the virtual map vision screen drawn based on the three-dimensional terrain data combined with the aircraft attitude information in the single graphics card mode is superimposed with the symbol screen showing the flight instrument information to form a synthetic virtual vision. However, when the real video screen is further superimposed, the symbol screen will inevitably be blocked. Drawing the virtual map screen and the symbol screen separately requires more hardware consumption, and the problem of virtual and real vision fusion under a single graphics card needs to be solved. Summary of the invention

[0003] The purpose of the present invention is to propose a comprehensive visual picture layered fusion method, which can solve the occlusion problem caused by the fusion of different pictures under a single graphics card.

[0004] The technical solution of the present invention is: the present application provides a method for layered fusion of comprehensive visual scenes, the method comprising:

[0005] Acquire a virtual map screen according to the terrain database and the flight data, and draw the virtual map screen and the symbol screen in blocks without overlapping each other to obtain a block-drawing screen;

[0006] Transmitting the block-drawing picture to a fusion module;

[0007] The fusion module segments the block drawing picture according to a fixed resolution to extract a virtual map picture and a symbol picture;

[0008] The extracted virtual map image is merged with the video image collected at the same time to obtain a merged image;

[0009] The extracted symbol picture is scaled according to a fixed resolution to obtain a transformed symbol picture, wherein the transformed symbol picture has the same size as the fused picture;

[0010] According to the preset background pixel range of the picture, the transformed symbol picture is pixel-segmented, and the background area is removed to obtain the effective symbol area;

[0011] The effective symbol area is superimposed on the fused picture to obtain a final layered fused comprehensive visual picture.

[0012] Preferably, the virtual map screen and the symbol screen are drawn in non-overlapping blocks, specifically including:

[0013] In the first preset area, the virtual map screen is drawn according to a fixed resolution;

[0014] In the second preset area, a symbol picture is drawn, and the first preset area and the second preset area do not overlap.

[0015] Preferably, the fixed resolution is 320 to 4096 pixels in the horizontal direction and 240 to 2160 pixels in the vertical direction.

[0016] Preferably,

[0017] If the first preset area is the left area of ​​the overall display screen, then the second preset area is the right area of ​​the overall display screen;

[0018] If the first preset area is the right area of ​​the overall display screen, then the second preset area is the left area of ​​the overall display screen;

[0019] If the first preset area is the upper area of ​​the entire display screen, the second preset area is the lower area of ​​the entire display screen;

[0020] If the first preset area is the lower area of ​​the entire display screen, the second preset area is the upper area of ​​the entire screen.

[0021] Preferably, obtaining a virtual map image according to a terrain database and flight data includes:

[0022] Based on the terrain database, combined with the aircraft's current posture and position, flight data information, and open source 3D rendering engine (OSG, OpenSceneGraph) technology, the terrain is dynamically drawn to generate the current virtual map screen in real time.

[0023] Preferably, the symbol screen includes an altitude instrument panel, an attitude instrument panel, and a flight navigation line.

[0024] Preferably, the extracted virtual map picture is merged with the video picture collected at the same time to obtain a merged picture, including:

[0025] Perform pre-processing operations on the video images;

[0026] Use image registration algorithm to align the video image with the virtual map image;

[0027] The multi-resolution analysis method is used to perform multi-resolution Gaussian sampling on the image, and the sub-images of different scales are fused at the pixel level to obtain a fused image.

[0028] Preferably, the extracted symbol image is scaled according to a fixed resolution, including:

[0029] The bilinear interpolation method is adopted. The new pixel value of the transformed picture is determined according to the adjacent original pixel value. A linear difference is performed in the horizontal and vertical directions respectively. The new pixel value of the transformed picture is obtained by interpolation of four adjacent pixels, thereby obtaining the transformed symbol picture.

[0030] The advantages of the present invention are as follows: in the order of drawing, segmentation and fusion, the drawing of the virtual map screen and the symbol screen is first completed on a single graphics card, and then after segmentation and extraction, layered fusion is performed in the order of the virtual map screen, the sensor video screen and the symbol screen, which can solve the occlusion problem caused by drawing the screen first and then fusing it without increasing hardware resources and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flow chart of a method for hierarchical fusion of comprehensive visual images. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0033] Embodiment 1

[0034] The present invention proposes a comprehensive visual picture layered fusion method, the method flow is as follows Figure 1 As shown, the following steps are included:

[0035] Step 101: obtaining a virtual map screen according to a terrain database and flight data, and drawing the virtual map screen and the symbol screen in non-overlapping blocks to obtain a block-drawing screen;

[0036] Specifically, in a first preset area, the virtual map screen is drawn according to a fixed resolution; in a second preset area, a symbol screen is drawn, and the first preset area and the second preset area do not overlap;

[0037] In actual applications, the overall display screen is divided left and right or up and down into a first preset area and a second preset area. The first preset area can be the left or right area of ​​the overall display screen, and the second preset area can be the right or left area of ​​the overall display screen, or the first preset area is the upper or lower area of ​​the overall display screen, and the second preset area is the lower or upper area of ​​the overall screen.

[0038] Among them, the comprehensive visual screen includes three layers of screens: virtual map screen, symbol screen and video screen.

[0039] Optionally, the virtual map screen is based on a terrain database, combined with information such as the current posture and position of the aircraft, flight data, etc., and uses an open source three-dimensional rendering engine OpenSceneGraph (OSG) technology to dynamically draw the terrain and generate the current virtual map screen in real time.

[0040] Optionally, the symbol screen displays various information such as aircraft instrument parameters in the form of drawn symbols based on information such as the current aircraft attitude position, and presets fixed background pixel values ​​for background area pixels in non-symbol areas for subsequent pixel segmentation, and draws them in the remaining area of ​​the fixed resolution area of ​​the virtual map screen. The symbol screen includes an altitude instrument panel, an attitude instrument panel, a flight navigation line, etc.

[0041] Step 102: transmitting the block-drawing picture to a fusion module;

[0042] Step 103: The fusion module segments the block drawing picture according to a fixed resolution to extract a virtual map picture and a symbol picture;

[0043] Among them, the range of fixed resolution is approximately 320 to 4096 in the horizontal direction and 240 to 2160 in the vertical direction, which is usually the commonly used resolution of video data, such as 320*240, 640*480, 720*480, 1024*768, 1280*720, 1920*1080, etc.

[0044] Step 104: Fusing the extracted virtual map image with the video image collected at the same time to obtain a fused image;

[0045] In practical applications, sensors are used to collect video images in real time, and the real-time matching video images from the sensors are fused with the virtual map images to obtain a fused image.

[0046] Specifically, the fusion process first performs preprocessing operations on the video screen, such as image defogging and detail enhancement, and then uses an image registration algorithm to align the video screen with the virtual map screen. For example, a two-stage registration algorithm is used to first use the local grayscale information around the feature points for initial matching, and the impact of various similarity measures on regional matching performance is analyzed through experiments. The matching performance and matching time are comprehensively considered. For homologous multi-phase images with large grayscale correlation, normalized grayscale cross-correlation is used, and for multi-spectral and multi-sensor images with weak grayscale correlation, regional mutual information is used. Then, a relaxation algorithm based on the Mahalanobis distance affine invariant constraint is used for precise matching. Finally, a multi-resolution analysis method is used to perform multi-resolution Gaussian sampling on the image, and sub-images of different scales are fused at the pixel level to obtain the final fused image.

[0047] Step 105: scaling the extracted symbol image according to a fixed resolution to obtain a transformed symbol image, wherein the transformed symbol image has the same size as the fused image;

[0048] Specifically, the scale transformation adopts a bilinear interpolation method. The new pixel value of the transformed picture is determined according to the adjacent original pixel value. A linear difference is performed in the horizontal and vertical directions respectively. The new pixel value of the transformed picture is obtained by interpolating four adjacent pixels, thereby obtaining the transformed symbol picture.

[0049] Step 106: segmenting the transformed symbol image into pixels according to a preset background pixel range, removing the background area, and obtaining a valid symbol area;

[0050] Specifically, the symbol screen is divided into a valid symbol area and an invalid background area. The background pixel value is fixed according to the background area set when the symbol screen is drawn, and the invalid pixel value range is set. The pixels that meet the range are deleted from the symbol screen for segmentation, and the valid symbol area is retained.

[0051] Step 107: superimpose the effective symbol area on the fused image to obtain a final layered fused comprehensive visual image.

[0052] In summary, the virtual map screen, the symbol screen and the sensor video screen are layered and fused in the order of the virtual map screen, the sensor video screen and the symbol screen to solve the occlusion problem caused by drawing first and then fusion under a single graphics card.

[0053] Embodiment 2

[0054] This scheme proposes a layered fusion method for comprehensive visual scenes, which can solve the problem of fusion of different comprehensive visual scenes under a single graphics card, including three layers of pictures: virtual map picture, symbol picture and video picture, such as Figure 1 As shown, the process is as follows:

[0055] Step 201: First, draw a virtual map screen and a symbol screen according to the terrain database and flight data, wherein the virtual map screen can be fixed at a resolution of 1024*768 and drawn on the left side of the output screen. According to the output screen resolution, such as 1900*1080, the symbol screen is drawn on the right side of the virtual map screen, with the same height and a width of the output screen width minus the fixed width of the virtual map screen, such as 1900-1024=876, which is used to display aircraft-related flight parameters. The background area can be set to black, and the corresponding RGB pixel value is (0,0,0). Make the drawn virtual map screen and symbol screen fill the output screen horizontally without overlapping each other.

[0056] Step 202: The synthetic scene composed of the virtual map screen and the symbol screen drawn in blocks is transmitted to the fusion module; the synthetic scene is segmented according to the fixed resolution of the virtual map screen, and the virtual map screen part is extracted. The real-time video screen is obtained from the sensor, and the matching video screen is superimposed on the virtual map screen with fixed coordinates and size to complete the first step of fusion.

[0057] Step 203: the right symbol picture extracted after segmentation in step 202 is scaled according to the fixed resolution of the virtual map picture, such as 1024*768, to obtain a picture size matching the virtual map picture;

[0058] Step 204: Perform pixel segmentation on the scale-transformed symbol image. According to the background pixel value set in the first step, such as RGB (0,0,0), extract the foreground pixel points of the symbol image and superimpose them on the fused image at the corresponding position, so that the flight data-related symbols are displayed on the fused image, completing the second fusion and obtaining the final integrated visual image.

[0059] In summary, the present application provides a method for layered fusion of comprehensive visual images. First, the virtual map image and the symbol image are drawn in non-overlapping blocks. The virtual map is drawn in the left area of ​​the overall image with a fixed resolution to form a virtual map image, and the remaining right area is used to draw symbol images such as instrument panels. After the two spliced ​​images are transmitted to the fusion module, they are segmented according to the above-mentioned fixed resolution. The real-time matching video image is first fused with the segmented virtual map image portion. The segmented symbol image is then scaled according to the above-mentioned fixed resolution, and pixel segmentation is performed according to the preset symbol image background pixel range. The background area is removed, and the retained effective symbol area is further superimposed on the first two layers of fused images to obtain the final layered fused comprehensive visual image. Thereby, the layered fusion of different comprehensive visual images without occlusion is achieved without adding hardware.

Claims

1. A method for layered fusion of comprehensive visual images, characterized in that: The method comprises: Acquire a virtual map screen according to the terrain database and the flight data, and draw the virtual map screen and the symbol screen in blocks without overlapping each other to obtain a block-drawing screen; Transmitting the block-drawing picture to a fusion module; The fusion module segments the block drawing picture according to a fixed resolution to extract a virtual map picture and a symbol picture; The extracted virtual map image is merged with the video image collected at the same time to obtain a merged image; The extracted symbol picture is scaled according to a fixed resolution to obtain a transformed symbol picture, wherein the transformed symbol picture has the same size as the fused picture; According to the preset background pixel range of the picture, the transformed symbol picture is pixel-segmented, and the background area is removed to obtain the effective symbol area; The effective symbol area is superimposed on the fused picture to obtain a final layered fused comprehensive visual picture.

2. The method according to claim 1, characterized in that Drawing the virtual map screen and the symbol screen in non-overlapping blocks specifically includes: In the first preset area, the virtual map screen is drawn according to a fixed resolution; In the second preset area, a symbol picture is drawn, and the first preset area and the second preset area do not overlap.

3. The method according to claim 1, characterized in that The fixed resolution is 320 to 4096 pixels in the horizontal direction and 240 to 2160 pixels in the vertical direction.

4. The method according to claim 1, characterized in that: If the first preset area is the left area of ​​the overall display screen, then the second preset area is the right area of ​​the overall display screen; If the first preset area is the right area of ​​the overall display screen, then the second preset area is the left area of ​​the overall display screen; If the first preset area is the upper area of ​​the entire display screen, the second preset area is the lower area of ​​the entire display screen; If the first preset area is the lower area of ​​the entire display screen, the second preset area is the upper area of ​​the entire screen.

5. The method according to claim 1, characterized in that: Get virtual map images based on terrain database and flight data, including: Based on the terrain database, combined with the aircraft's current attitude and position, flight data information, the open source 3D rendering engine technology OSG is used to dynamically draw the terrain and generate the current virtual map screen in real time.

6. The method according to claim 1, characterized in that The symbol screen includes an altitude instrument panel, an attitude instrument panel, and a flight guidance line.

7. The method according to claim 1, characterized in that The extracted virtual map image is merged with the video image collected at the same time to obtain a fused image, including: Perform pre-processing operations on the video images; Use image registration algorithm to align the video image with the virtual map image; The multi-resolution analysis method is used to perform multi-resolution Gaussian sampling on the image, and the sub-images of different scales are fused at the pixel level to obtain a fused image.

8. The method according to claim 1, characterized in that The extracted symbol image is scaled according to a fixed resolution, including: The bilinear interpolation method is adopted. The new pixel value of the transformed picture is determined according to the adjacent original pixel value. A linear difference is performed in the horizontal and vertical directions respectively. The new pixel value of the transformed picture is obtained by interpolation of four adjacent pixels, thereby obtaining the transformed symbol picture.

Images