A multi-depth fusion technology for ultra-high-definition panoramic images during AIT process of manned spacecraft

Abstract

In order to solve the problem that the existing panoramic image shooting method cannot take clear images at different depths of field, resulting in blurred areas in the stitched panoramic image, which affects the effect of the manned spacecraft panoramic image, the present invention provides a manned spacecraft AIT process ultra-high-definition panoramic image multi-depth fusion technology, including the following steps: S1: shooting multi-depth original images; S2: using image similarity recognition technology to group multi-depth original images; S3: panoramic image stitching; S4: using a blur recognition algorithm to identify the blurred area of ​​the panoramic image; S5: using a multi-depth fusion method to generate a clear panoramic image. The present invention shoots multi-depth images at different focal points at each position through the above steps, and generates a clear panoramic image through the multi-depth fusion technology.

Description

Technical Field

[0001] The present invention relates to a multi-depth fusion technology of ultra-high-definition panoramic images during an AIT process of a manned spacecraft, and belongs to the technical field of digital image processing. Background Art

[0002] Panoramic image stitching refers to the process of generating a high-resolution, wide-viewing angle panoramic image by combining a group of overlapping images in the same scene through image preprocessing, image registration and image fusion. Panoramic image stitching technology occupies an important position in the field of digital image processing.

[0003] The existing panoramic image shooting adopts the autofocus method or the single focus shooting method at each position. In the panoramic image shooting of the manned spacecraft AIT process, due to the complex structure inside and outside the cabin, the depth of field of different single equipment in the image is different. The existing shooting method cannot take clear images at different depths of field, resulting in blurred areas in the stitched panoramic image, which affects the effect of the manned spacecraft panoramic image. Summary of the invention

[0004] The technical problem to be solved by the present invention is: to overcome the shortcomings of the existing technology and propose a multi-depth-of-field fusion technology for ultra-high-definition panoramic images during the AIT process of a manned spacecraft, to shoot multi-depth-of-field images with different focal points at each position, and to generate a clear panoramic image through the multi-depth-of-field fusion technology.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A multi-depth fusion technology for ultra-high-definition panoramic images during the AIT process of a manned spacecraft includes the following steps: S1: Shooting multi-depth original images. S2: Grouping the multi-depth original images using image similarity recognition technology. S3: Stitching panoramic images. S4: Identifying the blurred area of ​​the panoramic image using a blur recognition algorithm. S5: Generating a clear panoramic image using a multi-depth fusion method.

[0007] S1: Shooting original images with multiple depths of field

[0008] Step S1 uses a SLR camera, a panoramic gimbal, and a tripod to shoot original images with multiple depths of field. When shooting at each position, the single-machine equipment with different depths of field is focused separately to obtain original images under different focusing conditions. Original images at different positions are shot by adjusting the pitch angle and horizontal angle of the panoramic gimbal. Assuming that the size of each original image is M×N, the original image data set with multiple depths of field shot is represented as {P1,P2,...,P n-1 ,P n}, where n represents the number of original images taken.

[0009] S2: Grouping original images with multiple depths of field using image similarity recognition technology

[0010] Step S2 uses image similarity recognition technology to group the original multi-depth images, and groups the multi-depth images at the same position into one group. The similarity of two adjacent images is calculated in sequence. When the similarity value is greater than or equal to the set threshold, the adjacent images are grouped together, and when the similarity value is less than the set threshold, the adjacent images are grouped into different groups. The original multi-depth image data set obtained after grouping is represented as {G1, G2, ..., G K-1 ,G K}, where K represents the number of locations photographed, m k Represents the number of multi-depth images taken at the kth shooting position.

[0011] S3: Panoramic Image Stitching

[0012] Step S3 generates a panoramic image by stitching the single-focus original images. First, the first original image is selected from each group after grouping, which is represented by {G 11 ,G 21 ,...,G (K-1)1 ,G K1}, generate a stitched panoramic image Q through image preprocessing, image registration and image fusion steps for the K original images, and record the group number and position coordinates of each pixel in Q in the original image. For the overlapping area of ​​the original image, record the first group number and corresponding position coordinates of the pixel.

[0013] S4: Using blur recognition algorithm to identify blur areas in panoramic images

[0014] Step S4 uses a non-reference image blur recognition algorithm to identify blur areas in the panoramic image. Local windows of different sizes are set in the panoramic image, and multiple image clarity evaluation indicators are calculated for the image in each local window. When the values ​​are all less than the corresponding set threshold, the corresponding local window is marked as a blur area. Multiple blur area frames are obtained by performing blur detection on each local window.

[0015] In view of the situation where there are overlapping areas between different fuzzy area frames, the present invention optimizes the identified fuzzy area frames. Assume that there are overlapping areas between fuzzy area frame U1 and fuzzy area frame U2, the area range of frame U1 is (x1, y1, w1, u1), and the area range of frame U2 is (x2, y2, w2, u2). When U1∪U2=U2, delete frame U1; when U1∪U2=U1, delete frame U2; when U1∪U2≠U2 and U1∪U2≠U1, calculate the intersection over union (IoU) of the two fuzzy area frames, and when IoU is greater than or equal to the set threshold ε, merge frame U1 and frame U2 into frame U3, which is the minimum rectangular frame that contains both frame U1 and frame U2; when IoU is less than the set threshold ε, retain frame U1 and frame U2 respectively.

[0016] Assume that the number of blurred regions after optimization is d, and the position of the dth blurred region in the panoramic image Q is (x d ,y d ,w d ,u d ), the position in the original image is represented by (x d ′,y d ′,w d ,u d ), the group number in the original image is represented by k d , where x d and d Indicates the position coordinates of the center point of the blurred area in the panoramic image, x d ′ and y d ′ represents the position coordinates of the center point of the blurred area in the original image, w d and u d Indicates the width and height of the blurred area.

[0017] S5: Generate clear panoramic images using multi-depth fusion method

[0018] Step S5 performs multi-depth fusion on each blurred area identified in step S4 in turn. If the maximum clarity of the corresponding image captured under other focusing conditions is greater than δ times the clarity of the blurred area, the image with the maximum clarity is fused into the panoramic image. The position of the dth blurred area in the original image is (x d ′,y d ′,w d ,u d ), extract the kth d m in group number k The corresponding area of ​​the original image (x d ′,y d ′,w d ,u d ), expressed as Calculate the image sequentially The no-reference image clarity is expressed as m k The image with the highest resolution among the images is denoted by c d , if c d If c is equal to 1, multi-depth fusion is not performed on the dth blurred area. d is not equal to 1, and Then the multi-depth fusion method is used to transform the c d The corresponding areas of the images are fused into the panoramic image to improve the clarity of the panoramic image.

[0019] In the multi-depth fusion method, in order to achieve the c d The corresponding area of ​​the image is smoothly fused with the panoramic image Q to extract the cth d The area of ​​the image is greater than (x d ′,y d ′,w d ,u d ), and the extracted image is represented as I. Image I and panoramic image Q are preprocessed and registered. After registration, a conversion model is constructed to convert image I to the coordinate system of panoramic image Q. Assume that the stitched panoramic image is Q′, and the size of Q′ is the same as that of panoramic image Q. In Q′, (x d ,y d ,w d ,u d ) area is the pixel value corresponding to the image I. The pixel value of the area other than the image I in Q′ is the pixel value corresponding to the panoramic image Q, and the pixel values ​​of other areas in Q′ are obtained by weighted fusion or multi-resolution analysis fusion method.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] (1) The present invention captures multi-depth images at each position and utilizes multi-depth fusion technology to solve the problem of image blur at non-focus positions, thereby improving the clarity of the panoramic image of the manned spacecraft.

[0022] (2) The present invention uses image similarity recognition technology to automatically group original images with multiple depths of field, and groups original images taken at the same position with different focusing conditions into one group. The image grouping efficiency is high and the grouping accuracy is high.

[0023] (3) The multi-depth-of-field fusion technology of the present invention first generates a panoramic image by stitching together the original images taken under a single focus, then identifies the blurred area in the panoramic image, and fuses the original images corresponding to the blurred area under other focus conditions into the panoramic image. Compared with the method of first performing depth-of-field fusion on the original images of the same position with different focus conditions, and then stitching the fused images, the present invention improves the quality of panoramic image stitching.

[0024] (4) The present invention adopts a non-reference image blur recognition algorithm to automatically identify the local blur area of ​​the panoramic image, thereby reducing the workload of blur recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The flowchart of the multi-depth fusion technology of ultra-high-definition panoramic images during the AIT process of manned spacecraft of the present invention;

[0026] Figure 2 Schematic diagram of grouping original images with multiple depths of field according to the present invention;

[0027] Figure 3 Schematic diagram of the optimization processing of the blur region frame in blur detection of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail with reference to the accompanying drawings. The present invention first captures original images with multiple depths of field, generates a panoramic image by stitching original images captured under a single focus, then identifies a blurred area in the panoramic image, and fuses the original images corresponding to the blurred area under other focus conditions into the panoramic image.

[0029] A multi-depth fusion technology for ultra-high-definition panoramic images during the AIT process of a manned spacecraft includes the following steps: S1: Shooting multi-depth original images. S2: Grouping multi-depth original images using image similarity recognition technology. S3: Stitching panoramic images. S4: Identifying the blurred area of ​​the panoramic image using a blur recognition algorithm. S5: Generating a clear panoramic image using a multi-depth fusion method. Figure 1 Shown is the flow chart of the ultra-high-definition panoramic image multi-depth fusion technology during the AIT process of manned spacecraft.

[0030] S1: Shooting original images with multiple depths of field

[0031] Step S1 uses a SLR camera, a panoramic gimbal, and a tripod to shoot original images with multiple depths of field. When shooting at each position, the single-machine equipment with different depths of field is focused separately to obtain original images under different focusing conditions. Original images at different positions are shot by adjusting the pitch angle and horizontal angle of the panoramic gimbal. Assuming that the size of each original image is M×N, the original image data set with multiple depths of field shot is represented as {P1,P2,...,P n-1 ,P n}, where n represents the number of original images taken.

[0032] S2: Grouping original images with multiple depths of field using image similarity recognition technology

[0033] Step S2 uses image similarity recognition technology to group the original multi-depth images, and groups the multi-depth images at the same position into one group. The similarity of two adjacent images is calculated in sequence. When the similarity value is greater than or equal to the set threshold, the adjacent images are grouped together, and when the similarity value is less than the set threshold, the adjacent images are grouped into different groups. The original multi-depth image data set obtained after grouping is represented as {G1, G2, ..., G K-1 ,G K}, where K represents the number of locations photographed, m k represents the number of multi-depth images taken at the kth shooting position. Figure 2 Shown is a schematic diagram of grouping original images with multiple depths of field.

[0034] Preferably, the similarity calculation method described in this embodiment first generates histogram data of two adjacent images, normalizes the histogram, and uses the Bhattacharyya coefficient algorithm to calculate the histogram data to obtain the similarity value of the two images. When the similarity value is greater than or equal to 0.90, the adjacent images are grouped together, and when the similarity value is less than 0.90, the adjacent images are grouped together.

[0035] S3: Panoramic Image Stitching

[0036] Step S3 generates a panoramic image by stitching the single-focus original images. First, the first original image is selected from each group after grouping, which is represented by {G 11 ,G 21 ,...,G (K-1)1 ,G K1}, generate a stitched panoramic image Q through image preprocessing, image registration and image fusion steps for the K original images, and record the group number and position coordinates of each pixel in Q in the original image. For the overlapping area of ​​the original image, record the first group number and corresponding position coordinates of the pixel.

[0037] S4: Using blur recognition algorithm to identify blur areas in panoramic images

[0038] Step S4 uses a non-reference image blur recognition algorithm to identify the blur area of ​​the panoramic image. Local windows of different sizes are set in the panoramic image, and multiple image clarity evaluation indicators are calculated for the image in each local window. When the values ​​are all less than the corresponding set threshold, the corresponding local window is marked as a blur area. Multiple blur area frames are obtained by performing blur detection on each local window. Preferably, this embodiment uses grayscale difference and edge strength as clarity evaluation indicators. Only when both values ​​are less than the corresponding set threshold, the local window is marked as a blur area.

[0039] In view of the situation where different fuzzy area frames have overlapping areas, the present invention optimizes the identified fuzzy area frames. Assume that there are overlapping areas between fuzzy area frames U1 and U2, the area range of frame U1 is (x1, y1, w1, u1), and the area range of frame U2 is (x2, y2, w2, u2). Figure 3 The figure shows a schematic diagram of the optimization process. When U1∪U2=U2, box U1 is deleted; when U1∪U2=U1, box U2 is deleted; when U1∪U2≠U2 and U1∪U2≠U1, the intersection over union (IoU) of the two fuzzy area boxes is calculated. When IoU is greater than or equal to the set threshold ε, box U1 and box U2 are merged into box U3, which is the minimum rectangular box containing both box U1 and box U2; when IoU is less than the set threshold ε, box U1 and box U2 are retained respectively. Preferably, the threshold ε in this embodiment is set to 0.3.

[0040] Assume that the number of blurred regions after optimization is D, and the position of the dth blurred region in the panoramic image Q is (x d ,y d ,w d ,u d ), the position in the original image is represented by (x d ′,y d ′,w d ,u d ), the group number in the original image is represented by k d , where x d and d Indicates the position coordinates of the center point of the blurred area in the panoramic image, x d ′ and y d ′ represents the position coordinates of the center point of the blurred area in the original image, w d and u d Indicates the width and height of the blurred area.

[0041] S5: Generate clear panoramic images using multi-depth fusion method

[0042] Step S5 performs multi-depth fusion on each blurred area identified in step S4 in turn. If the maximum clarity of the corresponding image captured under other focusing conditions is greater than δ times the clarity of the blurred area, the image with the maximum clarity is fused into the panoramic image. The position of the dth blurred area in the original image is (x d ′,y d ′,w d ,u d ), extract the kth d m in group number k The corresponding area of ​​the original image (x d ′,y d ′,w d ,u d ), expressed as Calculate the image sequentially The no-reference image clarity is expressed as Preferably, this embodiment uses an energy gradient function as the non-reference image clarity. k The image with the highest resolution among the images is denoted by c d , if c d If c is equal to 1, multi-depth fusion is not performed on the dth blurred area. d is not equal to 1, and Then the multi-depth fusion method is used to transform the c d The corresponding areas of the images are fused into the panoramic image to improve the clarity of the panoramic image. Preferably, in this embodiment, δ=1.5.

[0043] In the multi-depth fusion method, in order to achieve the c d The corresponding area of ​​the image is smoothly fused with the panoramic image Q to extract the cth d The area of ​​the image is greater than (x d ′,y d ′,w d ,u d ), and the extracted image is represented as I. Image I and panoramic image Q are preprocessed and registered. After registration, a conversion model is constructed to convert image I to the coordinate system of panoramic image Q. Assume that the stitched panoramic image is Q′, and the size of Q′ is the same as that of panoramic image Q. In Q′, (x d ,y d ,w d ,u d) area is the pixel value corresponding to image I. The pixel value of the area other than image I in Q′ corresponds to the pixel value of the panoramic image Q. The pixel values ​​of other areas in Q′ are obtained by weighted fusion or multi-resolution analysis fusion method. Preferably, this embodiment adopts a weighted fusion method, and the pixel value in Q′ is expressed as Q′(x,y)=aQ(x,y)+(1-a)I(x,y), where a represents the weighting coefficient, and the weighted fusion adopts a fade-in and fade-out method. If a changes from 0 to 1, the image pixel value gradually changes from I(x,y) to Q(x,y).

[0044] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A multi-depth fusion method for ultra-high-definition panoramic images during the AIT process of a manned spacecraft, characterized in that: The steps include: S1: capture original images with multiple depths of field; S2: grouping the original images with multiple depths of field using image similarity recognition technology; S3: panoramic image stitching; S4: using a blur recognition algorithm to identify blur areas of the panoramic image; Assume that the number of blurred regions after optimization is d, and the position of the dth blurred region in the panoramic image Q is (x d ,y d ,w d ,u d ), the position in the original image is represented by (x d ′,y d ′,w d ,u d ), the group number in the original image is represented by k d , where x d and d Indicates the position coordinates of the center point of the blurred area in the panoramic image, x d ′ and y d ′ represents the position coordinates of the center point of the blurred area in the original image, w d and u d Indicates the width and height of the blurred area; S5: Generate a clear panoramic image using a multi-depth fusion method; S5: Generate a clear panoramic image using a multi-depth fusion method, specifically, perform multi-depth fusion on each blurred area identified in step S4 in turn. If the maximum clarity of the corresponding image taken under other focusing conditions is greater than δ times the clarity of the blurred area, the image with the maximum clarity is fused into the panoramic image. The position of the dth blurred area in the original image is (x d ′,y d ′,w d ,u d ), extract the kth d m in group number k The corresponding area of ​​the original image (x d ′,y d ′,w d ,u d ), expressed as Calculate the image sequentially The no-reference image clarity is expressed as m k The image with the highest resolution among the images is denoted by c d , if c d If c is equal to 1, then multi-depth fusion is not performed on the dth blurred area. d is not equal to 1, and fc d >δf1, then the multi-depth fusion method is used to combine the c d The corresponding areas of the images are fused into the panoramic image to improve the clarity of the panoramic image; In the multi-depth fusion method, in order to achieve the c d The corresponding area of ​​the image is smoothly fused with the panoramic image Q to extract the cth d The area of ​​the image is greater than (x d ′,y d ′,w d ,u d ), and the extracted image is represented as I, image I and panoramic image Q are preprocessed and registered, and after registration, a conversion model is constructed to convert image I into the coordinate system of panoramic image Q. Assume that the stitched panoramic image is Q′, and the size of Q′ is consistent with that of panoramic image Q. In Q′, (x d ,y d ,w d ,u d ) area is the pixel value corresponding to image I, the pixel value of the area other than image I in Q′ is the pixel value corresponding to the panoramic image Q, and the pixel values ​​of other areas in Q′ are obtained by weighted fusion or multi-resolution analysis fusion method.

2. The method for multi-depth fusion of ultra-high-definition panoramic images during the AIT process of a manned spacecraft according to claim 1, characterized in that: S1: shooting original images with multiple depths of field, specifically using a single-lens reflex camera, a panoramic gimbal and a tripod to shoot original images with multiple depths of field. When shooting at each position, focusing is performed on the single-machine devices with different depths of field respectively to obtain original images under different focusing conditions. Original images at different positions are shot by adjusting the pitch angle and horizontal angle of the panoramic gimbal. Assuming that the size of each original image is M×N, the original image data set obtained by shooting multiple depths of field is represented as {P1, P2, ..., P n-1 ,P n }, where n represents the number of original images taken.

3. The method for multi-depth fusion of ultra-high-definition panoramic images during the AIT process of a manned spacecraft according to claim 1, characterized in that: S2: using image similarity recognition technology to group the original images with multiple depths of field, specifically using image similarity recognition technology to group the original images with multiple depths of field, grouping the images with multiple depths of field at the same position into one group, and calculating the similarity of two adjacent images in sequence. When the similarity value is greater than or equal to a set threshold, the adjacent images are grouped into one group, and when the similarity value is less than the set threshold, the adjacent images are grouped into different groups. The original image data set with multiple depths of field obtained after grouping is represented as {G1, G2, ..., G K-1 ,G K }, where K represents the number of locations photographed, , m k Represents the number of multi-depth images taken at the kth shooting position.

4. The method for multi-depth fusion of ultra-high-definition panoramic images during the AIT process of a manned spacecraft according to claim 3 is characterized in that: S3: panoramic image stitching, specifically, using single-focus original images to stitch together to generate a panoramic image, firstly select the first original image from each group after grouping, represented by {G 11 ,G 21 ,...,G (K-1)1 ,G K1 }, generate a stitched panoramic image Q through image preprocessing, image registration and image fusion steps for the K original images, and record the group number and position coordinates of each pixel in Q in the original image. For the overlapping area of ​​the original image, record the first group number and corresponding position coordinates of the pixel.

5. The method for multi-depth fusion of ultra-high-definition panoramic images during the AIT process of a manned spacecraft according to claim 4, characterized in that: The S4: using a blur recognition algorithm to identify blur areas of the panoramic image, specifically using a blur recognition algorithm without a reference image to identify blur areas of the panoramic image, setting local windows of different sizes in the panoramic image, and calculating multiple image clarity evaluation indexes for the image in each local window. When the values ​​are all less than the corresponding set thresholds, the corresponding local window is marked as a blur area, and multiple blur area frames are obtained by performing blur detection on each local window.

6. The method for fusion of ultra-high-definition panoramic images with multiple depths of field during the AIT process of a manned spacecraft according to claim 5, characterized in that: In view of the situation where there are overlapping areas of different blurred area frames, the identified blurred area frames are optimized. Assume that there are overlapping areas of blurred area frame U1 and blurred area frame U2. The area range of frame U1 is (x1, y1, w1, u1), where x1 and y1 represent the position coordinates of the center point of frame U1 in the panoramic image Q, w1 and u1 represent the width and height of frame U1, and the area range of frame U2 is (x2, y2, w2, u2), where x2 and y2 represent the position coordinates of the center point of frame U2 in the panoramic image Q. Position coordinates, w2 and u2 represent the width and height of box U2; when U1∪U2=U2, delete box U1; when U1∪U2=U1, delete box U2; when U1∪U2≠U2 and U1∪U2≠U1, calculate the intersection and union ratio of the two fuzzy area boxes, and when the intersection and union ratio is greater than or equal to the set threshold ε, merge box U1 and box U2 into box U3, and box U3 is the minimum rectangular box that contains both box U1 and box U2; when the intersection and union ratio is less than the set threshold ε, retain box U1 and box U2 respectively.

Images