A method for processing panoramic images of a manned spacecraft by combining close-up images of equipment

Abstract

The present invention provides a method for processing panoramic images of a manned spacecraft in combination with device close-up images. The original images to be stitched and the device close-up images are respectively captured. A panoramic image is generated using the captured original images to be stitched, and the fusion of the device close-up image and the panoramic image is achieved. Since different focal lengths are set when a single-lens reflex camera captures the device close-up image and the original image to be stitched, the present invention proposes a registration and fusion technology for the device close-up image and the panoramic image after stitching. When performing feature registration, a feature extraction method combining an edge detection operator and a Scale-invariant feature transform (SIFT) algorithm is adopted, and when performing image fusion, a fusion method combining the spatial domain and the frequency domain is adopted.

Description

Technical Field

[0001] The present invention relates to a panoramic image processing method for a manned spacecraft combined with device close-up images, belonging to the technical field of digital image processing. Background Art

[0002] Panoramic image stitching refers to generating a high-resolution and wide-angle panoramic image by performing image preprocessing, image registration, and image fusion on a set of images with overlapping regions in the same scene. The panoramic images of a manned spacecraft are used to record and display information such as the actual final installation state of equipment inside and outside the cabin, the external dimensions of products, the installation positions, the assembly relationships, and various soft structures inside and outside the cabin.

[0003] The panoramic images of a manned spacecraft need to clearly reflect the actual final installation state information of the equipment. When the focal length set by a single-lens reflex camera is small, the shooting range of a single image is wider, and the number of original images taken for panoramic stitching is less. At the same time, the difficulty of stitching adjacent original images is lower. However, in this case, the proportion of the equipment in the picture is small, which will affect the clarity of the equipment. When the focal length set by a single-lens reflex camera is large, the shooting range of a single image is narrow, and the proportion of the equipment in a single original image is large, which improves the clarity of the equipment. However, in this case, the number of original images taken for panoramic stitching is large, and the difficulty of feature registration of adjacent original images in areas with less equipment is large. Summary of the Invention

[0004] The technical problem to be solved by the present invention is: overcoming the deficiencies of the prior art, and proposing a panoramic image processing method for a manned spacecraft combined with device close-up images. The original images to be stitched and the device close-up images are taken at different focal lengths, and the panoramic image is generated by using the taken original images to be stitched, and the fusion of the device close-up image and the panoramic image is realized. Since different focal lengths are set when the single-lens reflex camera takes the device close-up image and the original image to be stitched, using the traditional image processing method to realize the fusion of the device close-up image and the panoramic image will affect the processing effect. The present invention proposes a registration and fusion technology for the device close-up image and the stitched panoramic image. When performing feature registration, a feature extraction method combining an edge detection operator and a Scale-invariant feature transform (SIFT) algorithm is adopted, and when performing image fusion, a fusion method combining the spatial domain and the frequency domain is adopted.

[0005] The object of the present invention is achieved by the following technical solutions:

[0006] S1: Take the original images to be stitched and the device close-up images

[0007] Step S1 includes step S11 and step S12.

[0008] S11: Shoot the original image to be stitched

[0009] In step S11, a single-lens reflex camera, a panoramic head, and a tripod are used to shoot the original image to be stitched. First, set the shooting parameters of the single-lens reflex camera (aperture, ISO, exposure time, focal length, etc.), and set the shooting angles in the pitch direction and horizontal direction of the panoramic head according to the field of view size of the single-lens reflex camera. Set the starting position and ending position of shooting according to the coverage range of the manned spacecraft, and complete the shooting of the original image to be stitched using the set parameters. Suppose the dataset of the original images to be stitched is represented as {P 1 , P 2 ,..., P N-1 , P N}, where N represents the number of positions where the original images to be stitched are shot.

[0010] S12: Shoot close-up images of the equipment

[0011] The panoramic image needs to clearly reflect the details such as the numbers, codes, and QR codes of the equipment. In step S12, close-up images of the equipment are supplemented for shooting. For each piece of equipment to be supplemented with shooting, first control the position of the panoramic head to make the equipment to be shot located at the center of the field of view, and then adjust the focal length of the single-lens reflex camera to make the equipment to be supplemented with shooting account for about 80% of the frame. Complete the shooting of M close-up images of the equipment by controlling the position of the panoramic head. Suppose the dataset of the close-up images of the equipment is represented as {T 1 , T 2 ,..., T M-1 , T M}, where M represents the number of pieces of equipment for which close-up images are shot.

[0012] S2: Stitch the original images to be stitched to generate a panoramic image

[0013] Step S2 includes step S21 and step S22.

[0014] S21: Image preprocessing

[0015] In step S21, preprocessing operations are performed on the original images to be stitched and the close-up images of the equipment, including geometric correction and brightness equalization processing.

[0016] S22: Panoramic image stitching

[0017] In step S22, the original images to be stitched are used to stitch and generate a panoramic image. The stitched panoramic image F is generated through traditional image registration and image fusion steps for the N preprocessed original images.

[0018] S3: Registration of the close-up images of the equipment and the stitched panoramic image

[0019] Step S3 performs the registration of each device close-up image with the stitched panoramic image F in sequence. The m-th device close-up image T m The registration step of the device close-up image with the stitched panoramic image includes step S31 and step S32.

[0020] S31: Use an edge detection operator to extract the edges of the device close-up image

[0021] In step S31, first use an edge detection operator to extract the edges of the device close-up image, and after extraction, an edge binary image is obtained, where the gray value of the edges in the edge binary image is 255 and the gray value of non-edges is 0. Since the images taken by a single-lens reflex camera have a high image resolution, the edges extracted by the edge detection operator cannot completely cover all the pixel points where the image edges are located. In addition, the edges extracted by the edge detection operator are prone to breakage, discontinuity, etc. Therefore, the present invention uses morphological dilation processing to process the edge binary image to obtain an optimized edge binary image E m .

[0022] S32: Feature point extraction and registration

[0023] Step S32 obtains the feature points and feature descriptors of the T m image based on the SIFT method. Assume that the position of the i-th feature point is (x i , y i ). To ensure that the extracted feature points are on the T m image edge and improve the accuracy of feature point registration, only when E m (x i , y i ) = 255, the feature point and the feature descriptor are retained. Then use the (Best Bin First, BBF) algorithm to obtain the matching point pairs of the image T m and all the original images to be stitched, and construct an image transformation model to transform the image T m into the coordinate system where the panoramic image F is located. The transformed image is denoted as P m .

[0024] Repeat steps S31 and S32 to achieve the registration of M device close-up images with the stitched panoramic image.

[0025] S4: Fusion of the device close-up image and the stitched panoramic image

[0026] Step S4 performs the fusion of each registered and transformed image with the stitched panoramic image F in sequence. For the m-th registered and transformed image P m , intercept the image Q m in the image F. Q m is composed of the image P mComposed of corresponding position pixel points. P m and Q m is the source fusion image. For P m and Q m perform Laplacian pyramid decomposition. Let P ml and Q ml be the l-th layer images obtained after Laplacian pyramid decomposition respectively. Laplacian pyramid image fusion is performed separately on each decomposed layer. Assume the fused image of the l-th layer is L ml (0 ≤ l ≤ K). When l = K, take the larger value of the two images as the fused top layer image:

[0027]

[0028] When 0 ≤ l < K, that is, for non-top layer decomposed images, use the linear weighted method to achieve fusion, where the weight coefficients are calculated in the spatial domain. Calculate the Brenner gradient, Tenengrad gradient, and energy gradient functions within the σ×u region centered on each pixel in P m and Q m (σ and u are odd numbers, and σ ≥ 3, u ≥ 3). Among them, the Brenner gradient, Tenengrad gradient, and energy gradient functions in image P m are denoted as B pm , T pm and E pm respectively, and the Brenner gradient, Tenengrad gradient, and energy gradient functions in image Q m are denoted as B qm , T qm and E qm .

[0029] When B pm (i, j) ≥ B qm (i, j), T pm (i, j) ≥ T qm (i, j) and E pm (i, j) ≥ E qm (i, j),

[0030] L ml (i, j) = P ml (i, j)

[0031] When B pm (i, j) < B qm (i, j), T pm (i, j) < T qm (i, j) and E pm (i, j) < E qm (i, j),

[0032] L ml (i, j) = Q ml (i, j)

[0033] In other cases, L ml (i, j) is obtained by using a linear weighting method,

[0034] L ml (i, j) = w p (i, j)P ml (i, j) + (1 - w p (i, j))Q ml (i, j)

[0035] where w p represents the fusion weight coefficient, and normalizes B pm , T pm and E pm respectively. After normalization, B pm ′, T pm ′ and E pm ′ are respectively expressed as:

[0036]

[0037] Therefore, L m0 , L m1 ,..., L m(K-1) , L mK constitute the fused Laplacian pyramid, and the final fused image R m is obtained by reconstructing the Laplacian pyramid, and the Q in the image F is replaced with the fused image R m image. m image.

[0038] Repeat step S4 to implement the fusion of the M registered transformed images and the panoramic image F to obtain the final panoramic image.

[0039] The present invention has the following beneficial effects compared with the prior art:

[0040] (1) The present invention supplements the shooting of the device close-up image while changing the focal length of the single-lens reflex camera, and realizes the fusion of the device close-up image and the panoramic image, more clearly reflecting the actual final installation state information of the device, and solving the problems encountered when the single-lens reflex camera shoots only at a small focal length and only at a large focal length.

[0041] (2) When registering the device close-up image with the stitched panoramic image in the present invention, a feature extraction method combining an edge detection operator and the SIFT algorithm is adopted, ensuring that only the feature points on the image edge are used for feature registration, thereby improving the accuracy of feature extraction and registration.

[0042] (3) When fusing the device close-up image with the stitched panoramic image in the present invention, a fusion method combining the spatial domain and the frequency domain is adopted. The images to be fused are decomposed by Laplacian pyramid. When constructing the frequency domain fusion weight, multiple sharpness evaluation indicators in the spatial domain are comprehensively considered, and finally the fused image is reconstructed, improving the sharpness of image fusion. Description of the Drawings

[0043] Figure 1 Flowchart of the panoramic image processing method of the present invention. Detailed Embodiments

[0044] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. The present invention first takes the original images to be stitched and the device close-up images at different focal lengths, and then uses the original images to be stitched to achieve panoramic image stitching. Since the single-lens reflex camera sets different focal lengths when taking the device close-up images and the original images to be stitched, using traditional image processing methods to fuse the device close-up images and the panoramic images will affect the processing effect. Therefore, the present invention proposes a registration and fusion technology for the device close-up images and the stitched panoramic images, adopting a feature extraction method combining an edge detection operator and the SIFT algorithm during feature registration, and a fusion method combining the spatial domain and the frequency domain during feature fusion.

[0045] A panoramic image processing method for a manned spacecraft combined with device close-up images includes the following steps: S1: Take the original images to be stitched and the device close-up images. S2: Stitch the original images to be stitched to generate a panoramic image. S3: Register the device close-up image with the stitched panoramic image. S4: Fuse the device close-up image with the stitched panoramic image. As Figure 1 shown is a flowchart of a panoramic image processing method for a manned spacecraft combined with device close-up images.

[0046] S1: Take the original images to be stitched and the device close-up images

[0047] Step S1 includes step S11 and step S12.

[0048] S11: Take the original images to be stitched

[0049] Step S11: Use a single-lens reflex camera, a panoramic head, and a tripod to capture the original images to be stitched. First, set the shooting parameters of the single-lens reflex camera (aperture, ISO, exposure time, focal length, etc.), and set the shooting angles in the pitch and horizontal directions of the panoramic head according to the field of view of the single-lens reflex camera. Set the starting and ending positions of the shooting according to the coverage range of the manned spacecraft, and use the set parameters to complete the shooting of the original images to be stitched. Assume that the dataset of the original images to be stitched is represented as {P 1 , P 2 ,..., P N-1 , P N}, where N represents the number of positions where the original images to be stitched are captured.

[0050] S12: Capture close-up images of the equipment

[0051] The panoramic image needs to clearly reflect the details such as the equipment number, code, and QR code. In step S12, close-up images of the equipment are supplemented. For each piece of equipment to be supplemented, first control the position of the panoramic head to place the equipment to be photographed at the center of the field of view, and then adjust the focal length of the single-lens reflex camera so that the supplemented equipment occupies about 80% of the frame. Control the position of the panoramic head to complete the shooting of M close-up images of the equipment. Assume that the dataset of the close-up images of the equipment is represented as {T 1 , T 2 ,..., T M-1 , T M}, where M represents the number of pieces of equipment for which close-ups are taken.

[0052] S2: Stitch the original images to be stitched to generate a panoramic image

[0053] Step S2 includes step S21 and step S22.

[0054] S21: Image preprocessing

[0055] In step S21, preprocessing operations are performed on the original images to be stitched and the close-up images of the equipment, including geometric correction and brightness equalization.

[0056] S22: Panoramic image stitching

[0057] In step S22, the original images to be stitched are used to stitch and generate a panoramic image. The stitched panoramic image F is generated from the N preprocessed original images through traditional image registration and image fusion steps.

[0058] Preferably, the scale-invariant feature transform (SIFT) algorithm is used for feature extraction, the best-bin-first (BBF) algorithm is used for feature registration, and the linear weighted fusion method is used for image fusion.

[0059] S3: Registration of the close-up images of the equipment and the stitched panoramic image

[0060] Step S3 performs registration of each device close-up image with the stitched panoramic image F in sequence. The m-th device close-up image T m The registration step of the device close-up image with the stitched panoramic image includes Step S31 and Step S32.

[0061] S31: Extract the edges of the device close-up image using an edge detection operator

[0062] In Step S31, first, the edges of the device close-up image are extracted using an edge detection operator, and after extraction, a binary edge image is obtained, where the gray value of the edges in the binary edge image is 255 and the gray value of non-edges is 0. Since the images taken by a single-lens reflex camera have a high image resolution, the edges extracted using the edge detection operator cannot completely cover all the pixel points where the image edges are located. In addition, the edges extracted using the edge detection operator are prone to breakage, discontinuity, etc. Therefore, the present invention processes the binary edge image using morphological dilation to obtain an optimized binary edge image E m .

[0063] Preferably, the edge detection operator is selected as the Roberts edge detection operator.

[0064] S32: Feature point extraction and registration

[0065] In Step S32, based on the SIFT method, the feature points and feature descriptors of the T m image are obtained. Assume that the position of the i-th feature point is (x i , y i ). To ensure that the extracted feature points are on the edge of the T m image and improve the accuracy of feature point registration, only when E m (x i , y i ) = 255, the feature point and the feature descriptor are retained. Then, using the BBF algorithm, the matching point pairs of the image T m and all the original images to be stitched are obtained, and an image transformation model is constructed to transform the image T m to the coordinate system where the panoramic image F is located, and the transformed image is denoted as P m .

[0066] Repeat Step S31 and Step S32 to achieve the registration of the M device close-up images with the stitched panoramic image.

[0067] S4: Fusion of the device close-up image and the stitched panoramic image

[0068] In Step S4, the fusion of each registered and transformed image with the stitched panoramic image F is performed in sequence. For the m-th registered and transformed image P m, intercept image Q from image F m , Q m is composed of the pixel points at the corresponding positions of image P m . P m and Q m are the source fusion images. Perform Laplacian pyramid decomposition on P m and Q m . Let P ml and Q ml be the l-th layer images obtained after Laplacian pyramid decomposition respectively. Laplacian pyramid image fusion is performed separately on each decomposed layer. Assume the fused image of the l-th layer is L ml (0 ≤ l ≤ K). When l = K, take the larger value of the two images as the fused top layer image:

[0069]

[0070] When 0 ≤ l < K, that is, for non-top layer decomposed images, use the linear weighted method to achieve fusion, where the weight coefficients are calculated in the spatial domain. Calculate the Brenner gradient, Tenengrad gradient, and energy gradient functions within the σ×u region centered on each pixel in P m and Q m (σ and u are odd numbers, and σ ≥ 3, u ≥ 3). Among them, the Brenner gradient, Tenengrad gradient, and energy gradient functions in image P m are denoted as B pm , T pm and E pm respectively, and the Brenner gradient, Tenengrad gradient, and energy gradient functions in image Q m are denoted as B qm , T qm and E qm .

[0071] When B pm (i, j) ≥ B qm (i, j), T pm (i, j) ≥ T qm (i, j) and E pm (i, j) ≥ E qm (i, j),

[0072] L ml (i, j) = P ml (i, j)

[0073] When B pm (i, j) < B qm (i, j), T pm (i, j) < T qm(i, j) and E pm (i, j) < E qm When (i, j),

[0074] L ml (i, j) = Q ml (i, j)

[0075] In other cases, L ml (i, j) is obtained by linear weighting,

[0076] L ml (i, j) = w p P ml (i, j) + (1 - w p )Q ml (i, j)

[0077] where w p represents the fusion weight coefficient, and normalizes B pm , T pm and E pm respectively. After normalization, B pm ′, T pm ′ and E pm ′ are respectively expressed as:

[0078]

[0079]

[0080] Therefore, L m0 , L m1 ,..., L m(K-1) , L mK constitute the fused Laplacian pyramid, and the final fused image R m is obtained by reconstructing the Laplacian pyramid, and the Q in the image F is replaced with the fused image R m image. m Image.

[0081] Repeat step S4 to implement the fusion of the M registered transformed images and the panoramic image F to obtain the final panoramic image.

[0082] Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solution of the present invention without departing from the spirit and scope of the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for processing a manned spacecraft panoramic image combined with a device close-up image, comprising the following steps: S1: Take the original image to be stitched and the close-up image of the device; S2: The original images to be stitched are stitched to generate a panoramic image; S3: Registration of the device close-up image with the stitched panoramic image. Registration of each device close-up image with the stitched panoramic image is performed in turn, including: Step S31: using an edge detection operator to extract the edge of the device close-up image, obtaining an edge binary image after extraction, and using morphological dilation processing to obtain an optimized edge binary image; Step S32: obtaining feature points and feature descriptors based on the SIFT method, and retaining the feature points and feature descriptors when the grayscale value of the feature points in the optimized edge binary image is 255; then using the BBF algorithm to obtain matching point pairs between the device close-up image and all the images to be stitched, transforming the device close-up image into the coordinate system of the panoramic image, and obtaining the image after registration transformation; S4: Fusion of the device close-up image and the stitched panoramic image, including: Performing Laplace pyramid decomposition on the image after the registration transformation and the corresponding area of ​​the spliced ​​panoramic image; At the top level of the pyramid, the larger pixel value of the two images is taken as the fusion result; For non-top-level decomposition images, the fusion weights are dynamically calculated based on the Brenner gradient, Tenengrad gradient and energy gradient functions in the spatial domain, and the fusion is achieved using a linear weighted method.

2. The method for processing panoramic images of manned spacecraft combined with equipment close-up images according to claim 1, It is characterized in that The step S1 captures the original image to be stitched and the close-up image of the device, including step S11 and step S12. Step S11: Shoot the original image to be stitched. Use a single-lens reflex camera, a panoramic head, and a tripod to shoot the original image to be stitched. First, set the shooting parameters of the single-lens reflex camera, and set the shooting angles in the pitch direction and the horizontal direction of the panoramic head according to the field of view of the single-lens reflex camera. Set the starting position and the ending position of the shooting according to the coverage range of the manned spacecraft, and complete the shooting of the original image to be stitched using the set parameters. Assume that the dataset of the original images to be stitched obtained by shooting is represented as {P 1 , P 2 ,..., P N-1 , P N}, where N represents the number of positions where the original images to be stitched are shot; Step S12: Take close-up images of the devices. For each additional device to be photographed, first control the position of the panoramic pan-tilt so that the device to be photographed is at the center of the field of view, and then adjust the focal length of the SLR camera so that the additional device occupies a certain proportion of the frame. Complete the shooting of M device close-up images by controlling the position of the panoramic pan-tilt. Assume that the device close-up image dataset is represented as {T 1 , T 2 ,..., T M-1 , T M}, where M represents the number of devices for which close-ups are taken.

3. The method for processing panoramic images of manned spacecraft combined with equipment close-up images according to claim 1, It is characterized in that The step S2 of stitching the original images to be stitched together to generate a panoramic image includes step S21 and step S22. Step S21: image preprocessing, performing preprocessing operations on the original image to be stitched and the device close-up image, including geometric correction and brightness equalization processing; Step S22: panoramic image stitching, using the original images to be stitched to generate a panoramic image, and generating a stitched panoramic image F through traditional image registration and image fusion steps for the N pre-processed original images.

4. The method for processing panoramic images of manned spacecraft combined with equipment close-up images according to claim 1, It is characterized in that The registration of the S3 device close-up image and the stitched panoramic image is specifically to perform the registration of each device close-up image and the stitched panoramic image F in sequence. The registration steps of the m-th device close-up image T m and the stitched panoramic image include step S31 and step S32; Step S31: Extract the edges of the device close-up image using an edge detection operator. First, use the edge detection operator to extract the edges of the device close-up image. After extraction, an edge binary image is obtained. Then, perform morphological dilation processing on the edge binary image to obtain an optimized edge binary image E m ; Step S32: Feature point extraction and registration. Based on the SIFT method, obtain the feature points and feature descriptors of image T m For the feature points and feature descriptors of the image, assume the position of the i-th feature point is (x i , y i ). Only when E m (x i , y i ) = 255, retain the feature point and its feature descriptor. Then, use the BBF algorithm to obtain the matching point pairs between image T m and all the original images to be stitched, and construct an image transformation model to transform image T m to the coordinate system where the panoramic image F is located. The transformed image is denoted as P m . Repeat Step S31 and Step S32 to achieve the registration of M device close-up images and the stitched panoramic image.

5. The method for processing panoramic images of manned spacecraft combined with equipment close-up images according to claim 1, It is characterized in that The fusion of the close-up image of the S4 device and the stitched panoramic image is to perform the fusion of each registered and transformed image and the stitched panoramic image F in sequence. For the image P after the m-th registration transformation m , an image Q is intercepted from the image F m , Q m is composed of the pixel points at the corresponding positions of the image P m , P m and Q m are the source fusion images. Perform Laplacian pyramid decomposition on P m and Q m . Let P ml and Q ml be the images at the l-th layer obtained after Laplacian pyramid decomposition respectively. Laplacian pyramid image fusion is performed separately on each decomposed layer. Assume that the fused image at the l-th layer is L ml (0 ≤ l ≤ K). When l = K, take the larger value of the two images as the fused top layer image: When \(0\leq l\lt K\), that is, for non-top-level decomposed images, a linear weighted method is used to achieve fusion, where the weight coefficients are calculated in the spatial domain, and \(P\) is calculated in the spatial domain. m and \(Q\) m In \(P\) and \(Q\), taking each pixel as the center, the Brenner gradient, Tenengrad gradient, and energy gradient functions within the range of the \(\sigma\times u\) region, where \(\sigma\) and \(u\) are odd numbers, and \(\sigma\geq3\), \(u\geq3\). The Brenner gradient, Tenengrad gradient, and energy gradient functions in image \(P\) m are respectively denoted as \(B\) pm , \(T\) pm and \(E\) pm , and the Brenner gradient, Tenengrad gradient, and energy gradient functions in image \(Q\) m are respectively denoted as \(B\) qm , \(T\) qm and \(E\) qm , When B pm (i,j) ≥ B qm (i,j), T pm (i,j) ≥ T qm (i,j) and E pm (i,j) ≥ E qm (i,j), then L ml (i, j) = P ml (i, j) When B pm (i,j) < B qm (i,j), T pm (i,j) < T qm (i,j) and E pm (i,j) < E qm (i,j), then L ml (i,j) = Q ml (i,j) In other cases, L ml (i, j) is obtained by using a linear weighting method. L ml (i,j) = w p (i,j)P ml (i,j)+(1 - w p (i,j))Q ml (i,j) where w p represents the fusion weight coefficient, and respectively normalizes B pm , T pm and E pm . After normalization, B pm ′, T pm ′ and E pm ′ are respectively expressed as: Therefore, L m0 , L m1 ,..., L m(K-1) , L mK constitute the fused Laplacian pyramid, and the final fused image R is obtained through the reconstruction of the Laplacian pyramid m , and the fused image R is used m to replace Q in the image F m image, and repeat step S4 to implement the fusion of the M registered transformed images and the panoramic image F to obtain the final panoramic image.

Images