Space target image reconstruction method, device and equipment facing dynamic observation scene

Abstract

The invention relates to a method, a device and equipment for reconstructing a space target image in a dynamic observation scene, and the method comprises the steps: sequentially dividing sequence images in a dynamic sequence multi-exposure data set into a plurality of groups of low-dynamic-range sub-image sequences in a manner that three images form one group; wherein the low dynamic range sub-image sequences comprise three sequence images with exposure degrees from low to high and different space target postures, generating corresponding static sub-image sequences, and training the space target image reconstruction network by using multiple groups of low dynamic range sub-image sequences and the corresponding static sub-image sequences to obtain a trained space target image reconstruction network. The space target image reconstruction network comprises a feature extraction unit, a feature deformable alignment unit and a feature fusion reconstruction unit, and an observation sequence image of a space target in a dynamic observation scene can generate a high dynamic range image of the space target by using the trained space target image reconstruction network. By adopting the method, efficient HDR image reconstruction in a dynamic scene can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of space target imaging, and in particular to a method, device and equipment for reconstructing space target images in a dynamic observation scene. Background Art

[0002] In recent years, with the gradual development of space resources, the importance of developing space-based optical space target situational awareness technology has gradually increased. Space-based imaging methods play an important role in spacecraft maintenance and debris removal tasks. Space situational awareness technology can effectively monitor space debris and ensure the normal operation and maintenance of on-orbit spacecraft. In space-based optical imaging, due to the complex space lighting environment, the high speed of the imaging platform and the observed target during the intersection process, the large changes in the lighting angle and scene brightness, therefore, rolling exposure mechanism imaging is often used in actual scenes. By changing the camera exposure in a very short time, a continuous multi-exposure image sequence of space targets is obtained, so that the image sequence contains various detailed information of the observed target at different exposure levels.

[0003] For the sequence of aerial images, although the exposure levels are different, the dynamic range of each image is limited. Dynamic range refers to the ratio of the maximum brightness (luminance) to the minimum brightness in an image or scene. The range of brightness in real scenes can reach 8 to 10 orders of magnitude, and the dynamic range that can be observed by the human visual system is also very wide, while the dynamic range captured by the camera during actual imaging is relatively low. Multi-exposure fusion (MEF) is the main method for obtaining high dynamic range (HDR) images. This method is based on the complementary and redundant information between low dynamic range (LDR) images with different exposure levels, and uses image fusion technology to accurately characterize high dynamic scenes in a single image, which can effectively enhance the image's detail information and improve the image's visual effect. Aerial multi-exposure fusion aims to enhance the texture details of the observed target from a subjective perspective and lay a good foundation for computer processing of subsequent space situational awareness tasks.

[0004] However, the traditional dynamic multi-exposure fusion method based on removing motion pixels is difficult to handle complex target overlapping areas and will lose a lot of effective information. In addition, due to the particularity of the subsequent tasks of space target situation awareness technology, such as 3D reconstruction and posture estimation, it is very sensitive to the changes in the multi-view geometric relationship of the observed target, and the traditional method based on registration is prone to changes in the representation of the observed target (such as the angle of the sailboard, etc.), which leads to problems such as inaccurate reconstruction of the space target image. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device and equipment for reconstructing spatial target images in dynamic observation scenes, which can achieve efficient HDR image reconstruction in dynamic scenes while better retaining the original multi-perspective geometric relationship of the observed target in response to the above technical problems.

[0006] A method for reconstructing a space target image in a dynamic observation scene, the method comprising:

[0007] Acquire a dynamic sequence multi-exposure dataset of a space target, wherein the dynamic sequence multi-exposure dataset includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures;

[0008] Sequentially dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of groups of low dynamic range sub-image sequences in groups of three, each of the low dynamic range sub-image sequences including three sequence images with exposure levels from low to high and with different spatial target postures, and generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences;

[0009] A space target image reconstruction network is trained using multiple groups of low dynamic range sub-image sequences and corresponding static sub-image sequences to obtain a trained space target image reconstruction network, wherein the space target image reconstruction network includes a feature extraction unit, a feature deformable alignment unit, and a feature fusion reconstruction unit, wherein the feature extraction unit transforms each of the sequence images from an image domain to a corresponding feature space to obtain a feature map, the feature deformable alignment unit aligns the non-reference image feature map with the reference image feature map to generate a corresponding aligned feature map, so as to eliminate and weaken the influence of the spatial information of the dynamic scene on the fusion effect, and the feature fusion reconstruction unit fuses and reconstructs the aligned feature maps corresponding to different exposure levels to obtain a high dynamic range image;

[0010] An observation sequence image of a space target in a dynamic observation scene is acquired, and a high dynamic range image of the space target is generated according to the observation sequence image using a trained space target image reconstruction network.

[0011] In one embodiment, when constructing the dynamic sequence multi-exposure dataset:

[0012] Three-dimensional models of different types of space targets are obtained, and simulation is performed based on the three-dimensional models. By adjusting the posture of the three-dimensional model at the initial position and performing rolling exposure rendering, a multi-exposure image sequence corresponding to the same space target at different initial postures is obtained.

[0013] In one embodiment, generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences includes:

[0014] The posture of the space target in the sequence image located in the middle of the low dynamic range sub-image sequence is used as the posture of the space target in the three sequence images in the corresponding static sub-image sequence. At the same time, the three sequence images in the static sub-image sequence retain the exposure degree of the three sequence images in the corresponding low dynamic range sub-image sequence.

[0015] In one embodiment, in the feature extraction unit:

[0016] Correcting the low dynamic range sub-image sequence by gamma transformation according to the sequence image located in the middle, to obtain a low dynamic range sub-image sequence in a high dynamic range image domain;

[0017] Connect each image in the low dynamic range sub-image sequence in the high dynamic range image domain into a multi-channel low dynamic range sub-image sequence according to the channel dimension;

[0018] Stride convolution is introduced into the multi-channel low dynamic range sub-image sequence to perform pyramid feature extraction, so as to obtain feature maps corresponding to each sequence image.

[0019] In one embodiment, in the feature deformable alignment unit:

[0020] The feature map corresponding to the medium exposure time image in the low dynamic range sub-image sequence is used as the reference feature map, and the other two feature maps are used as the non-reference feature maps;

[0021] Based on the reference feature map, the bias information of the other two non-reference feature maps is calculated respectively, and the non-reference feature map and the corresponding bias information are subjected to deformable convolution to obtain a spatially aligned aligned feature map;

[0022] The reference feature map is directly used as the aligned feature map.

[0023] In one embodiment, in the feature fusion reconstruction unit:

[0024] After merging the aligned feature maps, a DenseNet network is used to perform feature fusion and reconstruction to obtain the high dynamic range image.

[0025] In one embodiment, when training the space target image reconstruction network, the loss functions used include a structural similarity loss function, a mean square error loss function, an alignment loss function, and a feature point extraction loss function;

[0026] The alignment loss function is calculated by using an encoder to extract features from each low dynamic range image in the static sub-image sequence and obtaining a feature map, and a corresponding aligned feature map;

[0027] The feature point extraction loss function is calculated by using a scale-invariant feature transformation operator on the low dynamic sub-image sequence and the corresponding reconstructed high dynamic range image.

[0028] The present application also provides a device for reconstructing a space target image in a dynamic observation scene, the device comprising:

[0029] A dynamic sequence multi-exposure data set acquisition module is used to acquire a dynamic sequence multi-exposure data set of a space target, wherein the dynamic sequence multi-exposure data set includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures;

[0030] a static sub-image sequence generation module, for dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of low-dynamic sub-image sequences in groups of three, each of which includes three sequence images with different spatial target postures and exposure levels from low to high, and generating a corresponding static sub-image sequence according to each group of low-dynamic sub-image sequences;

[0031] a space target image reconstruction network training module, for dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of groups of low dynamic range sub-image sequences in groups of three, each of the low dynamic range sub-image sequences including three sequence images with different exposure levels from low to high and with different postures of the space target, and generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences;

[0032] The dynamic target image reconstruction module is used to obtain the observation sequence images of the space target in the dynamic observation scene, and use the trained space target image reconstruction network to generate a high dynamic range image of the space target according to the observation sequence images.

[0033] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire a dynamic sequence multi-exposure dataset of a space target, wherein the dynamic sequence multi-exposure dataset includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures;

[0035] Sequentially dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of groups of low dynamic range sub-image sequences in groups of three, each of the low dynamic range sub-image sequences including three sequence images with exposure levels from low to high and with different spatial target postures, and generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences;

[0036] A space target image reconstruction network is trained using multiple groups of low dynamic range sub-image sequences and corresponding static sub-image sequences to obtain a trained space target image reconstruction network, wherein the space target image reconstruction network includes a feature extraction unit, a feature deformable alignment unit, and a feature fusion reconstruction unit, wherein the feature extraction unit transforms each of the sequence images from an image domain to a corresponding feature space to obtain a feature map, the feature deformable alignment unit aligns the non-reference image feature map with the reference image feature map to generate a corresponding aligned feature map, so as to eliminate and weaken the influence of the spatial information of the dynamic scene on the fusion effect, and the feature fusion reconstruction unit fuses and reconstructs the aligned feature maps corresponding to different exposure levels to obtain a high dynamic range image;

[0037] An observation sequence image of a space target in a dynamic observation scene is acquired, and a high dynamic range image of the space target is generated according to the observation sequence image using a trained space target image reconstruction network.

[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0039] Acquire a dynamic sequence multi-exposure dataset of a space target, wherein the dynamic sequence multi-exposure dataset includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures;

[0040] Sequentially dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of groups of low dynamic range sub-image sequences in groups of three, each of the low dynamic range sub-image sequences including three sequence images with exposure levels from low to high and with different spatial target postures, and generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences;

[0041] A space target image reconstruction network is trained using multiple groups of low dynamic range sub-image sequences and corresponding static sub-image sequences to obtain a trained space target image reconstruction network, wherein the space target image reconstruction network includes a feature extraction unit, a feature deformable alignment unit, and a feature fusion reconstruction unit, wherein the feature extraction unit transforms each of the sequence images from an image domain to a corresponding feature space to obtain a feature map, the feature deformable alignment unit aligns the non-reference image feature map with the reference image feature map to generate a corresponding aligned feature map, so as to eliminate and weaken the influence of the spatial information of the dynamic scene on the fusion effect, and the feature fusion reconstruction unit fuses and reconstructs the aligned feature maps corresponding to different exposure levels to obtain a high dynamic range image;

[0042] An observation sequence image of a space target in a dynamic observation scene is acquired, and a high dynamic range image of the space target is generated according to the observation sequence image using a trained space target image reconstruction network.

[0043] The above-mentioned method, device and apparatus for reconstructing the image of a space target in a dynamic observation scene divides the sequence images in the dynamic sequence multi-exposure data set into a plurality of low dynamic range sub-image sequences in groups of three, each low dynamic range sub-image sequence includes three sequence images with different exposure levels from low to high and different postures of the space target, and generates a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences, and trains the space target image reconstruction network using the plurality of groups of low dynamic range sub-image sequences and the corresponding static sub-image sequences to obtain a trained space target image reconstruction network, which includes a feature extraction unit, a feature deformable alignment unit and a feature fusion reconstruction unit, and uses the trained space target image reconstruction network to generate a high dynamic range image of the space target from the observation sequence images of the space target in the dynamic observation scene. This method can realize efficient HDR image reconstruction in dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flowchart of a method for reconstructing a space target image in a dynamic observation scene in one embodiment;

[0045] Figure 2 A schematic diagram of a multi-exposure image sequence simulation step of a space target in one embodiment;

[0046] Figure 3 A schematic diagram of a portion of sequence images in dynamic sequence multi-exposure data in one embodiment;

[0047] Figure 4 A schematic diagram of the structure of a space target image reconstruction network in one embodiment;

[0048] Figure 5 is a schematic diagram of a low dynamic range sub-image sequence and a static sub-image sequence in one embodiment;

[0049] Figure 6 A schematic diagram showing the comparison of target HDR images obtained by using this method and other methods in an experiment;

[0050] Figure 7 A schematic diagram showing a comparison of SIFT feature point matching effects in a target HDR image obtained by using the present method and other methods in an embodiment;

[0051] Figure 8 It is a structural block diagram of a device for reconstructing a space target image in a dynamic observation scene in one embodiment;

[0052] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] In the existing space-based optical imaging technology, the traditional dynamic multi-exposure fusion method based on removing motion pixels is difficult to handle complex target overlapping areas and will lose a lot of effective information. At the same time, due to the certain particularity of the subsequent tasks of space target situation awareness technology, such as three-dimensional reconstruction and attitude estimation, it is very sensitive to the changes in the multi-view geometric relationship of the observed target, and the traditional method based on registration is prone to changes in the observed target representation (such as the angle of the sailboard, etc.). Further, when using deep learning to reconstruct space target images, a method using convolutional neural network to solve the ghost problem in dynamic scene multi-exposure fusion is proposed, but most of them use supervised learning mode and need true value HDR images as guidance. Unsupervised methods based on GAN networks may produce pseudo textures during reconstruction, while unsupervised methods based on CNN lack modules for processing a large number of motion pixels in the learning framework. When fusion reconstruction is performed, the feature map is usually directly added, which is not suitable for the situation where there are complex overlapping areas between air target image frames. At the same time, the camera and the still background of the existing deep learning multi-exposure general dataset are in a relatively static state, which is very different from the imaging scene of the space target image sequence, and the open source space target dataset is only generated based on a single exposure condition. Both cannot meet the actual application needs of the multi-exposure fusion problem of space target image sequences without true value in the real scene of space situational awareness.

[0055] In response to the above problems, Figure 1 As shown, a method for reconstructing a space target image in a dynamic observation scene is provided, which specifically includes the following steps:

[0056] Step S100: acquiring a dynamic sequence multi-exposure dataset of a space target, wherein the dynamic sequence multi-exposure dataset includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures.

[0057] Step S110, dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of groups of low dynamic range sub-image sequences in groups of three, each of the low dynamic range sub-image sequences including three sequence images with exposure levels from low to high and different spatial target postures, and generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences.

[0058] Step S120, using multiple groups of low dynamic range sub-image sequences and corresponding static sub-image sequences to train the space target image reconstruction network to obtain a trained space target image reconstruction network, the space target image reconstruction network includes a feature extraction unit, a feature deformable alignment unit and a feature fusion reconstruction unit, wherein the feature extraction unit transforms each sequence image from the image domain to the corresponding feature space to obtain a feature map, the feature deformable alignment unit aligns the non-reference image feature map with the reference image feature map to generate a corresponding aligned feature map to eliminate and weaken the influence of the spatial information of the dynamic scene on the fusion effect, and the feature fusion reconstruction unit fuses and reconstructs the aligned feature maps corresponding to different exposure levels to obtain a high dynamic range image.

[0059] Step S130, obtaining an observation sequence image of the space target in the dynamic observation scene, and using the trained space target image reconstruction network to generate a high dynamic range image of the space target according to the observation sequence image.

[0060] In this embodiment, an unsupervised multi-exposure fusion method for space-based optical images of space targets in dynamic observation scenarios is proposed, which achieves efficient HDR image reconstruction in dynamic scenarios while better preserving the original multi-view geometric relationship of the observed target. While constructing an unsupervised learning framework suitable for multi-exposure fusion of space-based optical images of space targets, a new rolling exposure dataset of space targets is proposed, and blender is used to simulate the observation imaging of space targets when orbits intersect, so as to solve the problem of too little training data when training neural networks.

[0061] In step S100 and step S110, it is a process of constructing a training data set, wherein the space target is an artificial satellite.

[0062] In step S100, when constructing a dynamic sequence multi-exposure data set: three-dimensional models of different types of space targets are obtained, and simulation is performed based on the three-dimensional models. By adjusting the posture of the three-dimensional model at the initial position and performing rolling exposure, multiple sequence images corresponding to the same space target with multiple exposures in different postures are obtained.

[0063] Considering that it is difficult to obtain actual observation images of space targets, most space target image datasets are generated through software rendering or semi-physical simulation, such as the BUAA-SID dataset and the SPEED+ dataset. However, the existing simulation datasets do not consider the actual situation of imaging the observation target through the rolling exposure mechanism when the orbits intersect naturally. At the same time, the illumination conditions of the observation target vary in different areas of the orbit, and the reflectivity of the materials of different components of the observation target also varies.

[0064] Therefore, in this embodiment, a multi-exposure image dataset of a space target for orbital intersection observation is created by simulation. The simulation steps for generating the space target image are as follows: Figure 2 shown.

[0065] Since the intersection time between the imaging platform and the observed target is short in actual situations, the posture of the space target does not change much from the beginning to the end of the observation. In order to obtain as much training data as possible, the initial position of the space target is adjusted to obtain a multi-exposure image sequence of the same target with different postures. This dataset is named "Multi-exposure Spacecraft Image Dataset" (MESDataset), which is a dynamic sequence multi-exposure dataset. Some images, such as Figure 3 shown.

[0066] Furthermore, in order to adapt to the subsequent neural network training, the space target is reconstructed in the neural network according to three sequence images of the same space target at different exposure levels and different postures. In step S110, the sequence images in the dynamic sequence multi-exposure data are divided into a plurality of low dynamic range sub-image sequences in groups of three. Each low dynamic range sub-image sequence includes three sequence images with different exposure levels from low to high and different postures of the space target. Figure 3 The multiple groups of low dynamic range sub-image sequences shown, for example, three low, medium and high exposure images located in the same row at the initial moment of the intersection observation, further show that the postures of the space targets in these three images are also different.

[0067] In one embodiment, in order to better train the neural network, the dynamic sequence multi-exposure data set includes 120 rolling exposure image sequences of 4 groups of different postures of 12 space target models, a total of 48 groups of image sequences, including 1920 groups of multi-exposure pairs, that is, 5760 images. In this embodiment, in order to subsequently train the neural network, a corresponding static sub-image sequence is also generated according to each group of low dynamic range sub-image sequences to calculate the alignment loss function, which will be introduced later.

[0068] Specifically, the posture of the space target in the sequence image in the middle of the low dynamic range sub-image sequence is used as the posture of the space target in the three sequence images in the corresponding static sub-image sequence, and at the same time, the three sequence images in the static sub-image sequence retain the exposure level of the three sequence images in the corresponding low dynamic range sub-image sequence. In this way, each group of low dynamic range sub-image sequences will correspond to a group of static sub-image sequences.

[0069] In this application, an unsupervised learning neural network framework is also proposed, namely, a space target image reconstruction network, whose structure is as follows: Figure 4As shown in the figure. From a functional perspective, this network consists of three parts: feature extraction unit, feature deformable alignment unit and feature fusion reconstruction unit. The feature extraction unit transforms the image from the image domain to the corresponding feature space. The feature deformable alignment unit is responsible for aligning the extracted feature maps one by one to eliminate and weaken the influence of the spatial information of the dynamic scene on the fusion effect. Finally, the feature fusion reconstruction unit realizes the information fusion and reconstruction of the alignment features corresponding to different exposures and outputs the final HDR image.

[0070] In this embodiment, in the feature extraction unit: the low dynamic range sub-image sequence is corrected according to the medium exposure time image by using gamma transformation to obtain a low dynamic range sub-image sequence in the high dynamic range image domain; then, each image in the low dynamic range sub-image sequence in the high dynamic range image domain is connected into a multi-channel low dynamic range sub-image sequence according to the channel dimension; Stride convolution is introduced into the multi-channel low dynamic range sub-image sequence to perform pyramid feature extraction, and a feature map corresponding to each sequence image is obtained.

[0071] Specifically, in general, although the LDR image sequence can distinguish the exposure degree and saturation area of ​​the image well, its measurement of the moving area is limited. The image in the linear space of the HDR domain has an absolute advantage in identifying uncalibrated and moving areas because it simulates the real illumination field and places the scene of the image sequence in a unified measurement. Therefore, the input dynamic scene LDR rolling exposure space target space-based optical image ordered sequence, that is, the low dynamic range sub-image sequence Apply gamma shift correction using the following formula:

[0072]

[0073] The corresponding image in the HDR domain is obtained by formula (1) Among them, t i represents the exposure time of the i-th image, and γ represents the gamma correction parameter.

[0074] Then, each image input in the image sequence is concatenated into a 6-channel input sequence according to the channel dimension Where [·;·] means connecting operators along the channel dimension.

[0075] Furthermore, for the input sequence Stride convolution is introduced for pyramid feature extraction, which is converted into a 64-channel feature map, expressed as:

[0076]

[0077] In this embodiment, in the feature deformable alignment unit: the feature map corresponding to the middle sequence image of the low dynamic range sub-image sequence is used as the reference feature map, and the other two feature maps are used as non-reference feature maps. Based on the reference feature map, the bias information of the other two non-reference feature maps is calculated respectively, and the non-reference feature map and the corresponding bias information are subjected to deformable convolution to obtain a spatially aligned aligned feature map, and the reference feature map is directly used as the aligned feature map.

[0078] The feature deformable alignment unit in this method introduces deformable convolution to adapt to the complex posture changes of the observed target in the empty image, and to retain its multi-view geometric information consistent with the reference image as much as possible during HDR reconstruction. The deformable convolution has the advantage of arbitrary sampling, and can realize the correction of basic geometric deformations such as translation, rotation, scaling, beveling and perspective, which is suitable for solving the problem of mismatching of the observed target image sequence in the space target monitoring scene. At the same time, its anisotropic bias information can fit a dense optical flow field, and adjust the direction and amplitude of the object deformation through the direction and amplitude of the bias to solve the non-rigid deformation situation.

[0079] Furthermore, the feature deformable alignment unit is constructed by using the non-reference feature E i Same as reference feature E r (i.e. E 2 ) together with the calculation of the bias information O i , and send the above information (i.e., non-reference feature map and corresponding bias information) to the corresponding deformable convolution dc i In (·), the spatially aligned feature map A is calculated. i :

[0080]

[0081] In formula (3), p is the position in the feature map, ω j is the weight in the convolution kernel Ω, Δp j and Δm j denote the learnable bias and modulation strength of the jth position in the convolution kernel Ω, respectively.

[0082] In this embodiment, in the feature fusion and reconstruction unit: after merging the aligned feature maps, a DenseNet network is used to perform feature fusion and reconstruction to obtain a high dynamic range image.

[0083] Furthermore, through the feature extraction unit and the feature deformable alignment unit, the feature map corresponding to each exposure length is obtained. For the dynamic information of this fixed exposure length lens, the texture information contained in the adjacent frames is also rich and important. Considering that the typical exposure gear is 3, the concatenation operation is used as the feature merging method to obtain the merged feature E c =[A1 ; E r ; A 3 ], and send it to DenseNet for fusion reconstruction to obtain the final HDR image, which is expressed as:

[0084] H = DenseNet (E c ) (4)

[0085] In this embodiment, when training the space target image reconstruction network, the loss functions used include a structural similarity index loss function, a mean square error loss function, an alignment loss function, and a feature point extraction loss function, which are expressed as:

[0086]

[0087] In formula (5), θ represents the parameters in the network, D represents the training data set, and α, β, λ, and μ are balance parameters greater than 0.

[0088] Due to the lack of true value images, SSIM (Structural Similarity Index Metric, SSIM) focuses on the changes in contrast and structure, and MSE (Mean Squared Error, MSE) focuses on the constraints of intensity distribution differences. Therefore, the similarity constraints of the fused image and the input image are implemented from two aspects: structural similarity and intensity distribution:

[0089]

[0090]

[0091] In formula (6) and formula (7), ω i The calculation of the information extraction and information preservation degree is performed using the VGG network, and then normalized by the softmax function. Can Figure 4 The weight calculation part in is expressed as f(·), and the weight The overall calculation process is expressed as:

[0092]

[0093] Furthermore, this method also proposes a corresponding feature alignment loss function for the feature deformable alignment unit, introduces a deformable convolution to fit the complex geometric deformation of the observed target, and does not change the representation of the observed target in the reference image while extracting the image exposure information. At the same time, a loss function based on the SIFT operator is designed to optimize the fusion result of the network, obtain better feature extraction effects, and make it more suitable for tasks such as posture estimation in space detection systems.

[0094] In this embodiment, in order to achieve better registration effect of spatial target image information and retain its multi-view geometric information as strictly as possible, a feature alignment loss function is also proposed. The function is calculated by using an encoder to extract features from each sequence image in the static sub-image sequence to obtain a feature map, and the corresponding aligned feature map.

[0095] Specifically, if it is known that the input dynamic LDR image sequence, i.e., the low dynamic range sub-image sequence {I 1 ,I 2 ,I 3 The corresponding static LDR image sequence is the static sub-image sequence Right now, The observed target posture and I 2 The image is the same, and the exposure level is the same i Therefore, we only need to limit the image features in dynamic scenes. After correction by the feature deformable alignment unit, its output feature The features extracted by the encoder from the corresponding static image sequence As close as possible to achieve feature alignment:

[0096]

[0097] However, for existing dynamic datasets, a large amount of data does not contain corresponding static scene data. Therefore, most of them propose a method of using reference images for exposure conversion to generate pseudo-static scene image sequences. And use self-supervision to learn and update network parameters. However, when preparing MESDataset, this method can directly produce a static data set to ensure the consistency of image information and exposure level. Take the rendered image of the Dawn model as an example, as shown in Figure 5, where Figure 5 (a)-(c) is a sequence of low dynamic range sub-images. Figure 5 (d)-(f) are the corresponding static sub-image sequences.

[0098] Therefore, the alignment loss function can be expressed as:

[0099]

[0100] In this embodiment, a feature point extraction loss function is also proposed to optimize the computer processing effect of the HDR reconstruction results in tasks such as subsequent posture estimation in space detection. The loss function is mainly constructed based on the low dynamic range sub-image sequence and the corresponding reconstructed high dynamic range image using the SIFT (Scale Invariant Feature Transform) operator.

[0101] The IFT operator is an algorithm used for feature extraction in image processing and computer vision. It can detect key points in different scale spaces and is invariant to image rotation, scaling, and brightness changes. The SIFT operator is used to detect key points for each exposure level image in the network. These key points should have unique local features and be insensitive to exposure changes.

[0102] For each set of dynamic LDR image sequences {I 1 ,I 2 ,I 3} That is, the low dynamic range sub-image sequence, and the corresponding reconstructed HDR image H to find its SIFT feature point set and number, using the following formula:

[0103]

[0104] K H =SIFT(H) (12)

[0105]

[0106] n H =num(K H ) (14)

[0107] In formulas (11) to (14), SIFT(·) represents the SIFT function for extracting image feature points. represents the set of feature points in the i-th LDR image, K H Represents the set of feature points in the HDR image. num(·) is used to find the number of feature points in the set. Represents the number of feature points in the i-th LDR image, n H Indicates the number of feature points in the HDR image.

[0108] Furthermore, in order to make the reconstructed HDR image of the space target have better computer processing effect in the subsequent space monitoring task, the fused image should contain as many non-repeated feature points as possible. Therefore, the intersection of the feature point set of the LDR image is calculated and repeated detection is performed, and the number is compared with the feature point set of the HDR image that has undergone repeated detection. Therefore, the constructed feature point extraction loss function is as follows:

[0109]

[0110] In formula (15), rd(·) represents the repeatability detection of the feature point set, It means the set with the largest number of feature points retained after repeatability detection of each feature point set in the LDR sequence. From the numerical calculation, it can be understood that when the number of non-repeating feature points in the HDR image is the same as the number of non-repeating feature points in the LDR image, When the number of non-repeated feature points in the HDR image is the same as the number of all non-repeated feature points in the LDR image, That is, the fewer the number of non-repeating feature points of the HDR image, the closer the value of the loss function is to 1, and the more non-repeating feature points of the HDR image, the closer the value of the loss function is to 0.

[0111] In this paper, the effectiveness of the proposed method is also demonstrated through experimental results. In the experiment, 16 rolling exposure observation sequences of 4 spatial targets in the MES dataset are used, totaling 640 sets of multi-exposure image pairs with a size of 800*800.

[0112] Figure 6 and Figure 7 The fusion results of the proposed method and the comparison algorithm on the MESDataset test set and the SIFT feature point extraction effect are respectively demonstrated.

[0113] from Figure 6 It can be seen that when multi-exposure fusion is performed on the space-based optical image sequence of space targets on the MES test set, except for the SPD-MEF algorithm and the method proposed in this paper, the HDR reconstruction results of other comparison algorithms have serious artifacts at the edge of the sailboard and the important parts in the middle. However, the fusion results of the SPD-MEF algorithm have uneven and incoherent brightness changes at the sailboard and the main body, while the fusion results of the method in this paper retain higher contrast and texture details.

[0114] from Figure 7 It can be seen that compared with the comparison algorithm, the proposed method can obtain better feature matching effect, especially at the outer sailboard with large posture changes, more feature points can be extracted, while retaining the subjective visual effect, it can provide better computer processing results for subsequent space situation awareness tasks.

[0115] In the above-mentioned space target image reconstruction method in dynamic observation scenes, a multi-exposure fusion method of space target space-based optical images in dynamic monitoring scenes based on unsupervised learning is proposed, and a space target rolling exposure dataset is proposed. This method can achieve high-quality HDR space target image reconstruction in dynamic scenes. By introducing deformable convolution in the feature alignment module and constructing feature alignment loss function and feature point extraction loss function, the original multi-view geometric relationship of the observed target can be effectively retained, and better feature extraction effect can be obtained to serve subsequent tasks such as posture estimation in the space situation awareness system.

[0116] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0117] In one embodiment, Figure 8 As shown, a space target image reconstruction device for dynamic observation scenes is provided, comprising: a dynamic sequence multi-exposure data acquisition module 200, a static sub-image sequence generation module 210, a space target image reconstruction network training module 220 and a dynamic target image reconstruction module 230, wherein:

[0118] A dynamic sequence multi-exposure data set acquisition module 200 is used to acquire a dynamic sequence multi-exposure data set of a space target, wherein the dynamic sequence multi-exposure data set includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures;

[0119] The static sub-image sequence generating module 210 is used to divide the sequence images in the dynamic sequence multi-exposure data set into a plurality of low-dynamic sub-image sequences in groups of three, each of which includes three sequence images with different spatial target postures from low to high exposure levels, and generate a corresponding static sub-image sequence according to each group of low-dynamic sub-image sequences;

[0120] The space target image reconstruction network training module 220 is used to divide the sequence images in the dynamic sequence multi-exposure data set into a plurality of low dynamic range sub-image sequences in groups of three, each of which includes three sequence images with different exposure levels from low to high and different postures of the space target, and generate a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences;

[0121] The dynamic target image reconstruction module 230 is used to obtain the observation sequence images of the space target in the dynamic observation scene, and generate a high dynamic range image of the space target according to the observation sequence images using the trained space target image reconstruction network.

[0122] Regarding the specific limitations of the space target image reconstruction device facing a dynamic observation scene, please refer to the limitations of the space target image reconstruction method facing a dynamic observation scene in the above text, which will not be repeated here. Each module in the above-mentioned space target image reconstruction device facing a dynamic observation scene can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0123] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for reconstructing a space target image in a dynamic observation scene is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0124] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0126] Acquire a dynamic sequence multi-exposure dataset of a space target, wherein the dynamic sequence multi-exposure dataset includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures;

[0127] Sequentially dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of groups of low dynamic range sub-image sequences in groups of three, each of the low dynamic range sub-image sequences including three sequence images with exposure levels from low to high and with different spatial target postures, and generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences;

[0128] A space target image reconstruction network is trained using multiple groups of low dynamic range sub-image sequences and corresponding static sub-image sequences to obtain a trained space target image reconstruction network, wherein the space target image reconstruction network includes a feature extraction unit, a feature deformable alignment unit, and a feature fusion reconstruction unit, wherein the feature extraction unit transforms each of the sequence images from an image domain to a corresponding feature space to obtain a feature map, the feature deformable alignment unit aligns the non-reference image feature map with the reference image feature map to generate a corresponding aligned feature map, so as to eliminate and weaken the influence of the spatial information of the dynamic scene on the fusion effect, and the feature fusion reconstruction unit fuses and reconstructs the aligned feature maps corresponding to different exposure levels to obtain a high dynamic range image;

[0129] An observation sequence image of a space target in a dynamic observation scene is acquired, and a high dynamic range image of the space target is generated according to the observation sequence image using a trained space target image reconstruction network.

[0130] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0131] Acquire a dynamic sequence multi-exposure dataset of a space target, wherein the dynamic sequence multi-exposure dataset includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures;

[0132] Sequentially dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of groups of low dynamic range sub-image sequences in groups of three, each of the low dynamic range sub-image sequences including three sequence images with exposure levels from low to high and with different spatial target postures, and generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences;

[0133] A space target image reconstruction network is trained using multiple groups of low dynamic range sub-image sequences and corresponding static sub-image sequences to obtain a trained space target image reconstruction network, wherein the space target image reconstruction network includes a feature extraction unit, a feature deformable alignment unit, and a feature fusion reconstruction unit, wherein the feature extraction unit transforms each of the sequence images from an image domain to a corresponding feature space to obtain a feature map, the feature deformable alignment unit aligns the non-reference image feature map with the reference image feature map to generate a corresponding aligned feature map, so as to eliminate and weaken the influence of the spatial information of the dynamic scene on the fusion effect, and the feature fusion reconstruction unit fuses and reconstructs the aligned feature maps corresponding to different exposure levels to obtain a high dynamic range image;

[0134] An observation sequence image of a space target in a dynamic observation scene is acquired, and a high dynamic range image of the space target is generated according to the observation sequence image using a trained space target image reconstruction network.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0136] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for reconstructing a space target image in a dynamic observation scene, characterized in that: The method comprises: Acquire a dynamic sequence multi-exposure dataset of a space target, wherein the dynamic sequence multi-exposure dataset includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures; Sequentially dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of groups of low dynamic range sub-image sequences in groups of three, each of the low dynamic range sub-image sequences including three sequence images with exposure levels from low to high and with different spatial target postures, and generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences; A space target image reconstruction network is trained using multiple groups of low dynamic range sub-image sequences and corresponding static sub-image sequences to obtain a trained space target image reconstruction network, wherein the space target image reconstruction network includes a feature extraction unit, a feature deformable alignment unit, and a feature fusion reconstruction unit, wherein the feature extraction unit transforms each of the sequence images from an image domain to a corresponding feature space to obtain a feature map, the feature deformable alignment unit aligns the non-reference image feature map with the reference image feature map to generate a corresponding aligned feature map, so as to eliminate and weaken the influence of the spatial information of the dynamic scene on the fusion effect, and the feature fusion reconstruction unit fuses and reconstructs the aligned feature maps corresponding to different exposure levels to obtain a high dynamic range image; An observation sequence image of a space target in a dynamic observation scene is acquired, and a high dynamic range image of the space target is generated according to the observation sequence image using a trained space target image reconstruction network.

2. The space target image reconstruction method according to claim 1, characterized in that: When constructing the dynamic sequence multi-exposure dataset: Three-dimensional models of different types of space targets are obtained, and simulation is performed based on the three-dimensional models. By adjusting the posture of the three-dimensional model at the initial position and performing rolling exposure rendering, a multi-exposure image sequence corresponding to the same space target at different initial postures is obtained.

3. The space target image reconstruction method according to claim 2, characterized in that: Generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences includes: The posture of the space target in the sequence image located in the middle of the low dynamic range sub-image sequence is used as the posture of the space target in the three sequence images in the corresponding static sub-image sequence. At the same time, the three sequence images in the static sub-image sequence retain the exposure degree of the three sequence images in the corresponding low dynamic range sub-image sequence.

4. The space target image reconstruction method according to claim 3, characterized in that: In the feature extraction unit: Correcting the low dynamic range sub-image sequence by gamma transformation according to the sequence image located in the middle, to obtain a low dynamic range sub-image sequence in a high dynamic range image domain; Connect each image in the low dynamic range sub-image sequence in the high dynamic range image domain into a multi-channel low dynamic range sub-image sequence according to the channel dimension; Stride convolution is introduced into the multi-channel low dynamic range sub-image sequence to perform pyramid feature extraction, so as to obtain feature maps corresponding to each sequence image.

5. The space target image reconstruction method according to claim 4, characterized in that: In the feature deformable alignment unit: Taking the feature map corresponding to the medium exposure time image in the low dynamic range sub-image sequence as the reference feature map, and the other two feature maps as the non-reference feature maps; Based on the reference feature map, the bias information of the other two non-reference feature maps is calculated respectively, and the non-reference feature map and the corresponding bias information are subjected to deformable convolution to obtain a spatially aligned aligned feature map; The reference feature map is directly used as the aligned feature map.

6. The space target image reconstruction method according to claim 5, characterized in that: In the feature fusion reconstruction unit: After merging the aligned feature maps, a DenseNet network is used to perform feature fusion and reconstruction to obtain the high dynamic range image.

7. The space target image reconstruction method according to any one of claims 1 to 6, characterized in that: When training the space target image reconstruction network, the loss functions used include a structural similarity loss function, a mean square error loss function, an alignment loss function, and a feature point extraction loss function; The alignment loss function is calculated by using an encoder to extract features from each low dynamic range image in the static sub-image sequence and obtaining a feature map, and a corresponding aligned feature map; The feature point extraction loss function is calculated by using a scale-invariant feature transformation operator on the low dynamic sub-image sequence and the corresponding reconstructed high dynamic range image.

8. A device for reconstructing images of space targets in dynamic observation scenes, characterized in that: The device comprises: A dynamic sequence multi-exposure data set acquisition module is used to acquire a dynamic sequence multi-exposure data set of a space target, wherein the dynamic sequence multi-exposure data set includes a plurality of orbital intersection observation image sequences of rolling exposure of a plurality of space targets at different initial postures; a static sub-image sequence generation module, for dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of low-dynamic sub-image sequences in groups of three, each of which includes three sequence images with different spatial target postures and exposure levels from low to high, and generating a corresponding static sub-image sequence according to each group of low-dynamic sub-image sequences; a space target image reconstruction network training module, for dividing the sequence images in the dynamic sequence multi-exposure data set into a plurality of groups of low dynamic range sub-image sequences in groups of three, each of the low dynamic range sub-image sequences including three sequence images with different exposure levels from low to high and with different postures of the space target, and generating a corresponding static sub-image sequence according to each group of low dynamic range sub-image sequences; The dynamic target image reconstruction module is used to obtain the observation sequence images of the space target in the dynamic observation scene, and use the trained space target image reconstruction network to generate a high dynamic range image of the space target according to the observation sequence images.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images