Event camera based rocket launch image reconstruction method and system

Abstract

The application discloses a rocket launch image reconstruction method and system based on an event camera, and belongs to the technical field of image reconstruction, and comprises the following steps: acquiring a rocket launch image sequence and an event stream; for the image captured at a moment, converting the image into a grayscale image, generating an overexposure area mask based on a grayscale threshold; counting the total number of events at each pixel within a time window to obtain an event density map, generating a binary image based on a high-frequency event threshold, and performing a close operation to obtain a high-frequency event mask; summing the intersection to obtain a plume mask; separating the plume image and the rocket body and background image, and separating the plume event stream and the rocket body and background event stream; reconstructing the plume region according to the plume event stream, reconstructing the rocket body and background region according to the rocket body and background image and the rocket body and background event stream, fusing the two reconstructed images to obtain a rocket launch image. The application can improve the reconstruction quality of the rocket launch image.

Description

Technical Field

[0001] This invention belongs to the field of image reconstruction technology, and more specifically, relates to a method and system for reconstructing rocket launch images based on an event camera. Background Technology

[0002] Rocket launches have an extremely high dynamic range (greater than or equal to 120 dB), and image reconstruction of rocket launches is crucial for safety assessment, fault diagnosis, process retrospection, and performance optimization during the launch process. Currently, imaging methods for rocket launches primarily rely on frame-based imaging. The extremely bright exhaust plume during a rocket launch can easily lead to localized overexposure in the image, resulting in the loss of information in the exhaust plume area. Conversely, reducing the exposure time can lead to excessive darkness in the rocket body area, especially in nighttime launch scenarios where low exposure times render the area outside the exhaust plume completely invisible.

[0003] Traditional frame cameras struggle to achieve high-precision global imaging within the ultra-high dynamic range of rocket launches. Event cameras, a novel neuromorphic sensor with asynchronous pixel triggering, offer high temporal resolution and a wide dynamic range, making them ideal for capturing the high-speed, transient dynamic characteristics of scenes with both extremely bright and dark areas during launch. However, the event stream output by event cameras lacks absolute intensity information, failing to provide static appearance and texture details of the scene, resulting in poor image readability. Therefore, a dual-mode fusion method using events and images can be employed to leverage their complementary strengths, thereby enabling image reconstruction of ultra-high dynamic rocket launches.

[0004] Existing techniques typically use exposure masks to distinguish underexposed / overexposed areas to reconstruct edge textures (Neur Img, HDRev, EHDRI-Net, etc.), or rely on event-assisted image color reconstruction (EvHDR-GS, CEvHDR, etc.). These existing reconstruction methods are primarily designed for reconstructing high-dynamic-range natural scene images; however, for ultra-high-dynamic scenes such as rocket launches, the quality of the fused reconstruction results is limited. Summary of the Invention

[0005] In view of the shortcomings of existing technologies and the need for improvement, this invention provides a method and system for rocket launch image reconstruction based on event cameras, with the aim of improving the reconstruction quality of rocket launch images.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for reconstructing rocket launch images based on an event camera is provided, comprising:

[0007] Acquire rocket launch image sequences and event streams captured by a coaxial system of frame cameras and event cameras;

[0008] For time Captured images After converting it to a grayscale image, the positions with grayscale values ​​greater than a preset grayscale threshold are set to 1, and the remaining positions are set to 0, thus obtaining the overexposed area mask. ;

[0009] In the time window The total number of events occurring at each pixel location is counted to obtain the event density map. Positions where the total number of events exceeds a preset event threshold are set to 1, and all other positions are set to 0. After obtaining a binary image of high-frequency events, a closing operation is performed to obtain a high-frequency event mask. ; , , Exposure time;

[0010] Please expose the region mask. and high-frequency event mask The intersection of these points yields the tail flame mask. ;

[0011] According to the tail flame mask From the image Separate the exhaust plume image and the rocket body and background image from the exhaust plume mask. From the time window Separate the tail flame event stream and the arrow body and background event stream from the event stream within the event stream;

[0012] The tail flame region is reconstructed based on the tail flame event stream to obtain a tail flame reconstruction image; the arrow body and background region are reconstructed based on the arrow body and background image and the arrow body and background event stream to obtain an arrow body and background reconstruction image.

[0013] According to the tail flame mask By fusing the reconstructed image of the exhaust plume and the arrow body with the reconstructed image of the background, the time can be obtained. Images of rocket launches.

[0014] Furthermore, the exhaust region is reconstructed based on the exhaust event flow to obtain a reconstructed exhaust image, including:

[0015] Input the exhaust event stream into the trained exhaust reconstruction network to obtain the exhaust reconstruction image;

[0016] The tail flame reconstruction network includes: a brightness mapping sub-network, a first deformable convolutional network, a first fusion module, and a coloring network;

[0017] A brightness mapping sub-network is used to accumulate events and suppress brightness in the exhaust event stream to obtain a brightness map of the exhaust region.

[0018] Deformable convolutional networks are used to extract detailed texture feature maps of the exhaust region from exhaust event streams.

[0019] The first fusion module is used to fuse the brightness map and detail texture feature map of the exhaust flame region to obtain a reconstructed grayscale image of the exhaust flame region.

[0020] A coloring network is used to color the reconstructed grayscale image of the exhaust flame region to obtain the exhaust flame reconstructed image.

[0021] Furthermore, the formula for calculating brightness suppression is as follows:

[0022]

[0023] in, Indicates time The cumulative results of the events are normalized to The result afterwards, For gamma correction function, This indicates the brightness after suppression.

[0024] Furthermore, based on the arrow body and background image and the arrow body and background event stream, the arrow body and background region is reconstructed to obtain a reconstructed image of the arrow body and background, including:

[0025] Input the arrow body and background image and the arrow body and background event stream into the trained arrow body and background reconstruction network to obtain the arrow body and background reconstructed image;

[0026] The rocket body and background reconstruction network includes: an illumination conversion module, a convolutional layer, a second deformable convolutional network, and a second fusion module;

[0027] The illumination conversion module is used to calculate the time difference between each pixel in the arrow body and the background image. The true illuminance value is obtained, thereby converting the arrow body and background image into a true illuminance map;

[0028] Convolutional layers are used to extract the brightness features of the arrow body and the background from the real illumination map;

[0029] The second deformable convolutional network is used to extract edge texture features of the arrow body and background from the event stream of the arrow body and background;

[0030] The second fusion module is used to fuse the brightness features and edge texture features of the arrow body and the background to obtain a reconstructed image of the arrow body and the background.

[0031] Furthermore, for the pixels in the arrow body and background image At any moment True illuminance value The calculation methods include:

[0032] S1: Select an unexposed image as the initial image. ;

[0033] S2: Calculate the initial image medium pixel Illuminance value As a pixel Initial illuminance value:

[0034]

[0035] in, Represents the initial image medium pixel Pixel value at; For the inverse camera response function of the frame camera;

[0036] S3: Calculate pixels through event overlay At any moment True illuminance value :

[0037]

[0038] in, The preset event logarithmic brightness threshold; Indicates at time , pixels The polarity of events at that location.

[0039] Furthermore,

[0040]

[0041] in, This is a preset constant.

[0042] Furthermore, based on the exhaust mask The fusion of the reconstructed exhaust image and the arrow body image with the reconstructed background image is achieved by the third fusion module. Furthermore, the training loss function of the third fusion module includes brightness consistency constraints. Brightness consistency constraint The calculation expression is as follows:

[0043]

[0044] in, This is the left boundary region. This is the right boundary region. The average pixel value in the region; The number of boundary regions selected.

[0045] According to another aspect of the present invention, a computer-readable storage medium is provided, including a stored computer program that, when executed by a processor, implements the rocket launch image reconstruction method based on an event camera provided by the present invention.

[0046] According to another aspect of the present invention, an electronic device is provided, comprising:

[0047] A computer-readable storage medium for storing computer programs;

[0048] And a processor for reading a computer program stored in a computer-readable storage medium to implement the rocket launch image reconstruction method based on an event camera provided by the present invention.

[0049] According to another aspect of the present invention, a rocket launch image reconstruction system based on an event camera is provided, comprising: a co-optical axis acquisition system formed by a frame camera and an event camera being set together on the same optical axis, and an electronic device provided by the present invention.

[0050] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0051] (1) The present invention found that in the ultra-high dynamic rocket launch scene, the tail flame part is not only always in a state of complete overexposure, but its brightness is far greater than that of the rocket body and the background part. At the same time, the tail flame part of the image cannot provide information, while the rocket body and the background part of the image are rich in information. Therefore, it is necessary to process the two parts separately in different ways. Based on this, the present invention generates a mask for the overexposure area based on the frame image, generates a mask for the high-frequency event area based on the event information, and uses the intersection of the two area masks as the tail flame mask. Thus, the tail flame part can be accurately separated from other parts. Then, the tail flame part and the rocket body and the background part obtained by separation are reconstructed separately, and the reconstruction results of the two parts are merged to achieve high-quality reconstruction of rocket launch images.

[0052] (2) The present invention found that the initial brightness of the tail flame during the rocket launch process is high, and then it turns to even higher brightness. If most of the tail flame is too bright, it will lead to low contrast of the reconstruction result and poor reconstruction quality. Based on this, when reconstructing the tail flame part, the present invention performs brightness suppression on the basis of obtaining brightness information through event accumulation. The brightness map of the tail flame area generated in the end is controlled in range and has distinct layers. It can maintain a reasonable distribution of global brightness and avoid distortion due to excessive brightness of local pixels, thereby further improving the quality of the reconstructed rocket launch image.

[0053] (3) The present invention found that there are small overexposed areas between the arrow body and the background, and the image can provide rich information. Based on this, when reconstructing the arrow body and the background, the present invention takes the image as the main factor, selects the initial image and calculates the illuminance value, obtains the real illuminance image by superimposing events, and then combines the detailed texture information of the events to reconstruct the arrow body and the background with balanced brightness and preserved details.

[0054] (4) In the process of merging the tail flame part to reconstruct the image and the arrow body and background part to reconstruct the image, the present invention adds brightness consistency constraints to avoid excessive distribution differences on both sides of the edge of the merged area. The final synthesized image can be improved in terms of luminous consistency and structural continuity, and avoids the appearance of unnatural boundaries. Attached Figure Description

[0055] Figure 1 A flowchart of a rocket launch image reconstruction method based on an event camera provided in an embodiment of the present invention.

[0056] Figure 2 This is a schematic diagram of a rocket launch image reconstruction method provided in an embodiment of the present invention.

[0057] Figure 3 This is a schematic diagram of a frame image and an overexposure mask provided in an embodiment of the present invention.

[0058] Figure 4 This is a schematic diagram of a high-frequency event mask provided in an embodiment of the present invention.

[0059] Figure 5 A schematic diagram of the exhaust mask provided in an embodiment of the present invention.

[0060] Figure 6 This is a schematic diagram of the result of tail flame reconstruction based on event data provided in an embodiment of the present invention.

[0061] Figure 7 This is a schematic diagram illustrating the result of image-driven reconstruction of the arrow body and background, provided in an embodiment of the present invention.

[0062] Figure 8 This is a schematic diagram showing the result of the rocket launch image reconstruction method provided in an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0064] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0065] To leverage the strengths of both event and image modalities for image reconstruction of ultra-high dynamic range (UHR) rocket launch scenes and improve imaging quality, this invention provides a rocket launch image reconstruction method and system based on an event camera. The overall concept involves accurately generating a mask for the exhaust plume region based on its brightness variation characteristics within the UHR rocket launch scene. This allows for the accurate separation and reconstruction of the exhaust plume from the rocket body and background, effectively improving the reconstruction quality of UHR rocket launch images. Furthermore, the reconstruction processes for the exhaust plume, the rocket body and background, and the fusion of the two reconstruction results are further optimized.

[0066] The following is an example.

[0067] Example 1:

[0068] A rocket launch image reconstruction method based on an event camera, such as Figure 1 and Figure 2 As shown, it includes:

[0069] Acquire rocket launch image sequences and event streams captured by a coaxial system of frame cameras and event cameras;

[0070] For time Captured images After converting it to a grayscale image, the positions with grayscale values ​​greater than a preset grayscale threshold are set to 1, and the remaining positions are set to 0, thus obtaining the overexposed area mask. ;

[0071] In the time window The total number of events occurring at each pixel location is counted to obtain the event density map. Positions where the total number of events exceeds a preset event threshold are set to 1, and all other positions are set to 0. After obtaining a binary image of high-frequency events, a closing operation is performed to obtain a high-frequency event mask. ; , , Exposure time;

[0072] Please expose the region mask. and high-frequency event mask The intersection of these points yields the tail flame mask. ;

[0073] According to the tail flame mask From the image Separate the exhaust plume image and the rocket body and background image from the exhaust plume mask. From the time window Separate the tail flame event stream and the arrow body and background event stream from the event stream within the event stream;

[0074] The tail flame region is reconstructed based on the tail flame event stream to obtain a tail flame reconstruction image; the arrow body and background region are reconstructed based on the arrow body and background image and the arrow body and background event stream to obtain an arrow body and background reconstruction image.

[0075] According to the tail flame mask By fusing the reconstructed image of the exhaust plume and the arrow body with the reconstructed image of the background, the time can be obtained. Images of rocket launches.

[0076] This embodiment generates a mask for overexposed areas based on frame images and a mask for high-frequency event areas based on event information. The intersection of the two area masks is used as the tail flame mask, which can accurately separate the tail flame from other parts. Then, the separated tail flame and rocket body and background parts are reconstructed separately, and the two reconstruction results are merged to achieve high-quality reconstruction of rocket launch images.

[0077] The following provides further explanation of each step.

[0078] In this embodiment, the rocket launch image sequence and event stream are obtained by capturing the rocket launch process using a conventional frame camera and event camera coaxial acquisition system. Optionally, the frame camera outputs the image sequence at a fixed frame rate (e.g., 30 frames / second), and the event camera asynchronously outputs the brightness change event of each pixel, in the format of... ,in For pixel coordinates, For timestamps, Polarity (indicating an increase or decrease in brightness).

[0079] This embodiment generates an overexposed area mask. The result diagram is as follows Figure 3 As shown, specifically, for time... Captured images Convert it to grayscale. Set a grayscale threshold based on experience. , The range of values ​​is Optionally, in this embodiment, By thresholding, the grayscale image is... Medium grayscale values ​​greater than the grayscale threshold By setting one position to 1 and the rest to 0, the overexposed area mask can be obtained. ,Right now:

[0080] like ,but ;

[0081] otherwise, .

[0082] Figure 4 This is a schematic diagram illustrating the result of generating a high-frequency event mask in this embodiment, using time... Set a short time window centered on the target. ,in, For frame image capture time Subtract 1 / 2 exposure time , For frame image capture time Add 1 / 2 exposure time Project the event data within this window onto a cumulative grid. Above, the event accumulation is completed, and the accumulated grid is formed. This is the time density plot. Each pixel value The total number of events occurring within this window is calculated using the following formula:

[0083]

[0084] in, For indicator functions, when equal to the pixel position of the event If the value is 1, then the value is 0; otherwise, the value is 0. It is a moment The event pixel.

[0085] Since the exhaust plume is a fluid, while the arrow body and background are rigid bodies, the frequency of events triggered by the fluid is much higher than that of events triggered by the rigid body. Therefore, an event count threshold is set based on experience. (For example, This is used to distinguish between high-frequency and low-frequency event regions. It is based on an event quantity threshold. This can generate a preliminary binary map of high-frequency events. , specifically, if At that time, ;otherwise, Binary graphs of high-frequency events A closing operation (dilation followed by erosion) is performed to fill the small holes inside the high-frequency event region and smooth the boundaries, resulting in the final high-frequency event mask. .

[0086] Figure 5 The image shown illustrates the masking of overexposed areas in this embodiment. and high-frequency event mask Generate a tail flame mask by performing intersection operations. The result is illustrated in the diagram. Specifically, the overexposed area is masked. and high-frequency event mask Perform a logical AND operation on these two masks to obtain the accurate exhaust mask. .

[0087] This embodiment is based on The tail flame portion, along with the remaining arrow body and background portion, is separated from the frame image and event stream for further processing.

[0088] Considering that the image's exhaust flame is almost completely overexposed during the reconstruction process, making it impossible to extract detailed texture information, brightness and detailed texture can only be reconstructed through event information. Therefore, this embodiment only reconstructs the exhaust flame based on the exhaust flame event stream.

[0089] In this embodiment, the reconstruction of the exhaust flame is specifically accomplished through a pre-trained exhaust flame reconstruction network. During reconstruction, the exhaust flame event stream is input into the exhaust flame reconstruction network to obtain the exhaust flame reconstruction image.

[0090] In this embodiment, the tail flame reconstruction network includes: a brightness mapping sub-network, a first deformable convolutional network, a first fusion module, and a coloring network;

[0091] A brightness mapping sub-network is used to accumulate events and suppress brightness in the exhaust event stream to obtain a brightness map of the exhaust region.

[0092] Deformable convolutional networks are used to extract detailed texture feature maps of the exhaust region from exhaust event streams.

[0093] The first fusion module is used to fuse the brightness map and detail texture feature map of the exhaust flame region to obtain a reconstructed grayscale image of the exhaust flame region.

[0094] A coloring network is used to color the reconstructed grayscale image of the exhaust flame region to obtain the exhaust flame reconstructed image.

[0095] Optionally, in this embodiment, the temporal accumulation in the luminance mapping sub-network is specifically implemented by a lightweight convolutional network (CNN) or a multilayer perceptron (MLP). This embodiment further reveals that the initial brightness of the rocket's exhaust plume during launch is high, then rapidly transitions to even higher brightness. Therefore, the number of events increases rapidly over time, and direct integration would lead to excessive local brightness enhancement and reduced overall contrast. Thus, the luminance mapping sub-network introduces a brightness suppression mechanism during the brightness reconstruction of the exhaust plume, as expressed by the following formula:

[0096]

[0097] in, Indicates time The cumulative results of the events are normalized to The result afterwards, This is the gamma correction function. This represents the corrected event density, i.e., the suppressed brightness.

[0098] In this embodiment, the brightness mapping subnetwork performs brightness suppression when reconstructing the tail flame. Based on the brightness information obtained through event accumulation, the brightness map of the tail flame region is controlled in range and has distinct layers. It can maintain a reasonable distribution of global brightness and avoid distortion due to excessive brightness of local pixels, thereby further improving the quality of the reconstructed rocket launch image.

[0099] Building upon this foundation, detailed texture features are further extracted from event data within the same time window. Since the exhaust plume is a rapidly changing fluid structure with complex local geometry, traditional convolutional kernels struggle to capture it effectively. Therefore, Deformable Convolution (DeformConv) is employed to extract event-driven high-frequency detailed features. This process adaptively adjusts the convolution sampling position based on the spatiotemporal distribution differences of events, more accurately representing the edges and detailed changes of the exhaust plume. Finally, the suppressed low-frequency brightness information is fused with the high-frequency texture features extracted by DeformConv to obtain a grayscale exhaust plume image that possesses both overall brightness levels and detailed textures.

[0100] Optionally, in this embodiment, the pre-trained colorization network can employ existing colorization methods. Specifically, to obtain a realistically colored rocket exhaust image, this embodiment first collects rocket exhaust images, then creates a color-grayscale image pairwise dataset, and trains the colorization network on the images in this dataset. This allows the network to learn the mapping relationship from brightness to the natural rocket exhaust color (such as a white-yellow-red color model), ultimately resulting in a high-quality reconstructed exhaust image. .

[0101] Figure 6 This is a schematic diagram of the result of tail flame reconstruction based on event data in this embodiment.

[0102] Considering that there are small overexposed areas between the arrow body and the background, and that the image can provide rich information, this embodiment takes the image as the main factor in the process of reconstructing the arrow body and the background, while using event information to provide brightness changes and edge texture information to assist the image in high dynamic range reconstruction.

[0103] In this embodiment, the reconstruction of the arrow body and the background is achieved through a pre-trained tail flame reconstruction network. Specifically, the arrow body and background image and the arrow body and background event stream are input into the arrow body and background reconstruction network to obtain the arrow body and background reconstructed image.

[0104] In this embodiment, the arrow body and background reconstruction network includes: an illumination conversion module, a convolutional layer, a second deformable convolutional network, and a second fusion module;

[0105] The illumination conversion module is used to calculate the time difference between each pixel in the arrow body and the background image. The true illuminance value is obtained, thereby converting the arrow body and background image into a true illuminance map;

[0106] Convolutional layers are used to extract the brightness features of the arrow body and the background from the real illumination map;

[0107] The second deformable convolutional network is used to extract edge texture features of the arrow body and background from the event stream of the arrow body and background;

[0108] The second fusion module is used to fuse the brightness features and edge texture features of the arrow body and the background to obtain a reconstructed image of the arrow body and the background.

[0109] Figure 7 This is a schematic diagram illustrating the image-driven reconstruction of the arrow body and background in this embodiment. Specifically, after the arrow body and background images and the arrow body and background event stream are input into the illumination conversion module, the pixels in the arrow body and background images are... At any moment True illuminance value The calculation methods include:

[0110] S1: Select an unexposed image as the initial image. ;

[0111] S2: Calculate the initial image medium pixel Illuminance value As a pixel Initial illuminance value:

[0112]

[0113] in, Represents the initial image medium pixel Pixel value at; Let be the inverse camera response function of the frame camera; for evolutionary calculation, in this embodiment, the frame camera response function is... Approximately the Gamma function, the corresponding expression is:

[0114]

[0115] in, As a preset constant, in this embodiment, ;

[0116] S3: Calculate pixels through event overlay At any moment True illuminance value :

[0117]

[0118] in, The preset event logarithmic brightness threshold is usually obtained through experience or calibration (generally around 0.1). (0.5 logarithmic unit range). Indicates at time , pixels The polarity of events at that location.

[0119] Then, standard convolutional layers are used to extract global brightness distribution and low-frequency information, i.e., brightness features, from the ground illumination map of each frame. Simultaneously, a second deformable convolutional network (DeformConv) is used to extract high-frequency information, i.e., edge texture features, from the event stream of the arrow and background. The extracted brightness and edge texture features are combined in a second fusion module (such as the skip connection structure of U-Net) to finally reconstruct a brightness-balanced and detail-enhanced reconstructed image of the arrow and background. .

[0120] Figure 8 The diagram shown illustrates the result obtained by the rocket launch image reconstruction method in this embodiment. Specifically, in this embodiment, the third fusion module reconstructs the exhaust plume image. Arrow body and background reconstructed image According to the tail flame shield The images are merged to generate preliminary high dynamic range rocket launch images. :

[0121]

[0122] Preliminary high dynamic range rocket launch images can be observed. Brightness jumps or discontinuities occur at the boundaries between the two regions. To address this, during the training of the third fusion module, a brightness consistency constraint is applied to the boundary regions; specifically, in the initial high dynamic range rocket launch images... The left and right boundaries are each selected from two sizes (e.g., 1). 1, 2 2) The same local area, i.e., the left boundary area and right boundary area The average pixel value within each region is calculated, and the third fusion module is constrained in its processing of the initial high dynamic range rocket launch image. Edge restoration is performed to minimize the difference in the mean pixel value between the left and right boundary regions. Accordingly, the training loss function of the third fusion module includes brightness consistency constraints. Brightness consistency constraint The calculation expression is as follows:

[0123]

[0124] in, This is the left boundary region. This is the right boundary region. The average pixel value in the region; The number of left / right boundary regions selected.

[0125] In this embodiment, during the process of merging the reconstructed images of the exhaust plume and the rocket body and background, a brightness consistency constraint is added to avoid excessive differences in distribution on both sides of the edge of the merged area. The synthesized image is improved in both brightness consistency and structural continuity, avoiding the appearance of unnatural boundaries, and finally obtaining a complete rocket launch image with consistent brightness. (i.e., reconstructing HDR images).

[0126] Example 2:

[0127] A computer-readable storage medium includes a stored computer program that, when executed by a processor, implements the rocket launch image reconstruction method based on an event camera provided in Embodiment 1 above.

[0128] Example 3:

[0129] An electronic device, comprising:

[0130] A computer-readable storage medium for storing computer programs;

[0131] And a processor for reading a computer program stored in a computer-readable storage medium to implement the rocket launch image reconstruction method based on an event camera provided in Embodiment 1 above.

[0132] Example 4:

[0133] A rocket launch image reconstruction system based on an event camera includes: a co-optical axis acquisition system formed by a frame camera and an event camera with the same optical axis, and the electronic equipment provided in Embodiment 3 above.

[0134] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for reconstructing rocket launch images based on an event camera, characterized in that, include: Acquire rocket launch image sequences and event streams captured by a coaxial system of frame cameras and event cameras; For time Captured images After converting it to a grayscale image, the positions with grayscale values ​​greater than a preset grayscale threshold are set to 1, and the remaining positions are set to 0, thus obtaining the overexposed area mask. ; In the time window The total number of events occurring at each pixel location is counted to obtain the event density map. Positions where the total number of events exceeds a preset event threshold are set to 1, and all other positions are set to 0. After obtaining a binary image of high-frequency events, a closing operation is performed to obtain a high-frequency event mask. ; , , Exposure time; Find the mask of the exposed area. and the high-frequency event mask The intersection of these points yields the tail flame mask. ; According to the tail flame mask From the image Separate the exhaust plume image and the rocket body and background image from the exhaust plume mask. From the time window Separate the tail flame event stream and the arrow body and background event stream from the event stream within the event stream; The tail flame region is reconstructed based on the tail flame event stream to obtain a tail flame reconstruction image; the arrow body and background region are reconstructed based on the arrow body and background image and the arrow body and background event stream to obtain an arrow body and background reconstruction image. According to the tail flame mask By fusing the reconstructed image of the exhaust plume and the arrow body with the reconstructed image of the background, the time can be obtained. Images of rocket launches; The exhaust region is reconstructed based on the exhaust event stream to obtain a reconstructed exhaust image, including: Input the exhaust event stream into the trained exhaust reconstruction network to obtain the exhaust reconstruction image; The tail flame reconstruction network includes: a brightness mapping sub-network, a first deformable convolutional network, a first fusion module, and a coloring network; The brightness mapping sub-network is used to accumulate events and suppress brightness in the tail flame event stream to obtain a brightness map of the tail flame region. The deformable convolutional network is used to extract detailed texture feature maps of the exhaust region from the exhaust event stream; The first fusion module is used to fuse the brightness map and detail texture feature map of the exhaust flame region to obtain a reconstructed grayscale image of the exhaust flame region; The coloring network is used to color the reconstructed grayscale image of the exhaust flame region to obtain the exhaust flame reconstructed image. Based on the arrow body and background images and the arrow body and background event stream, the arrow body and background regions are reconstructed to obtain reconstructed images of the arrow body and background, including: Input the arrow body and background image and the arrow body and background event stream into the trained arrow body and background reconstruction network to obtain the arrow body and background reconstructed image; The arrow body and background reconstruction network includes: an illumination conversion module, a convolutional layer, a second deformable convolutional network, and a second fusion module; The illumination conversion module is used to calculate the time difference between each pixel in the arrow body and the background image. The true illuminance value is obtained, thereby converting the arrow body and background image into a true illuminance map; The convolutional layer is used to extract the brightness features of the arrow body and the background from the real illumination map; The second deformable convolutional network is used to extract edge texture features of the arrow body and background from the event stream of the arrow body and background; The second fusion module is used to fuse the brightness features and edge texture features of the arrow body and the background to obtain a reconstructed image of the arrow body and the background.

2. The rocket launch image reconstruction method based on an event camera as described in claim 1, characterized in that, The formula for calculating the brightness suppression is as follows: in, Indicates time The cumulative results of the events are normalized to The result afterwards, For gamma correction function, This indicates the brightness after suppression.

3. The rocket launch image reconstruction method based on an event camera as described in claim 1, characterized in that, For pixels in the arrow body and background image At any given moment True illuminance value The calculation methods include: S1: Select an unexposed image as the initial image. ; S2: Calculate the initial image medium pixel Illuminance value As a pixel Initial illuminance value: in, Represents the initial image medium pixel Pixel value at; For the inverse camera response function of the frame camera; S3: Calculate pixels through event overlay At any moment True illuminance value : in, The preset event logarithmic brightness threshold; Indicates at time Pixels The polarity of events at that location.

4. The rocket launch image reconstruction method based on an event camera as described in claim 3, characterized in that, in, This is a preset constant.

5. The rocket launch image reconstruction method based on an event camera as described in any one of claims 1 to 4, characterized in that, According to the tail flame mask The fusion of the reconstructed exhaust image and the rocket body image with the reconstructed background image is achieved by a third fusion module, and the training loss function of the third fusion module includes a brightness consistency constraint. The brightness consistency constraint The calculation expression is as follows: in, The left boundary region, This is the right boundary region. The average pixel value in the region; The number of boundary regions selected.

6. A computer-readable storage medium, characterized in that, The system includes a stored computer program, which, when executed by a processor, implements the rocket launch image reconstruction method based on an event camera as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, include: A computer-readable storage medium for storing computer programs; And a processor for reading the computer program stored in the computer-readable storage medium to implement the rocket launch image reconstruction method based on an event camera as described in any one of claims 1 to 5.

8. A rocket launch image reconstruction system based on an event camera, characterized in that, include: A co-optical axis acquisition system formed by a frame camera and an event camera being set together on the same optical axis, and the electronic device as described in claim 7.

Images