A lunar flag fluttering demonstration image processing method, demonstration system and device

Abstract

The present invention discloses a lunar flag fluttering demonstration image processing method, demonstration system, and equipment, relating to the field of aerospace engineering technology. The present invention comprises the following steps: receiving a flag fluttering video transmitted from the lunar surface; performing frame processing on the received video, using a filtering algorithm to perform bilateral filtering on the image frames to remove image noise while maintaining image edges and details; performing brightness enhancement processing on the image frames after noise removal, extracting image features in a modulation vector generation module to generate a modulation vector, which is passed through a fully connected layer to generate modulation parameters, which are modulated with intermediate features generated by an image feature extraction module to obtain feature modulation results; and processing the feature modulation results through an activation function to obtain a brightness-enhanced image frame. The present invention can improve the quality and brightness of flag images by removing image noise and enhancing brightness, which helps reduce noise and detail loss in the image and reduces the risk of system errors.

Description

Technical Field

[0001] The present invention relates to the field of aerospace engineering technology, and in particular to an image processing method, a demonstration system and equipment for a lunar flag fluttering demonstration. Background Art

[0002] A ground-based demonstration system is a system used in aerospace engineering to simulate, test, and verify spacecraft, satellites, or other aerospace equipment and payloads. It is a complex system built on the ground, typically consisting of multiple subsystems and components, including hardware, software, and network equipment. These systems are designed and built according to the specific mission requirements and goals to simulate and replicate all aspects of the mission and its operating conditions. However, the overall quality and clarity of video images transmitted from the lunar surface are currently poor, increasing the risk of system errors. To address the system risks and coordinate the subsequent planning of the entire demonstration system, we propose a lunar flag-flying demonstration image processing method, demonstration system, and equipment. Summary of the Invention

[0003] The purpose of the present invention is to provide a lunar flag flying demonstration image processing method, demonstration system and equipment, which can accurately demonstrate the entire process of payload operation and effectively reduce system risks.

[0004] To achieve the above-mentioned object, according to a first aspect of the present invention, the present invention provides the following technical solution: a method for processing a lunar flag flying demonstration image, comprising:

[0005] Receive flag-flying videos transmitted from the moon;

[0006] The received video is framed and a filtering algorithm is used to perform bilateral filtering on the image frames to remove image noise while maintaining the edges and details of the image.

[0007] Perform image feature processing on the image frame after noise removal to extract the feature representation of the image, then generate a modulation vector for feature modulation, and finally multiply the feature modulation result processed by the activation function with the original image to obtain the image frame after brightness enhancement processing;

[0008] The processed image frames are subjected to binocular camera correction to obtain the corrected images, and then the corrected left and right image frames are synthesized in sequence to display the video.

[0009] Furthermore, to perform bilateral filtering on the image, it is necessary to process the image frames one by one, as follows:

[0010]

[0011] Among them, I outThe pixel value of the image at position (x, y) after (x, y) filtering, I in (s, t) represents the pixel value of the original image at position (s, t), W1(x, y) represents the normalized weight coefficient,

[0012] S is the set of pixel positions that define the filter window, which contains the set of pixels in the image that are adjacent to the pixel at position (x,y); sw(x,y,s,t) represents the spatial distance weight, which measures the distance between position (s,t) and position (x,y); iw(x,y,s,t) represents the pixel value similarity weight, which measures the similarity between the pixel values at position (s,t) and position (x,y).

[0013] Furthermore, the image frame after noise removal is subjected to image feature processing, specifically as follows:

[0014] Given an input image frame I, a three-dimensional tensor of H×W×C, where H and W represent the height and width of the image respectively, and C represents the number of color channels;

[0015] Among them, in the image feature extraction module, the convolution layer is mainly composed of 1×1 convolution, denoted as B, and each convolution layer processes each pixel in the image independently; for each pixel (x, y) and channel c, there is Weis i is the weight of the i-th convolutional layer, b i represents the bias term, and N is the number of convolutional layers.

[0016] Furthermore, in the modulation vector generation module, the feature representation of the image is extracted to generate the modulation vector, as follows:

[0017] The network extracts the global features of the image and generates a modulation vector, which is expressed as V. Then: V = gen(I) = f(W c ×I+b c ), where gen represents the modulation vector generation network, W c and b c Represents the weight and bias of the network, and f represents the activation function;

[0018] With the modulation vector, feature modulation can be performed, and then: mod(B)=(α×B i )+β, where α and β represent the modulation parameters obtained by the modulation vector V after the full connection layer conversion, B i Represents the i-th feature modulation layer. Further, the feature modulation result is processed by the activation function to obtain the image frame after brightness enhancement, which is specifically expressed as:

[0019] I out =g(mod(b)), where g represents the activation function.

[0020] Furthermore, the image feature extraction module and the modulation vector generation module are jointly optimized to learn and update parameters by minimizing the reconstruction loss function L, which is specifically expressed as:

[0021] L(I,I out )=||I out -B(I)⊙mod(B;V)||1

[0022] where ⊙ represents element-wise multiplication.

[0023] Furthermore, the processed image frames are subjected to binocular camera correction to obtain the corrected images, as follows:

[0024] Through the camera calibration experiment, the camera intrinsic parameter matrix, distortion coefficient, rotation matrix and translation vector parameters are obtained. The cv2.stereoRectify() function in OpenCV is called to calculate the binocular correction matrix. The cv2.initUndistortRectifyMap() function is used to calculate the correction mapping relationship between the left and right cameras. The cv2.remap() function is used to map the left and right camera image frames according to the mapping relationship to obtain the final corrected image.

[0025] According to a second aspect of the present invention, the present invention provides a lunar flag flying demonstration system, comprising:

[0026] Receiving module: used to receive the flag-flying video transmitted from the moon;

[0027] Image noise removal module: used to process the received video into frames, and use filtering algorithms to perform bilateral filtering on the image frames to remove image noise while maintaining the edges and details of the image;

[0028] Image brightness enhancement module: It is used to process the image features of the image frame after noise removal, extract the feature representation of the image, and then generate a modulation vector for feature modulation. Finally, the feature modulation result processed by the activation function is multiplied with the original image to obtain the image frame after brightness enhancement processing;

[0029] Binocular correction and 3D synthesis module: used to perform binocular camera correction on the processed image frames to obtain the corrected image, and then synthesize the corrected left and right image frames in sequence to display the video.

[0030] Furthermore, the image brightness enhancement module includes two parts, as follows:

[0031] Image feature extraction module: The convolutional neural network is mainly composed of 1×1 convolution, denoted as B. Each convolution layer processes each pixel in the image independently; for each pixel (x, y) and channel c, there is Weis i is the weight of the i-th convolutional layer, b i represents the bias term, N is the number of convolutional layers;

[0032] Modulation vector generation module: The network extracts global features of the image to generate a modulation vector, which is expressed as:

[0033] V=gen(I)=f(W c ×I+b c )

[0034] Where gen represents the modulation vector generation network, W c and b c Represents the weight and bias of the network, and f represents the activation function;

[0035] With the modulation vector, feature modulation can be performed, specifically:

[0036] mod(B)=(α×B i )+β

[0037] Where α and β represent the modulation parameters obtained by converting the modulation vector V through the fully connected layer, B i represents the i-th feature modulation layer.

[0038] According to a third aspect of the present invention, the present invention provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The device is characterized in that the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, it adopts the method for processing a lunar flag fluttering demonstration image.

[0039] The present invention has at least the following beneficial effects:

[0040] (1) The present invention can improve the quality and clarity of flag images by removing image noise and enhancing image brightness, which helps to reduce noise and detail loss in the image and reduces the risk of system errors.

[0041] (2) The present invention can accurately demonstrate the entire process of lunar payload work through the flag flying demonstration, which helps to improve people's understanding and awareness of lunar missions.

[0042] (3) The present invention utilizes binocular correction and 3D synthesis to generate a display video with good visual effects. Through correction and synthesis, the distortion and inconsistency of the image can be eliminated, making the display video more realistic and lifelike.

[0043] (4) The present invention is not only used in the lunar flag flying demonstration, but also provides technical support for the construction of a deep space exploration ground demonstration system, which has positive significance for promoting scientific knowledge and promoting popular science education.

[0044] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0046] Figure 2 Schematic diagram of the structure of the image brightness enhancement module of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0048] See also Figure 1-2 The present invention provides a technical solution: a method for processing a lunar flag fluttering demonstration image, comprising:

[0049] S1. Receive the flag-flying video transmitted from the lunar surface;

[0050] S2. Frame the received video and perform bilateral filtering on the image frames using a filtering algorithm to remove image noise while preserving image edges and details;

[0051] Taking into account the similarity between pixel space and pixel values, bilateral filtering is performed on the image. Bilateral filtering can effectively remove noise while maintaining the edges and details of the image. For a single frame image I, after bilateral filtering, it can be expressed as:

[0052]

[0053] Among them, I out The pixel value of the image at position (x, y) after (x, y) filtering, I in (s, t) represents the pixel value of the original image at position (s, t), W1(x, y) represents the normalized weight coefficient,

[0054] S is the set of pixel positions that defines the filter window, which contains the set of pixels adjacent to the pixel at position (x,y) in the image. sw(x,y,s,t) represents the spatial distance weight, which measures the distance between position (s,t) and position (x,y). iw(x,y,s,t) represents the pixel value similarity weight, which measures the similarity between the pixel values at position (s,t) and position (x,y). Both the spatial distance weight and the pixel value similarity weight are calculated using Gaussian functions.

[0055] S3. Performing image feature processing on the image frame after noise removal, extracting the feature representation of the image, and then generating a modulation vector for feature modulation. Finally, the feature modulation result processed by the activation function is multiplied by the original image to obtain the image frame after brightness enhancement processing;

[0056] Given an input image frame I, a three-dimensional tensor of H×W×C, where H and W represent the height and width of the image respectively, and C represents the number of color channels;

[0057] Among them, in image feature extraction, the convolutional neural network is mainly composed of 1×1 convolution, represented as B. Each convolution layer processes each pixel in the image independently. For each pixel (x, y) and channel c, there is Weis i is the weight of the i-th convolutional layer, b i represents the bias term, N is the number of convolutional layers;

[0058] In the modulation vector generation, the network extracts the global features of the image to generate the modulation vector, which is expressed as V. Then: V = gen(I) = f(W c ×I+b c ), where gen represents the modulation vector generation network, W c and b c Represents the weight and bias of the network, and f represents the activation function;

[0059] With the modulation vector, feature modulation can be performed, and then: mod(B)=(α×B i )+β, where α and β represent the modulation parameters obtained by converting the modulation vector V, B i Represents the i-th feature modulation layer. ;

[0060] Then, the output of the entire image brightness enhancement network can be expressed as:

[0061] I out =g(mod(b)), where g represents an activation function. In the technical solution of this embodiment, the LeakyReLu activation function is used;

[0062] S4. Perform binocular camera correction on the processed image frames to obtain a corrected image, and then synthesize the corrected left and right image frames in sequence to display the video;

[0063] The camera intrinsic parameters used in binocular camera correction come from the camera calibration experiment. During the entire correction process, the left and right camera image frames are corrected in sequence. According to the camera intrinsic parameter matrix, distortion coefficient, rotation matrix, translation vector and other parameters, the cv2.stereoRectify() function in OpenCV is called to calculate the binocular correction matrix. The cv2.initUndistortRectifyMap() function is used to calculate the correction mapping relationship between the left and right cameras. The cv2.remap() function is used to map the left and right camera image frames according to the mapping relationship to obtain the final corrected image.

[0064] After correction, the left and right image frames are synthesized into videos in sequence, and a display video is synthesized according to the left and right 3D format. The left and right 3D represents the left and right videos, and the widths are scaled by half respectively. That is, the synthesized display video is the same size as the original video. It should be noted that the synthesized display video can be imported into a 3D polarized screen or AR device for playback and display.

[0065] Furthermore, the binocular camera is specifically a camera installed on the left and right sides of the lunar rover, which is used to capture binocular images.

[0066] According to a second aspect of the present invention, the present invention provides a lunar flag flying demonstration system, comprising:

[0067] Receiving module: used to receive the flag-flying video transmitted from the moon;

[0068] Image noise removal module: used to process the received video into frames, and use filtering algorithms to perform bilateral filtering on the image frames to remove image noise while maintaining the edges and details of the image;

[0069] Image brightness enhancement module: It is used to process the image features of the image frame after noise removal, extract the feature representation of the image, and then generate a modulation vector for feature modulation. Finally, the feature modulation result processed by the activation function is multiplied with the original image to obtain the image frame after brightness enhancement processing;

[0070] Binocular correction and 3D synthesis module: used to perform binocular camera correction on the processed image frames to obtain the corrected image, and then synthesize the corrected left and right image frames in sequence to display the video.

[0071] Furthermore, the image brightness enhancement module includes two parts, as follows:

[0072] Image feature extraction module: The convolutional neural network is mainly composed of 1×1 convolution, denoted as B. Each convolution layer processes each pixel in the image independently; for each pixel (x, y) and channel c, there is Weis i is the weight of the i-th convolutional layer, b i represents the bias term, N is the number of convolutional layers;

[0073] Modulation vector generation module: The network extracts global features of the image to generate a modulation vector, which is expressed as:

[0074] V=gen(I)=f(W c ×I+b c )

[0075] Where gen represents the modulation vector generation network, W c and b c Represents the weight and bias of the network, and f represents the activation function;

[0076] With the modulation vector, feature modulation can be performed, specifically:

[0077] mod(B)=(α×B i )+β

[0078] Where α and β represent the modulation parameters obtained by converting the modulation vector V, B i represents the i-th feature modulation layer.

[0079] Specifically, the above-mentioned receiving module, image noise removal module, image brightness enhancement module, and binocular correction and 3D synthesis module can be embedded in a computer processing system. The computer calls the above-mentioned modules to complete the task of equivalent modeling based on the above-mentioned image processing method for the demonstration of a flag flying on the moon. The above-mentioned receiving module, image noise removal module, image brightness enhancement module, and binocular correction and 3D synthesis module can perform operations according to the specific steps provided in the above-mentioned image processing method for the demonstration of a flag flying on the moon.

[0080] It should be noted that it should be understood that the division of the various modules of the above system is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or they can be physically separated. Moreover, these modules can be implemented entirely in the form of software called by a processing element; or entirely in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the image noise removal module can be a separately established processing element, or it can be integrated into a chip of the above device. In addition, it can be stored in the memory of the above device in the form of program code, and called by a processing element of the above device to perform the functions of the above signal processing module. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0081] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0082] According to a third aspect of the present invention, the present invention provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The device is characterized in that the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, it adopts the method for processing a lunar flag fluttering demonstration image.

[0083] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer or a cloud server, and the terminal device includes but is not limited to a processor and a memory. For example, the terminal device can also include input and output devices, network access devices and buses, etc.

[0084] Furthermore, the processor can adopt a central processing unit (CPU). Of course, depending on the actual usage, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0085] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0086] For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on", "installed on", "fixed on" or "set on" another element, it can be directly on the other element or there can be a central element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there can be a central element at the same time. The terms "vertical", "horizontal", "up", "down", "left", "right" and similar expressions used herein are for illustrative purposes only and are not intended to be the only embodiment.

[0087] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0088] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present disclosure. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Claims

1. A method for processing a lunar flag flying demonstration image, characterized in that: include: Receive flag-flying videos transmitted from the moon; The received video is framed and a filtering algorithm is used to perform bilateral filtering on the image frames to remove image noise while maintaining the edges and details of the image. Image feature extraction is performed on the image frame after noise removal to generate a modulation vector, which is expressed as: ,in represents the modulation vector generation network, and represents the weights and biases of the network, represents the activation function of the modulation vector generation network; The modulation vector is processed by the fully connected layer to generate modulation parameters, and then the feature modulation result is obtained based on the modulation parameters: ,in and represents the modulation vector The modulation parameters obtained by the conversion of the fully connected layer, Indicates the feature modulation layers; The feature modulation result is processed by the activation function to obtain the image frame with brightness enhancement, as follows: , where g represents the LeakyReLu activation function; The image frame after brightness enhancement is rectified by the binocular camera to obtain a rectified image, and then the left and right image frames after correction are synthesized in sequence to display the video.

2. The method for processing a lunar flag flying demonstration image according to claim 1, characterized in that: To perform bilateral filtering on an image, it is necessary to process the image frames one by one, as follows: in, Indicates that the image after filtering is at position The pixel value at Indicates that the original image is at position The pixel value at represents the normalized weight coefficient, is a set of pixel positions that define the filter window, which contains all pixels in the image that are at position A group of pixels adjacent to the pixel at position; Represents spatial distance weight, measuring location and location the distance between them; Represents the pixel value similarity weight, which measures the position and location The similarity of pixel values.

3. The method for processing a lunar flag flying demonstration image according to claim 2, characterized in that: Image feature extraction is performed on the image frame after noise removal, as follows: Given an input image frame ,one A three-dimensional tensor C, where and Represents the height and width of the image respectively, Indicates the number of color channels; Among them, in image feature extraction, the convolutional neural network is mainly composed of 1x1 convolution, which is expressed as , each convolutional layer processes each pixel in the image independently; for each pixel and channel c, then we have , It is The weights of the convolutional layers, represents the bias term, is the number of convolutional layers.

4. The method for processing a lunar flag flying demonstration image according to claim 3, characterized in that: The image feature extraction module and the modulation vector generation module are jointly optimized to minimize the reconstruction loss function To learn and update parameters, it is specifically expressed as: in Represents element-wise multiplication.

5. The method for processing a lunar flag flying demonstration image according to claim 1, characterized in that: The processed image frames are subjected to binocular camera correction to obtain the corrected images, as follows: Through the camera calibration experiment, the camera intrinsic parameter matrix, distortion coefficient, rotation matrix and translation vector parameters are obtained. The cv2.stereoRectify() function in OpenCV is called to pass in the intrinsic parameter matrices of the left and right cameras, the distortion parameters of the left and right cameras, the image size, the rotation matrix of the left camera relative to the right camera, and the translation vector of the left camera relative to the right camera for binocular correction. The cv2.initUndistortRectifyMap() function is used to pass in the camera intrinsic parameter matrix, camera distortion coefficient, camera rotation matrix, corrected camera intrinsic parameter matrix, input image size and output parameter numerical format to calculate the correction mapping relationship between the left and right cameras. The cv2.remap() function is used to pass in the original image, x pixel coordinate remapping table, y pixel coordinate remapping table, interpolation method and boundary mode to map the left and right camera image frames according to the mapping relationship to obtain the final corrected image.

6. A terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor loads and executes the computer program, the method for processing a lunar flag fluttering demonstration image according to any one of claims 1 to 5 is adopted.

Images