Shader-based integrated spatio-temporal information and visual image information integration method

Abstract

This invention belongs to the field of airborne integrated visual information processing and discloses a shader-based method for integrating integrated visual spatiotemporal information with visual image information. The method includes the following steps: creating a data integration rendering layer; adding drawing nodes to the rendering layer as carriers for transmitting integrated visual spatiotemporal information; segmenting and converting the integrated visual spatiotemporal information into RGB (8-bit depth) data; and using a GLSL fragment shader to write the data pixel-by-pixel to a specified location, thus completing the integration of integrated visual spatiotemporal information with visual image information. This invention integrates spatiotemporal information related to integrated visual information into the visual image in pixel form through technical means. This method can be used for extended functions such as time synchronization and data alignment between asynchronous processes of integrated visual information processing.

Description

Technical Field

[0001] This invention belongs to the field of airborne integrated visual information processing technology, and relates to a method for integrating integrated visual spatiotemporal information and visual image information based on shaders. Background Technology

[0002] Integrated visual technology fuses multi-source heterogeneous information, such as geographic information data, integrated navigation data, helmet attitude data, photoelectric image data, and flight instrument parameters, into a comprehensive, all-element flight visual image. This image is used for tasks such as flight-assisted navigation in adverse visual environments and obstacle avoidance in low-altitude flight.

[0003] Unlike video images, which reside in CPU memory and can be directly accessed by applications through memory read / write operations, visual images are generated by a 3D rendering engine through a series of graphical transformations at a variable frame rate. These visual images are stored in the GPU's frame buffer, and applications cannot directly and efficiently read / write data from the frame buffer to integrate important rendering state information in specific areas of the visual image using pixels without affecting the overall content. This invention addresses these issues by implementing a method for integrating comprehensive spatiotemporal information of the visual image with visual image information based on a programmable shader. Summary of the Invention

[0004] (I) Purpose of the Invention

[0005] The purpose of this invention is to provide a method for integrating comprehensive spatiotemporal information and visual image information based on shaders. Based on programmable shader technology, it integrates comprehensive spatiotemporal information related to the view into the visual image. The spatiotemporal information includes at least the rendering frame number, rendering timestamp, navigation data of the current frame, and photoelectric aiming line data of the current frame. During asynchronous process interaction, the spatiotemporal information integrated in the current frame is used to achieve functions such as time synchronization, alignment of photoelectric images and visual images, and geographic coordinate calculation.

[0006] (II) Technical Solution

[0007] To address the aforementioned technical problems, this invention provides a method for integrating visual images with spatiotemporal information based on a programmable shader, comprising the following steps:

[0008] Step 1: Create a POST_RENDER integrated rendering layer, clear the depth buffer information, and adjust the rendering order of the POST_RENDER rendering layer after the main scene rendering is complete. Set the POST_RENDER rendering layer to absolute view coordinates, and the view matrix to an identity matrix, so that it is not affected by the view transformation of the parent node. The POST_RENDER rendering layer is only completed at the end of each frame during the rendering process, and it is the foundation for integrating spatiotemporal information on the view image.

[0009] Step 2: Add a quadrilateral drawing node to the created POST_RENDER integrated rendering layer. The length and width of the quadrilateral should be exactly the same as the viewport of the integrated view rendering.

[0010] Step 3: Acquire spatiotemporal information related to the overall view frame by frame (including at least the rendering frame number, rendering timestamp, navigation data of the current frame, and photoelectric aiming line data of the current frame), and convert the above information from integer, long integer, or floating-point data into a character data stream UNSIGNED CHAR;

[0011] Step 4: Save the UNSIGNED CHAR type data converted in Step 3 as an RGB three-dimensional vector data format. The conversion formula is as follows:

[0012]

[0013] Where g_buf[i] is the character data converted from the comprehensive visual spatiotemporal information, and i represents the pixel position index that the comprehensive visual spatiotemporal information can be converted into RGB, i = 0, 1, 2...;

[0014] Step 5: Use the Uniform variable of the GLSL shading language to pass the RGB 3D vector transformed in Step 4 to the gl_FragColor variable in the fragment shader, as shown in the following equation:

[0015]

[0016] Where j = 0, 1, 2...; j represents the index of the fragment coordinate position to be written;

[0017] Step 6: The RGB 3D vector data converted from the integrated spatiotemporal information of the scene is output to the color buffer line by line and pixel by pixel through the fragment shader as the final output of the scene image. At this time, the spatiotemporal information data of the integrated scene will be integrated into the scene image.

[0018] (III) Beneficial Effects

[0019] The shader-based method for integrating spatiotemporal information and image information of the aforementioned technical solution can integrate state information related to the overall view into the image. This does not affect the image display effect, and through data transmission between different modules, it enables asynchronous interaction, time synchronization, data alignment, and processing of important delayed data between different processes. Furthermore, this invention, combined with existing optoelectronic pod products, can further extend the additional functions of our optoelectronic pods and improve their information capabilities. Attached Figure Description

[0020] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0022] This invention addresses the phenomena described in the background art by designing a shader-based method for integrating spatiotemporal information and visual image information of the integrated scene. This method resolves issues related to data interaction, time synchronization, and data alignment between asynchronous integrated scene programs. It implements corresponding measures for different rendering stages during the 3D rendering process, ultimately integrating the scene data information into the visual image.

[0023] like Figure 1 As shown, the 3D rendering process 100 includes the following steps: DATA preparation (geographic information data, navigation-driven data, photoelectric image data preparation), APP application configuration stage (rendering context settings, window attribute configuration, etc.), EVENT event handling stage (external manipulation), UPDATE update stage (observation matrix update, database update, etc.), CULL clipping stage (view frustum clipping, rendering data construction, etc.), and DRAW drawing stage (sending drawing data to the OpenGL pipeline). After the above stages of processing, the data is stored in the FRAME BUFFER frame buffer, which can be directly sent to the monitor through the display interface, or transmitted to other systems to complete asynchronous data interaction.

[0024] To integrate the spatiotemporal information of the scene with the scene image, embodiments of the present invention implement corresponding measures for different rendering stages, including the following steps:

[0025] Step 101: In the APP stage of the rendering pipeline, create a POST_RENDER rendering layer, clear the depth buffer information, and adjust the rendering order of the POST_RENDER rendering layer after the main scene rendering is complete. Set the POST_RENDER layer to absolute view coordinates, and the view matrix to an identity matrix, meaning it is not affected by the view transformation of the parent node. The POST_RENDER rendering layer implemented in Step 101 is only completed at the end of each frame's drawing process, and it forms the basis for integrating spatiotemporal information onto the view image.

[0026] Step 102: Add a quadrilateral drawing node to the created POST_RENDER rendering layer. The size (length and width) of the quadrilateral should be exactly the same as the rendering viewport.

[0027] Step 103: In the CULL stage of the rendering process, obtain spatiotemporal information related to the overall view frame by frame (including at least the rendering frame number, rendering timestamp, navigation data of the current frame, and photoelectric aiming line data of the current frame), and convert the above information from integer, long integer or floating-point data into character data stream (UNSIGNED CHAR).

[0028] Taking one frame of data as an example, the following is an explanation: When the overall visual rendering reaches frame 2001, the current frame timestamp is 1696398535000 milliseconds. The navigation data for the current frame are as follows: longitude 110.77132212 latitude 33.52223736 altitude 596.686 pitch angle 3.367 roll angle 1.373 yaw angle 11.488. The electro-optical aiming line data for the current frame are as follows: azimuth 0.29 pitch angle 19.85. The spatiotemporal information is converted to UNSIGNED CHAR type and the data is represented as: 00 00 00 0000 00 07D1|00 00 01 8A F9 3B 69 58|42 06 5C C5|13FB 16 65|E9 14|01 50|00 89|04 7C|00 1D|07C1, a total of 36 bytes, represented as g_buf

[36] .

[0029] Step 104: Save the UNSIGNED CHAR type data converted in Step 103 as an RGB three-dimensional vector data format. The conversion formula is as follows:

[0030]

[0031] Where g_buf[i] is the character data converted from the comprehensive visual spatiotemporal information, and i represents the pixel position index that the comprehensive visual spatiotemporal information can be converted into RGB, i = 0, 1, 2, ..., 12.

[0032] The 36 bytes of data in the example in step 103 can be decomposed into 12 groups of RGB three-dimensional vector data after being converted by the above formula.

[0033] Step 105: Use the Uniform uniform variables of the GLSL shading language to pass the 12 sets of RGB 3D vectors obtained in step 104 to gl_FragColor in the fragment shader, as shown in the following formula.

[0034]

[0035] If gl_FragCoord.y = 0, the above formula can integrate spatiotemporal information into the first row of pixels in the lower left corner of the scene image. If gl_FragCoord.y = screen_heigh, the above formula can integrate spatiotemporal information into the first row of pixels in the upper left corner of the scene image.

[0036] Step 106: The fragment shader outputs the spatiotemporal information-converted RGB 3D vector data to the color buffer through fragment-by-fragment processing, as the final visual image output.

[0037] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for integrating visual images and spatiotemporal information based on programmable shaders, characterized in that, Includes the following steps: Step 1: Create a POST_RENDER integrated rendering layer, clear the depth buffer information, and adjust the rendering order of the POST_RENDER rendering layer after the main scene rendering is completed. Set the POST_RENDER rendering layer to absolute view coordinates and the view matrix to the identity matrix. Step 2: Add a quadrilateral drawing node to the created POST_RENDER integrated rendering layer; Step 3: Acquire spatiotemporal information related to the overall visual scene frame by frame, and convert the above information from integer, long integer or floating-point data into a character data stream UNSIGNED CHAR; Step 4: Save the UNSIGNED CHAR type data converted in Step 3 as an RGB three-dimensional vector data format; Step 5: Use the Uniform variable of the GLSL shading language to pass the RGB 3D vector transformed in Step 4 to the gl_FragColor variable in the fragment shader; Step 6: The RGB 3D vector data converted from the integrated spatiotemporal information of the scene is output to the color buffer line by line and pixel by pixel through the fragment shader as the final output scene image. At this time, the scene image integrates the spatiotemporal information data of the integrated scene.

2. The method for integrating visual images and spatiotemporal information based on programmable shaders as described in claim 1, characterized in that, In step 1, the POST_RENDER rendering layer is completed at the end of the rendering process for each frame.

3. The method for integrating visual images and spatiotemporal information based on programmable shaders as described in claim 2, characterized in that, In step 2, the length and width dimensions of the quadrilateral are completely consistent with the viewport of the integrated visual rendering.

4. The method for integrating visual images and spatiotemporal information based on programmable shaders as described in claim 3, characterized in that, In step 3, the spatiotemporal information related to the overall scene is integrated, including the rendering frame number, rendering timestamp, navigation data of the current frame, and photoelectric aiming line data of the current frame.

5. The method for integrating visual images and spatiotemporal information based on programmable shaders as described in claim 4, characterized in that, In step 4, the conversion formula between UNSIGNED CHAR type data and RGB three-dimensional vector data format is as follows: Where g_buf[i] is the character data converted from the comprehensive visual spatiotemporal information, and i represents the pixel position index that the comprehensive visual spatiotemporal information can be converted into RGB, i = 0, 1, 2....

6. The method for integrating visual images and spatiotemporal information based on programmable shaders as described in claim 5, characterized in that, In step 5, the RGB 3D vector transformed in step 4 is passed to the gl_FragColor variable in the fragment shader using the Uniform uniform variable of the GLSL shading language, as shown in the equation: Where j = 0, 1, 2, ...; j represents the index of the fragment coordinate position to be written.

7. The method for integrating visual images and spatiotemporal information based on programmable shaders as described in claim 6, characterized in that, The integration method implements corresponding measures for different rendering stages during the 3D rendering process. The 3D rendering process includes DATA preparation, APP application configuration, EVENT event handling, UPDATE update, CULL clipping, and DRAW drawing. Step 1 is used for the APP application configuration stage, and step 3 is used for the CULL clipping stage.

8. The method for integrating visual images and spatiotemporal information based on programmable shaders as described in claim 7, characterized in that, Step 3, used in the CULL stage of the rendering process, is illustrated with a single frame of data: When the overall visual rendering reaches frame 2001, the current frame timestamp is 1696398535000 milliseconds. The navigation data for the current frame are: longitude 110.77132212, latitude 33.52223736, altitude 596.686, pitch angle 3.367, roll angle 1.373, and yaw angle 11.

488. The electro-optical aiming line data for the current frame are: azimuth 0.29 and pitch angle 19.

85. The data after converting the spatiotemporal information to UNSIGNED CHAR type is represented as: 00 00 00 00 00 00 07D1|00 00 01 8A F9 3B 6958|42 06 5C C5|13 FB 16 65|E9 14|01 50|00 89|04 7C|00 1D|07 C1, a total of 36 bytes, is represented as g_buf[36].

9. The method for integrating visual images and spatiotemporal information based on programmable shaders as described in claim 8, characterized in that, When step 3 is used in the CULL stage of the rendering process, in step 4, i = 0, 1, 2, ..., 12. The 36-byte data in the example in step 3 is decomposed into 12 groups of RGB three-dimensional vector data after conversion.

10. The method for integrating visual images and spatiotemporal information based on programmable shaders as described in claim 9, characterized in that, When step 3 is used in the CULL stage of the rendering process, in step 5, the 12 sets of RGB 3D vectors obtained in step 4 are passed to gl_FragColor in the fragment shader using the Uniform consistent variables of the GLSL shading language, as shown in the following formula: If gl_fragCoord.y = 0, the above formula integrates the spatiotemporal information into the first row of pixels in the lower left corner of the visual image. If gl_FragCoord.y = screen_heigh, the above formula integrates the spatiotemporal information into the first row of pixels in the upper left corner of the visual image.

Images