A virtual scene test platform fusing video stream feedback, a creation method and a test method

Abstract

The application discloses a kind of virtual scene test platform of fusion video stream feedback, creating method and test method, test platform includes: push stream end: for obtaining the frame buffer resource of virtual camera component in virtual scene, integrated virtual engine rendering and video encoding method, realize virtual scene video stream push;RTSP server: for receiving the H.264 coded video stream from push stream end, as streaming media relay node, support the real-time access and distribution of multiple concurrent video stream, remote transmission is carried out by RTSP protocol;Multiple pull stream end: for receiving the H.264 video stream that RTSP server transmits respectively, utilize FFmpeg to decode restoration, and the running picture of test object in real-time virtual test scene is displayed by OpenCV.The virtual scene test platform of fusion video stream feedback of the application, realize the high fidelity real-time video output of virtual scene effectively support the test and verification of perception type algorithm.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, virtual reality and intelligent equipment testing, and relates to a virtual scene testing platform, creation method and testing method that integrates video stream feedback. Background Technology

[0002] With the widespread application of virtual testing platforms in fields such as autonomous driving, drone navigation, and intelligent robots, the construction and real-time feedback of virtual scenarios have become particularly important. Traditional virtual scenario testing methods typically input data and evaluate it in a local or simulation environment, providing test results in a controlled environment. However, these platforms have limitations in terms of real-time performance, feedback accuracy, and synchronization with the actual operating environment, especially in complex tasks such as target locking, environmental perception, and autonomous navigation, where they struggle to accurately simulate real-world scenarios. Furthermore, traditional testing platforms often fail to provide accurate real-time feedback on key performance indicators such as test latency and frame rate control, limiting their effectiveness in the development of intelligent equipment. Summary of the Invention

[0003] To address the aforementioned problems, the technical solution adopted by this invention is: a virtual scene testing platform integrating video stream feedback, comprising: Push end: Used to obtain frame buffer resources of virtual camera components in virtual scenes, integrate virtual engine rendering and video encoding methods, and realize the push of virtual scene video stream; RTSP server: Used to receive H.264 encoded video streams from the streaming end, acting as a streaming media relay node, supporting real-time access and distribution of multiple concurrent video streams, and remote transmission via the RTSP protocol; Multiple streaming ends: used to receive H.264 video streams transmitted from RTSP servers, decode and restore them using FFmpeg, and display the running screen of the test object in the real-time virtual test scene using OpenCV.

[0004] Furthermore: the process of acquiring frame buffer resources of the virtual camera component in the virtual scene, integrating virtual engine rendering and video encoding methods, and realizing the virtual scene video stream push is as follows: Used to obtain the frame buffer resources of the virtual camera component in the virtual scene and generate virtual scene rendering source data in BGR format; OpenCV is used to encapsulate, process, and preview the BGR format virtual scene rendering source data, and output the image matrix to be processed. Use FFmpeg's pixel format conversion module to convert the image matrix to be processed from BGR format to YUV420P format; The YUV420P format image matrix is ​​compressed using an H.264 encoder to generate an H.264 encoded video stream. The encoded video stream is pushed using the RTSP protocol.

[0005] Furthermore: the real-time operation process of the streaming end: reads the BGR format pixel data of the virtual camera through the graphics rendering interface and stores it in the OpenCV data structure; Use the libswscale library to convert BGR format to YUV420P format; The YUV420P image data is compressed and encoded using the libavcodec library using H.264. The libavformat library is used to encapsulate the encoded data packets and push them to the streaming media server via the RTSP protocol. A method for creating a virtual scene testing platform that integrates video stream feedback includes the following steps: The construction process of the push-stream end is as follows: In UE4, a virtual test scene is built, test objects are placed, virtual camera components are configured for the test objects, and frame buffer resources of the virtual camera components are obtained through the graphics rendering interface; OpenCV's storage data structure is initialized for image data management; a connection to the RTSP server is established using FFmpeg's libavformat library; the H.264 encoder is initialized and encoding parameters are configured using FFmpeg's libavcodec library; the pixel format converter is initialized using FFmpeg's libswscale library, and the streaming end is created.

[0006] Deploy an independent RTSP streaming media server to receive H.264 encoded streams from the push client, manage and distribute streaming media based on the RTSP protocol, and support multiple client pull streams; A connection to the RTSP server is established using FFmpeg's libavformat library to pull real-time video streams; the H.264 stream is decoded using the libavcodec library to restore it to image frame data; and the decoded image data is rendered to the display window using OpenCV, thus realizing the creation and operation of the streaming client.

[0007] Furthermore, it also includes obtaining scene rendering source data in UE4, the process of which is as follows: Step 1.1: Open the engine plugin manager, activate the "Composure" plugin group, and restart the engine to ensure that the plugin configuration takes effect; Step 1.2: Create a test object and a virtual camera component in the 3D scene view. Select "Create → Camera → CineCameraActor" in the scene editor menu bar, and deploy the camera entity at the predetermined observation point in the scene coordinate system. Adjust its focal length, field of view and depth of field parameters to adapt to the target rendering scene. Step 1.3: Build a compositing pipeline to achieve layered image processing. In the engine interface, call the "Composure" function panel, create a new blank compositing lens node, and add a "CG Layer" child node by right-clicking in the hierarchy tree. Bind the CineCameraActor in the world outline view to the CG Layer, and establish the relationship between the camera and the compositing lens through the "Target Camera Actor" parameter in the properties panel. Step 1.4: Configure the material mapping channel for the composited shot to optimize the output quality; in the "Transform / Compositing Passes" module of the composited shot, add a "compositing element material channel" and create a post-processing material; Step 1.5: Set the output format and storage path of the rendering target. In the "Output" module of the compositing lens, select the "Rendering Target Compositing Output" mode, create the rendering target resource, and associate it with the compositing lens node. Through the engine's built-in frame buffer management mechanism, the rendered image is updated frame by frame and persistently stored.

[0008] Furthermore: the material includes TextureSampleParameter2D type input nodes, the parameter names of the input nodes strictly match the CG Layer names, and the material output is connected to a self-illuminating channel; Enable sRGB color space conversion and anti-aliasing filtering in the material properties.

[0009] Furthermore, the H.264 encoded stream received by the streaming end is managed and distributed based on the RTSP protocol, and the process of supporting multiple clients pulling the stream is as follows: Step 2.1: Read the pixel data of the rendering target through the UE4 graphics rendering interface, store the BGR format pixel data into the OpenCV data structure; use FFmpeg's libswscale library to perform pixel format conversion, convert the BGR format to YUV420P format to meet the input requirements of the H.264 encoder. Step 2.2: Use FFmpeg's libavcodec library to perform H.264 encoding. Input the YUV420P format video frame data into the encoder for compression through the avcodec_send_frame function, and extract the encoded data packets through the avcodec_receive_packet function for subsequent transmission. Step 2.3: Use FFmpeg's libavformat library to encapsulate the encoded data, use the av_interleaved_write_frame function to organize the encoded data packets in chronological order, and push the encapsulated H.264 video stream to the streaming media server for transmission via the RTSP protocol.

[0010] Furthermore, the process of establishing a connection with the RTSP server through FFmpeg's libavformat library to pull the real-time video stream; using the libavcodec library to decode the H.264 stream and restore it to image frame data; and rendering the decoded image data to the display window using OpenCV, is as follows: FFmpeg parses video stream data, traverses multiple data streams contained in the video stream, identifies the video stream index, retrieves the decoding tool that matches the video stream, establishes the parameters and context information required for decoding, and completes the initialization of the decoding unit; The system receives video data packets from the RTSP server and parses them to determine whether they belong to video stream data. If they are video stream data, they are input to the decoding unit for processing, and the decoded video frame data is obtained. The acquired YUV420P format video frames are converted to BGR format, which can be processed by OpenCV, for subsequent display. The video frame data in BGR format is stored in the cv::Mat data structure of OpenCV, and the cv::imshow function is called to display the video screen. This function uses the graphical user interface API of the operating system to transfer the BGR pixel data in memory to the graphics card and render it to the display window, so as to realize the visualization of the test object running screen in the real-time virtual test scene.

[0011] The testing method for a virtual scene testing platform that integrates video stream feedback includes the following steps: The test subject was subjected to multi-functional testing of target locking, environmental awareness, and autonomous navigation on a platform that integrates UE4 and video streaming; Through simulation and calculation using multiple sensors, the drone achieves comprehensive environmental perception. The simulation environment based on UE4 allows test objects to load and run input-based multi-functional navigation and obstacle avoidance algorithms for testing.

[0012] This invention provides a virtual scene testing platform that integrates video stream feedback. This platform combines virtual reality (VR) technology with video stream data, enabling test subjects to perform target locking, environmental perception, and autonomous navigation tests in a highly simulated virtual environment, and optimizing their perception and decision-making capabilities. Compared to traditional UE scene rendering testing processes, this invention offers the following improvements: It uses OpenCV to acquire BGR format images for encoding; it uses FFmpeg for H.264 encoding and employs the RTSP protocol to control the video stream for remote transmission; the client pulls the video stream, decodes it, and displays it for playback, simulating a remote viewing experience; and the test subject can perform multifunctional tests on the platform that integrates UE and video stream. Through video stream transmission technology, this invention integrates the UE4 virtual scene with external video streams, improving the realism and adaptability of the test, while supporting a precise feedback mechanism with controllable latency and adjustable frame rate to meet different testing needs.

[0013] Compared with traditional methods, this invention has the following significant advantages: 1) The virtual scene testing platform, which integrates video stream feedback, significantly improves the realism of the scene through real-time video output, while also enabling real-time monitoring of system performance, thus enhancing the authenticity and adaptability of the test; 2) Employing streaming media transmission technology, the rendering output of the virtual scene can be transmitted to the streaming end in real time. The streaming end can pull and play the video stream in real time, and supports controllable latency and adjustable frame rate to meet the needs of different testing scenarios; 3) It supports the testing of equipment equipped with video streams, such as drones, robots, and other automated equipment, and can test their target locking, environmental perception, path planning, and autonomous navigation capabilities in a virtual environment. Compared with the traditional UE4 method that relies solely on computer-generated scenes, this platform significantly improves the diversity and stability of the test, making it suitable for the research, development, optimization, and deployment of intelligent equipment. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the structure of a virtual scene testing platform; Figure 2 This is a rendered target image captured in real time by the virtual camera of the present invention; Figure 3This is a diagram showing the video stream output result of the present invention; Figure 4 This is a flowchart illustrating the overall construction process of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Figure 1 This is a schematic diagram of the structure of a virtual scene testing platform; A virtual scene testing platform integrating video stream feedback, comprising: Push end: Used to obtain frame buffer resources of virtual camera components in virtual scenes, integrate virtual engine rendering and video encoding methods, and realize the push of virtual scene video stream; RTSP server: Used to receive H.264 encoded video streams from the streaming end, acting as a streaming media relay node, supporting real-time access and distribution of multiple concurrent video streams, and remote transmission via the RTSP protocol; Multiple streaming ends: used to receive H.264 video streams transmitted from RTSP servers, decode and restore them using FFmpeg, and display the running screen of the test object in the real-time virtual test scene using OpenCV.

[0019] The process of acquiring frame buffer resources of virtual camera components in a virtual scene, integrating virtual engine rendering and video encoding methods, and realizing the push of virtual scene video streams is as follows: S11: Used to obtain the frame buffer resources of the virtual camera component in the virtual scene and generate virtual scene rendering source data in BGR format; S12: Use OpenCV to encapsulate, process, and preview the BGR format virtual scene rendering source data, and output the image matrix to be processed; In this project, OpenCV is primarily responsible for image data containerization, local preview, and image processing.

[0020] S13: After the UE rendering engine completes a frame, OpenCV encapsulates the raw pixel data in BGR format into an image matrix to be processed. S14: Creates a display window to provide a real-time preview of the camera, facilitating development and debugging; S15: OpenCV is also responsible for image enhancement processing, overlaying millisecond-precision time watermarks on the image to provide accurate time stamps for video streams.

[0021] These display and processing operations are all completed before the video is encoded and streamed, and what is being processed is the uncompressed raw footage.

[0022] Figure 2 This is a rendered target image captured in real time by the virtual camera of the present invention; Figure 3 This is a diagram showing the video stream output results of the present invention; the topmost image is the drone scene in the 3D scene of the UE, the Display window displays the UE rendering source data before the push stream is encoded, and the VideoStream window displays the pull stream image; Use FFmpeg's pixel format conversion module to convert the image matrix to be processed from BGR format to YUV420P format; Among them, FFmpeg undertakes the core tasks of video encoding, format conversion, and network streaming.

[0023] First, configure the H.264 encoder parameters. Then, convert the BGR format image data to be processed, encapsulated in OpenCV, to the YUV420P format required by the H.264 encoder. Next, perform H.264 compression encoding. Finally, push the encoded data to the streaming media server according to the RTSP protocol. The image matrix, converted to YUV420P format, is then compressed using an H.264 encoder to generate an H.264 encoded video stream. The encoded video stream is pushed using the RTSP protocol.

[0024] The real-time operation process of the streaming end is as follows: read the BGR format pixel data of the virtual camera through the graphics rendering interface and store it in the OpenCV data structure; Use the libswscale library to convert BGR format to YUV420P format; The YUV420P image data is compressed and encoded using the libavcodec library using H.264. The libavformat library is used to encapsulate the encoded data packets and push them to the streaming media server via the RTSP protocol.

[0025] Figure 4This is a flowchart illustrating the overall construction process of the present invention; The method for creating a virtual scene testing platform that integrates video stream feedback, according to any one of the descriptions, includes the following steps: S1: The construction process of the push end is as follows: In UE4, a virtual test scene is built, test objects are placed, and virtual camera components are configured on the test objects. The frame buffer resources of the virtual camera components are obtained through the graphics rendering interface. The OpenCV storage data structure is initialized for image data management. The FFmpeg libavformat library is used to establish a connection with the RTSP server. The FFmpeg libavcodec library is used to initialize the H.264 encoder and configure encoding parameters (resolution, bitrate, GOP size, encoding preset, etc.). The FFmpeg libswscale library is used to initialize the pixel format converter, completing the creation of the streaming end. The test object can be a drone or an unmanned surface vessel. S2: Deploy an independent RTSP streaming media server to receive H.264 encoded streams from the streaming client, manage and distribute streaming media based on the RTSP protocol, and support multiple client streaming. S3: Establish a connection with the RTSP server through FFmpeg's libavformat library to pull real-time video streams; use the libavcodec library to decode the H.264 stream and restore it to image frame data; use OpenCV to render the decoded image data to the display window, thus realizing the creation and operation of the streaming end.

[0026] Furthermore, the method also includes obtaining scene rendering source data in UE4, as follows: Step 1.1: Open the engine plugin manager, activate the "Composure" plugin group, and restart the engine to ensure that the plugin configuration takes effect; Step 1.2: Create a test object and a virtual camera component in the 3D scene view. Select "Create → Camera → CineCameraActor" in the scene editor menu bar, and deploy the camera entity at the predetermined observation point in the scene coordinate system. Adjust its focal length, field of view and depth of field parameters to adapt to the target rendering scene. Step 1.3: Build a compositing pipeline to achieve layered image processing. In the engine interface, call the "Composure" function panel, create a new blank compositing lens node, and add a "CG Layer" child node by right-clicking in the hierarchy tree. Bind the CineCameraActor in the world outline view to the CG Layer, and establish the relationship between the camera and the compositing lens through the "Target Camera Actor" parameter in the properties panel. Step 1.4: Configure the material mapping channel for the composited shot to optimize the output quality; in the "Transform / Compositing Passes" module of the composited shot, add a "compositing element material channel" and create a post-processing material; Step 1.5: Set the output format and storage path of the rendering target. In the "Output" module of the compositing lens, select the "Rendering Target Compositing Output" mode, create the rendering target resource, and associate it with the compositing lens node. Through the engine's built-in frame buffer management mechanism, the rendered image is updated frame by frame and persistently stored.

[0027] Furthermore, the material includes TextureSampleParameter2D type input nodes, the parameter names of the input nodes strictly match the CG Layer names, and the material output is connected to a self-illuminating channel; Enable sRGB color space conversion and anti-aliasing filtering in the material properties.

[0028] Furthermore, the H.264 encoded stream received by the streaming end is managed and distributed based on the RTSP protocol, and the process of supporting multiple clients pulling the stream is as follows: Step 2.1: Use OpenCV to read the pixel data of the UE4 rendering target and store it as a BGR format video frame. Then, convert the format of the acquired video frame from BGR to YUV420P format to meet the input requirements of FFmpeg for H.264 encoding. Step 2.2: Perform H.264 encoding using FFmpeg. First, establish the context required for encoding and set the encoding parameters, including but not limited to resolution, frame rate, and bit rate. Then, input the YUV420P format video frame data into the encoding unit for compression processing and extract the encoded data packets for subsequent transmission. Where: Encoding unit = H.264 encoder algorithm + encoder context (parameter configuration) + frame buffer + packet buffer; Step 2.3: To enable remote push of the video stream, the encoded data is encapsulated using FFmpeg, and the encoded data packets are organized in chronological order and transmitted to the RTSP server for the client to pull and play.

[0029] Furthermore: the process of establishing a connection with the RTSP server through FFmpeg's libavformat library to pull the real-time video stream; using the libavcodec library to decode the H.264 stream and restore it to image frame data; and rendering the decoded image data to the display window using OpenCV, thus realizing the creation and operation of the streaming client, is as follows: Before pulling the video stream, the network environment is first initialized and a connection with the RTSP server is established. After the connection is successful, the video stream information is parsed to obtain basic parameters such as frame rate, resolution, and encoding format.

[0030] FFmpeg parses video stream data, traverses multiple data streams contained in the video stream, identifies the video stream index, retrieves the decoding tool that matches the video stream, establishes the parameters and context information required for decoding, and completes the initialization of the decoding unit; The decoding unit consists of: H.264 decoder algorithm + decoder context (parameter configuration) + packet buffer + frame buffer. The system receives video data packets from the RTSP server and parses them to determine whether they belong to video stream data. If they are video stream data, they are input to the decoding unit for processing, and the decoded video frame data is obtained. The acquired YUV420P format video frames are converted to BGR format, which can be processed by OpenCV, for subsequent display. The video frame data in BGR format is stored in the cv::Mat data structure of OpenCV, and the cv::imshow function is called to display the video screen. This function uses the graphical user interface API of the operating system to transfer the BGR pixel data in memory to the graphics card and render it to the display window, so as to realize the visualization of the test object running screen in the real-time virtual test scene.

[0031] Furthermore, a testing method for a virtual scene testing platform that integrates video stream feedback includes the following steps: S41: The test subject is subjected to multi-functional testing of target locking, environmental awareness and autonomous navigation on a platform that integrates UE4 and video stream; S42: Through simulation and calculation using multiple sensors, the drone achieves comprehensive environmental perception. S43: The test platform provides a UE4-based simulation environment for test objects (such as drones) to load and run input multi-functional algorithms (such as navigation, obstacle avoidance, etc.).

[0032] Steps S41 / S42 / S43 are executed in parallel; The test object is subjected to multi-functional tests such as target locking, environmental awareness, and autonomous navigation on a platform that integrates UE4 and video streams. It supports controllable latency and adjustable frame rate to adapt to different test scenario requirements. The specific process is as follows: S411: Uses Unreal Engine's AI perception system to identify targets and controls the drone to perform locking operations via behavior trees.

[0033] S412: Calls OpenCV to process the video stream, combines deep learning algorithms for target recognition and tracking, and improves target detection accuracy.

[0034] S413: Utilizes light projection to calculate the spatial distance between the UAV and the target, optimizing target tracking stability and ensuring the continuity and accuracy of tracking.

[0035] The process involves simulating and calculating using multiple sensors to achieve comprehensive environmental perception for the drone. The specific steps are as follows: S421: Uses a depth camera to collect environmental information, identify passable areas, and assist the drone in path planning.

[0036] S422: Based on a collision detection mechanism, it identifies obstacles and adjusts the drone's flight path in real time to improve its autonomous obstacle avoidance capabilities.

[0037] The testing platform provides a UE4-based simulation environment for test objects (such as drones) to load and run input multi-functional algorithms (such as navigation, obstacle avoidance, etc.). The specific steps are as follows: S431: By integrating UE4's NavMesh system and 3D environment model, it provides the test object with complete scene information and passable area data; S432: Supports flexible configuration of test environment parameters, such as obstacle density and dynamic elements, to meet different test requirements; S433: The test object can load input multi-functional algorithms and run independently in the platform environment. The platform does not interfere with the execution of the algorithm, but only provides data collection, test recording and visualization support.

[0038] Through simulation and calculation using multiple sensors, the drone achieves comprehensive environmental perception. The simulation environment based on UE4 allows test objects to load and run input-based multi-functional navigation and obstacle avoidance algorithms for testing.

[0039] Example 1 A method for constructing a virtual scene testing platform that integrates video stream feedback includes the following steps: S101: The construction process of the streaming end is as follows: Build a virtual test scene in UE4, place test objects, configure virtual camera components on the test objects, and obtain the frame buffer resources of the virtual camera components through the graphics rendering interface; initialize the OpenCV storage data structure for image data management; establish a connection with the RTSP server using the FFmpeg libavformat library; initialize the H.264 encoder and configure the encoding parameters using the FFmpeg libavcodec library; initialize the pixel format converter using the FFmpeg libswscale library, and complete the creation of the streaming end.

[0040] S102: Deploy an independent RTSP streaming media server to receive H.264 encoded streams from the streaming client, manage and distribute streaming media based on the RTSP protocol, and support multiple client streaming. S103: Establish a connection with the RTSP server through FFmpeg's libavformat library to pull real-time video streams; use the libavcodec library to decode the H.264 stream and restore it to image frame data; use OpenCV to render the decoded image data to the display window, thus realizing the creation and operation of the streaming end.

[0041] Step S101 specifically includes: Create a virtual camera component with configurable parameters in the logic control thread; Obtain the frame buffer resource descriptor on the GPU side through the render target access method of the graphics device interface; The pixel data migration interface is called to transfer the rendered image in the video memory to the system memory, forming an image matrix to be processed.

[0042] Step S102 includes: Raw video frame data is acquired through the image acquisition interface; Activate the color space conversion module to convert BGR format to the standard video encoding YUV420P format; Call the frame submission interface of the video encoder to send the converted data into the encoding pipeline; The encoded video stream is pushed to the streaming media distribution node via the network transmission interface.

[0043] In step S103, a streaming media client C++ streaming media player is built. It controls the pulling of real-time video streams through the RTSP protocol. The video data is transmitted via RTP, decoded using FFmpeg, and displayed in real-time using OpenCV.

[0044] In step S103, the test subject performs target locking, environmental perception, and autonomous navigation capability tests. During target locking, the AI ​​perception system and deep learning algorithms identify and track the target. In the environmental perception phase, LIDAR sensors and depth cameras collect environmental information, and a collision detection mechanism optimizes path planning. In the autonomous navigation phase, NavMesh and A-path search algorithms enable dynamic obstacle avoidance and intelligent decision-making. Throughout the testing process, controllable latency and adjustable frame rate are supported to meet the needs of different testing scenarios.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual scene testing platform integrating video stream feedback, characterized in that: include: Push end: Used to obtain frame buffer resources of virtual camera components in virtual scenes, integrate virtual engine rendering and video encoding methods, and realize the push of virtual scene video stream; RTSP server: Used to receive H.264 encoded video streams from the streaming end, acting as a streaming media relay node, supporting real-time access and distribution of multiple concurrent video streams, and remote transmission via the RTSP protocol; Multiple streaming ends: used to receive H.264 video streams transmitted from RTSP servers, decode and restore them using FFmpeg, and display the running screen of the test object in the real-time virtual test scene using OpenCV.

2. The virtual scene testing platform integrating video stream feedback according to claim 1, characterized in that: The process of acquiring frame buffer resources of virtual camera components in a virtual scene, integrating virtual engine rendering and video encoding methods, and realizing the push of virtual scene video streams is as follows: Used to obtain the frame buffer resources of the virtual camera component in the virtual scene and generate virtual scene rendering source data in BGR format; OpenCV is used to encapsulate, process, and preview the BGR format virtual scene rendering source data, and output the image matrix to be processed. Use FFmpeg's pixel format conversion module to convert the image matrix to be processed from BGR format to YUV420P format; The YUV420P format image matrix is ​​compressed using an H.264 encoder to generate an H.264 encoded video stream. The encoded video stream is pushed using the RTSP protocol.

3. The virtual scene testing platform integrating video stream feedback according to claim 1, characterized in that, The real-time operation process of the streaming end is as follows: The BGR format pixel data of the virtual camera is read through the graphics rendering interface and stored in the OpenCV data structure; Use the libswscale library to convert BGR format to YUV420P format; The YUV420P image data is compressed and encoded using the libavcodec library using H.

264. The libavformat library is used to encapsulate the encoded data packets and push them to the streaming media server via the RTSP protocol.

4. The method for creating a virtual scene testing platform integrating video stream feedback according to any one of claims 1-3, characterized in that, Includes the following steps: The construction process of the push-stream end is as follows: In UE4, a virtual test scene is built, test objects are placed, virtual camera components are configured for the test objects, and the frame buffer resources of the virtual camera components are obtained through the graphics rendering interface. Initialize OpenCV's storage data structures for image data management; Use FFmpeg's libavformat library to establish a connection with the RTSP server; Initialize the H.264 encoder and configure the encoding parameters using FFmpeg's libavcodec library; The pixel format converter is initialized using FFmpeg's libswscale library to complete the creation of the streaming end; Deploy an independent RTSP streaming media server to receive H.264 encoded streams from the push client, manage and distribute streaming media based on the RTSP protocol, and support multiple client pull streams; Establish a connection with the RTSP server using FFmpeg's libavformat library to pull real-time video streams; The H.264 stream is decoded using the libavcodec library and restored to image frame data; the decoded image data is then rendered to the display window using OpenCV, thus enabling the creation and operation of the streaming client.

5. The method for creating a virtual scene testing platform integrating video stream feedback according to claim 4, characterized in that, This also includes obtaining scene rendering source data from UE4, the process of which is as follows: Step 1.1: Open the engine plugin manager, activate the "Composure" plugin group, and restart the engine to ensure that the plugin configuration takes effect; Step 1.2: Create a test object and a virtual camera component in the 3D scene view. Select "Create → Camera → CineCameraActor" in the scene editor menu bar, and deploy the camera entity at the predetermined observation point in the scene coordinate system. Adjust its focal length, field of view and depth of field parameters to adapt to the target rendering scene. Step 1.3: Build a compositing rendering pipeline to achieve layered image processing. In the engine interface, call the "Composure" function panel, create a new blank compositing lens node, and add a "CG Layer" child node by right-clicking in the hierarchy tree. Bind the CineCameraActor in the world outline view to the CG Layer, and establish the relationship between the camera and the compositing lens through the "Target Camera Actor" parameter in the properties panel. Step 1.4: Configure the material mapping channel for the composited shot to optimize the output quality of the image; in the "Transform / Compositing Passes" module of the composited shot, add the "compositing element material channel" and create post-processing materials; Step 1.5: Set the output format and storage path of the rendering target. In the "Output" module of the compositing lens, select the "Rendering Target Compositing Output" mode, create the rendering target resource, and associate it with the compositing lens node. Through the engine's built-in frame buffer management mechanism, the rendered image is updated frame by frame and persistently stored.

6. The method for creating a virtual scene testing platform integrating video stream feedback according to claim 5, characterized in that: The material contains TextureSampleParameter2D type input nodes, the parameter names of the input nodes strictly match the CGLayer names, and the material output is connected to the self-illuminating channel; Enable sRGB color space conversion and anti-aliasing filtering in the material properties.

7. The method for creating a virtual scene testing platform integrating video stream feedback according to claim 4, characterized in that: The H.264 encoded stream received by the receiving end is managed and distributed based on the RTSP protocol, and the process of supporting multiple clients pulling the stream is as follows: Step 2.1: Read the pixel data of the rendering target through the UE4 graphics rendering interface and store the pixel data in BGR format into the OpenCV data structure; The libswscale library of FFmpeg is used to perform pixel format conversion, converting the BGR format to YUV420P format to meet the input requirements of the H.264 encoder; Step 2.2: Use FFmpeg's libavcodec library to perform H.264 encoding. Input the YUV420P format video frame data into the encoder for compression through the avcodec_send_frame function, and extract the encoded data packets through the avcodec_receive_packet function for subsequent transmission. Step 2.3: Use FFmpeg's libavformat library to encapsulate the encoded data, use the av_interleaved_write_frame function to organize the encoded data packets in chronological order, and push the encapsulated H.264 video stream to the streaming media server for transmission via the RTSP protocol.

8. The method for creating a virtual scene testing platform integrating video stream feedback according to claim 3, characterized in that: The process of creating and running the streaming client involves establishing a connection with the RTSP server using FFmpeg's libavformat library to pull the real-time video stream; decoding the H.264 stream using the libavcodec library to restore it to image frame data; and rendering the decoded image data to the display window using OpenCV. FFmpeg parses video stream data, traverses multiple data streams contained in the video stream, identifies the video stream index, retrieves the decoding tool that matches the video stream, establishes the parameters and context information required for decoding, and completes the initialization of the decoding unit; Receive video data packets from the RTSP server and parse the video data packets to determine whether they belong to video stream data; If it is video stream data, it is input to the decoding unit for processing, and the decoded video frame data is obtained; The acquired YUV420P format video frames are converted to BGR format, which can be processed by OpenCV, for subsequent display. The video frame data in BGR format is stored in the cv::Mat data structure of OpenCV, and the cv::imshow function is called to display the video screen. This function uses the graphical user interface API of the operating system to transfer the BGR pixel data in memory to the graphics card and render it to the display window, so as to realize the visualization of the test object running screen in the real-time virtual test scene.

9. A testing method for a virtual scene testing platform integrating video stream feedback according to any one of claims 1-3, characterized in that: Includes the following steps: The test subject was subjected to multi-functional testing of target locking, environmental awareness, and autonomous navigation on a platform that integrates UE4 and video streaming; Through simulation and calculation using multiple sensors, the drone achieves comprehensive environmental perception. The simulation environment based on UE4 allows test objects to load and run input-based multi-functional navigation and obstacle avoidance algorithms for testing.

Images