A method for data generation and automatic annotation of highly realistic virtual environments

Abstract

The present invention discloses a method for generating and automatically annotating data for a high-fidelity virtual environment, which relates to the fields of data generation and drone simulation technology. The method comprises the following steps: using an Unreal Engine to render a virtual simulation environment, obtaining and exporting a scene camera rendered image; performing real-time image encoding and single-frame image transmission on the scene camera rendered image through a customized network protocol and transmission control protocol; utilizing the environmental projection principle and target key point coordinate information to convert the target world coordinates into camera coordinates, extract target annotation information, and complete the overall data acquisition and annotation tasks. The present invention can achieve real-time and stable extraction of simulated images without dropping frames, realize low-latency single-frame image data transmission tasks, solve the drawback that the Unreal Engine can only transfer coordinates to the screen coordinate system, and support image acquisition and automatic annotation tasks for multiple cameras.

Description

Technical Field

[0001] The present invention relates to the technical field of data generation and unmanned aerial vehicle simulation, and more particularly to a method for generating and automatically labeling data for a high-fidelity virtual environment. Background Art

[0002] Visual algorithms are a hot topic in the current drone industry, and visual data is crucial for these algorithms. Currently, due to concerns about experimental costs and safety, both the autonomous driving and drone sectors are placing significant emphasis on the development and application of digital twin simulation systems. Common digital twin platforms include Unity and Unreal Engine, which are representative of game rendering engines, and Tai Chi and Blender, which are representative of interface modeling. In recent years, due to the high realism and user-friendly interface of Unreal Engine's game rendering engine, researchers have proposed a variety of simulation platforms, including CARLA, AIRSIM, and URDFSim. However, as a game engine, Unreal's overall data communication and image processing interfaces are not well-developed, making it difficult to support high-definition image acquisition and transmission.

[0003] There are currently several ways to obtain visual data from Unreal Engine:

[0004] (1) HDMI image acquisition card: In various simulation systems, the simulation environment can be constructed according to the needs and rendered into realistic image data. The image data is transmitted to the HDMI image acquisition card through the HDMI interface. This method obtains the image of the entire screen and requires secondary processing to intercept the useful part as visual image data;

[0005] (2) Window capture: Any application with a window can obtain the main window handle through the operating system API and obtain the device context for screenshots. In essence, it captures an image at a specified position and size on the current screen.

[0006] (3) PixelStreaming plug-in of Unreal Engine: This plug-in allows playing Unreal projects from any Internet-connected device with a screen (such as computers, mobile phones, tablets, etc.). The principle is that the plug-in obtains the final rendered picture of the frame from the rendering thread of Unreal Engine through a callback function, and encodes the final result of each rendered frame using H.264 video compression, packages these video frames together with the game audio into a media stream, and sends the stream to one or more connected browsers through a direct point-to-point connection.

[0007] In recent years, Unreal Engine-based simulation systems have become increasingly popular for visual clustering algorithms. Unreal Engine can provide a highly realistic virtual simulation environment, providing more realistic simulation data for visual clustering algorithms. However, there are various limitations in the export method of simulated visual image data, including:

[0008] (1) When the display resolution is insufficient or the simulation main window is not full screen, the quality of the captured rendered image is low, or the image needs to be cropped, making it difficult to use effectively and unable to meet the requirements of the visual clustering algorithm;

[0009] (2) The exported image data needs to be transmitted to the algorithm module in real time and effectively. The commonly used UDP-based image transmission can ensure that the data is transmitted to the algorithm module in real time, but packet loss will cause serious distortion of the image data, increasing the difficulty of algorithm verification.

[0010] (3) The visual image data obtained through simulation needs to be processed through image recognition algorithms to extract image information. As training data, it also needs to be manually labeled, which is extremely labor-intensive. In addition, the scale of manual labeling is not uniform and it is easy to miss or mislabel the data, which affects the performance of the visual algorithm.

[0011] Therefore, how to solve the problems of difficulty in collecting image data in special environments, high cost and non-uniform scale of drone visual information annotation is a technical problem that technicians in this field urgently need to solve. Summary of the Invention

[0012] In view of this, the present invention provides a data generation and automatic annotation method for a high-fidelity virtual environment, in which the image quality is fixed and controllable, which can ensure the efficiency and effectiveness of image data transmission and complete the overall data collection and annotation tasks.

[0013] In order to achieve the above object, the present invention provides the following technical solutions:

[0014] A method for generating and automatically labeling data for a highly realistic virtual environment comprises the following steps:

[0015] Use Unreal Engine to render the virtual simulation environment, obtain the scene camera rendering image and export it;

[0016] Through customized network protocols and transmission control protocols, the rendered images of the view camera are encoded in real time and transmitted as single frames.

[0017] By using the environmental projection principle and the target key point coordinate information, the target world coordinates are converted into camera coordinates, the target annotation information is extracted, and the automatic annotation task is completed.

[0018] The technical effect achieved by the above technical solution is: solving the current shortcomings of Unreal Engine, such as insufficient image interface, easy packet loss in single-frame image transmission, large data transmission volume, high transmission delay, and the fact that Unreal Engine can only transfer coordinates to the screen coordinate system, and supporting multi-camera image acquisition and automatic labeling tasks.

[0019] Optionally, Unreal Engine includes a normal camera module and a movie camera module, including:

[0020] The ordinary camera module converts the collected image data into a 2D pixel map and performs post-processing, exports the low-motion image and stores it; the adjustable parameters of the ordinary camera module include aspect ratio and horizontal field of view;

[0021] The adjustable parameters of the movie camera module include pixel size, lens focal length, lens aperture, aspect ratio, horizontal field of view, tracking settings, and camera focal length.

[0022] Optionally, Unreal Engine is used to render a virtual simulation environment, which specifically includes the following steps:

[0023] Use the synthetic compression plug-in to convert the image of the movie camera into texture materials in real time;

[0024] Send the texture material to the background rendering thread for separate rendering to obtain image data.

[0025] Optionally, before converting the image from the movie camera into a texture material in real time, the method further includes:

[0026] Import the image from the movie camera into the CG layer for display, and then import the rendering data in the CG layer into a texture material for saving.

[0027] Optionally, the texture material is sent to the background rendering thread for separate rendering, which specifically includes the following steps:

[0028] Use the backend rendering thread to render the texture material separately in the background using the graphics processor, and automatically start a releasable image encoding thread after the rendering data is completed;

[0029] The original image data in the texture material is compressed into a regular image readable format, and the forwarding or storage task is completed according to the image encoding thread.

[0030] Optionally, before performing conventional image readable compression, the method further includes:

[0031] use The hyperparameters of the texture material are gamma corrected to complete the color restoration task of the image;

[0032] Among them, the formula for gamma correction is:

[0033] ;

[0034] Where: L Indicates the correction value, R Indicates the red channel value before correction, G Indicates the green channel value before correction. B Indicates the blue channel value before correction.

[0035] The technical effects achieved by the above technical solution are: a dedicated high-dynamic image data acquisition method has been developed for the Unreal Engine, which is divided into two parts: image acquisition and compression encoding. It can solve the problems of high-dynamic image data generated by movie cameras that cannot be exported, software delays and freezes, and enable multiple nodes to execute image acquisition tasks simultaneously within the hardware's permitted range; gamma correction is used to solve the problem that the overall picture contrast and brightness of high-dynamic images are significantly lower than the actual rendering data when exported, which can achieve better working results.

[0036] Optionally, performing real-time image encoding and single-frame image transmission on the scene camera rendered image specifically includes the following steps:

[0037] Design an image link protocol, and set the identification bits of the image link protocol to the protocol start bit, the current received image frame number, the current image split packet number, the sequence number of the current received packet in the current image packet, the current packet valid data length, the current transmission data type, the image encoding data, and the protocol check bit in sequence;

[0038] When a data packet is lost, a response is immediately generated using the transmission control protocol, the number of packets split from the current image, and the identification bit of the sequence number of the current received packet in the current image packet, directly omitting the current decoded image;

[0039] When the check bit is read, first calculate whether the data length is consistent with the identification bit of the current packet's valid data length to determine whether the decoding is complete, and then determine whether the protocol check bit is consistent.

[0040] The technical effect achieved by the above technical solution is: it can ensure the stability and real-time performance of image data transmission and support real-time information transmission tasks.

[0041] Optionally, convert the target world coordinates to camera coordinates, specifically including the following steps:

[0042] The transformation matrix composed of a rotation matrix in the form of homogeneous coordinates and a translation vector represents the conversion relationship between the world coordinate system, the body coordinate system and the camera coordinate system; when the pitch angle corresponding to the coordinate system is known , roll angle , yaw angle When , the expression of the rotation matrix is:

[0043] ;

[0044] World coordinate system midpoint and in the camera coordinate system The corresponding relationship between them is expressed as:

[0045] ;

[0046] Where: C represents the camera intrinsic parameter matrix, r Represents the camera center offset vector;

[0047] Using the camera's position vector and the rotation matrix And the projection matrix obtained by correcting the camera's own parameters , calculate the visual projection matrix under the current viewing angle :

[0048] ;

[0049] Where: Indicates that the camera is in the system x Position offset in the direction, Indicates that the camera is in the system y Position offset in the direction, Indicates that the camera is in the system z Position offset of direction;

[0050] Will and the position of the target in the world coordinate system Calculate and get the coordinates of the target in the camera coordinate system ,in, It is used to normalize the calculated XY data. The final coordinate projection result is:

[0051] ;

[0052] Where: and To obtain the length and width of the image; The pixel coordinates of the world point in the current image.

[0053] Optionally, when performing the automatic labeling task, the method further includes:

[0054] The maximum intersection-over-union ratio and the annotation area between all the annotation results of the current image are calculated. When the calculation result does not meet the preset threshold, the annotation result is selected or discarded to complete the automatic annotation task of the current image.

[0055] Optionally, the method further includes:

[0056] Calculate the current trigger frame number through external trigger software and send it to the rendering part of Unreal Engine;

[0057] After the Unreal Engine completes the image reading and automatic annotation tasks, it transmits the data to the external trigger software through the network port;

[0058] The external trigger software aligns the obtained images and results using the frame number and stores them in the corresponding hard disk space, completing the complete data labeling task.

[0059] Through the above technical solutions, it can be seen that compared with the prior art, the present invention discloses a method for generating and automatically annotating data of a highly restored virtual environment, which has the following beneficial effects:

[0060] (1) The present invention designs a real-time image reading and encoding method for high-dynamic images, which solves the problem of insufficient image interface in the current Unreal Engine and can achieve real-time and stable extraction of simulated images without frame drops;

[0061] (2) This invention proposes a new image transmission protocol, FrameLink (image link protocol), which solves the problems of easy packet loss, large data transmission volume, and high transmission delay in single-frame image transmission, and can realize the task of low-latency single-frame image data transmission;

[0062] (3) The present invention proposes a method for automatic annotation of simulation platform data based on Unreal Engine camera projection parameters and projection rules. This method can solve the disadvantage that Unreal Engine can only transfer coordinates to the screen coordinate system. It is more compatible with the high-dynamic image reading method of this technical solution and can simultaneously support image acquisition and automatic annotation tasks of multiple cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0064] Figure 1 A flow chart of the method for generating and automatically annotating data for a highly reductive virtual environment provided by the present invention;

[0065] Figure 2 A high dynamic image reading flow chart provided by the present invention;

[0066] Figure 3 A schematic diagram of the image link protocol provided by the present invention;

[0067] Figure 4 A schematic diagram of the camera projection model provided by the present invention;

[0068] Figure 5 This is a schematic diagram of the external trigger marking and data transmission results provided by the present invention. DETAILED DESCRIPTION

[0069] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0070] The embodiment of the present invention discloses a method for generating and automatically labeling data of a high-fidelity virtual environment. Figure 1 As shown, the following steps are included:

[0071] Use Unreal Engine to render a highly realistic virtual simulation environment, obtain and export the rendered image from the view camera;

[0072] Through customized network protocols and TCP transmission (Transmission Control Protocol), the rendered images of the view camera are encoded in real time and transmitted as single frames.

[0073] By using the environmental projection principle and the target key point coordinate information, the target world coordinates are converted into camera coordinates, the target annotation information is extracted, and the automatic annotation task is completed.

[0074] Figure 1 The steps shown can be summarized as follows: a method for exporting and storing high-dynamic images based on the Unreal Engine camera system, a method for real-time image encoding and single-frame image transmission based on TCP, and an automatic labeling method based on the Unreal Engine view camera projection matrix. Figure 1 The technical solution shown is described in detail.

[0075] As one of the two leading game engines in the industry, Unreal Engine can perform high-precision modeling and real-time dynamic rendering, and is widely used in scenes such as film and television CG production. It can maintain extremely high realism and scene restoration, and is very suitable for using this environment for data collection to enrich the data sets used by intelligent algorithms in high-intelligence platforms such as drones, unmanned vehicles, and robots.

[0076] Furthermore, the Unreal Engine includes a normal camera module and a movie camera module, where:

[0077] Ordinary camera modules convert captured image data into 2D pixel maps and perform post-processing such as depth and infrared to export and store low-motion images. This method, which is built into the Unreal Engine, has limited adjustable parameters, including only aspect ratio and horizontal field of view. Due to its built-in post-processing capabilities, images can be significantly distorted during export, making it unsuitable for visual dataset acquisition.

[0078] The movie camera module has many adjustable parameters, including pixel size, lens focal length, lens aperture, aspect ratio, horizontal field of view, tracking settings, and camera focal length. It can greatly restore the real camera parameters and obtain images without distortion. However, since the obtained images are high-dynamic images, they cannot be exported or post-processed, and therefore cannot support dataset production.

[0079] Furthermore, in order to better complete the image data acquisition task, this embodiment develops a set of dedicated HDR (high dynamic range) image data acquisition methods for Unreal Engine. The overall solution process is as follows: Figure 2 As shown, the image data capture method can be divided into two steps, namely image acquisition and compression encoding.

[0080] Specifically, the film post-processing plug-in Composure is used to convert cinema camera images into texture assets in real time. Prior to this, the cinema camera image must first be imported into a CG layer for display, and then the rendered data from the CG layer must be imported into a texture asset for storage. This method solves the problem of HDR image data generated by cinema cameras being unable to be exported. The capture process operates in the software's main thread, performing only data reading tasks without CPU or GPU processing. This ensures high real-time performance and low algorithm latency.

[0081] The compression and encoding steps for the raw image data are more complex. This is because the Unreal Engine's built-in functions cannot directly process and store texture materials. Directly exporting the raw data to the hard disk will perform a large amount of image processing operations directly in the main thread, blocking the main thread. When exporting images, the program's running frame rate may even drop to less than 10 frames, which cannot meet normal usage requirements. Therefore, this embodiment performs the following processing in the post-processing step:

[0082] The RHI (backend rendering engine) thread renders textures independently in the background using the GPU (graphics processing unit). Once the rendering data is complete, a releasable image encoding thread is automatically started. The read image data undergoes standard image-readable compression. Subsequently, tasks initiated by the thread can be forwarded or stored. This resolves software latency and lag issues, and allows multiple nodes to simultaneously execute image acquisition tasks within the hardware's capabilities.

[0083] In addition, the overall image contrast and brightness of the HDR image when exported will be significantly lower than the actual rendering data. To further solve this problem, this embodiment uses The hyperparameters of the texture material are gamma corrected to complete the color restoration task of the image;

[0084] Among them, the formula for gamma correction is:

[0085] ;

[0086] Where: L Indicates the correction value, R Indicates the red channel value before correction, G Indicates the green channel value before correction. B Indicates the blue channel value before correction;

[0087] This method uses the Unreal Engine's unique, automatically releasable independent thread startup mode to directly release the corresponding rendering thread after completing the rendering of a single image. Therefore, the computer's memory and video memory will not continue to grow during the overall work process, but will remain in a basically stable state, and the overall working effect is good.

[0088] Furthermore, the TCP-based real-time image encoding and single-frame image transmission method is as follows:

[0089] Unreal Engine itself provides a PixelStreaming plug-in, which is used to transmit the display interface and control instructions of the current project using the TCP network and H.264. This method directly reads the rendering data of the rendering thread and encodes the rendered video data in H.264. This is similar to the idea of ​​this embodiment. However, as an automatic annotation method, this technical solution itself needs to align the frame number of the image with the annotation information according to the timestamp. At this time, the overall effect of the encoding method of the video data is difficult to achieve the ideal effect. For this reason, this embodiment specially designs a FrameLink (image link) protocol to complete the real-time image forwarding task based on the TCP method. The specific protocol content is as follows. Figure 3 and as shown in Table 1.

[0090] Table 1 FrameLink (Image Link Protocol) details

[0091] Flag Data Type Content Description HD uint8 Protocol start bit, default is 0x86 SEQ uint16 Current received image frame number PN uint8 Current number of image split packages PS uint8 The sequence number of the current received packet in the current image packet LEN uint16 Valid data length of the current packet MSG uint8 Current transfer data type PAYLOAD uint8* Image coding data FCS uint8 Protocol check digit, which is the final result after data accumulation

[0092] Utilizing TCP transmission stability and the PN and PS check bits, an immediate response is generated when a packet is lost, bypassing the currently decoded image. LEN and PAYLOAD definitions ensure that the corresponding data length and number of bits are read. When the check bit is read, the data length is first calculated to ensure consistency with LEN to ensure decoding is complete, and then the consistency of the check bit is determined. This simple protocol transmission method ensures stable and real-time image data transmission, supporting real-time information delivery tasks.

[0093] Furthermore, the automatic annotation method based on the Unreal Engine view camera projection matrix is ​​as follows:

[0094] Figure 4 This is a diagram showing the relationship between the four coordinate systems commonly used in the field of computer vision, namely the pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system. This diagram can well show the relative relationship between the five coordinate systems used in computer vision image processing (among which the world coordinate system: ; Body coordinate system: ;Camera coordinate system: ; Image coordinate system: ;Pixel coordinate system: It is worth mentioning that the camera coordinate system takes the camera optical center as its origin, and the body coordinate system takes the center of mass of the drone as its origin. When the drone adopts the strapdown guidance method, the camera coordinate system and the body coordinate system can obtain a transformation matrix in three-dimensional space that satisfies the special Euclidean group, realizing the conversion of pixel point information to local space information.

[0095] The principle of automatic annotation is based on this projection relationship. It projects the target's world coordinates to pixel coordinates. All three-dimensional coordinate systems belong to the Euler coordinate system, which includes three degrees of freedom for translation and three degrees of freedom for rotation. Rotation is generally represented using methods such as rotation vectors, quaternions, rotation matrices, and Gibbs vectors. The current pose is represented by combining the rotation matrix with the translation vector representing translation.

[0096] Specifically, the transformation matrix composed of a rotation matrix in the form of homogeneous coordinates and a translation vector represents the transformation relationship between the world coordinate system, the body coordinate system, and the camera coordinate system; when the pitch angle corresponding to the coordinate system is known, , roll angle , yaw angle When , the expression of the rotation matrix is:

[0097] ;

[0098] World coordinate system midpoint and in the camera coordinate system The corresponding relationship between them is expressed as:

[0099] ;

[0100] Where: C represents the camera intrinsic parameter matrix, r Represents the camera center offset vector;

[0101] In Unreal Engine, since the camera itself needs to use the camera's projection relationship when rendering, the camera's position vector can be directly used. and the rotation matrix And the projection matrix obtained by correcting the camera's own parameters , calculate the visual projection matrix under the current viewing angle :

[0102] ;

[0103] Where: Indicates that the camera is in the system x Position offset in the direction, Indicates that the camera is in the system y Position offset in the direction, Indicates that the camera is in the system z Position offset of direction;

[0104] Afterwards, directly and the position of the target in the world coordinate system Calculate and get the coordinates of the target in the camera coordinate system ,in, It is used to normalize the calculated XY data. The final coordinate projection result is:

[0105] ;

[0106] Where: and To obtain the length and width of the image; The pixel coordinates of the world point in the current image.

[0107] Using this method, we can complete multiple automatic labeling tasks such as target bounding box labeling and key point labeling. Taking the target bounding box labeling as an example, the world coordinates of the eight spatial vertices of a target are summarized into a list, and the coordinates of each vertex are converted. x and yThe two maximum values ​​in the direction are compared with the image's length and width, and zero, respectively, to obtain the final annotation result. Next, to account for overlapping and small object annotation, the maximum IoU (Intersection over Union) and the annotation area among all annotation results for the current image are calculated. If the value does not meet the preset threshold, the annotation result is selected or discarded, completing the automatic annotation task for the current image. This method can distinguish target objects by category and generate more realistic annotation information based on the aspect ratio parameters of different objects.

[0108] Furthermore, in order to better match the automatic annotation task and image delivery results, the automatic annotation method needs to use an external UDP trigger method to align the image and annotation results. The overall solution flow chart is as follows Figure 5 As shown:

[0109] Calculate the current trigger frame number through external trigger software and send it to the rendering part of Unreal Engine;

[0110] After the Unreal Engine completes the image reading and automatic annotation tasks, it transmits the data to the external trigger software through the network port;

[0111] The external trigger software aligns the obtained images and results using the frame number and stores them in the corresponding hard disk space, completing the complete data labeling task.

[0112] This embodiment designs a data generation and automatic annotation method for a high-fidelity virtual environment. This method uses the Unreal Engine to render a high-fidelity virtual simulation environment. To address the problem of insufficient resolution in capturing simulated rendered images, a HDR image export and storage method is designed to directly obtain and export images rendered by a scene camera, with fixed and controllable image quality. To address the problem of unguaranteed effectiveness of image data transmission, a real-time image encoding and single-frame image transmission method is designed. Through a custom network protocol and reliable TCP transmission, the efficiency and effectiveness of image data transmission are guaranteed. To address the cost of manual image annotation, an automatic annotation method based on scene image projection is designed. By utilizing the principle of environmental projection and the coordinate information of target key points, the target world coordinates are converted to camera coordinates, and the target annotation information is extracted to complete the overall data acquisition and annotation tasks.

[0113] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be apparent to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating and automatically labeling data of a highly realistic virtual environment, characterized in that: The following steps are involved: Use Unreal Engine to render the virtual simulation environment, obtain the scene camera rendering image and export it; Through the customized network protocol and transmission control protocol, the real-time image encoding and single-frame image transmission of the scene camera rendering image are carried out. Specifically, the image link protocol is designed, and the identification bits of the image link protocol are set in sequence to the protocol start bit, the current received image frame number, the current image split packet number, the sequence number of the current received packet in the current image packet, the current packet valid data length, the current transmission data type, the image encoding data, and the protocol check bit; When a data packet is lost, the system uses the transmission control protocol, the number of packets split from the current image, and the identification bit of the sequence number of the current received packet in the current image packet to generate an immediate response, directly omitting the current decoded image. When the check bit is read, the system first calculates whether the data length is consistent with the identification bit of the valid data length of the current packet to determine whether the decoding is complete, and then determines whether the protocol check bit is consistent. Using the environmental projection principle and the target key point coordinate information, the target world coordinates are converted to camera coordinates, the target annotation information is extracted, and the automatic annotation task is completed; The method further comprises: Calculate the current trigger frame number through external trigger software and send it to the rendering part of Unreal Engine; After the Unreal Engine completes the image reading and automatic annotation tasks, it transmits them to the external trigger software through the network port; the external trigger software aligns the obtained image and results using the frame number and stores them in the corresponding hard disk space, completing the complete data annotation task.

2. The method for generating and automatically labeling data of a high-fidelity virtual environment according to claim 1, characterized in that: Unreal Engine includes a normal camera module and a movie camera module, including: The ordinary camera module converts the collected image data into a 2D pixel map and performs post-processing, exports the low-motion image and stores it; the adjustable parameters of the ordinary camera module include aspect ratio and horizontal field of view; The adjustable parameters of the movie camera module include pixel size, lens focal length, lens aperture, aspect ratio, horizontal field of view, tracking settings, and camera focal length.

3. The method for generating and automatically labeling data of a high-fidelity virtual environment according to claim 2, characterized in that: Using Unreal Engine to render a virtual simulation environment includes the following steps: Use the synthetic compression plug-in to convert the image of the movie camera into texture materials in real time; Send the texture material to the background rendering thread for separate rendering to obtain image data.

4. The method for generating and automatically labeling data of a highly realistic virtual environment according to claim 3, wherein: Before converting the image from the movie camera into a texture material in real time, the method further includes: Import the image from the movie camera into the CG layer for display, and then import the rendering data in the CG layer into a texture material for saving.

5. The method for generating and automatically labeling data of a high-fidelity virtual environment according to claim 3, characterized in that: Send the texture material to the background rendering thread for separate rendering, which includes the following steps: Use the backend rendering thread to render the texture material separately in the background using the graphics processor, and automatically start a releasable image encoding thread after the rendering data is completed; The original image data in the texture material is compressed into a regular image readable format, and the forwarding or storage task is completed according to the image encoding thread.

6. The method for generating and automatically labeling data of a highly realistic virtual environment according to claim 5, characterized in that: Before performing conventional image readable compression, the method further comprises: Use the hyperparameter γ=2.2 to perform gamma correction on the texture material to complete the color restoration task of the image; Among them, the formula for gamma correction is: Where: L represents the correction value, R represents the red channel value before correction, G represents the green channel value before correction, and B represents the blue channel value before correction.

7. The method for generating and automatically labeling data of a high-fidelity virtual environment according to claim 1, characterized in that: Convert the target world coordinates to the camera coordinates, including the following steps: The transformation matrix composed of a rotation matrix in homogeneous coordinate form and a translation vector represents the conversion relationship between the world coordinate system, the body coordinate system, and the camera coordinate system. When the pitch angle θ, roll angle φ, and yaw angle ψ corresponding to the coordinate system are known, the expression of the rotation matrix is: The midpoint P(X W ,Y W ,Z W ) and P(X C ,Y C ,Z C ) is expressed as: Where: C represents the camera intrinsic parameter matrix, r represents the camera center offset vector; Using the camera's position vector L C and the rotation matrix R C And the projection matrix P obtained by correcting the camera's own parameters C , calculate the visual projection matrix M under the current viewing angle vp : Where: L cx Indicates the position offset of the camera in the x direction in the machine system, L cy Indicates the position offset of the camera in the y direction in the machine system, L cz Indicates the position offset of the camera in the z direction in the machine system; M vp and the target's position in the world coordinate system L t Calculate and get the coordinates of the target in the camera coordinate system [X C ,Y C ,Z C ,W C ], where W C It is used to normalize the calculated XY data. The final coordinate projection result is: Where: s w and s h To obtain the length and width of the image; (x, y) is the pixel coordinates of the world point coordinates in the current image.

8. The method for generating and automatically labeling data of a highly realistic virtual environment according to claim 1, wherein: When performing the automatic labeling task, the method further includes: The maximum intersection-over-union ratio and the annotation area between all the annotation results of the current image are calculated. When the calculation result does not meet the preset threshold, the annotation result is selected or discarded to complete the automatic annotation task of the current image.

Images