Methods, apparatus, devices, and storage media for constructing datasets based on 3D games.

Abstract

This application discloses a method, apparatus, device, and storage medium for constructing a dataset based on 3D games. The method first obtains a compiled game data scraping script for the target 3D game; runs the target 3D game and obtains game screen data based on the game data scraping script, wherein the game screen data includes image data and non-image data; then cleans the game screen data to obtain sample data to be labeled; finally, a data labeling system is constructed, and the sample data is labeled using the data labeling system. It can be seen that this application achieves the acquisition of synthetic data from 3D games, which is more efficient than data acquisition in the real world. Furthermore, the synthetic game data can extract corresponding semantic information from the game, which can greatly simplify the data labeling process, enabling data obtained from games to be used for model training.

Description

Technical Field

[0001] This invention relates to the field of computer graphics, and in particular to a method, apparatus, device, and storage medium for constructing datasets based on 3D games. Background Technology

[0002] With the continuous development of technology and the increasing demands of people, deep learning technology has become a hot topic in today's society. Deep learning is a new research direction in the field of machine learning. It learns the inherent patterns and representational levels of sample data to acquire features for downstream tasks, enabling machines to perform analytical and discriminative tasks like humans. With the rapid development of social media platforms, people generate massive amounts of data every day. This vast amount of data has also made training deep learning models more feasible. Currently, deep learning models have shown significant advantages in tasks such as text classification, image classification, and machine translation, with results far exceeding those of previous traditional methods.

[0003] Data is crucial in the training of deep learning models; in a sense, the quantity and quality of data determine the final performance of the model. However, in the real world, building a dataset is extremely expensive, often requiring a large number of people to spend a significant amount of time collecting, filtering, and labeling the data. Summary of the Invention

[0004] Based on this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for constructing a dataset based on 3D games, which can effectively solve the problems of high cost of collecting images from real environments and difficulty in obtaining accurate depth data.

[0005] Firstly, a method for constructing a dataset based on 3D games is provided, which includes:

[0006] A script to retrieve compiled game data from a target 3D game;

[0007] Run the target 3D game, and obtain game screen data based on the game data capture script. The game screen data includes image data and non-image data.

[0008] The game screen data is cleaned to obtain sample data to be labeled;

[0009] A data annotation system is constructed, and the sample data is annotated using the data annotation system.

[0010] Optionally, constructing a data annotation system and annotating the sample data using the data annotation system includes:

[0011] A data annotation system based on mobile terminals is constructed, the data annotation system including a user login component and a data annotation component;

[0012] The user login component is used to enable users to log in to the data labeling system by entering their account and password on the user login page;

[0013] The data annotation component is used to score and describe the sample data displayed on the data annotation page.

[0014] Optionally, after labeling the sample data, the method further includes:

[0015] The monocular depth map estimation model, the green screen-free matting model, the text generation layout model, and the human skeleton key point prediction model were trained using labeled sample data.

[0016] Optionally, the game screen data includes game screenshots, depth maps, and template images; the non-image data includes camera parameter information, object 2D bounding box data, object 3D bounding box data, object 3D model names, and human 3D skeleton key point information.

[0017] Optionally, the game screen data further includes:

[0018] The ternary graph of the current frame is obtained based on the template graph using the dilation and erosion method, which specifically includes using cv2.dilate and cv2.erode in OpenCV.

[0019] Optionally, obtaining game screen data based on the game data capture script includes:

[0020] When the current game page quality is judged to meet the requirements by a human, the current game screen data is automatically captured according to the game data capture script; wherein, the game page quality includes the number of entities in the screen, the relative positions of the entities in the screen, and the behavior and actions of the characters in the screen.

[0021] Optionally, obtaining game screen data based on the game data capture script includes:

[0022] Use the GTAVisionExport tool to retrieve image data from the cache;

[0023] This method uses the ScriptHookVDotNet game script library to extract non-image data from the GTA V game.

[0024] Secondly, a dataset construction device based on 3D games is provided, the device comprising:

[0025] The compilation module is used to obtain compiled game data scraping scripts for the target 3D game;

[0026] The acquisition module is used to run the target 3D game and obtain game screen data based on the game data capture script. The game screen data includes image data and non-image data.

[0027] The preprocessing module is used to clean the game screen data to obtain sample data to be labeled;

[0028] The annotation module is used to build a data annotation system and to annotate the sample data through the data annotation system.

[0029] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the dataset construction method based on any of the first aspects described above.

[0030] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the dataset construction method based on any of the first aspects described above.

[0031] The technical solution provided in this application first obtains the compiled game data capture script of the target 3D game; runs the target 3D game, and obtains game screen data according to the game data capture script, wherein the game screen data includes image data and non-image data; then cleans the game screen data to obtain sample data to be labeled; finally, constructs a data labeling system, and labels the sample data through the data labeling system. It can be seen that obtaining data from games in this application has two advantages over obtaining real data from the real world:

[0032] 1. Data acquisition from games can often be automated, greatly simplifying the data collection process;

[0033] 2. Game data can often be used to obtain relevant semantic information from the game, which greatly simplifies the data annotation process. Attached Figure Description

[0034] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0035] Figure 1A flowchart illustrating a dataset construction method based on 3D games, provided in this application embodiment;

[0036] Figure 2 A flowchart illustrating the steps of a method for constructing a dataset based on 3D games, as provided in this application embodiment;

[0037] Figure 3 This is a schematic diagram of the user login component in the data annotation system provided in the embodiments of this application;

[0038] Figure 4 This is a schematic diagram of the data annotation component in the data annotation system provided in the embodiments of this application;

[0039] Figure 5 A block diagram of a dataset construction device based on 3D games provided in an embodiment of this application. Detailed Implementation

[0040] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] With the rapid development of the gaming industry, more and more scholars have turned their attention to games in recent years, attempting to extract data from games for model training. Extracting data from games has two advantages over obtaining real-world data: 1. Data extraction from games can often be automated, greatly simplifying the data collection process; 2. Game data often contains semantic information, significantly simplifying data annotation and other processes.

[0042] This application proposes a 3D game dataset for various tasks. The dataset includes original game images, depth maps, template images, trimap images, camera parameter information, 2D object bounding boxes, 3D object bounding boxes, 3D object model names, 3D human skeleton keypoint information, and manually annotated image captions. The dataset construction process is as follows: Figure 1 As shown.

[0043] This application proposes a dataset construction method for 3D games, which can collect 3D game datasets for various tasks. This method mainly acquires two types of data: image data and non-image data. The method captures game screenshots, depth maps, and template maps by intercepting communication between the game and hardware, and obtains the corresponding trimap maps through post-processing. The method utilizes ScriptHookVDotNet to acquire non-image data such as camera coordinate information, camera Euler angle information, camera field of view, object coordinate information, corresponding forward vectors, object model names, and 3D skeleton keypoint coordinates of the human body.

[0044] To facilitate understanding of this embodiment, a detailed description of a 3D game-based dataset construction method disclosed in this application embodiment will be provided first. Please refer to... Figure 2 The diagram illustrates a flowchart of a dataset construction method based on 3D games provided in an embodiment of this application. The method may include the following steps:

[0045] Step 101: Obtain the compiled game data capture script for the target 3D game.

[0046] In this embodiment of the application, the target 3D game can be GTAV (Grand Theft Auto V). The GTAVisionExport tool can be used to obtain game screenshots, depth maps and template maps from the cache, and the corresponding ternary map can be obtained based on the template map through dilation and erosion.

[0047] Specifically:

[0048] Step 1011: Obtain the GTAVisionExport project source code files from GitHub for use in subsequent compilation scripts.

[0049] GitHub is a hosting platform for open-source and private software projects. It hosts various open-source code. Select the source code files of the project named GTAVisionExport and copy them. The specific location is (git clone https: / / github.com / umautobots / GTAVisionExport).

[0050] Step 1012: Download the Eigen3 template library and place it in the folder directory corresponding to the GTAV game.

[0051] Enigen3 is specifically an open-source template library that supports linear algebra operations, matrix and vector operations, numerical analysis, and related algorithms.

[0052] Step 1013: Compile the GTAVisionExport source code using CMake to obtain the game data scraping script for GTAVisionExport.

[0053] CMake is a cross-platform installation (compilation) tool. The specific compilation process on the client side using CMake includes:

[0054] ① After downloading CMake, first run the cmake-gui program from the Windows Start menu.

[0055] In the opened Windows Tools interface, find "BrowseSource," click and select the GTAVisionExport / native folder, which is the folder directory corresponding to the GTAV game. Click "BrowseBuild," create the GTAVisionExport / native / build folder and select it. Click "Configure," select the project generator "VisualStudio15 2017 Win64," and keep the option "use default native compilers." After the failure dialog box pops up, change EIGEN3_INCLUDE_DIR to point to the Eigen3 folder, run "Configure" again, and then run "Generate." CMake has now generated the Visual Studio (development tool) solution into the project file GTAVisionExport / build.

[0056] ② Open the project file in Visual Studio and select Release from the Solution Configurations drop-down list.

[0057] ③ In Visual Studio, edit the GTAVisionNative project properties / configuration properties / c / c++ / additionalincludedirs to add the GTAVisionExport / native / src folder. Then, edit the GTAVisionNative project properties / configuration properties / linker / input / additional dependencies to add: "..\..\deps\libMinHook.x64.lib". Press F6 to build the solution. The compiled game data scraping script will be saved in "GTAVisionExport\native\build\src\Release". Copy GTAVisionNative.asi and GTAVisionNative.lib to the GTAV game folder.

[0058] Step 102: Run the target 3D game and obtain the game screen data from the script based on the game data capture.

[0059] The game screen data includes image data and non-image data. The game screen data includes game screenshots, depth maps, and template images; the non-image data includes camera parameter information, object 2D bounding box data, object 3D bounding box data, object 3D model names, and human 3D skeleton key point information.

[0060] In this embodiment of the application, the steps following step 1013 specifically include:

[0061] Step 1021: After obtaining the compiled game script file, run the GTAV game.

[0062] The data in the dataset proposed in this invention were all manually obtained from game story animations. During data collection, the following factors needed to be considered: 1. The richness of entities in the current game screen, and whether it includes different entities. 2. Whether the positions of different entities are reasonable, and whether there is occlusion, etc. The richness of the behaviors and actions of different entities in the current game screen. After confirming that the game quality of the current game page meets the requirements, simply press the "Z" key on the keyboard, and the script will automatically capture various types of data for the current frame. The script will automatically create corresponding raw files (color.raw, stencil.raw, depth.raw) for the original image, depth image, and template image of the current frame in the game folder.

[0063] In an optional embodiment of this application, after performing step 1021 of capturing image data, the method further includes:

[0064] Step 1022: Obtain the corresponding ternary diagram based on the template diagram using the principle of expansion and erosion.

[0065] Specifically, the raw file in step 1021 is converted to JPG, and the corresponding ternary image is obtained based on the template image using the principles of dilation and erosion. Dilation refers to taking the maximum value within the rectangular neighborhood of each position as the output grayscale value of that position, while erosion refers to taking the minimum value within the neighborhood specified by the structuring element as the output grayscale value of that position. In this embodiment, the cv2.dilate and cv2.erode methods from OpenCV are specifically used.

[0066] This application uses the ScriptHookVDotNet game script library to obtain non-image data from the GTAV game, which mainly includes camera data (camera coordinates, rotation angle, etc.), entity data (coordinates of the entity, model name, etc.), and human skeleton data (relative coordinates of 55 skeleton key points).

[0067] Specifically, we use the camera class to obtain the world coordinates, rotation angle, and other attributes of the camera currently used in the game. We use the player class to obtain all people and entities within a circle with a radius of 50f centered on the current character. Then, we obtain the world coordinates, screen coordinates, memory address, and other information corresponding to each entity. We use the memory address to obtain the name of the corresponding model file and parse the object category based on the name. For each person, this application uses the skeleton subclass to obtain the relative coordinates (relative to the root node of the skeleton) of 55 skeletal key points for each person. The skeletal key points include: the root node of the skeleton (generally referring to the tailbone), head, neck, pelvis, spine (containing 4 points), and the left and right symmetrical lower legs, collarbone, fingers (containing 15 points), feet, forearms, hands, thighs, toes, and upper arms.

[0068] Step 103: Clean the game screen data to obtain the sample data to be labeled.

[0069] In the embodiments of this application, the data in the proposed dataset were initially screened manually before annotation, mainly removing data with large errors and messy images, and simplifying the data (deleting data corresponding to useless entities in the image) to facilitate subsequent data annotation.

[0070] Step 104: Construct a data annotation system and use the data annotation system to annotate the sample data.

[0071] In this embodiment of the application, to simplify the data annotation process, the present invention constructs a simple mobile data annotation system, allowing users to annotate data anytime, anywhere via their mobile phones. The system mainly includes two pages: user login and data annotation (e.g., ...). Figure 3 and Figure 4 As shown in the diagram, a MySQL database is finally used to store the data.

[0072] In the data annotation system, the data annotation process mainly includes two parts: describing the content of the image in one sentence and scoring the current data quality. The following criteria must be met when describing the image:

[0073] 1. The objects circled in green in the image are known data objects. The image is described based on these objects (the 3D model file name of the object is in the upper left corner, which can be used as a reference).

[0074] 2. Some unimportant details are omitted in the description, such as what kind of clothes the person is wearing;

[0075] 3. Avoid using phrases like "There is a..." or "There is...";

[0076] 4. Do not describe past or future events;

[0077] 5. Do not name people or props without authorization;

[0078] 6. Since many data points have short time intervals, the images are very similar. To ensure the diversity of data annotation, different description methods are used when describing similar data.

[0079] For example:

[0080] A man and a woman are standing face to face, chatting.

[0081] A man stood upright, while another woman spoke to him angrily.

[0082] 7. When describing the content of the image, include as many objects as possible. If some objects are not closely related to the main content of the image, they can be ignored.

[0083] When labeling and scoring data quality, the main influencing factors include:

[0084] 1. Is the data deviation in the image serious (judged based on the 2D bounding box)?

[0085] 2. Are there too many irrelevant elements in the image?

[0086] The entire annotation process involved 6 people, working in pairs. Each pair independently annotated 1,111 data points, resulting in 3,333 image descriptions (three descriptions for each data point). The average of the three quality scores was used as the final quality annotation.

[0087] In an optional embodiment of this application, after labeling the sample data, the method further includes:

[0088] The training of models applied to multiple sub-tasks mainly includes the following tasks:

[0089] 1. Monocular Depth Map Estimation

[0090] The original images and depth maps in the dataset proposed in this application embodiment can be used for monocular depth map estimation tasks.

[0091] 2. No green screen keying

[0092] The original images and ternary images in the dataset proposed in this application embodiment can be used for green screen keying tasks.

[0093] 3. Text generation layout

[0094] The dataset proposed in this application includes text descriptions of image content and 2D and 3D coordinates of objects in the images, which can be used for 2D and 3D layout generation tasks.

[0095] 4. Prediction of key points in the human skeleton

[0096] The dataset proposed in this application contains 2D and 3D coordinate information of 55 human skeletal key points, which can be used for 2D and 3D skeletal key point prediction tasks.

[0097] Please refer to Figure 5 The diagram illustrates a block diagram of a dataset construction apparatus 200 based on a 3D game, as provided in an embodiment of this application. Figure 5 As shown, the device 200 may include: a compilation module 201, an acquisition module 202, a preprocessing module 203, and an annotation module 204.

[0098] Compilation module 201 is used to obtain the compiled game data capture script of the target 3D game;

[0099] The acquisition module 202 is used to run the target 3D game and obtain game screen data according to the game data capture script. The game screen data includes image data and non-image data.

[0100] Preprocessing module 203 is used to clean the game screen data to obtain sample data to be labeled;

[0101] The annotation module 204 is used to construct a data annotation system and to annotate the sample data through the data annotation system.

[0102] Specific limitations regarding the 3D game-based dataset construction device can be found in the limitations of the 3D game-based dataset construction method described above, and will not be repeated here. Each module in the aforementioned 3D game-based dataset construction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0103] In one embodiment, an electronic device, which may be a computer, is provided. The electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to construct datasets based on 3D games. The network interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for constructing datasets based on 3D games.

[0104] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method for constructing a dataset based on a 3D game.

[0105] The computer-readable storage medium provided in this embodiment is similar in principle and technical effect to the method embodiment described above, and will not be repeated here.

[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0107] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for constructing a dataset based on 3D games, characterized in that, The method includes: A script to retrieve compiled game data from a target 3D game; Run the target 3D game, and obtain game screen data based on the game data capture script. The game screen data includes image data and non-image data. The game screen data is cleaned to obtain sample data to be labeled; Construct a data annotation system, and use the data annotation system to annotate the sample data; The construction of the data annotation system, and the annotation of the sample data through the data annotation system, includes: Construct a data annotation system based on mobile terminals, the data annotation system including a user login component and a data annotation component; The user login component is used to enable users to log in to the data labeling system by entering their account and password on the user login page; The data annotation component is used to score and describe the sample data displayed on the data annotation page; After labeling the sample data, the method further includes: The monocular depth map estimation model, the green screen-free matting model, the text generation layout model, and the human skeleton key point prediction model were trained using labeled sample data.

2. The method according to claim 1, characterized in that, The game screen data includes game screenshots, depth maps, and template images; the non-image data includes camera parameter information, 2D bounding box data of objects, 3D bounding box data of objects, 3D model names of objects, and key point information of the human 3D skeleton.

3. The method according to claim 2, characterized in that, The game screen data also includes: The ternary graph of the current frame is obtained based on the template graph using the dilation and erosion method, which specifically includes using cv2.dilate and cv2.erode in OpenCV.

4. The method according to claim 1, characterized in that, The process of obtaining game screen data based on the game data capture script includes: When the current game page quality is judged to meet the requirements by a human, the current game screen data is automatically captured according to the game data capture script; wherein, the game page quality includes the number of entities in the screen, the relative positions of the entities in the screen, and the behavior and actions of the characters in the screen.

5. The method according to claim 1, wherein the target 3D game is GTAV, characterized in that, The process of obtaining game screen data based on the game data capture script includes: Use the GTAVisionExport tool to retrieve image data from the cache; This method uses the ScriptHookVDotNet game script library to extract non-image data from the GTA V game.

6. A dataset construction device based on 3D games, characterized in that, The device includes: The compilation module is used to obtain compiled game data scraping scripts for the target 3D game; The acquisition module is used to run the target 3D game and obtain game screen data based on the game data capture script. The game screen data includes image data and non-image data. The preprocessing module is used to clean the game screen data to obtain sample data to be labeled; The annotation module is used to build a data annotation system and to annotate the sample data through the data annotation system; The construction of the data annotation system, and the annotation of the sample data through the data annotation system, includes: A data annotation system based on mobile terminals is constructed, the data annotation system including a user login component and a data annotation component; The user login component is used to enable users to log in to the data labeling system by entering their account and password on the user login page; The data annotation component is used to score and describe the sample data displayed on the data annotation page; After labeling the sample data, the method further includes: The monocular depth map estimation model, the green screen-free matting model, the text generation layout model, and the human skeleton key point prediction model were trained using labeled sample data.

7. A device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the dataset construction method based on any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the dataset construction method based on any one of claims 1 to 5.

Images