A method, apparatus, device, and storage medium for data acquisition based on 3D games.

Abstract

This application discloses a method, apparatus, device, and storage medium for data acquisition based on 3D games. The method includes: first, acquiring a compiled game data capture script for a target 3D game; running the target 3D game and, based on the data quality of the current game screen, using the capture script to obtain image data of the current game screen, including the original image of the current frame, depth map, and template image; then, acquiring non-image data of the current game screen based on the game script library of the target 3D game, including camera data, entity data, and human skeleton data; finally, matching the image data and non-image data of the current game screen. It can be seen that this application achieves the acquisition of synthetic data from 3D games, which is more efficient than data acquisition in the real world. Furthermore, the game synthetic data can extract corresponding semantic information from the game, greatly simplifying the data annotation process.

Description

Technical Field

[0001] This invention relates to the field of computer graphics, and in particular to a data acquisition method, apparatus, device, and storage medium based on 3D games. Background Technology

[0002] With the continuous advancement of technology, deep learning, as a new research direction in the field of machine learning, has become a hot topic in today's world. Deep learning technology enables machines to have analytical and learning capabilities like humans by learning the inherent patterns and hierarchies of data samples, thus far surpassing previous methods in many tasks such as human-computer dialogue and image recognition.

[0003] However, data plays a crucial role in the training process of deep learning. Since neural network models often have large parameters, if the amount of data is small, the network parameters cannot be trained well, and therefore the results will not be good. However, building a large-scale dataset is very expensive, requiring a lot of human and financial resources in both the data acquisition and data labeling stages. Summary of the Invention

[0004] Based on this, embodiments of this application provide a data acquisition method, apparatus, device, and storage medium based on 3D games, which can effectively solve the problems of high cost of collecting images from real environments and difficulty in obtaining accurate depth data.

[0005] Firstly, a data acquisition method based on 3D games is provided, the method comprising:

[0006] A script to retrieve compiled game data from a target 3D game;

[0007] Run the target 3D game, and use a capture script to obtain the image data of the current game screen based on the data quality of the current game screen. The image data includes the original image of the current frame, the depth map, and the template image.

[0008] Based on the game script library of the target 3D game, non-image data of the current game screen is obtained, including camera data, entity data, and human skeleton data.

[0009] The image data of the current game screen is matched with the non-image data.

[0010] Optionally, the image data further includes:

[0011] The ternary graph of the current frame is obtained based on the template graph using the dilatation and erosion method.

[0012] Optionally, the dilatational erosion method specifically includes using cv2.dilate and cv2.erode in OpenCV.

[0013] Optionally, obtaining the image data of the current game screen using a capture script based on the data quality of the current game screen includes:

[0014] The quality of the current game screen data is judged by humans; the game quality includes the number of entities in the screen, the relative positions of the entities in the screen, and the behavior and actions of the characters in the screen.

[0015] Optionally, the camera data includes camera coordinates and rotation angle; the entity data includes the coordinates of the model entity and the name of the model entity; and the human skeleton data includes the relative coordinates of 55 skeletal key points.

[0016] Optionally, the target 3D game is GTA V, and the GTAVisionExport tool is used to retrieve image data from the cache;

[0017] This method uses the ScriptHookVDotNet game script library to extract non-image data from the GTA V game.

[0018] Secondly, a data acquisition device based on 3D games is provided, the device comprising:

[0019] The compilation module is used to obtain compiled game data scraping scripts for the target 3D game;

[0020] The first acquisition module is used to run the target 3D game and obtain the image data of the current game screen using a capture script based on the data quality of the current game screen. The image data includes the original image of the current frame, the depth map, and the template image.

[0021] The second acquisition module is used to acquire non-image data of the current game screen based on the game script library of the target 3D game. The non-image data includes camera data, entity data, and human skeleton data.

[0022] The processing module is used to match the image data of the current game screen with the non-image data.

[0023] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the data acquisition method based on 3D games as described in any of the first aspects above.

[0024] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the data acquisition method based on any of the first aspects described above.

[0025] The technical solution provided in this application first obtains the compiled game data capture script of the target 3D game; runs the target 3D game, and uses the capture script to obtain the image data of the current game screen based on the data quality of the current game screen, including the original image of the current frame, the depth map, and the template image; then, based on the game script library of the target 3D game, it obtains the non-image data of the current game screen, including camera data, entity data, and human skeleton data; finally, it matches the image data and non-image data of the current game screen. It can be seen that this application achieves the acquisition of synthetic data from 3D games, which has two advantages compared to constructing real data in the real world:

[0026] 1. Data acquisition in the real world is very complex, but obtaining synthetic data from 3D games is often relatively simple;

[0027] 2. Game-generated data can often obtain relevant semantic information from the game, which can greatly simplify the data annotation process and even eliminate the need for additional manual annotation. Attached Figure Description

[0028] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0029] Figure 1 A flowchart illustrating the steps of a data acquisition method based on 3D games provided in this application embodiment;

[0030] Figure 2 This is a schematic diagram of image data provided in an embodiment of this application;

[0031] Figure 3 A data acquisition flowchart is provided for one optional embodiment of this application;

[0032] Figure 4 A block diagram of a data acquisition device based on 3D games provided in an embodiment of this application. Detailed Implementation

[0033] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] With the continuous development of computer hardware and the gaming industry, 3D game graphics are becoming increasingly realistic, and game characters and items are becoming more and more like the real world. This provides more possibilities for training neural networks. Obtaining synthetic data from 3D games has two advantages over constructing real data in the real world: 1. Data collection in the real world is very complex, while obtaining synthetic data from 3D games is often relatively simple; 2. Synthetic data from games can often extract relevant semantic information from the game, which can greatly simplify the data annotation process and even eliminate the need for additional manual annotation.

[0035] This application proposes a data acquisition method for 3D games, which mainly acquires two types of data: image data and other data (non-image data). The method captures game screenshots, depth maps, and template images by intercepting communication between the game and hardware, and obtains the corresponding trimap images through post-processing. The method utilizes ScriptHookVDotNet to acquire non-image data such as camera coordinates, camera Euler angles, camera field of view, object coordinates, corresponding forward vectors, object model names, and 3D skeleton keypoint coordinates of the human body.

[0036] To facilitate understanding of this embodiment, a data acquisition method based on 3D games disclosed in this application embodiment will first be described in detail.

[0037] Please refer to Figure 1 The diagram illustrates a flowchart of a data acquisition method based on 3D games provided in an embodiment of this application. The method may include the following steps:

[0038] Step 101: Obtain the compiled game data capture script for the target 3D game.

[0039] In this embodiment of the application, the target 3D game can be GTAV (Grand Theft Auto V). The GTAVisionExport tool can be used to obtain game screenshots, depth maps and template maps from the cache, and the corresponding ternary map can be obtained based on the template map through dilation and erosion.

[0040] Specifically:

[0041] Step 1011: Obtain the GTAVisionExport project source code files from GitHub for use in subsequent compilation scripts.

[0042] GitHub is a hosting platform for open-source and private software projects. It hosts various open-source code. Select the source code file of the project named GTAVisionExport and copy it. The specific location is (git clone https: / / github.com / umautobots / GTAVisionExport).

[0043] Step 1012: Download the Eigen3 template library and place it in the folder directory corresponding to the GTAV game.

[0044] Enigen3 is specifically an open-source template library that supports linear algebra operations, matrix and vector operations, numerical analysis, and related algorithms.

[0045] Step 1013: Compile the GTAVisionExport source code using CMake to obtain the game data scraping script for GTAVisionExport.

[0046] CMake is a cross-platform installation (compilation) tool. The specific compilation process on the client side using CMake includes:

[0047] ① After downloading CMake, first run the cmake-gui program from the Windows Start menu.

[0048] In the opened Windows Tools interface, find "BrowseSource," click and select the GTAVisionExport / native folder, which is the folder directory corresponding to the GTAV game. Click "BrowseBuild," create the GTAVisionExport / native / build folder and select it. Click "Configure," select the project generator "VisualStudio15 2017 Win64," and keep the option "use default native compilers." After the failure dialog box pops up, change EIGEN3_INCLUDE_DIR to point to the Eigen3 folder, run "Configure" again, and then run "Generate." CMake has now generated the Visual Studio (development tool) solution into the project file GTAVisionExport / build.

[0049] ② Open the project file in Visual Studio and select Release from the Solution Configurations drop-down list.

[0050] ③ In Visual Studio, edit the GTAVisionNative project properties / configuration properties / c / c++ / additionalincludedirs to add the GTAVisionExport / native / src folder. Then, edit the GTAVisionNative project properties / configuration properties / linker / input / additional dependencies to add: "..\..\deps\libMinHook.x64.lib". Press F6 to build the solution. The compiled game data scraping script will be saved in "GTAVisionExport\native\build\src\Release". Copy GTAVisionNative.asi and GTAVisionNative.lib to the GTAV game folder.

[0051] Step 102: Run the target 3D game and use a capture script to obtain the image data of the current game screen based on the data quality of the current game screen.

[0052] The image data includes the original image of the current frame (game screenshot), a depth map, and a template image. In this embodiment, step 1013 specifically includes the following:

[0053] Step 1021: Use the compiled game data capture script file to capture image data of the target game GTAV.

[0054] Specifically, when running the GTA V game, the user manually judges the quality of the current game screen data. Data quality is mainly determined by the following factors: the number of entities in the screen, the relative positions of entities (whether they are occluded), and the actions of characters in the screen. Locate the position in the game where you want to capture the frame, and then press the "l" key (lowercase "L") to capture the frame. The script will automatically create corresponding raw files (color.raw, stencil.raw, depth.raw) for the current frame's original image, depth image, and template image in the game folder.

[0055] In an optional embodiment of this application, after performing step 1021 of capturing image data, the method further includes:

[0056] Step 1022: Obtain the corresponding ternary diagram based on the template diagram using the principle of expansion and erosion.

[0057] Specifically, the raw file in step 1021 is converted to JPG, and the corresponding ternary image is obtained based on the template image using the principles of dilation and erosion. Dilation refers to taking the maximum value within the rectangular neighborhood of each position as the output grayscale value of that position, while erosion refers to taking the minimum value within the neighborhood specified by the structuring element as the output grayscale value of that position. In this embodiment, the cv2.dilate and cv2.erode methods from OpenCV are specifically used.

[0058] like Figure 2 The image data obtained in this application is an example image, including the original image (game screenshot), depth image, template image and ternary image.

[0059] Step 103: Obtain non-image data of the current game screen based on the game script library of the target 3D game.

[0060] Non-image data includes camera data, entity data, and human skeleton data.

[0061] This application uses the ScriptHookVDotNet game script library to obtain non-image data from the GTAV game, which mainly includes camera data (camera coordinates, rotation angle, etc.), entity data (coordinates of the entity, model name, etc.), and human skeleton data (relative coordinates of 55 skeleton key points).

[0062] Specifically, we use the camera class to obtain the world coordinates, rotation angle, and other attributes of the camera currently used in the game, and the player class to obtain all people and entities within a circle with a radius of 50f centered on the current character. Then, we obtain the world coordinates, screen coordinates, memory address, and other information corresponding to each entity, and use the memory address to obtain the name of its corresponding model file, and parse the object category based on the name.

[0063] In this embodiment of the application, the skeleton subclass is used to obtain the relative coordinates (relative to the root node of the skeleton) of 55 skeleton key points for each person. The skeleton key points include: the root node of the skeleton (generally referring to the tailbone), head, neck, pelvis, spine (containing 4 points) and the left and right symmetrical lower legs, clavicle, fingers (containing 15 points), feet, forearms, hands, thighs, toes, and upper arms.

[0064] Step 104: Match the image data of the current game screen with the non-image data.

[0065] In this embodiment of the application, the image data and non-image data of the current game screen are filtered and processed to obtain the final matching data.

[0066] Please refer to Figure 4This diagram illustrates a block diagram of a 3D game-based data acquisition device 200 provided in an embodiment of this application. Figure 4 As shown, the device 200 may include: a compilation module 201, a first acquisition module 202, a second acquisition module 203, and a processing module 204.

[0067] Compilation module 201 is used to obtain the compiled game data capture script of the target 3D game;

[0068] The first acquisition module 202 is used to run the target 3D game and obtain the image data of the current game screen using a capture script based on the data quality of the current game screen. The image data includes the original image of the current frame, the depth map, and the template image.

[0069] The second acquisition module 203 is used to acquire non-image data of the current game screen based on the game script library of the target 3D game. The non-image data includes camera data, entity data, and human skeleton data.

[0070] The processing module 204 is used to match the image data of the current game screen with non-image data.

[0071] Specific limitations regarding the data acquisition device for 3D games can be found in the limitations of the data acquisition method for 3D games described above, and will not be repeated here. Each module in the aforementioned data acquisition device for 3D games can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0072] In one embodiment, an electronic device, which may be a computer, is provided. The electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for data acquisition based on 3D games. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data acquisition method based on 3D games.

[0073] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described data acquisition method based on 3D games.

[0074] The computer-readable storage medium provided in this embodiment is similar in principle and technical effect to the method embodiment described above, and will not be repeated here.

[0075] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M ​​forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data acquisition method based on 3D games, characterized in that, The method includes: A script to retrieve compiled game data from a target 3D game; Run the target 3D game, and use a capture script to obtain the image data of the current game screen based on the data quality of the current game screen. The image data includes the original image of the current frame, the depth map, and the template image. Based on the game script library of the target 3D game, non-image data of the current game screen is obtained, including camera data, entity data, and human skeleton data. Match the image data of the current game screen with the non-image data; The image data also includes: obtaining a ternary image of the current frame based on the template image using the dilation and erosion method; The dilatational erosion method specifically includes using cv2.dilate and cv2.erode in OpenCV; The step of obtaining image data of the current game screen using a capture script based on the data quality of the current game screen includes: judging the data quality of the current game screen by human judgment; the data quality of the current game screen includes the number of entities in the screen, the relative positions of entities in the screen, and the behavior and actions of characters in the screen.

2. The method according to claim 1, characterized in that, The camera data includes camera coordinates and rotation angle; the entity data includes the coordinates of the model entity and the name of the model entity; the human skeleton data includes the relative coordinates of 55 skeletal key points.

3. The method according to claim 1, wherein the target 3D game is GTA V, characterized in that, Use the GTAVisionExport tool to retrieve image data from the cache; This method uses the ScriptHookVDotNet game script library to extract non-image data from the GTA V game.

4. A data acquisition device based on 3D games, characterized in that, The device includes: The compilation module is used to obtain compiled game data scraping scripts for the target 3D game; The first acquisition module is used to run the target 3D game and obtain the image data of the current game screen using a capture script based on the data quality of the current game screen. The image data includes the original image of the current frame, the depth map, and the template image. The second acquisition module is used to acquire non-image data of the current game screen based on the game script library of the target 3D game. The non-image data includes camera data, entity data, and human skeleton data. The processing module is used to match the image data of the current game screen with the non-image data; The image data also includes: obtaining a ternary image of the current frame based on the template image using the dilation and erosion method; The dilatational erosion method specifically includes using cv2.dilate and cv2.erode in OpenCV; The step of obtaining image data of the current game screen using a capture script based on the data quality of the current game screen includes: judging the data quality of the current game screen by human judgment; the data quality of the current game screen includes the number of entities in the screen, the relative positions of entities in the screen, and the behavior and actions of characters in the screen.

5. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the data acquisition method based on any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the data acquisition method based on 3D games as described in any one of claims 1 to 3.

Images