Game picture processing method and device, computer equipment and storage medium
Through graphics processor rendering and post-processing, combined with direct memory access and single-instruction encoding, the problem of low cross-platform game screen processing performance is solved, efficient game screen post-processing and recording is achieved, and development complexity is reduced.
Patent Information
- Application Number
- CN202310605604.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-26
AI Technical Summary
The audio and video suites in the existing game engines are targeted at specific platforms, resulting in the need of a lot of manpower to plug-in compatibility and development across platforms, which cannot meet the performance requirements of real-time game screen post-processing and recording, and are poorly maintained.
The graphics processor is used to render and post-process the processed game screen to the central processor by direct memory access, and video encoding is performed with a single instruction to achieve game video file generation in the target format.
It improves the efficiency and performance of game screen processing, realizes real-time post-processing, acquisition and recording across platforms, reduces the need for plug-in compatibility and development, and improves maintainability.
Smart Images

Figure CN119015697A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for processing game images. Background Art
[0002] With the development of computer technology and Internet technology, the development and use of game engines play an important role in fields such as computer vision, making various types of game screen post-processing tools widely concerned by the public.
[0003] In the current game screen processing method, the audio and video suites in the commonly used game engines are all targeted at a specific platform. For example, there is an independent plug-in for a separate mobile operating system and an independent plug-in for a separate desktop operating system. The deployment and use of such independent plug-ins are also independent. During use, the plug-in switches need to be configured separately on different platforms. If these independent suites are used in cross-platform projects, a lot of manpower is required for related development and testing such as plug-in compatibility and plug-in splicing, which is very error-prone and has poor maintainability, which easily leads to low efficiency in game screen processing. Summary of the invention
[0004] Based on this, it is necessary to provide a game screen processing method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems, which can effectively improve the efficiency of game screen processing.
[0005] In the first aspect, the present application provides a method for processing a game screen. The method comprises: in the process of interacting with a game application, rendering the screen through a graphics processor to obtain a game screen; post-processing the game screen through the graphics processor to obtain a processed game screen; transmitting the processed game screen to a central processing unit based on a direct memory access method; and encoding the processed game screen through the central processing unit according to a single instruction to obtain a game video file in a target format.
[0006] In the second aspect, the present application also provides a game screen processing device. The device includes: a rendering module, which is used to render the screen through a graphics processor in the process of interacting with the game application to obtain a game screen; a processing module, which is used to post-process the game screen through the graphics processor to obtain a processed game screen; a transmission module, which is used to transmit the processed game screen to a central processing unit based on a direct memory access method; an encoding module, which is used to perform video encoding on the processed game screen according to a single instruction through the central processing unit to obtain a game video file in a target format.
[0007] In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: in the process of interacting with the game application, the graphics processor performs screen rendering to obtain a game screen; the graphics processor performs post-processing on the game screen to obtain a processed game screen; the processed game screen is transmitted to a central processing unit based on a direct memory access method; the central processing unit performs video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format.
[0008] In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented: in the process of interacting with the game application, a graphics processor is used to render a screen to obtain a game screen; the graphics processor is used to post-process the game screen to obtain a processed game screen; the processed game screen is transmitted to a central processing unit based on a direct memory access method; the central processing unit is used to perform video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format.
[0009] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, which implements the following steps when executed by a processor: in the process of interacting with a game application, a graphics processor is used to render a screen to obtain a game screen; the graphics processor is used to post-process the game screen to obtain a processed game screen; the processed game screen is transmitted to a central processing unit based on a direct memory access method; the central processing unit is used to perform video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format.
[0010] The above-mentioned game screen processing method, device, computer equipment, storage medium and computer program product obtain the game screen by rendering the screen through a graphics processor during the interaction between the user and the game application; post-process the game screen through the graphics processor to obtain the processed game screen; transmit the processed game screen to the central processing unit based on direct memory access; and use the central processing unit to perform video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format. Since the game screen is rendered by the graphics processor during the interaction between the user and the game application, the rendered game screen can be further post-processed by the graphics processor to obtain the processed game screen. At the same time, the processed game screen can be transmitted to the central processing unit based on the direct memory access method, so that the central processing unit can perform video encoding on the processed game screen according to a single instruction, thereby accelerating the performance of obtaining the game video file in the target format. Not only high-performance real-time screen post-processing operations are achieved, but also the game screen post-processing, game screen acquisition and transmission, and game screen format conversion are deeply accelerated. This solves the problem that different platform projects in the traditional method need to use independently developed kits, which cannot meet the performance requirements of real-time game screen post-processing, recording and encoding. The method provided by the present application does not require a lot of manpower for plug-in compatibility, plug-in splicing and other related development and testing, and can achieve compatibility with different platform projects to post-process, acquire, transmit and record the game screen in real time, thereby effectively improving the efficiency of game screen processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 This is an application environment diagram of a method for processing a game screen in one embodiment;
[0012] Figure 2 1 is a flowchart of a method for processing a game screen in one embodiment;
[0013] Figure 3 A schematic flow chart of steps of post-processing a game screen by a graphics processor to obtain a processed game screen in one embodiment;
[0014] Figure 4 is a schematic diagram of the basic structure of a post-processing chain in one embodiment;
[0015] Figure 5 A processing schematic diagram of a post-processing chain method in one embodiment;
[0016] Figure 6 A schematic diagram of using PBO to transfer GPU image data to CPU memory in one embodiment;
[0017] Figure 7A schematic diagram of a CPU processing a processed game screen in one embodiment;
[0018] Figure 8 A schematic diagram of rendering a game screen by a graphics processor in one embodiment;
[0019] Fig. 9 is a schematic diagram of a product side in one embodiment;
[0020] Fig.10 It is a schematic diagram of the overall process of a solution for real-time post-processing, collection and recording of cross-platform game screens on Unreal Engine mobile terminals in one embodiment;
[0021] Fig.11 A schematic diagram of transmitting GPU image data to CPU memory in a conventional manner in one embodiment;
[0022] Fig.12 A schematic diagram of using PBO to optimize the performance of transferring game images from GPU memory to CPU memory in one embodiment;
[0023] Fig.13 is a structural block diagram of a game screen processing device in one embodiment;
[0024] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0026] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.
[0027] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.
[0028] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.
[0029] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0030] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.
[0031] The game screen processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, can be integrated on the server 104, or can be placed on the cloud or other network servers. In the process of the user interacting with the game application in the terminal 102, the terminal 102 renders the screen through the graphics processor to obtain the game screen; the terminal 102 post-processes the game screen through the graphics processor to obtain the processed game screen; the terminal 102 transmits the processed game screen to the central processing unit based on the direct memory access method; the terminal 102 encodes the processed game screen through the central processing unit according to a single instruction to obtain a game video file in the target format. Furthermore, the terminal 102 can upload the obtained game video file in the target format to the server 104, so that the server 104 can reuse the game video file in the target format. For example, the server 104 can automatically obtain the highlights in the game video file for secondary display.
[0032] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an IoT device, and a portable wearable device. The IoT device may be a smart speaker, a smart TV, a smart air conditioner, and a smart vehicle-mounted device, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc.
[0033] The server 104 may be an independent physical server or a service node in a blockchain system. A peer-to-peer (P2P) network is formed between the service nodes in the blockchain system. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP).
[0034] In addition, server 104 can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN, Content Delivery NetworkCDN), as well as big data and artificial intelligence platforms.
[0035] The terminal 102 and the server 104 may be connected via Bluetooth, USB (Universal Serial Bus) or a network or other communication connection methods, which are not limited in this application.
[0036] In one embodiment, Figure 2 As shown, a method for processing a game screen is provided, which can be executed by a server or a terminal alone, or by a server and a terminal together. Figure 1 The terminal in is used as an example to illustrate, including the following steps:
[0037] Step 202, during the interaction with the game application, the graphics processor performs screen rendering to obtain the game screen.
[0038] Among them, game applications refer to different types of game applications (APP, Application), and the game applications in this application can be different types of game applications installed in the terminal used by the user. For example, the game applications in this application may include at least one of action game applications, adventure game applications, simulation game applications, role-playing game applications and other game applications.
[0039] A graphics processing unit (GPU), also known as a display core, visual processor, or display chip, is a microprocessor that is specifically used to perform image and graphics-related calculations on terminals such as smartphones, tablet computers, laptops, and desktop computers.
[0040] Game screen refers to the frame that can be displayed on the screen. The game screen in this application may include frame images corresponding to different moments. For example, a game scene is composed of continuous shots, describing the same short game story. Usually, a continuous shot sequence of any length can be regarded as a game video short. Each game scene in this application may contain continuous frame data of different frame numbers. For example, each game scene may contain the first frame of the game screen at the first moment and the second frame of the game screen at the second moment, etc.
[0041] The central processing unit (CPU) is the computing and control core of the computer system and is the final execution unit for information processing and program running.
[0042] Specifically, during the process of user interaction with the game application in the terminal, the terminal can render each frame of the game screen through the graphics processor. That is, when the user interacts with the game application, after the central processing unit in the terminal submits the drawing command, the GPU can asynchronously execute the steps of rendering the game screen. That is, when the GPU in the terminal receives the drawing instruction, the GPU starts rendering the game screen.
[0043] For example, the terminal responds to an interactive operation triggered by the user, and the central processing unit in the terminal submits a drawing command. When the GPU in the terminal receives the drawing instruction, the GPU starts to render the game screen corresponding to the interactive operation triggered by the user. For another example, the game screen in the game application at the current moment is the 6th frame of the game screen. The user selects an interactive product in the 6th frame of the game screen, so that the terminal responds to the product selection operation and renders the next frame of the game screen through the graphics processor. That is, the terminal can render the screen through the graphics processor to obtain the 7th frame of the game screen.
[0044] Step 204, post-processing the game screen by a graphics processor to obtain a processed game screen.
[0045] Among them, post-processing corresponds to pre-processing. Post-processing refers to the next step after pre-processing. It is the work performed before the final processing and improvement, or the step performed after a certain stage of work. It will be interpreted differently when applied in different industries or fields. The post-processing in this application may refer to the post-processing of the game screen, that is, the post-processing in this application can be understood as post-processing for the image. That is, the post-processing (Post-ProcessEffect) in this application can be a technology for reprocessing the rendered screen. For each camera, there is an opportunity to perform secondary processing on the entire screen before drawing it to the user window, and then present the processed screen to the user. Post-processing can easily produce global effects, but it will also bring performance consumption.
[0046] The processed game screen refers to each game frame screen obtained after a series of post-processing. For example, a certain frame of rendered game screen is sharpened, and the game screen after the sharpening process is the processed game screen.
[0047] Specifically, during the process of user interaction with the game application in the terminal, the terminal renders the screen through the graphics processor. After obtaining the game screen, the terminal needs to use various post-processing techniques in the process of recording the above game screen as a finished video, such as cropping, flipping the screen, adjusting the saturation and brightness of the screen, blurring, sharpening the screen and other post-processing operations.
[0048] In the traditional way, post-processing technology uses CPU for calculation, such as the post-processing interface in FFmpeg (an open source computer program), which has poor performance and cannot be used in real time on mobile devices. Therefore, in the embodiment of the present application, the post-processing method on the CPU side is transplanted to the GPU, making full use of the parallel computing power of the GPU and greatly improving the post-processing performance. At the same time, the method provided in the present application can provide an organization method of the post-processing chain, which can flexibly configure and combine various picture post-processing technologies.
[0049] In an embodiment of the present application, during the process of interaction between the user and the game application in the terminal, after the terminal renders each frame of the game screen through the graphics processor, the terminal can perform real-time post-processing on each rendered frame of the game screen through the graphics processor to obtain each processed frame of the game screen, and save the data of each processed frame of the game screen in the GPU memory.
[0050] For example, the terminal is a mobile device. The mobile device in this application can also be referred to as a mobile device. For example, the mobile device includes a smart phone and a tablet computer. Since the performance and memory of the mobile device are relatively limited, to ensure that the entire processing and recording process of the game screen can run in real time, each processing step needs to be fully optimized. For example, how to flexibly configure and combine various game screen post-processing methods and ensure performance is one of the technical difficulties. Therefore, in an embodiment of the present application, during the process of interaction between a user and a game application in a mobile device, after the mobile device renders each frame of the game screen through the GPU, the mobile device can call the post-processing chain through the GPU, wherein the basic structure of the post-processing chain may include an input screen node, an image post-processing method, and an output screen node. After the input screen node processes the input game screen using the image post-processing method, the corresponding processed game screen is output through the output screen node, that is, the mobile device can flexibly configure and combine various screen post-processing technologies by calling the post-processing chain. For example, assuming that the mobile device needs to perform post-processing operations on a certain frame of the game screen, including: first cropping, then adjusting the brightness, the mobile device can call the post-processing chain through the GPU, and input the frame of the game screen into the first input screen node of the post-processing chain for cropping, and output the first output screen; further, the mobile device inputs the first output screen into the second input screen node of the post-processing chain for brightness processing, and outputs the final processed game screen. That is, in the post-processing chain of the embodiment of the present application, the processed image output by the output image node of the previous post-processing chain can be used as an input parameter and input into the input image node of the next post-processing chain for further processing. Therefore, a variety of image post-processing methods can be flexibly expanded and used.
[0051] Step 206, transmitting the processed game screen to the central processing unit based on direct memory access.
[0052] Among them, direct memory access (DMA) is a function provided by some computer bus architectures, which enables data to be sent directly from an attached device (such as a disk drive) to the memory of the computer motherboard.
[0053] Specifically, during the process of user interaction with the game application in the terminal, the terminal renders the screen through the graphics processor to obtain the game screen, and post-processes the game screen through the graphics processor. After obtaining the processed game screen, the terminal can transmit the processed game screen to the central processing unit based on direct memory access, so that the central processing unit can directly obtain each processed game screen from its own memory.
[0054] In the traditional method, the terminal needs to transfer the game screen in the GPU memory back to the CPU memory during the acquisition process. This step is prone to performance problems because after the CPU submits the drawing command, the GPU performs rendering asynchronously and the CPU does not get the picture immediately. For example, assuming that the CPU submits a drawing command at the 5th second, the GPU starts rendering the picture at the 8th second after receiving the drawing command, and the GPU does not return the rendered game picture to the CPU until the 10th second. Therefore, during the time period from the 8th second to the 10th second, the CPU is always waiting for the picture drawing result. If the CPU side forcibly waits for the GPU to complete the game picture drawing, it is easy to cause serious performance problems. Therefore, in response to such problems, the method provided in this application can transfer the processed game screen in the GPU memory to the CPU memory based on direct memory access.
[0055] For example, the terminal is a mobile device. Since the performance and memory of the mobile device are relatively limited, in order to ensure that the entire processing and recording process of the game screen can run in real time, it is necessary to fully optimize each processing step. For example, how to efficiently collect the game screen is also one of the technical difficulties. Therefore, in the embodiment of the present application, in the process of the user interacting with the game application in the mobile device, the mobile device performs post-processing on each frame of the rendered game screen through the GPU, and after the data of each frame of the processed game screen is saved in the GPU memory, the mobile device can optimize the performance of transmitting the game screen from the GPU memory to the CPU memory by using the pixel buffer object (PBO). Among them, the pixel buffer object (PBO, Pixel Buffer Object) refers to a buffer object for storing pixel data. That is, the mobile device can use the pixel buffer object (PBO) to achieve high-performance real-time acquisition of the game screen. That is, the mobile device in the present application can quickly transmit pixel data between the GPU and the CPU based on the direct memory access method through PBO, without occupying the CPU's operating cycle, and it is asynchronous transmission, so it can achieve fast upload and download of pixel data.
[0056] Step 208, through the central processing unit, video encoding is performed on the processed game screen according to a single instruction to obtain a game video file in a target format.
[0057] Among them, a single instruction refers to an instruction that operates on multiple data simultaneously based on a single instruction. In simple terms, one instruction can process multiple data at the same time. For example, the single instruction in this application can be a NEON instruction, which optimizes application performance through vectorized computing. It is usually used in scenarios that require a lot of computing, such as image processing, audio and video processing, and data parallel processing.
[0058] Video encoding refers to the formatting of recorded videos to facilitate transmission over the network. For example, the video encoding in this application mainly includes two steps: compressing and encoding the video and audio separately, and packaging the audio and video.
[0059] The target format of the game video file refers to the encapsulation of the encoded game data to obtain a file in a specific encapsulation format, and the encapsulated file is the target format of the game video file. Among them, the target format refers to the target encapsulation format, and the encapsulation format refers to packaging the video compression data and the audio compression data, and merging the video compression data and the audio compression data into one file in a specific way. It can be understood that some encapsulation formats support almost all audio and video encodings, while some encapsulation formats only support a few audio and video encodings. For example, the target format in this application can be the AVI format.
[0060] Specifically, after the terminal transmits the processed game screen to the central processing unit based on direct memory access, the terminal can use the central processing unit to perform video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format. For example, the terminal can use the central processing unit to convert the data format of the processed game screen according to NEON instructions and encoding formats to obtain a converted game screen. Furthermore, the terminal can use the central processing unit to encode the converted game screen according to encoding parameters to obtain an encoded game screen, and encapsulate the encoded game screen to obtain a final game video file that conforms to the target encapsulation format.
[0061] For example, the terminal is a mobile terminal device. The mobile terminal device in this application can also be referred to as a mobile device, for example, the mobile terminal device includes a smart phone and a tablet computer. Since the performance and memory of the mobile terminal device are relatively limited, to ensure that the entire processing and recording process of the game screen can run in real time, it is necessary to fully optimize each processing step therein. For example, how to ensure the performance of data format conversion is also one of the technical difficulties. Therefore, in an embodiment of the present application, during the interaction between the user and the game application in the mobile terminal device, after the mobile terminal device transmits the processed game screen to the CPU based on the direct memory access method, the CPU in the mobile terminal device can accelerate the conversion process through the NEON instruction, that is, a single instruction processes multiple pixel data at the same time, improves performance, for example, the CPU in the mobile terminal device can convert the data format of the processed game screen through the NEON instruction to obtain a game screen that conforms to the encoding format, and further, the CPU in the mobile terminal device can encode the obtained game screen that conforms to the encoding format according to the encoding parameters to obtain the encoded game screen, and encapsulate the encoded game screen according to the target encapsulation format, so as to obtain the game video file that finally conforms to the target encapsulation format.
[0062] It can be understood that the target format in the present application can also be called a target encapsulation format. The mobile device can adapt the target encapsulation format based on the game style parameters of the game application. For example, the resolution parameter required in the game style parameters of a shooting game application is low, but the frame rate parameter requirement is high. Therefore, the mobile device can adapt the target encapsulation format to the first encapsulation format based on the resolution and frame rate in the game style parameters of the shooting game application. For example, the first encapsulation format may be an AVI format.
[0063] In this embodiment, during the interaction with the game application, the graphics processor performs screen rendering to obtain the game screen; the graphics processor performs post-processing on the game screen to obtain the processed game screen; the processed game screen is transmitted to the central processing unit based on direct memory access; the central processing unit performs video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format. Since the game screen is rendered by the graphics processor during the interaction between the user and the game application, the rendered game screen can be further post-processed by the graphics processor to obtain the processed game screen. At the same time, the processed game screen can be transmitted to the central processing unit based on the direct memory access method, so that the central processing unit can perform video encoding on the processed game screen according to a single instruction, thereby accelerating the performance of obtaining the game video file in the target format. Not only high-performance real-time screen post-processing operations are achieved, but also the game screen post-processing, game screen acquisition and transmission, and game screen format conversion are deeply accelerated. This solves the problem that different platform projects in the traditional method need to use independently developed kits, which cannot meet the performance requirements of real-time game screen post-processing, recording and encoding. The method provided by the present application does not require a lot of manpower for plug-in compatibility, plug-in splicing and other related development and testing, and can achieve compatibility with different platform projects to post-process, acquire, transmit and record the game screen in real time, thereby effectively improving the efficiency of game screen processing.
[0064] In one embodiment, Figure 3 As shown, the step of post-processing the game screen by a graphics processor to obtain the processed game screen includes:
[0065] Step 302, calling a post-processing chain through a graphics processor; wherein the post-processing chain comprises an input screen node, an image post-processing method, and an output screen node;
[0066] Step 304, performing image processing on the game screen according to an image post-processing method by inputting a screen node;
[0067] Step 306, outputting the processed game screen obtained by the input screen node through the output screen node.
[0068] Among them, the post-processing chain refers to a method of flexibly configuring and combining various image post-processing methods through a chain-like organization method, such as Figure 4 As shown in FIG. 1 , it is a schematic diagram of the basic structure of the post-processing chain. That is, the basic structure of the post-processing chain in this application can be as follows: Figure 4 As shown in , it includes: an input screen node, a post-processing method and an output screen node, wherein the input screen node processes the game screen using the post-processing method and then outputs the processed game screen through the output screen node.
[0069] An input screen node refers to a node used to process an input game screen. For example, the input screen node in the present application may be the first node in a post-processing chain.
[0070] The image post-processing method refers to various methods for post-processing an image. For example, the image post-processing method in the present application may include at least one of cropping, flipping the image, adjusting the saturation and brightness of the image, blurring, sharpening the image, etc.
[0071] The output screen node refers to a node used to output the processed game screen. For example, the output screen node in the present application may be the last node in the post-processing chain.
[0072] Specifically, in the process of interaction between the user and the game application in the terminal, the terminal performs post-processing on each rendered frame of the game screen through the GPU to obtain each processed frame of the game screen, and after saving the data of each processed frame of the game screen in the GPU memory, the terminal can call the post-processing chain through the GPU; wherein, the composition of the post-processing chain includes an input screen node, an image post-processing method, and an output screen node; the GPU in the terminal performs image processing on each frame of the game screen according to the image post-processing method through the input screen node, and outputs each frame of the processed game screen through the output screen node. It can be understood that when the GPU in this application performs image post-processing on each frame of the game screen, the post-processing chain called can be different, that is, the image post-processing operation performed by the GPU on each frame of the game screen in this application can be the same or different.
[0073] For example, the terminal is a mobile device. In the process of the user interacting with the game application in the mobile device, after the mobile device renders each frame of the game screen through the GPU, the mobile device can call the post-processing chain through the GPU. Assuming that the image post-processing method included in the post-processing chain A is a cropping post-processing method, and the image post-processing method included in the post-processing chain B is a sharpening post-processing method, the mobile device can first call the post-processing chain A through the GPU, and input the frame of the game screen into the input screen node of the post-processing chain A for cropping, and output the cropped game screen through the output screen node; further, the mobile device calls the post-processing chain B through the GPU, and inputs the cropped game screen into the input screen node in the post-processing chain B for sharpening, and outputs the final processed game screen through the output screen node. That is, in the post-processing chain of the embodiment of the present application, the picture output by the output picture node of the previous post-processing chain can be used as the input picture of the input picture node of the next post-processing chain, so it is possible to flexibly expand and use a variety of post-processing methods, thereby realizing flexible configuration and combination of various picture post-processing technologies, while making full use of the parallel computing capabilities of the GPU to greatly improve the real-time post-processing performance.
[0074] In one embodiment, the image post-processing method includes cropping, brightness processing and sharpening processing; the step of performing image processing on the game screen according to the image post-processing method by inputting a screen node includes:
[0075] Input the game screen to the first input screen node of the post-processing chain for cropping, and output the first output screen;
[0076] Input the first output picture into the second input picture node of the post-processing chain for brightness processing, and output a second output picture;
[0077] The second output image is input into the third input image node of the post-processing chain for sharpening processing, and the processed game image is output.
[0078] Among them, the first input picture node, the second input picture node, and the third input picture node are only used to distinguish different input picture nodes. For example, the first input picture node in the present application may be a node for cropping post-processing, the second input picture node may be a node for brightness post-processing, and the third input picture node may be a node for sharpening post-processing.
[0079] The first output screen, the second output screen, and the processed game screen are only used to distinguish the screens output by different output screen nodes. For example, the first output screen in the present application may be output by the first output screen node, the second output screen may be output by the second output screen node, and the processed game screen may be output by the third output screen node.
[0080] Specifically, the terminal is a mobile device as an example. In the process of the user interacting with the game application in the mobile device, after the mobile device renders each frame of the game screen through the GPU, the mobile device can call the post-processing chain through the GPU, wherein the basic structure of the post-processing chain may include an input screen node, an image post-processing method, and an output screen node.
[0081] like Figure 5 As shown in FIG. 1 , a schematic diagram of the post-processing chain is shown. Assume that the image post-processing method included in the post-processing chain A is as follows: Figure 5 The cropping post-processing method, brightness post-processing method and sharpening post-processing method shown in , the mobile terminal device can call the post-processing chain A through the GPU, and input the frame game screen f4 as shown in Figure 5 The first input screen node of the post-processing chain A shown in FIG. 1 is used to perform cropping processing, and the cropped game screen f41 is output through the first output screen node; further, the mobile terminal device inputs the cropped game screen f41 into the Figure 5 The second input picture node in the post-processing chain A shown in FIG. 1 performs brightness processing, and outputs the brightness-processed game picture f42 through the second output picture node; further, the mobile terminal device inputs the cropped game picture f42 into the Figure 5 The third input screen node in the post-processing chain A shown in the figure is sharpened, and the sharpened game screen is outputted through the third output screen node, so that the final processed game screen can be obtained. That is, in the post-processing chain of the embodiment of the present application, the game screen outputted by the previous output screen node can be used as input data to input the next input screen node, so it is possible to flexibly expand and use a variety of post-processing methods, thereby realizing flexible configuration and combination of various screen post-processing technologies, and at the same time making full use of the parallel computing capability of the GPU, greatly improving the real-time post-processing performance.
[0082] In one embodiment, a virtual scene corresponding to a game application has dynamic objects; the step of rendering a game screen by a graphics processor includes:
[0083] Determine the position of dynamic objects during motion;
[0084] Get the lighting information of the corresponding light probe according to the position;
[0085] Render dynamic objects according to the lighting information to obtain a globally illuminated game screen.
[0086] Among them, a virtual scene refers to a virtual scene environment generated by a computer, which can provide a multimedia virtual world. Users can control the operable virtual objects in the virtual scene through operating devices or operating interfaces, observe virtual objects such as objects, animals, people, and scenery in the virtual scene from the perspective of virtual objects, or interact with virtual objects such as objects, animals, people, and scenery in the virtual scene or other virtual objects through virtual objects. Virtual scenes are usually generated by applications in computer devices such as terminals and displayed based on the hardware in the terminals (such as screens).
[0087] Dynamic objects refer to virtual objects such as objects, animals, and characters in the virtual scene corresponding to the game application. For example, the dynamic objects in this application can be characters or animals in the virtual scene corresponding to the game application. Characters can be characters in the virtual scene. Animals can be pets or monsters in the virtual scene.
[0088] Specifically, during the process of interaction between the user and the game application in the terminal, the terminal can render the screen through the graphics processor to obtain each frame of the game screen, that is, the terminal can determine the position of the dynamic objects in the virtual scene corresponding to the game application during the movement, and obtain the lighting information of the corresponding lighting probe based on the position; further, the terminal can render the dynamic objects according to the obtained lighting information to obtain a frame of the game screen with global illumination.
[0089] It can be understood that in some cases, the terminal can make two layers of lighting probes for virtual objects in the virtual scene corresponding to the game application (static objects such as mountains, trees and buildings), and also make two layers of lighting probes for virtual characters (dynamic objects) in the virtual scene. Finally, the two types of lighting probes are integrated, and the information file storing the lighting probes is modified through code to obtain the final lighting probe position information available to the engine, so as to perform lighting rendering processing on the virtual objects and virtual characters in the virtual scene during the game, and obtain a globally illuminated game screen, thereby achieving a good global illumination effect with the highest cost-effectiveness, effectively improving the rendering efficiency and rendering effect of the game screen.
[0090] In one embodiment, after the game screen is post-processed by a graphics processor to obtain a processed game screen, the method further includes:
[0091] The processed game screen is stored in the memory of the graphics processor;
[0092] The steps of transmitting the processed game screen to the central processing unit based on the direct memory access method include:
[0093] Based on the direct memory access method, each processed game screen saved in the memory of the graphics processor is transferred to the pixel buffer object, and each processed game screen is transferred to the memory of the central processing unit through the pixel buffer object, so that the central processing unit obtains each processed game screen from the memory of the central processing unit.
[0094] Specifically, the terminal performs post-processing on the game screen through the GPU, and after obtaining the processed game screen, the terminal can save the processed game screen in the memory of the GPU; further, the terminal can transfer each processed game screen saved in the memory of the graphics processor to the pixel buffer object based on direct memory access, and transfer each processed game screen to the memory of the CPU through the pixel buffer object, so that the CPU can directly obtain each processed game screen from its own memory.
[0095] For example, the terminal is a mobile device. Figure 6 As shown in the figure, it is a schematic diagram of using PBO to transfer GPU image data to CPU memory. Figure 6 In the schematic diagram shown, the mobile device can use PBO to transfer pixel data to the CPU memory. For example, the GPU in the mobile device post-processes the game screen. After obtaining the processed game screen, the mobile device can save the processed game screen in the GPU memory, and transfer the data of each processed game screen saved in the GPU memory to the pixel buffer object (PBO) through asynchronous DMA through the GPU, and then send the data of each processed game screen directly to the CPU memory through the pixel buffer object (PBO), so that the CPU can quickly obtain each processed game screen from the CPU memory, and perform encoding and format conversion, so as to quickly and accurately obtain the game video file in the target format.
[0096] In this embodiment, PBO is used to optimize the performance of transmitting the game screen from the GPU memory to the CPU memory. PBO can use direct memory access to quickly transmit pixel data between the GPU and the CPU without occupying the CPU's operating cycle. Moreover, it is asynchronous transmission, so it can quickly upload and download pixel data, thereby speeding up the performance and being able to run the post-processing, acquisition and recording functions of the cross-platform game screen in real time with high performance, thereby effectively improving the efficiency of game screen processing.
[0097] In one embodiment, the steps of performing video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format by a central processing unit include:
[0098] Obtain encoding parameters corresponding to the game application;
[0099] By means of a central processing unit, data format conversion is performed on the processed game screen according to the encoding format in the single instruction and the encoding parameter to obtain a converted game screen;
[0100] Encoding the converted game screen according to the compression quality parameter, encoding time and bit rate in the encoding parameters to obtain the encoded game screen;
[0101] Encapsulate the encoded game screen to obtain a game video file in the target format.
[0102] Among them, the encoding parameters refer to various parameters used for encoding, for example, the encoding parameters in this application may include encoding time, encoding format, bit rate, compression quality parameters, resolution and other parameters. The encoding format may include different types of encoding formats, for example, the encoding format includes but is not limited to at least one of H264, H265, etc. It can be understood that the encoding parameters in this application can be pre-set or determined by real-time interaction with the user, and no specific restrictions are made here.
[0103] Specifically, after the terminal transmits the processed game screen to the central processing unit based on the direct memory access method, the terminal can obtain the encoding parameters corresponding to the game application, and through the central processing unit, convert the data format of each processed game screen according to the NEON instruction and the encoding format in the encoding parameters, so as to obtain the converted game screen that conforms to the encoding format; further, the terminal can encode the converted game screen that conforms to the encoding format according to the compression quality parameters, encoding time and bit rate in the encoding parameters to obtain the encoded game screen, and encapsulate the encoded game screen according to the target encapsulation format, so as to quickly and accurately obtain the game video file in the target encapsulation format.
[0104] For example, the terminal is a mobile device. Figure 7 As shown in FIG. 1 , it is a schematic diagram of how the CPU processes the processed game screen. Figure 7In the schematic diagram shown, after the CPU in the mobile device obtains the processed game screen provided by the GPU in the previous stage, the mobile device can configure various encoding parameters by real-time interaction with the user, or can adapt various encoding parameters according to the parameters corresponding to the game application; further, the CPU in the mobile device can convert the data format of the processed game screen through NEON instructions based on the encoding format to obtain a game screen that conforms to the encoding format, and further, the CPU in the mobile device can encode the obtained game screen that conforms to the encoding format according to the encoding parameters to obtain the encoded game screen, and encapsulate the encoded game screen according to the target encapsulation format, so as to obtain the final game video file that conforms to the target encapsulation format. For example, the target encapsulation format can include dozens of common audio and video formats such as AVI and mp4.
[0105] Since data format conversion is one of the performance bottlenecks, it is necessary to focus on optimizing and accelerating it. The CPU format conversion solution used in the traditional method has poor performance and does not fully reflect the SIMD (Single Instruction Multiple Data) attribute. In the embodiment of the present application, the conversion process is accelerated by NEON instructions, and a single instruction processes multiple pixel data at the same time, which can effectively improve the performance of data format conversion, thereby effectively improving the efficiency of game screen processing.
[0106] In one embodiment, before obtaining the encoding parameters corresponding to the game application, the method further includes:
[0107] In response to an editing operation triggered on the configuration interface, generating encoding parameters for the game application; or,
[0108] In response to a selection operation triggered in the configuration interface, compression quality parameters of the game application are generated.
[0109] Among them, the compression quality parameter is one of the encoding parameters.
[0110] The configuration interface refers to an interface used to configure encoding parameters. For example, the configuration interface in the present application can be displayed on the screen in the form of a secondary pop-up window.
[0111] Specifically, before the terminal obtains the encoding parameters corresponding to the game application, the terminal can configure the encoding parameters by interacting with the user in real time. For example, the user can edit the values of each encoding parameter in the configuration interface, and the terminal generates the values of each encoding parameter corresponding to the game application in response to the editing operation triggered by the user in the configuration interface; or, the user can edit the value of the compression quality parameter in the configuration interface, and the terminal generates the value of the compression quality parameter of the game application in response to the selection operation triggered in the configuration interface. That is, in an embodiment of the present application, the user can be provided with real-time settings of the encoding parameters of the game screen corresponding to the game application, or only a real-time setting window of the compression quality parameters can be provided for the user to set. Therefore, the game screen post-processing step, the screen acquisition step, and the data format conversion step are deeply accelerated to achieve high-performance real-time operation to post-process, acquire and record the cross-platform game screen, solving the problem that the real-time game screen of the mobile terminal cannot be post-processed and recorded in the traditional way.
[0112] In one embodiment, the step of encapsulating the encoded game screen to obtain a game video file in a target format includes:
[0113] Adaptive target format based on game style parameters of game applications;
[0114] The format of the encoded game screen is converted based on the target format to obtain a game video file in the target format.
[0115] The target format refers to the target encapsulation format. The target encapsulation format in this application includes but is not limited to AVI, mp4 and other formats.
[0116] Specifically, the terminal is a mobile terminal device as an example for description. Figure 7 As shown in FIG. 1 , it is a schematic diagram of how the CPU processes the processed game screen. Figure 7In the schematic diagram shown, after the CPU in the mobile terminal device obtains the processed game screen provided by the GPU in the previous stage, the mobile terminal device can configure various encoding parameters by real-time interaction with the user, or can adapt various encoding parameters according to the parameters corresponding to the game application; further, the CPU in the mobile terminal device can convert the data format of the processed game screen through NEON instructions based on the encoding format to obtain a game screen that conforms to the encoding format, and according to the encoding parameters, the obtained game screen that conforms to the encoding format is data encoded. After obtaining the encoded game screen, the CPU in the mobile terminal device can adapt the target format based on the game style parameters of the game application, and transform the format of the encoded game screen based on the target format to obtain a game video file in the target format. For example, the resolution parameter required in the game style parameters of the game application is low, but the frame rate parameter is required to be high. Therefore, the mobile terminal device can adapt the target encapsulation format to the first encapsulation format based on the resolution parameter and frame rate parameter in the game style parameters of the game application. For example, the first encapsulation format can be AVI format, that is, the CPU in the mobile terminal device can adapt the target encapsulation format to AVI format based on the game style parameters of the game application, and transform the format of the encoded game screen to obtain a game video file in AVI format. Therefore, the mobile device can adapt the target encapsulation format based on the parameters of game applications of different game styles (or game types), and accelerate the conversion process through NEON instructions, which can effectively improve the performance of the conversion process, so that the post-processing, acquisition and recording solutions for cross-platform game screens provided in the embodiments of the present application can run in real time with high performance.
[0117] In one embodiment, the single instruction includes a NEON instruction; and the steps of performing video encoding on the processed game screen according to the single instruction to obtain a game video file in a target format by a central processing unit include:
[0118] Through the central processing unit, at least two pixel data in the processed game screen are processed in parallel according to the NEON instruction to obtain a game video file in a target format.
[0119] Specifically, the terminal is a mobile device as an example for explanation. After the mobile device transmits the processed game screen to the central processing unit based on direct memory access, the mobile device processes at least two pixel data in the processed game screen in parallel through the central processing unit according to the NEON instruction to obtain a game video file in the target format. Since data format conversion is a performance bottleneck in the processing process, it is necessary to focus on optimizing acceleration. The CPU format conversion solution used in the traditional method has poor performance and does not fully reflect the SIMD attribute. In the embodiment of the present application, the conversion process is accelerated by NEON instructions, and a single instruction processes multiple pixel data at the same time, which can effectively improve the performance of the conversion process, thereby effectively improving the efficiency of game screen processing.
[0120] In one embodiment, after the processed game screen is encoded by the central processing unit according to a single instruction to obtain a game video file in a target format, the method further includes:
[0121] Use the game video in the game video file as the video to be segmented;
[0122] Detecting the shot boundaries of the video to be segmented to obtain the video shots of the video to be segmented;
[0123] Extracting target shot features from the video shots of the video to be segmented;
[0124] The trained scene segmentation model is used to identify the scene boundary of the target shot features to obtain the video scene of the video to be segmented;
[0125] Select the video scene of interest from the video scenes of the video to be segmented, and share the video scene of interest with the target object.
[0126] A video shot sequence refers to a sequence consisting of at least two video shots, one shot contains continuous frames taken by a camera without interruption, and a scene consists of continuous shots, describing the same short game story. Usually, a continuous shot sequence of any length can be regarded as a game video short. The video shot sequence in this application contains at least two video shots, and each video shot can contain continuous video frames of different numbers of frames.
[0127] The lens feature extraction model refers to a neural network model used to extract lens features. For example, the ResNet50 network can be used as the basic network framework of the lens feature extraction model in this application.
[0128] The scene segmentation model refers to a model used to divide a video into several independent sub-videos in the time domain according to different scenes in the video. For example, in this application, Bi-LSTM and three-layer fully connected layers can be used as the basic network framework of the scene segmentation model.
[0129] The video to be segmented refers to the video selected by the user for video scene segmentation. The video to be segmented may contain multiple video shots. For example, the video to be segmented in this application contains at least two video shots, and each video shot contains multiple continuous video frame images. For example, the video to be segmented is: shot 1→shot 2→shot 3, that is, the video to be segmented contains 3 video shots.
[0130] Specifically, the terminal uses a central processing unit to perform video encoding on the processed game screen according to a single instruction. After obtaining a game video file in a target format, the terminal can use the game video in the obtained game video file as the video to be segmented, detect the shot boundary of the video to be segmented, obtain the video shot of the video to be segmented, and use the shot feature extraction model to extract the target shot feature from the video shot of the video to be segmented; further, the terminal can use the trained scene segmentation model to identify the scene boundary of the target shot feature to obtain the video scene of the video to be segmented; the terminal can select the video scene of interest from the video scene of the video to be segmented, and share the video scene of interest to the target object. For example, the terminal can select a highlight scene from the game video scene of the video to be segmented, and share the highlight scene with the target object. That is, the trained scene segmentation model in this application can be applied to the long video scene segmentation task, which is usually defined as the video stripping capability on the product side, that is, given a long video, the scene segmentation model trained in this application can automatically divide the video into several independent sub-videos in the time domain according to the different scenes in the video. In addition, the trained scene segmentation model in the present application can also be applied to video shot retrieval, that is, for a given shot, shots of similar scenes are automatically retrieved and sorted according to similarity.
[0131] For example, taking the target video shot sequence with an input length of B*Shot-Len*N as an example, B in B*Shot-Len*N is the training batch size, Shot-Len is the number of shots processed in a single batch, and N is the dimension of the shot feature. Using the preset model training method, each input shot will be classified into scene boundaries, that is, it is determined whether each shot in the input target video shot sequence is a scene boundary. If a shot in the target video shot sequence is determined to be a scene boundary, the output sequence value corresponding to the shot is 1. If a shot in the target video shot sequence is determined to be a non-scene boundary, the output sequence value corresponding to the shot is 0, that is, when the sequence value of the output sequence is 0, it indicates a non-scene boundary; when the sequence value of the output sequence is 1, it indicates a scene boundary. That is, after the Bi-LSTM model sequence modeling in this application, each sequence value in the output sequence represents whether the boundary of the shot is a scene boundary.
[0132] For example, the mobile terminal device is used as an example for explanation. The mobile terminal device can use the game video in the obtained game video file as the video A to be segmented. Assuming that the video A to be segmented includes: shot 1→shot 2→shot 3, that is, the video to be segmented contains 3 video shots, then the mobile terminal device inputs the video A to be segmented into the trained scene segmentation model, and the output sequence is 110, which means that the shot 1 in the video A to be segmented is the scene boundary, the shot 2 is also the scene boundary, and the shot 3 is a non-scene boundary. Then the mobile terminal device performs video scene segmentation on the video A to be segmented: shot 1 is a video scene marked as scene 1, and shot 2→shot 3 is another video scene marked as scene 2; further, the mobile terminal device can select scene 2 in the game scene from the video scenes (scene 1 and scene 2) of the video to be segmented as a wonderful game scene, and share the wonderful game scene, i.e., scene 2, with the target object, so that the target object can directly watch the game screen corresponding to scene 2 in the game video generated in real time through the user's sharing. Therefore, by providing a cross-platform game screen post-processing, acquisition and recording solution on the mobile terminal, the problem that the mobile terminal cannot post-process and record the game screen in real time in the traditional way is solved. It can perform high-performance real-time post-processing, acquisition, format conversion, encoding and packaging of the rendered game screen, and quickly and accurately generate game video files in the target packaging format, effectively improving the efficiency of game screen processing, and also effectively improving the efficiency of the reuse of game video files.
[0133] In one embodiment, during the process of interacting with the game application, the step of rendering the game screen by a graphics processor includes:
[0134] Displaying a virtual environment, wherein a first virtual object and a second virtual object are displayed in the virtual environment;
[0135] Obtaining a role conversion instruction for the first virtual object;
[0136] In response to the role conversion instruction, the first virtual object is rendered by the graphics processor to obtain a game screen including a target virtual object; wherein the first virtual object in the game screen is converted into the target virtual object; the target virtual object is a virtual object with the same role attributes as the second virtual object.
[0137] Among them, a virtual environment refers to a virtual scene environment generated by a computer, such as a game scene in a game application. The virtual environment provides a multimedia virtual world, where users can control virtual objects through controls on the operation interface, or directly control operable virtual objects in the virtual environment, and observe objects, characters, scenery, etc. in the virtual environment from the perspective of virtual objects, and interact with other virtual objects in the virtual environment through virtual objects. For example, Figure 8 As shown in FIG. 1 , it is a schematic diagram of rendering the game screen through a graphics processor. Figure 8 In the virtual scene corresponding to the game application shown, the user terminal can attack the virtual character 2 by operating the virtual character 1.
[0138] The first virtual object may refer to a virtual object that is movable in the virtual environment and operated by the user himself. The second virtual object may refer to a virtual object that is movable in the virtual environment and operated by the other party (such as a computer program or other user), or may refer to an immovable virtual prop in the virtual environment. The movable objects may be virtual characters, virtual animals, and cartoon characters, etc., and the virtual props may be virtual hats, decorations, chairs, boxes, and cans, etc. Each of the virtual objects has its own shape and volume in the virtual environment, and occupies a part of the space in the virtual environment.
[0139] Specifically, in an interactive virtual scene, a user can interact with virtual objects controlled by another party (such as other users or computer programs) by controlling one or more virtual objects. For example, a user controls a game character to attack a game character controlled by another party to achieve the purpose of upgrading.
[0140] During the process of user interaction with a game application in a terminal, the terminal displays a virtual environment corresponding to the game application, in which a first virtual object and a second virtual object are displayed; the terminal can obtain a role conversion instruction for the first virtual object, and the terminal responds to the role conversion instruction by rendering the first virtual object through a graphics processor to obtain a game screen including a target virtual object; wherein the first virtual object in the game screen is converted into a target virtual object; and the target virtual object is a virtual object with the same role attributes as the second virtual object.
[0141] For example, the terminal is a mobile device. During the interaction between the user and the game application A in the mobile device, the mobile device displays the virtual environment corresponding to the game application A. The first virtual object is virtual character 1 and the second virtual object is virtual character 2 in the virtual environment. The mobile device can obtain the role conversion instruction of the operating user for the virtual character 1. Then, the mobile device responds to the role conversion instruction and renders the virtual character 1 through the GPU to obtain the following image: Figure 8The game screen shown includes a target virtual object; that is, the mobile terminal device responds to the role conversion instruction triggered by the operating user, converts virtual character 1 into a target virtual object with the same role as virtual character 2 through the GPU, and gives the target virtual object the same role attributes as virtual character 2. As a result, the user can perform role conversion on the first virtual object in the virtual scene corresponding to the game application, so as to convert the first virtual object into a target virtual object with the same role as the second virtual object, and obtain the same role attributes as the second virtual object and can interact with it, so that the user can not only use the role attributes of the first virtual object itself to attack the second virtual object of the enemy camp, but also use the same role attributes as the second virtual object to attack the second virtual object, which greatly improves the interactive effect, enhances the user experience, and can effectively improve user stickiness.
[0142] The present application also provides an application scenario, and the application scenario applies the above-mentioned game screen processing method. Specifically, the application of the game screen processing method in the application scenario is as follows:
[0143] In the process of user interaction with the game application, the above-mentioned game screen processing method can be used. For example, in a game scene of a certain game application, each user can select his own virtual character in the game, and interact with the virtual character controlled by the other party (such as other users or computer programs) by controlling one or more virtual characters. For example, the user controls a game character to attack the game character controlled by the other party to achieve the purpose of upgrading. That is, in the process of user interaction with the game, after the mobile terminal device receives the instruction, the mobile terminal device can render the corresponding game screen through the GPU, and perform post-processing such as real-time cropping, blurring, flipping, and superimposing on the game screen. Operation, and real-time and high-performance collection of post-processed game screens, so that the mobile device can encode the post-processed game screens and encapsulate them into game video files that conform to the target encapsulation format. As a result, when users use different mobile devices to interact with game applications, a solution is provided for post-processing, collecting and recording game screens in real time and efficiently on the UE (Unreal Engine) mobile terminal, which can ensure that the entire processing and recording process runs in real time, and fully utilize the parallel computing capabilities of the GPU to greatly improve the post-processing performance, solving the problem that the traditional method cannot achieve real-time post-processing and recording of UE (Unreal Engine) mobile terminal game screens. At the same time, a processing method for the GPU post-processing chain is proposed, which can efficiently and freely configure and combine various game screen post-processing technologies, and can be used in other 3D rendering fields, thereby effectively improving the efficiency of game screen processing.
[0144] The method provided in the embodiment of the present application can be applied to scenarios where users interact with various applications. The following takes the scenario where users interact with game applications as an example to illustrate the game screen processing method provided in the embodiment of the present application.
[0145] Among them, Unreal Engine (UE) is a 3A next-generation game engine developed by a certain company.
[0146] Plugins: In UE, plugins are collections of code and data that developers can enable or disable on a per-project basis in the editor. Plugins can add runtime gameplay features, modify built-in engine features (or add new features), create new file types, and extend the functionality of the editor using new menus, toolbar commands, and sub-modes. Many existing UE subsystems can be extended using plugins.
[0147] FFmpeg: It is a commonly used open source software for video processing. It is powerful and widely used. It is widely used in video websites and commercial software. It is also the implementation of standard encoding and decoding of many audio and video formats.
[0148] Editor: It is a UE visual operation tool that integrates UE functions into a visual interface, making it convenient for users to quickly edit scenes. It also integrates various tools and serves as a bridge for users to use the engine.
[0149] Recording: The video maker records some continuous scene clips.
[0150] Encoding: Format the recorded video to facilitate transmission on the network. It mainly includes two steps: compressing and encoding the video and audio separately, and packaging the audio and video.
[0151] Encapsulation format: It is to package the video compression data and audio compression data and merge them into one file in a specific way.
[0152] AVI (Audio Video Interleave): It can only encapsulate one video track and one audio track, but cannot encapsulate text. It does not have any control function, so it cannot realize streaming media. Its file extension is .avi.
[0153] YUV: It is a color encoding method that uses brightness and chromaticity to specify the color of a pixel. Among them, Y represents brightness (Luminance, Luma), U and V represent chromaticity (Chrominance, Chroma), and chromaticity defines two aspects of color: hue and saturation.
[0154] NEON instruction: is a technology based on SIMD concept, which can operate on multiple data simultaneously based on a single instruction. The NEON instruction it uses is similar to the MMX / SSE / AVX instructions under Intel CPU. It optimizes application performance through vectorized calculation. It is usually used in image processing, audio and video processing, data parallel processing and other scenarios that require a lot of computing.
[0155] Shader: A computer program that runs on the GPU.
[0156] PBO: Pixel Buffer Object.
[0157] SIMD (Single Instruction Multiple Data) is a technology that uses one controller to control multiple processors and simultaneously performs the same operation on each data in a set of data (also called "data vector"), thereby achieving spatial parallelism.
[0158] In the traditional way, the audio and video suite in UE: that is, the audio and video suites currently available in UE are all for a specific platform, such as an independent plug-in for the iOS system, an independent plug-in for the Android system, and an independent plug-in for the Windows system. These plug-ins mainly provide video recording functions on their respective platforms.
[0159] External cross-platform audio and video suite: FFmpeg can provide cross-platform video post-processing, recording and encoding functions.
[0160] That is, in the traditional way, for the audio and video suite in UE: this independent plug-in deployment and use are also independent. During use, it is necessary to configure the plug-in switch separately through different platforms, and combine the development and call of different plug-in features and capability matrices to combine and implement related functions such as camera stream acquisition and video recording. If cross-platform projects use these independent suites, they need to spend a lot of manpower for plug-in compatibility, plug-in splicing and other related development and testing, and maintain the combination code of different plug-ins on different platforms, which is very easy to make mistakes and has poor maintainability. In addition, the functions of these plug-ins are also scattered, and most of them do not support secondary customized development. In addition, the current audio and video plug-in of UE does not have the function of post-processing the video. After the user obtains the recorded video, it is necessary to use professional post-processing tools to post-process the video. The game screen acquisition function of UE also needs additional development. Although the external cross-platform audio and video suite FFmpeg can provide cross-platform post-processing, video recording and encoding capabilities, its video post-processing, recording and encoding are completed using the CPU, which cannot meet the performance requirements of post-processing, recording and encoding of real-time game screens.
[0161] Therefore, in order to solve the above problems, the present application provides a solution for real-time post-processing, collection and recording of cross-platform game screens on Unreal Engine mobile terminals, which can post-process, collect and record game screens in real time and efficiently on UE mobile terminals, solving the problem of the lack of real-time game screen post-processing tools and recording tools on UE mobile terminals in the current market. At the same time, it also realizes flexible configuration and combination of various game screen post-processing methods, and can ensure performance, as well as efficient collection of game screens, and ensure the performance advantages of data format conversion, thereby effectively improving the efficiency of game screen processing.
[0162] On the product side, Fig. 9 The following is a schematic diagram of the product side. After the user opens the game application, the specific steps include:
[0163] 1. The user interacts with the game. After receiving the command, the mobile device renders the screen through the graphics processor to obtain the game screen;
[0164] 2. The mobile device performs post-processing operations such as cropping, blurring, flipping, and overlaying on each frame of the rendered game screen in real time;
[0165] 3. Mobile devices can collect post-processed game images in real time and with high performance;
[0166] 4. The mobile device encodes the post-processed game screen and then packages it into a video file.
[0167] On the technical side, 1.1 Overview of the overall process
[0168] like Fig.10 As shown in the figure, it is a schematic diagram of the overall process of the solution for real-time post-processing, acquisition and recording of cross-platform game screens on Unreal Engine mobile terminals. Fig.10 As shown in , on different mobile platforms, the method provided in this application can realize post-processing, acquisition, format conversion, encoding and packaging of the game screen rendered in real time by Unreal Engine. For example, Fig.10 The mobile operating system 1 and the mobile operating system 2 shown in the figure are two different mobile platforms. Since the performance and memory of mobile devices are relatively limited, to ensure that the entire processing and recording process above can run in real time, it is necessary to fully optimize each module. There are three technical difficulties: 1. How to flexibly configure and combine various game screen post-processing methods and ensure performance; 2. How to efficiently collect game screens; 3. How to ensure the performance of data format conversion.
[0169] 1.2 Key Steps
[0170] 1.2.1 Game screen post-processing
[0171] In the process of recording the game screen as a finished video by the mobile device, various post-processing technologies need to be used. Such as cropping, flipping the picture; adjusting the saturation and brightness of the picture; blurring, sharpening the picture, etc. Traditional post-processing technology uses the CPU for calculation (such as the post-processing interface in FFmpeg), which has poor performance and cannot be used in real time on the mobile terminal. Therefore, in the embodiment of the present application, the post-processing method on the CPU side is transplanted to the GPU, making full use of the parallel computing power of the GPU and greatly improving the post-processing performance. At the same time, a post-processing chain organization method is provided, which can realize flexible configuration and combination of various screen post-processing technologies.
[0172] like Figure 4 The following is the basic structure of a post-processing chain: input screen node, post-processing method and output screen node. After the input screen node processes the screen using the post-processing method, the processed screen is output through the output screen node.
[0173] The game screen output by the output screen node of the previous post-processing chain can become the input screen of the input screen node of the next post-processing chain, so it can be flexibly expanded to use multiple post-processing methods. For example, if you want to crop the game screen first, then adjust the brightness, and finally sharpen it. Then the mobile device can follow the following steps: Figure 5 The post-processing chain shown in .
[0174] 1.2.2 Game Screen Collection
[0175] On the mobile device, after each frame of the game screen is rendered using the GPU, the game screen is saved in the GPU memory. However, FFmpeg can only encode the video data in the CPU memory, and the game screen in the GPU memory needs to be transferred back to the CPU memory during the acquisition process. This step is prone to performance problems, because after the CPU submits the drawing command, the GPU performs the rendering asynchronously, and the CPU does not get the screen result immediately. If the CPU side forcibly waits for the GPU to complete the drawing of the game screen, it will cause serious performance problems. In response to this problem, the mobile device in the embodiment of the present application uses PBO to optimize the performance of transferring the game screen from the GPU memory to the CPU memory.
[0176] like Fig.11 As shown in FIG. 1 , it is a schematic diagram of transferring GPU image data to CPU memory in the traditional way. Fig.12 The figure shows a schematic diagram of using PBO to optimize the performance of transferring game images from GPU memory to CPU memory. Fig.12The pixel buffer object (PBO) shown in the figure can flexibly interact with the frame buffer in the GPU. For example, the pixel buffer object (PBO) can obtain the rendered packed game screen from the frame buffer of the GPU, and unpack the obtained packed game screen to obtain each frame of the unpacked game screen (also called texture object), and store each frame of the unpacked game screen in the CPU. At the same time, Fig.12 The pixel buffer object (PBO) shown in can also obtain texture objects from the CPU and upload texture objects to the GPU. Fig.11 The transfer method shown in the figure is the traditional method of transferring image data from GPU memory to CPU memory without using PBO. Fig.11 The glReadPixel in the function represents a function that can be used to obtain image data. In the traditional way, the image data is transferred from the GPU memory to the CPU memory through the glReadPixel function, such as Fig.12 As shown in the PBO can use DMA technology to quickly transfer pixel data between GPU and CPU without occupying the CPU's operating cycle, and it is asynchronous DMA transmission, so it can quickly upload and download pixel data. For details, please refer to Figure 6 Schematic diagram of using PBO to transfer GPU image data to CPU memory.
[0177] 1.2.3 Video Recording
[0178] like Figure 7 As shown in , after the mobile device obtains the game screen provided in the previous stage, various encoding parameters can be configured. These encoding parameters include, but are not limited to: encoding time, encoding format, bit rate, resolution, etc. Further, the mobile device converts and encodes the data format of the game screen, and finally the mobile device encapsulates the encoded game screen into an output format file. Since data format conversion is one of the performance bottlenecks, it is necessary to focus on optimization and acceleration. The traditional CPU format conversion solution has poor performance and does not fully utilize the SIMD properties of modern CPUs. Therefore, in the implementation of this application, the conversion process can be accelerated by NEON instructions, that is, a single instruction can process multiple pixel data at the same time, thereby improving the performance of the conversion process. It can be understood that the encoding process in this application may include two steps: converting the data format and then encoding the data of the converted format.
[0179] The beneficial effects of the technical solution of this application include:
[0180] (1) It provides a cross-platform game screen post-processing, acquisition and recording solution on the UE mobile terminal, solving the problem of the lack of real-time game screen post-processing and recording tools on the UE mobile terminal in the current market;
[0181] (2) The game screen post-processing module, screen acquisition module, and data format conversion module have all been deeply accelerated, so the method provided by this application can run in real time with high performance. These modules are key modules for UE mobile terminal screen post-processing and recording, and one or more of them can be used separately in relevant business scenarios;
[0182] (3) The GPU post-processing chain technology is proposed, which can efficiently and freely configure and combine various game screen post-processing technologies, and can also be expanded to other 3D rendering fields.
[0183] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0184] Based on the same inventive concept, the embodiment of the present application also provides a game screen processing device for implementing the game screen processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more game screen processing device embodiments provided below can refer to the limitations of the game screen processing method above, and will not be repeated here.
[0185] In one embodiment, Fig.13 As shown, a game screen processing device is provided, including: a rendering module 1302, a processing module 1304, a transmission module 1306 and an encoding module 1308, wherein:
[0186] The rendering module 1302 is used to render the screen through the graphics processor to obtain the game screen during the interaction with the game application.
[0187] The processing module 1304 is used to post-process the game screen through a graphics processor to obtain a processed game screen.
[0188] The transmission module 1306 is used to transmit the processed game screen to the central processing unit based on the direct memory access method.
[0189] The encoding module 1308 is used to perform video encoding on the processed game screen according to a single instruction through the central processing unit to obtain a game video file in a target format.
[0190] In one embodiment, the device also includes: a calling module and an output module, the calling module is used to call the post-processing chain through the graphics processor; wherein the post-processing chain is composed of an input screen node, an image post-processing method and an output screen node; the processing module is also used to perform image processing on the game screen according to the image post-processing method through the input screen node; the output module is used to output the processed game screen obtained by the input screen node through the output screen node.
[0191] In one embodiment, the image post-processing method includes cropping, brightness processing and sharpening processing; the processing module is also used to input the game screen into the first input screen node of the post-processing chain for cropping processing, and output a first output screen; input the first output screen into the second input screen node of the post-processing chain for brightness processing, and output a second output screen; input the second output screen into the third input screen node of the post-processing chain for sharpening processing, and output the processed game screen.
[0192] In one embodiment, the virtual scene corresponding to the game application has a dynamic object; the device also includes: a determination module and an acquisition module, the determination module is used to determine the position of the dynamic object during the movement; the acquisition module is used to obtain the lighting information of the corresponding lighting probe according to the position; the rendering module is also used to render the dynamic object according to the lighting information to obtain a globally illuminated game screen.
[0193] In one embodiment, the device also includes: a saving module, used to save the processed game screen in the memory of the graphics processor; the transmission module is also used to transfer each processed game screen saved in the memory of the graphics processor to a pixel buffer object based on a direct memory access method, and transmit each processed game screen to the memory of the central processing unit through the pixel buffer object, so that the central processing unit obtains each processed game screen from the memory of the central processing unit.
[0194] In one embodiment, the device also includes: a conversion module and a packaging module, the acquisition module is also used to obtain the encoding parameters corresponding to the game application; the conversion module is used to convert the data format of the processed game screen through the central processor according to a single instruction and the encoding format in the encoding parameters to obtain the converted game screen; the encoding module is also used to encode the converted game screen according to the compression quality parameters, encoding time and bit rate in the encoding parameters to obtain the encoded game screen; the packaging module is used to package the encoded game screen to obtain the game video file in the target format.
[0195] In one embodiment, the device also includes: a generation module for generating encoding parameters of the game application in response to an editing operation triggered in the configuration interface; or, generating compression quality parameters of the game application in response to a selection operation triggered in the configuration interface.
[0196] In one embodiment, the device also includes: an adaptive module for adapting the target format based on the game style parameters of the game application; the encapsulation module is also used to convert the format of the encoded game screen based on the target format to obtain the game video file in the target format.
[0197] In one embodiment, the single instruction includes a NEON instruction; the processing module is also used to process at least two pixel data in the processed game screen in parallel through the central processing unit according to the NEON instruction to obtain the game video file in the target format.
[0198] In one embodiment, the device also includes: a detection module, an extraction module, an identification module and a selection module, the detection module is used to take the game video in the game video file as the video to be segmented; detect the shot boundaries of the video to be segmented to obtain the video shots of the video to be segmented; the extraction module is used to extract target shot features from the video shots of the video to be segmented; the identification module is used to perform scene boundary recognition on the target shot features through a trained scene segmentation model to obtain the video scene of the video to be segmented; the selection module is used to select a video scene of interest from the video scene of the video to be segmented, and share the video scene of interest with the target object.
[0199] In one embodiment, the device also includes: a display module, used to display a virtual environment, in which a first virtual object and a second virtual object are displayed; the acquisition module is also used to obtain a role conversion instruction for the first virtual object; the rendering module is also used to respond to the role conversion instruction, render the first virtual object through a graphics processor, and obtain a game screen including a target virtual object; wherein the first virtual object in the game screen is converted into a target virtual object; and the target virtual object is a virtual object with the same role attributes as the second virtual object.
[0200] Each module in the above-mentioned game screen processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0201] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. In this embodiment, the computer device is described as a terminal. The internal structure diagram thereof may be as follows: Fig.14 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a game screen processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.
[0202] Those skilled in the art will understand that Fig.14The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0203] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.
[0204] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0205] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0206] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0207] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0208] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0209] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for processing a game screen, characterized in that: The method comprises: In the process of interacting with the game application, the graphics processor is used to render the screen to obtain the game screen; Post-processing the game screen by the graphics processor to obtain a processed game screen; Transmitting the processed game screen to a central processing unit based on a direct memory access method; The central processing unit performs video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format.
2. The method according to claim 1, characterized in that The post-processing the game screen by the graphics processor to obtain a processed game screen includes: Calling a post-processing chain through the graphics processor; wherein the post-processing chain comprises an input screen node, an image post-processing method, and an output screen node; Through the input screen node, performing image processing on the game screen according to the image post-processing method; The processed game screen obtained by the input screen node is output through the output screen node.
3. The method according to claim 2, characterized in that The image post-processing method includes cropping, brightness processing and sharpening processing; The step of performing image processing on the game screen according to the image post-processing method through the input screen node includes: Input the game screen into the first input screen node of the post-processing chain for cropping, and output a first output screen; Inputting the first output picture into the second input picture node of the post-processing chain for brightness processing, and outputting a second output picture; The second output picture is input into the third input picture node of the post-processing chain for sharpening processing, and the processed game picture is output.
4. The method according to claim 1, characterized in that: The virtual scene corresponding to the game application has dynamic objects; the rendering of the picture by the graphics processor to obtain the game picture includes: Determining the position of the dynamic object during the motion; Obtaining lighting information corresponding to the lighting probe according to the position; The dynamic object is rendered according to the lighting information to obtain a game screen with global lighting.
5. The method according to claim 1, characterized in that After the graphics processor post-processes the game screen to obtain a processed game screen, the method further includes: Saving the processed game screen in the memory of the graphics processor; The method of transmitting the processed game screen to a central processing unit based on a direct memory access method includes: Based on the direct memory access method, each processed game screen saved in the memory of the graphics processor is transferred to the pixel buffer object, and each processed game screen is transferred to the memory of the central processing unit through the pixel buffer object, so that the central processing unit obtains each processed game screen from the memory of the central processing unit.
6. The method according to claim 1, characterized in that The central processor performs video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format, including: Obtaining encoding parameters corresponding to the game application; By means of the central processor, data format conversion is performed on the processed game screen according to the single instruction and the coding format in the coding parameter to obtain a converted game screen; Encoding the converted game screen according to the compression quality parameter, encoding time and bit rate in the encoding parameters to obtain an encoded game screen; The encoded game screen is packaged to obtain the game video file in the target format.
7. The method according to claim 6, characterized in that Before obtaining the encoding parameters corresponding to the game application, the method further includes: In response to an editing operation triggered on a configuration interface, generating encoding parameters of the game application; or, In response to a selection operation triggered in the configuration interface, compression quality parameters of the game application are generated.
8. The method according to claim 6, characterized in that The step of encapsulating the encoded game screen to obtain the game video file in a target format includes: Adapting the target format based on game style parameters of the game application; The format of the encoded game screen is converted based on the target format to obtain the game video file in the target format.
9. The method according to claim 1, characterized in that: The single instruction comprises a NEON instruction; The central processor performs video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format, including: The central processing unit processes at least two pixel data in the processed game screen in parallel according to the NEON instruction to obtain the game video file in the target format.
10. The method according to claim 1, characterized in that After the central processor performs video encoding on the processed game screen according to a single instruction to obtain a game video file in a target format, the method further includes: Using the game video in the game video file as the video to be segmented; Detecting the shot boundaries of the video to be segmented to obtain the video shots of the video to be segmented; Extracting target shot features from the video shots of the video to be segmented; Performing scene boundary recognition on the target shot features through the trained scene segmentation model to obtain the video scene of the video to be segmented; Selecting a video scene of interest from the video scenes of the video to be segmented, and sharing the video scene of interest with a target object.
11. The method according to any one of claims 1 to 10, characterized in that: In the process of interacting with the game application, the graphics processor is used to render the game screen, and the game screen is obtained, including: Displaying a virtual environment, wherein a first virtual object and a second virtual object are displayed in the virtual environment; Obtaining a role conversion instruction for the first virtual object; In response to the role conversion instruction, the first virtual object is rendered by a graphics processor to obtain a game screen including a target virtual object; wherein the first virtual object in the game screen is converted into a target virtual object; and the target virtual object is a virtual object with the same role attributes as the second virtual object.
12. A game screen processing device, characterized in that: The device comprises: A rendering module is used to render the screen through a graphics processor to obtain the game screen during the interaction with the game application; A processing module, used for post-processing the game screen through the graphics processor to obtain a processed game screen; A transmission module, used for transmitting the processed game screen to a central processing unit based on a direct memory access method; The encoding module is used to perform video encoding on the processed game screen according to a single instruction through the central processing unit to obtain a game video file in a target format.
13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.
Citation Information
Cited By
Game resource interaction method and device, equipment and medium
CN121490403A
