Image processing method, device, computer device, and computer program
By using a video card shader to parallelize the decoding of compressed texture blocks, the method addresses CPU resource inefficiencies in image processing, achieving faster and more efficient decoding of compressed texture resources.
Patent Information
- Application Number
- JP2023538900
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-05-17
- Filing Date
- 2022-04-08
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2042-04-08
AI Technical Summary
Current image processing technologies face inefficiencies in decoding compressed texture resources, particularly due to the high resource consumption of central processing units (CPUs) and the need for parallel decoding of multiple texture blocks.
The method employs a video card shader to pre-arrange independent workgroups for decoding compressed texture blocks in parallel, utilizing a compressed texture format like ASTC or ETC, thereby assigning each block to a corresponding workgroup for efficient decoding without involving the CPU.
This approach enhances decoding efficiency by reducing CPU resource occupation and improving overall processing speed through parallel decoding, while also allowing for flexible decoding methods adaptable to different hardware configurations.
Smart Images

Figure 0007582634000002 
Figure 0007582634000003 
Figure 0007582634000004
Abstract
Description
[Technical field]
[0001] This application claims priority to a Chinese patent application filed with the China Patent Office on May 17, 2021, bearing application number 202110536966.7 and titled "Image processing method, apparatus, computer device and storage medium", the entire contents of which are incorporated into this application by reference.
[0002] The present application relates to the field of Internet technology, and more particularly to image processing technology. [Background technology]
[0003] At present, in the process of creating an image, an image creator generally adopts a compressed texture format to compress and encode the image, obtain a compressed texture resource of the image, and then deliver the compressed texture resource to the outside, in order to save the memory occupied by the image and the network resource required for transmitting the image. After obtaining the compressed texture resource of the image, any one of the external devices generally needs to perform a decoding process on the compressed texture resource. Therefore, how to decode the compressed texture resource has become a focus of research. Summary of the Invention [Means for solving the problem]
[0004] The embodiments of the present application provide an image processing method, apparatus, computer device and storage medium, which can realize the decoding of a target compressed texture resource and effectively improve the decoding efficiency.
[0005] On the other hand, an embodiment of the present application provides an image processing method executed by a computer device, the method including the steps of: obtaining a target compressed texture resource of a target image, the target compressed texture resource being obtained by encoding the target image using a compressed texture format, the target compressed texture resource including a plurality of compressed texture blocks; allocating a plurality of target work groups for decoding to the plurality of compressed texture blocks in a video card shader, and allocating each compressed texture block to a corresponding target work group; and invoking each target work group in the video card shader in parallel, and performing a decoding process on the received compressed texture blocks based on the compressed texture format, thereby obtaining target texture data of the target image, the target texture data including decoded data corresponding to each of the compressed texture blocks.
[0006] On the other hand, an embodiment of the present application provides an image processing device arranged on a computer device, the device obtaining a target compressed texture resource of a target image, the target compressed texture resource being obtained by encoding the target image using a compressed texture format, the target compressed texture resource including a plurality of compressed texture blocks, and a processing unit used to assign a plurality of target work groups for decoding to the plurality of compressed texture blocks in a video card shader and assign each compressed texture block to a corresponding target work group, the processing unit is further used to obtain target texture data of the target image by invoking each target work group in the video card shader in parallel and performing a decoding process on the received compressed texture blocks based on the compressed texture format, the target texture data including decoded data corresponding to each of the compressed texture blocks.
[0007] Meanwhile, an embodiment of the present application provides a computer device, the computer device including an input interface and an output interface, and further including a video card shader and a computer storage medium, the computer storage medium storing one or more instructions, the one or more instructions being suitable for performing the image processing method of the above-mentioned aspects.
[0008] Meanwhile, an embodiment of the present application provides a computer storage medium having one or more instructions stored thereon, the one or more instructions being suitable for performing the image processing method according to the above aspects.
[0009] Meanwhile, an embodiment of the present application provides a computer program product, which, when executed, is used to perform the image processing method in the above aspects.
[0010] The embodiment of the present application introduces a video card shader, in which a plurality of mutually independent work groups are pre-arranged in the video card shader, and the target compressed texture resource obtained by adopting the compressed texture format to encode the target image includes a plurality of compressed texture blocks. After obtaining the target compressed texture resource of the target image, the video card shader can assign one target work group for decoding to each of the compressed texture blocks, and assign each compressed texture block to the corresponding target work group. Then, the method of calling each target work group in the video card shader in parallel can perform decoding processing on the compressed texture block received according to the compressed texture format. This processing method can not only realize the decoding processing of the target compressed texture resource, but also realize the parallel decoding of each compressed texture block, thereby effectively improving the decoding efficiency. In addition, by realizing the entire decoding process by the video card shader, the entire decoding process does not require the involvement of the central processing unit, thus effectively saving the resources of the central processing unit, thereby effectively reducing the resource occupation rate of the central processing unit and improving the operating efficiency of the computer device. [Brief description of the drawings]
[0011] In order to more clearly describe the technical solutions in the embodiments of the present invention, the following briefly describes the drawings necessary for the description of the embodiments. Obviously, the drawings in the following description are only the embodiments of the present invention, and those skilled in the art can further obtain other drawings according to the drawings provided under the premise of not exerting any labor worthy of inventive step. [Figure 1a] 4 is a process chart of the decoding of an image processing method provided by an embodiment of the present application; [Figure 1b] 4 is a process chart of video card shader decoding provided by an embodiment of the present application; [Diagram 2] 1 is a process chart of an image processing method provided by an embodiment of the present application. [Figure 3a]FIG. 1 is a schematic structural diagram of a blockchain provided by an embodiment of the present application. [Figure 3b] FIG. 1 is a schematic diagram of storing target texture data in a blockchain provided by an embodiment of the present application; [Figure 4] 4 is a process chart of an image processing method provided by another embodiment of the present application. [Diagram 5] FIG. 2 is a schematic diagram of spatial mapping provided by an embodiment of the present application. [Figure 6a] FIG. 2 is a structural schematic diagram of a cloud game server provided by an embodiment of the present application; [Figure 6b] 4 is a process chart of an image processing method provided by another embodiment of the present application. [Figure 7] FIG. 2 is a schematic diagram of a software tool packet invocation provided by an embodiment of the present application. [Figure 8] FIG. 1 is a structural schematic diagram of an image processing device provided by an embodiment of the present application; [Figure 9] FIG. 2 is a structural schematic diagram of a computer device provided by an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] The following clearly and completely describes the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application.
[0013] With the development of Internet technology, Artificial Intelligence (AI) technology will develop better accordingly. The so-called artificial intelligence technology refers to digital computers or machine simulations controlled by digital computers, theories, methods, technologies and application systems that extend and expand human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science, which mainly understands the substance of intelligence to manufacture new intelligent devices that can react in a manner similar to human intelligence, so that the intelligent devices have various functions such as sensing, reasoning and decision-making. Correspondingly, AI technology is an integrated discipline, which mainly includes several directions such as Computer Vision (CV), voice processing technology, natural language processing technology and Machine Learning (ML) / Deep Learning.
[0014] Here, computer vision technology is the science that studies how to make things "see" on devices, and more specifically, it refers to the use of cameras and computers to perform machine vision such as identification, tracking, and measurement of targets on behalf of human eyes, and then graphic processing and computer processing to make the image suitable for human eye observation or device detection. As theories and technologies related to computer vision research in the field of science, it aims to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / action identification, three-dimensional object reconstruction, 3D (three-dimensional) technology, virtual reality, augmented reality, synchronous positioning and map construction, and more generally includes biometric identification technologies such as face recognition and fingerprint recognition.
[0015] Based on the image processing technology in the computer vision technology mentioned above, the embodiment of the present application provides a technical solution for image processing to realize the decoding of the compressed texture resource of any image. The so-called compressed texture resource is the encoding result obtained by adopting a compressed texture format to encode an image. Here, the compressed texture format may be ASTC (Adaptive Scalable Texture Compression) format or ETC format (compressed texture format), etc. ASTC and ETC generally perform pixel compression and storage in units of compressed texture blocks (abbreviated as block), and the pixel numbers of different block compressions range from 4x4 to 12x12, and the blocks are independent of each other. That is, the compressed texture resource obtained by adopting a compressed texture format to encode an image can generally include multiple compressed texture blocks. It should be noted that the meaning of "multiple" mentioned in the embodiment of the present application refers to at least two.
[0016] In a specific implementation, the technical solution of the image processing may be executed in a computer device, and the computer device may be a terminal or a server. The terminals mentioned herein include, but are not limited to, smartphones, tablet computers, laptops, desktop computers, smart watches, smart TVs, etc. Various clients (applications, APPs), such as game clients, multimedia clients, social clients, etc., can be operated in the terminal. The servers mentioned herein may be independent physical servers, server clusters or distributed systems composed of multiple physical servers, and may further be cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs (Content Delivery Networks), and big data and artificial intelligence platforms.
[0017] In addition, the computer device may be located outside the blockchain network or within the blockchain network, but is not limited thereto. The so-called blockchain network is a network composed of a peer-to-peer network (P2P network) and a blockchain, where the blockchain refers to a new application mode of computer technology such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc., and is essentially a decentralized database, and data blocks (or called blocks) generated in association with cryptographic methods. When the computer device is located within the blockchain network or has a communication connection with the blockchain network, the computer device can upload and store internal data in the blockchain of the blockchain network, thereby preventing tampering with the internal data of the computer device and improving the security of the internal data.
[0018] In a possible implementation, the computer device may include at least a video card shader (Graphics Processing Unit, GPU), a video card computing shader (hereinafter abbreviated as video card shader) and a computing shader decoding pipeline (hereinafter abbreviated as decoding pipeline), etc. Here, (1) a video card is a device that is centered on a GPU and is responsible for outputting display images. (2) a video card shader is a processing unit operated by a video card, which is connected to an input video card shader cache block for data input and an output video card shader cache block for data output. The video card shader may include M work groups with decoding processing capabilities, and the so-called work groups can be understood as sub-processing units with decoding processing capabilities in the video card shader, where each work group is independent of each other, and M is an integer greater than 1. (3) a decoding pipeline is a processing unit for instructing the decoding process of the video card shader, that is, the video card shader in the embodiment of this application operates according to the process defined in the decoding pipeline. The computer device can further include a rendering pipeline, which can be called a rendering pipeline, and is a parallel processing unit that processes graphics signals inside a video card independently of each other. When the computer device includes a rendering pipeline, it can obtain texture data of any image by decoding through an image processing technical solution, and then realize the rendering display of the any image according to the texture data decoded by the rendering pipeline.
[0019] The image processing technical solution provided by the embodiment of the present application is mainly to decode the compressed texture resource of any image by the video card shader in a computer device. As shown in Figure 1a, the decoding process of the image processing technical solution is outlined as follows: first, the decoding pipeline in the computer device can be initialized and a shader decoded file can be loaded; after the shader decoded file is successfully loaded, the shader decoded file can be transmitted to the video card shader of the computer device. After the compressed texture resource of any image and the correlation decoding parameters of the compressed texture resource are input to the video card shader in the computer device through the input shader cache block, the video card shader can respectively assign a target work group (i.e., the work group to which the compressed texture block is assigned) to each compressed texture block in the compressed texture resource.
[0020] Then, each compressed texture block can be assigned to a corresponding target work group, and each target work group performs decoding processing on the received compressed texture block according to the correlation decoding parameters and the shader decoding file in parallel, thereby realizing the decoding processing of the compressed texture resource for any image, and obtaining the texture data of the any image. For example, take a 4×4 block (compressed texture block) as an example, and indicate each compressed texture block with "B", as shown in FIG. 1b, the compressed texture block 11 in the compressed texture resource can be assigned to the target work group 21 of the video card shader to perform decoding processing, thereby obtaining the decoded data identified by 31. The compressed texture block 12 in the compressed texture resource can be assigned to the target work group 22 of the video card shader to perform decoding processing, thereby obtaining the decoded data identified by 32, and so on. After obtaining the texture data of any image, the video card shader can output the texture data of the any image by the output shader cache block. In a possible embodiment, during the process of decoding a compressed texture resource in a video card shader, if there is a failure in decoding one or more compressed texture blocks, the video card shader can output error data related to the decoding process through an output shader cache block, so that relevant technical personnel can timely analyze the cause of the decoding failure based on the error data.
[0021] Through practice, the technical solution of image processing provided by the embodiments of this application has at least the following beneficial effects: (1) By implementing the entire decoding process by introducing the video card shader, the entire decoding process does not require the involvement of the central processing unit, thus effectively saving the resources of the central processing unit, thereby effectively reducing the resource occupation of the central processing unit and improving the operating efficiency of the computer device; (2) By the method of invoking each target work group in the video card shader in parallel, the parallel decoding of each compressed texture block can be realized, thereby effectively improving the decoding efficiency.
[0022] Based on the above description, an embodiment of the present application provides an image processing method, which may be executed in the above-mentioned computer device. The computer device may include a rendering pipeline, a decoding pipeline and a video card shader, and the video card shader operates according to a process defined in the decoding pipeline. As shown in Figure 2, the image processing method may include the following S201 to S204.
[0023] In S201, a target compressed texture resource of a target image is obtained.
[0024] Here, the target image may be any of the following images: a target game screen image in a target game, a video image of any movie, a selfie image of any user, a landscape image of any tourist spot, etc. The target compressed texture resource is obtained by adopting a compressed texture format to encode the target image, and the target compressed texture resource includes a plurality of compressed texture blocks. It should be noted that the target image includes texture data in a texture format, and the texture format here may be, for example, an RGBA format, where R in RGBA represents the red channel, G represents the green channel, B represents the blue channel, and A represents the transparent channel. The process of adopting a compressed texture format to encode the target image can be understood as a process of converting the texture data of the target image from the texture format to the compressed texture format. Accordingly, the essence of decoding the target compressed texture resource can be understood as a process of converting the target compressed texture resource from the compressed texture format to the texture format.
[0025] In a possible embodiment, if the target compressed texture resource of the target image is not pre-stored in the local space of the computer device, the embodiment of S201 is as follows, a resource acquisition request is sent to another device in which the target compressed texture resource is stored, requesting the other device to return the target compressed texture resource, and then the computer device can receive the target compressed texture resource. If the target compressed texture resource of the target image is pre-stored in the local space of the computer device, the computer device can realize the acquisition of the target compressed texture resource by loading the target compressed texture resource from the local space by the internal rendering pipeline, that is, the embodiment of S201 is as follows, the rendering pipeline loads the target compressed texture resource of the target image from the local space of the computer device.
[0026] At S202, multiple compressed texture blocks are assigned multiple target work groups for decoding in a video card shader.
[0027] As can be seen from the above description, a video card shader can include M work groups, each of which has a decoding processing capability, and each work group has an identification number. In a specific implementation, a plurality of compressed texture blocks can be traversed to determine a current compressed texture block that is currently traversed. Considering that the number of work groups and the number of compressed texture blocks in a video card shader can be the same or different, and if the number of work groups in a video card shader is less than the number of compressed texture blocks, when allocating a target work group to a current compressed texture block, it may result in no work group being allocated to the video card shader. Based on this, in order to reasonably allocate a target work group to a current compressed texture block that is currently traversed, it can first detect whether there is a remaining work group that is not allocated among the M work groups in a video card shader, and then adopt different allocation logics to allocate a target work group to a current compressed texture block according to the detection result, specifically as follows:
[0028] If there are remaining workgroups that are not assigned among the M workgroups in the video card shader, one target workgroup for decoding can be assigned to the current compressed texture block in the remaining workgroups. Here, the number of remaining workgroups may be one or more. In one embodiment, one remaining workgroup can be randomly selected from the remaining workgroups as the target workgroup for decoding the current compressed texture block. In another embodiment, the remaining workgroup with the smallest number identification can be selected from the remaining workgroups in ascending order of number identification as the target workgroup for decoding the current compressed texture block. In another embodiment, if there is a difference in the decoding capabilities of each workgroup in the video card shader, the remaining workgroup with the highest decoding capability can be selected from the remaining workgroups, or the remaining workgroup with the decoding capability greater than the capability threshold can be selected as the target workgroup for decoding the current compressed texture block based on the decoding capabilities of the remaining workgroups.
[0029] If each of the M workgroups is assigned a compressed texture block, one target workgroup for decoding can be assigned to the current compressed texture block in the M workgroups. In one embodiment, one workgroup can be randomly selected from the M workgroups as the target workgroup for decoding the current compressed texture block. In another embodiment, the workgroup with the least number of texture blocks can be selected from the M workgroups as the target workgroup for decoding the current compressed texture block based on the number of texture blocks of the compressed texture block assigned to each workgroup. This embodiment can effectively equalize the number of texture blocks in each workgroup, reduce the time waiting for the current compressed texture block to be decoded, and thereby improve the decoding efficiency. In another embodiment, if there is a difference in the decoding capabilities of each workgroup in the video card shader, the workgroup with the highest decoding capability can be selected from the M workgroups, or the workgroup with the decoding capability greater than a capability threshold can be selected as the target workgroup for decoding the current compressed texture block, based on the decoding capabilities of each workgroup.
[0030] The following should be explained: (1) As can be seen from the above description, one or more compressed texture blocks can be assigned to one target workgroup. Here, when the number of workgroups in the video card shader (i.e. M) is equal to or greater than the number of compressed texture blocks in the target compressed texture resource, one compressed texture block can be assigned to one target workgroup. When the number of workgroups in the video card shader (i.e. M) is less than the number of compressed texture blocks in the target compressed texture resource, at least two compressed texture blocks can be assigned to some or all target workgroups. (2) As can be seen from the above description, the embodiment of the present application traverses each compressed texture block in the target compressed texture resource in sequence, thereby assigning a target workgroup to each compressed texture block in sequence. However, in other specific realizations, a batch allocation method can be adopted, and a target workgroup can be assigned to each compressed texture block at the same time, limiting this step.
[0031] For example, suppose the number of compressed texture blocks in the target compressed texture resource is N, and N is an integer greater than 1. When M (the number of work groups in the video card shader) is greater than or equal to N, N target work groups for decoding can be selected from M work groups of the video card shader in a batch, and N target work groups can be assigned to N compressed texture blocks according to the principle that one target work group corresponds to one compressed texture block. When M is less than N, M compressed texture blocks can be selected from the target compressed texture resource in a batch, with all M work groups as target work groups, and M target work groups can be assigned to the selected M compressed texture blocks according to the principle that one target work group corresponds to one compressed texture block. When the number of unselected compressed texture blocks in the N compressed texture blocks (i.e., N to M) is less than M, N to M target work groups can be selected from M work groups, and N to M target work groups can be assigned to the unselected N to M compressed texture blocks according to the principle that one target work group corresponds to one compressed texture block. If N~M is equal to or greater than M, M compressed texture blocks can be selected in batches from the unselected N~M compressed texture blocks, with all M work groups as target work groups, and then the M target work groups can be assigned to the selected M compressed texture blocks according to the principle that one target work group corresponds to one compressed texture block. By this analogy, the process stops until all compressed texture blocks are assigned to a target work group.
[0032] At S203, each compressed texture block is assigned to a corresponding target workgroup.
[0033] In S204, each target work group in the video card shader is called in parallel, and a decoding process is performed on the received compressed texture block based on the compressed texture format to obtain target texture data of the target image, where the target texture data includes decoded data corresponding to each compressed texture block.
[0034] In a possible embodiment, a method in which any target workgroup performs a decoding process on a received compressed texture block is as follows: obtain a shader decode file, and obtain target decoding parameters for a target compressed texture resource, where the shader decode file can include multiple algorithm instructions for realizing decoding compressed texture resources in different compressed texture formats, and the target decoding parameters of the target compressed texture resource can be used to indicate the compressed texture format adopted by the target image; and obtain algorithm instructions for decoding the target compressed texture resource from the shader decode file according to the target decoding parameters, and perform a decoding process on the corresponding compressed texture block using the obtained algorithm instructions to obtain decoded data corresponding to the corresponding compressed texture block.
[0035] For example, the shader decode file may include a first algorithm instruction for realizing the decoding of a compressed texture resource in the ASTC format, a second algorithm instruction for realizing the decoding of a compressed texture resource in the ETC format, etc. When a target decoding parameter of a target compressed texture resource is used to indicate that the compressed texture format adopted by a target image is the ASTC format, a first algorithm instruction can be obtained from a shader file based on the target decoding parameter, and a decoding process can be performed on the corresponding compressed texture block using the first algorithm instruction. When a target decoding parameter of a target compressed texture resource is used to indicate that the compressed texture format adopted by a target image is the ETC format, a second algorithm instruction can be obtained from a shader file based on the target decoding parameter, and a decoding process can be performed on the corresponding compressed texture block using the second algorithm instruction.
[0036] To facilitate the decoding process, the specific processes of the decoding process mentioned above can be packaged into a decoding interface (glDispatchCompute) in one decoding pipeline, so that any target workgroup can directly call the decoding interface to perform decoding processing on the corresponding compressed texture block.
[0037] In a possible embodiment, after the computer device obtains the target texture data, the computer device further uploads the target texture data to the blockchain, so as to prevent the target texture data from being tampered with. Here, the blockchain is composed of a number of blocks, and as shown in FIG. 3a, the genesis block includes a block head and a block body, the block head stores the input information characteristic value, version number, timestamp and difficulty value, and the block body stores the input information. The block next to the genesis block takes the genesis block as the parent block, and the next block similarly includes a block head and a block body, and the block head stores the input information characteristic value of the current block, the block head characteristic value of the parent block, version number, timestamp and difficulty value. By this analogy, the block data stored in each block in the blockchain are all related to the block data stored in the parent block, ensuring the safety of the input information in the block. Based on this, a specific embodiment of uploading the target texture data to the blockchain is as follows:
[0038] When the computer device is located in the block chain network, it can first add the target texture data to the block body of the target block, and perform a hash operation on the target texture data in the block body to obtain a Merkle hash value. Then, a random number is generated using a random algorithm, and the calculated Merkle hash value, the random number, the version number, the previous block hash value, the current timestamp, and the current difficulty value are used to construct the block head of the target block, as shown in FIG. 3b. Here, the version number is the version information of the relevant block protocol in the block chain, the previous block hash value is the characteristic value of the block head of the previous block, the current timestamp is the system time when the block head is constructed, and the current difficulty value is the calculation difficulty value, which is a certain value within a certain time period and is determined again after the certain time period has passed. Then, a characteristic value algorithm (e.g., SHA256 algorithm) can be adopted to perform one or more hash operations on the content included in the block head to obtain the characteristic value of the block head of the target block, and the number of hash operations here can be determined based on the difficulty of the calculation, and the greater the difficulty of the calculation, the greater the number of hash operations. After obtaining the target block according to the above steps, the target block can be broadcast to each consensus node in the blockchain network to carry out a consensus process, and after the consensus process, the target block is added to the blockchain. As can be understood, if the computer device is not located in the blockchain network, the target texture data can be stored in the blockchain by sending the target texture data to any consensus node in the blockchain network and having the above steps carried out by the any consensus node.
[0039] In a possible embodiment, the computer device can further adopt cloud technology to perform cloud storage processing for the target texture data. Here, cloud technology is a custodian technology that integrates a series of resources such as hardware, software, and network in a wide area network or a local area network to realize data calculation, storage, processing, and sharing. Cloud technology can form a resource pool based on the general term of network technology, information technology, integration technology, management platform technology, application technology, etc. applied in the cloud computing business mode, and is applied according to demand, flexibly, and conveniently. Cloud computing technology is an important support. Accordingly, cloud storage is a new concept developed by extending the cloud computing concept, and a distributed cloud storage system (hereinafter referred to as a storage system) is a storage system that integrates a large number of different types of storage devices (storage devices are also called storage nodes) in a network through application software or application interface to cooperate with each other through functions such as cluster application, mesh technology, and distributed storage file system, and provides a common data storage and service access function to the outside.
[0040] The embodiment of the present application introduces a video card shader, and pre-arranges multiple mutually independent work groups in the video card shader, and adopts a compressed texture format to encode a target image to obtain a target compressed texture resource that includes multiple compressed texture blocks. After obtaining the target compressed texture resource of the target image, the video card shader can assign each compressed texture block to one target work group for decoding, and assign each compressed texture block to the corresponding target work group. Then, the video card shader can perform a decoding process on the compressed texture block received according to the compressed texture format by calling each target work group in parallel. This processing method can not only realize the decoding process of the target compressed texture resource, but also realize the parallel decoding of each compressed texture block, thereby effectively improving the decoding efficiency. In addition, by realizing the entire decoding process by the video card shader, the entire decoding process does not require the involvement of the central processing unit, and thus can effectively save the resources of the central processing unit, thereby effectively reducing the resource occupation rate of the central processing unit and improving the operating efficiency of the computer device.
[0041] As shown in Figure 4, it is a flowchart of another image processing method provided by an embodiment of the present application, and the image processing method may be executed in the above-mentioned computer device. The computer device may include a rendering pipeline, a decoding pipeline and a video card shader, and the video card shader operates according to a process defined in the decoding pipeline. As shown in Figure 4, the image processing method may include the following S401 to S407.
[0042] At S401, the rendering pipeline loads a target compressed texture resource of a target image from the local space of the computing device.
[0043] In a possible embodiment, the rendering pipeline can load a target compressed texture resource of a target image from the local space of a computing device by calling a texture resource loading interface (e.g., glTexCompressedImage2D (an interface for loading a compressed texture resource)).
[0044] At S402, the rendering pipeline allocates a target compressed texture resource to the decoding pipeline.
[0045] In one embodiment, the rendering pipeline can directly assign the target compressed texture resource to the decoding pipeline after successfully loading the target compressed texture resource. In another embodiment, a video card decoding mode (or called video card decoding function) can be set in the computer device, and the so-called video card decoding mode is a mode in which the video card shader performs decoding. When the video card decoding mode is in an on state, it indicates that the video card shader supports decoding, and when the video card decoding mode is in an off state, it indicates that the video card shader does not support decoding, and at this time the video card driving central processing unit (abbreviated as video card driving CPU) can decode. In this way, by setting the video card decoding mode, it is possible to realize that the user can turn on or off the video card decoding mode according to his / her needs, so that it is possible to flexibly select whether or not the video card shader needs to perform decoding. Based on this, if the computer device has a video card decoding mode, the rendering pipeline can first detect the mode state of the video card decoding mode before executing S402, and the mode state includes a mode on state or a mode off state. If the mode state is the mode-on state, the rendering pipeline executes the step of allocating the target compressed texture resource to the decoding pipeline, i.e., executes S402. If the mode state is the mode-off state, the rendering pipeline calls the video card driving central processing unit to perform decoding processing on each compressed texture block in the target compressed texture resource, thereby obtaining the target texture data of the target image, i.e., in this case, after obtaining the target texture data by the video card driving central processing unit, the rendering pipeline can directly execute S407, i.e., in this case, there is no need to execute S402 to S406.
[0046] In S403, after the decoding pipeline successfully receives the target compressed texture resource, the decoding pipeline sends a decoding notification to the video card shader to notify the video card shader to perform S404.
[0047] At S404, the video card shader assigns a target work group for decoding to each compressed texture block in the plurality of compressed texture blocks.
[0048] As will be appreciated, one or more compressed texture blocks can be assigned to a destination workgroup.
[0049] At S405, each compressed texture block is assigned to a corresponding destination workgroup by a video card shader.
[0050] In S406, the video card shader calls each target work group in the video card shader in parallel, and performs decoding processing on the corresponding compressed texture blocks to obtain target texture data of the target image, where the target texture data includes decoded data corresponding to each compressed texture block.
[0051] In S407, the rendering pipeline performs image rendering based on the target texture data to obtain a target image.
[0052] As can be seen from the above description, the target texture data may be decoded by the video card shader in S404 to S406, or may be decoded by the rendering pipeline by calling the video card-driven central processing unit. When the target texture data is decoded by the rendering pipeline by calling the video card-driven central processing unit, an embodiment of S407 is as follows, and the rendering pipeline directly calls a rendering interface (e.g., glDraw (an interface for image rendering) to perform image rendering based on the target texture data and obtain a target image. When the target texture data is decoded by the video card shader, the following description can be referred to for a specific embodiment of S407.
[0053] In a possible embodiment, the rendering pipeline may have a first physical texture video memory space, and the decoding pipeline may have a second physical texture video memory space, where the first texture video memory space is a video memory space for storing data to be processed by the rendering pipeline or data processed by the rendering pipeline, and the second texture video memory space is a video memory space for storing data to be processed by the decoding pipeline or data processed by the decoding pipeline, and the so-called video memory space is a storage space located on the video card. In this case, after any target work group in the video card shader performs a decoding process on the received compressed texture block, the decoded data can be cached in the second texture video memory space, and after all the decoded data corresponding to each compressed texture block is cached in the second texture video memory space, i.e., after the target texture data of the target image is cached in the second texture video memory space, the decoding pipeline can copy the target texture data of the target image from the second texture video memory space to the first texture video memory space by a central processing unit (CPU). Or, after the target texture data of the target image is cached in the second texture video memory space, the decoding pipeline can notify the rendering pipeline, and the rendering pipeline copies the target texture data of the target image from the second texture video memory space to the first texture video memory space in the CPU. In this case, the embodiment of S407 is as follows: the rendering pipeline performs image rendering according to the target texture data in the first texture video memory space to obtain the target image.
[0054] Practice shows that copying target texture data by CPU may cause a performance loss of 1-5 ms. In order to fully utilize the advantages of the decoding pipeline and reduce the performance loss, the embodiment of the present application proposes a sharing method for the decoding pipeline and the rendering pipeline, in which the physical space is the same but the virtual space is different, so that the rendering pipeline uses the same physical texture memory space to store compressed texture resources and other data, and the decoding pipeline uses the texture memory space to cache the target texture data obtained by the decoding process by the video card shader, thus avoiding the performance loss caused by the CPU copying the target texture data. Based on this, the embodiment of the present application proposes another implementation method, and the specific description is as follows:
[0055] In another implementation, the rendering pipeline can have a virtual first video card address space, and the decryption pipeline can have a virtual second video card address space, where the first video card address space and the second video card address space have a mapping relationship with the same texture storage space, and the texture storage space is essentially a physical storage space (i.e., a storage space that actually exists). In this case, after any target work group in the video card shader performs a decoding process on the corresponding compressed texture block, the decryption data decrypted by any target work group can be cached in the texture storage space based on the mapping relationship between the second video card address space and the texture storage space. In this case, the embodiment of S407 is as follows: after all the decryption data corresponding to each compressed texture block is stored in the texture storage space, the rendering pipeline performs image rendering based on the target texture data in the texture storage space based on the mapping relationship between the first video card address space and the texture storage space to obtain the target image. Preferably, in order for the rendering pipeline to perform image rendering in a timely manner, after the video card shader stores all the decoded data corresponding to each compressed texture block in the texture memory space, the decoding pipeline can send a decoding completion notification to the rendering pipeline, thereby notifying the rendering pipeline to perform image rendering based on the texture data in the texture memory space.
[0056] Here, the specific manner of mapping the first video card address space and the second video card address space to the same physical texture storage space is as follows: the rendering pipeline allocates texture storage space to the target image in the local space of the computer device, the rendering pipeline maps the first video card address space to the texture storage space, and transmits the spatial identification of the texture storage space (such as the spatial address, spatial serial number, etc. of the texture storage space) to the decoding pipeline, and the decoding pipeline maps the second video card address space to the texture storage space according to the spatial identification. It should be noted that the timing at which the embodiment of the present application performs the spatial mapping operation on the rendering pipeline and the decoding pipeline is not limited. For example, the rendering pipeline may perform the texture storage space allocation and spatial mapping operation in the process of loading the target compressed texture resource of the target image, and may perform the texture storage space allocation and mapping operation before or after loading the target texture resource. For example, the decoding pipeline may perform the spatial mapping operation immediately after receiving the spatial identification transmitted from the rendering pipeline, or may perform the spatial mapping operation after receiving the assigned target compressed texture resource by the rendering pipeline; further, the spatial identification and the target compressed texture resource may be sent to the decoding pipeline by the rendering pipeline simultaneously, or may be sent to the decoding pipeline respectively, which is not limited thereto.
[0057] The specific manner of mapping the first video card address space and the second video card address space to the same physical texture storage space will be described in more detail below with reference to the schematic diagram of spatial mapping shown in Figure 5. As shown in Figure 5, in the process of loading the target compressed texture resource of the target image through glTexCompressedImage2D (interface for loading compressed texture resource), the rendering pipeline allocates texture storage space (Physical Pages), maps the texture storage space to the first video card address space of the rendering pipeline, and sends a spatial identification of the texture storage space to the decoding pipeline to trigger the decoding pipeline to decode the target compressed texture resource to the texture storage space. The decoding pipeline receives the spatial identification sent by the rendering pipeline, and can map the texture storage space indicated by the spatial identification to the second video card address space of the decoding pipeline through astc_cs_decode (interface for preparing for decoding).
[0058] In a possible embodiment, if all or some of the compressed texture blocks in the target compressed texture resource fail to be decoded by the video card shader, the video card driving central processing unit (i.e., the video card driving CPU) can be called to perform a decoding process on each compressed texture block in the target compressed texture resource to obtain the target texture data of the target image.
[0059] The embodiment of the present application realizes the entire decoding process by the video card shader, so that the entire decoding process does not require the involvement of the central processing unit, and thus can effectively save the resources of the central processing unit, thereby effectively reducing the resource occupation rate of the central processing unit and improving the operation efficiency of the computer device. In addition, the method of calling each target work group in the video card shader in parallel can effectively improve the decoding efficiency by performing the decoding process on the corresponding compressed texture block. In addition, the method of sharing the same physical texture memory space by the decoding pipeline and the rendering pipeline can effectively reduce the performance loss caused by the CPU copying data, and can improve the immediacy of image rendering and the processing efficiency of the entire image processing process.
[0060] In the method embodiment shown in FIG. 4 above, when the target image is a target game screen image in a target game, the target game will usually have a game resource packet, which is used to store various data resources required in the process of running the target game. Therefore, the game developer can adopt a compressed texture format to encode the game screen image of each frame in the target game, and after obtaining the compressed texture resource of the game screen image of each frame, perform offline decoding in advance on the compressed texture resource of the game screen of one frame or multiple frames in the target game, and use each decoded texture data and image identification of the corresponding game screen image to construct an offline decoding dataset of the target game, and add the offline decoding dataset to the game resource packet of the target game.
[0061] In this case, the rendering pipeline can obtain the offline decoded data set of the target game before executing S402. In a possible implementation, the offline decoded data set of the target game can be obtained from a game resource packet of the target game, the offline decoded data set includes image identifications of game screen images of one or more frames in the target game, and texture data associated with each image identification, and the texture data associated with any one of the image identifications is obtained by offline decoding the compressed texture resource of the game screen image indicated by any one of the image identifications. Then, the rendering pipeline uses the image identification of the target image to perform an identification hit process on the image identification in the offline decoded data set, thereby detecting whether the offline decoded data set includes the target texture data of the target game screen image. If the hit is not successful, the rendering pipeline executes S402. If the hit is successful, the rendering pipeline loads the texture data associated with the hit image identification from the offline decoded data set, and the loaded texture data is used as the target texture data of the target image, thus effectively shortening the acquisition time of the target texture data and improving the decoding efficiency.
[0062] In a possible embodiment, as can be seen from the relevant description of S402 above, the rendering pipeline can first perform the step of detecting the mode state of the video card decoding mode before performing S402. In this case, the rendering pipeline can perform the operation of identification hit processing before or after detecting the mode state of the video card decoding mode. Taking the operation of performing identification hit processing before detecting the mode state of the video card decoding mode as an example, the processing logic of the rendering pipeline is as follows: the rendering pipeline uses the image identification of the target image to perform identification hit processing on the image identification in the offline decoding data set; if it is not successful, the rendering pipeline performs the step of detecting the mode state of the video card decoding mode; if it is successful, the rendering pipeline loads the texture data associated with the hit image identification from the offline decoding data set, and the loaded texture data is the target texture data of the target image.
[0063] As can be seen from the above description, in the image processing process, there may be a situation where the rendering pipeline calls the video card driven central processing unit to perform a decoding process on each compressed texture block in the target compressed texture resource. For example, after the rendering pipeline detects that the mode state of the video card decoding mode is in the off state, it can call the video card driven central processing unit to perform a decoding process on each compressed texture block in the target compressed texture resource. Also, for example, if all or some compressed texture blocks in the target compressed texture resource fail to be decoded by the video card shader, the rendering pipeline can call the video card driven central processing unit to perform a decoding process on each compressed texture block in the target compressed texture resource. Furthermore, considering that the video card driven central processing unit may have a decoding ability to decode the target compressed texture resource, or may not have the decoding ability, if the video card driven central processing unit does not have the decoding ability, calling the video card driven central processing unit to perform a decoding process on the target compressed texture resource will result in a failure in decoding the target compressed texture resource. Therefore, if the video card driven central processing unit does not have the decoding ability, the rendering pipeline can directly call the game engine central processing unit (i.e., the game engine CPU) of the target game to decode the target compressed texture resource. This processing method can ensure that the target compressed texture resource is successfully decoded, thereby ensuring that the target image is successfully rendered and displayed, while avoiding the problem of the video card driving central processing unit wasting processing resources by performing unnecessary decoding processes.
[0064] Based on this, before the rendering pipeline calls the video card driven central processing unit to perform a decoding process on each compressed texture block in the target compressed texture resource, the rendering pipeline can further detect the decoding capability of the video card driven central processing unit (i.e., the video card driven CPU) for the target compressed texture resource. If it is detected that the video card driven central processing unit has the decoding capability, the rendering pipeline executes the step of calling the video card driven central processing unit to perform a decoding process on each compressed texture block in the target compressed texture resource. If it is detected that the video card driven central processing unit does not have the decoding capability, the rendering pipeline executes the step of calling the game engine central processing unit of the target game to perform a decoding process on each compressed texture block in the target compressed texture resource, thereby obtaining the target texture data of the target image. For illustrative purposes, the embodiment of the present application does not limit the specific time point at which the rendering pipeline executes the detection of the decoding capability of the video card driven central processing unit for the target compressed texture resource. For example, in another embodiment, the rendering pipeline may execute the step of decoding capability detection before obtaining the offline decoding data set of the target game. In this case, the processing logic of the rendering pipeline is as follows: detect the decoding capability of the video card driven central processing unit (i.e., the video card driven CPU) for the target compressed texture resource; if it is detected that the video card driven central processing unit has the decoding capability, the rendering pipeline executes the step of obtaining an offline decoded data set of the target game; if it is detected that the video card driven central processing unit does not have the decoding capability, the rendering pipeline calls the game engine central processing unit of the target game to perform decoding processing on each compressed texture block in the target compressed texture resource, thereby obtaining target texture data of the target image.
[0065] It should be noted that the target game mentioned above may be a normal game or a cloud game. The so-called normal game is a game that is directly operated on a game client installed in a terminal device used by a player user. The so-called cloud game may be called gaming on demand, which is a gaming method based on cloud computing, where cloud computing is an internet-based computing method. In a cloud game scene, the game is operated on a cloud game server, not on a game client of a player user. The cloud game server encodes a game screen related to the game scene into a video stream, and transmits the video stream to the game client of the player user via a network for playback. Here, when the target game is a normal game, the computer device mentioned above is a terminal device used by a player user, and when the target game is a cloud game, the computer device mentioned above is a cloud game server that operates the cloud game.
[0066] As shown in FIG. 6a, the cloud game server may include a device hardware layer, a virtualization layer (Hypervisor), and one or more containers. Here, the device hardware layer may include, but is not limited to, a CPU (Central Processing Unit), a video card, a network communication module, and a local space for storage, and the video card may include a video card shader, a decoding pipeline, and a rendering pipeline. The virtualization layer is mainly used in a software manner to realize a virtual environment that is completely similar to a set of physical host environments, and the cloud game server may deploy and run one or more containers through the virtualization layer, and the so-called container is a type of virtualization at the operating system level, which is used to bear the operating system. Each container is connected to one or more game clients, and each container is used to run one or more cloud games, and each container may transmit and display the game screen of any cloud game to the connected game client in the process of running any cloud game.
[0067] Based on the above description, the embodiment of the present application further puts forward the image processing method shown in Fig. 6b. In the embodiment of the present application, the target game is mainly cloud game, that is, the computer device is a cloud game server, that is, the image processing method shown in Fig. 6b can be executed in the cloud game server, specifically, in the video card in the cloud game server. As shown in Fig. 6b, the outline process of the image processing method is as follows:
[0068] First, the decoding pipeline can be initialized (created), for example, in the initialization process of the target game, an independent thread can be created and the decoding pipeline can be initialized by the independent thread. Then, the shader decoding file can be loaded, and the video card shader creates a decoding manager, which is used to wait and receive the target compressed texture resource of the target image.
[0069] After the rendering pipeline is loaded with the target compressed texture resource, it can detect whether the video card driving CPU has the decoding capability. If not, the rendering pipeline calls the game engine CPU to perform the decoding process on the target compressed texture resource, thereby obtaining the target texture data of the target image. If yes, the rendering pipeline can further detect whether the image identification of the target image hits the offline decoding dataset, thereby detecting whether the offline decoding dataset contains the target texture data of the target game screen image. If the hit is successful, the rendering pipeline loads the target texture data of the target image from the offline decoding dataset; if the hit is not successful, the rendering pipeline can detect whether the video card shader supports the decoding process by detecting the mode state of the video card decoding mode. If it is detected that the video card shader does not support the decoding process, the rendering pipeline calls the video card driving CPU to perform the decoding process on the target compressed texture resource, thereby obtaining the target texture data of the target image; if it is detected that the video card shader does not support the decoding process, the rendering pipeline assigns the target compressed texture resource of the target image to the decoding pipeline.
[0070] Correspondingly, after receiving the target compressed texture resource allocated by the rendering pipeline, the decoding pipeline can send the target compressed texture resource to the video card shader by the decoding manager, and notify the video card shader to perform decoding processing on the target compressed texture resource. For example, the video card shader can allocate a target work group for decoding to each compressed texture block in the target compressed texture resource, and further assign each compressed texture block to the corresponding target work group, and then call each target work group in the video card shader in parallel to perform decoding processing on the received compressed texture block, thereby obtaining the target texture data of the target image. The decoding pipeline can detect in real time or periodically whether the video card shader has successfully decoded, and if the decoding is successful, notify the rendering pipeline to perform image rendering based on the target texture data of the target image, and if the decoding is unsuccessful, notify the rendering pipeline to perform decoding processing on the target compressed texture resource by the video card driving CPU, and after obtaining the target texture data by decoding, perform image rendering based on the target texture data of the target image.
[0071] With respect to the relevant description of the method embodiments shown in the above Figs. 2, 4 and 6b, the following two points should be mentioned.
[0072] 1) The embodiment of the present application may adopt a method of being individually packaged in an SDK (Software Development Kit, software development tool packet), and packaging processing is performed for the decoding process performed by the video card shader mentioned in the method embodiment shown in the above figures 2, 4 and 6b. That is, the decoding process performed by the video card shader is individually packaged in one software tool packet (named cs decoder library), so that when it is necessary to decode the compressed texture resource of any image, the software tool packet is called to perform the decoding process. Here, the principle diagram of the calling of the software tool coding is as shown in figure 7. After loading the target game, the sys vulkan library (system vulkan rendering interface) is first called, then the sys vulkan library is used to call the GPU vulkan library (GPU vulkan rendering interface), and the GPU vulkan library is used to call the cs decoder library, thereby realizing the decoding process. Alternatively, the decoding process can be achieved by first calling the sys EGL / GELS libraries (system EGL / GELS rendering interfaces), then calling the GPU EGL / GELS libraries (GPU EGL / GELS rendering interfaces) using the sys EGL / GELS libraries, and finally calling the cs decoder library using the GPU EGL / GELS libraries.
[0073] Here, the above-mentioned Vulkan is a cross-platform 2D and 3D drawing application interface, GELS is an abbreviation of OpenGL ES (OpenGL for Embedded Systems), which is a subset of the three-dimensional graphics API of OpenGL (a cross-language, cross-platform application programming interface for rendering 2D and 3D vector graphics), and EGL is an intermediate interface layer between the OpenGL ES rendering interface and the local window system (native platform window system). The method of packaging the SDK separately can have the following beneficial effects: (1) Avoid introducing potential rendering risks due to over-correction of video card driving. (2) Be independent of the rendering pipeline and avoid destruction of the game's rendering state machine. (3) Support the openGLES rendering interface and the Vulkan rendering interface while driving the video card in the video card shader, avoiding duplicate construction.
[0074] 2) As can be seen from the above, when the target image is a target game screen image in a target game, the embodiment of the present application can provide the following several decoding methods for the target compressed resource of the target image: (1) decoding by game engine CPU, (2) decoding by video card driving CPU, (3) decoding by video card shader, (4) decoding by offline decoding data set, where the essence of decoding method (1) and decoding method (2) is both decoding by CPU. The embodiment of the present application adopts compressed texture resources of different sizes in ASTC format, and tests the decoding time length required for three kinds of decoding methods, namely decoding by CPU, decoding by offline decoding data set, and decoding by video card shader, and the test results are shown in Table 1 below.
[0075] [Table 1]
[0076] Here, "ms" in Table 1 indicates milliseconds. As can be seen from Table 1, the decoding method using the offline decoding dataset requires the shortest decoding time and has the optimal decoding performance. The decoding time required by the decoding method using the video card shader is much shorter than the decoding time required by the decoding method using the CPU. In other words, the decoding performance of the decoding method using the video card shader is superior to the decoding performance of the decoding method using the CPU.
[0077] The embodiment of the present application utilizes the powerful floating-point arithmetic and parallel computing ability of the video card in the cloud game server to realize the real-time rapid decoding of the target compressed texture resource in the compressed texture format in the target game, thereby realizing the rapid rendering and display of the target image. In this way, the smoothness of the game is effectively improved, the occupation rate of CPU resources is reduced, and the problem of game lag and low frame rate caused by the CPU decoding being too slow, and the problem of the cloud game server's operation cost of limited game parallel circuit number caused by the CPU resource occupied by a single-pass game being too large, are solved. In addition, by performing the entire decoding process in the cloud game server, the entire decoding process is not limited by the terminal type and operating system, and can be fully enabled, and the embodiment of the present application supports the decoding process of the target compressed texture resource by the video card shader, so that the game developer does not need to make special modifications and adaptations to the implementation logic of the cloud game to create an offline decoding data set that meets the decoding requirements of the cloud game server, and has high applicability and compatibility, and at the same time, the problem of the storage memory of the game resource package of the cloud game being too large due to the offline decoding of the compressed texture resource of each game screen image can be solved.
[0078] Based on the description of the embodiment of the image processing method above, the embodiment of the present application further discloses an image processing device, and the image processing device may be a computer program (including program code) operated by a computer device. The image processing device can execute the method shown in FIG. 2, FIG. 4 or FIG. 6b. As shown in FIG. 8, the image processing device can operate an acquisition unit 801 used to acquire a target compressed texture resource of a target image, the target compressed texture resource being obtained by adopting a compressed texture format to encode the target image, the target compressed texture resource including a plurality of compressed texture blocks, and a processing unit 802 used to assign a plurality of target work groups for decoding to the plurality of compressed texture blocks in a video card shader, and assign each compressed texture block to a corresponding target work group, the processing unit 802 is further used to acquire target texture data of the target image by calling each target work group in the video card shader in parallel, and performing a decoding process on the received compressed texture block according to the compressed texture format, the target texture data including decoded data corresponding to each compressed texture block.
[0079] In one embodiment, a workgroup is a sub-processing unit having decoding processing capability in the video card shader, each workgroup is independent of each other, and any one of the target workgroups adopts an algorithm instruction for decoding the target compressed texture resource to perform decoding processing on the received compressed texture block.
[0080] In one embodiment, the video card shader is installed in a computing device, which further includes a rendering pipeline and a decoding pipeline, and the video card shader operates according to a process defined in the decoding pipeline, and correspondingly, the processing unit 802 is further used to perform image rendering through the rendering pipeline based on the target texture data and obtain the target image.
[0081] In another embodiment, the rendering pipeline has a virtual first video card address space and the decoding pipeline has a virtual second video card address space, and the first video card address space and the second video card address space have a mapping relationship with the same texture memory space.
[0082] In another embodiment, the processing unit 802 is further used to perform a decoding process on a compressed texture block received by any one of the target workgroups in the video card shader, and then cache the decoded data obtained by decoding the any one of the target workgroups in the texture memory space based on the mapping relationship between the second video card address space and the texture memory space.
[0083] Correspondingly, when the processing unit 802 is used in a rendering pipeline to perform image rendering based on the target texture data and obtain the target image, specifically, after all of the decoded data corresponding to each compressed texture block is stored in the texture memory space, the processing unit 802 is used in the rendering pipeline to perform image rendering based on the target texture data in the texture memory space and obtain the target image based on the mapping relationship between the first video card address space and the texture memory space.
[0084] In another embodiment, the processing unit 802 is further used for allocating the texture memory space to the target image in a local space of the computing device by the rendering pipeline, mapping the first video card address space to the texture memory space by the rendering pipeline and transmitting a spatial identification of the texture memory space to the decoding pipeline, and mapping the second video card address space to the texture memory space by the decoding pipeline based on the spatial identification.
[0085] In another embodiment, when the acquisition unit 801 is used to acquire a target compressed texture resource of a target image, it is specifically used for the rendering pipeline to load the target compressed texture resource of the target image from the local space of the computer device, and correspondingly, the processing unit 802 is further used for the rendering pipeline to allocate the target compressed texture resource to the decoding pipeline, and after the decoding pipeline successfully receives the target compressed texture resource, the decoding pipeline sends a decoding notification to the video card shader to notify the video card shader to perform the step of allocating multiple target work groups for decoding to the multiple compressed texture blocks in the video card shader.
[0086] In another embodiment, the computer device has a video card decoding mode, which is a mode in which decoding processing is performed by the video card shader, and correspondingly, the processing unit 802 further includes: the rendering pipeline detects a mode state of the video card decoding mode, the mode state including a mode on state or a mode off state, and if the mode state is the mode on state, the rendering pipeline performs a step of allocating the target compressed texture resource to the decoding pipeline, and if the mode state is the mode off state, the rendering pipeline is used to obtain target texture data of the target image by calling a video card driving central processing unit to perform decoding processing on each compressed texture block in the target compressed texture resource.
[0087] In another embodiment, the target image is a target game screen image in a target game, and correspondingly, the processing unit 802 further obtains an offline decoded dataset of the target game by the rendering pipeline, the offline decoded dataset including image identifications of game screen images of one frame or multiple frames in the target game and texture data associated with each image identification, the texture data associated with any one of the image identifications being obtained by offline decoding a compressed texture resource of a game screen image indicated by any one of the image identifications, and based on the image identification of the target image, the rendering pipeline performs an identification hit process on the image identification in the offline decoded dataset, and if the hit is not successful, the rendering pipeline performs a step of detecting a mode state of the video card decoding mode, and if the hit is successful, the rendering pipeline loads texture data associated with the hit image identification from the offline decoded dataset, and the loaded texture data is used to set the target texture data of the target image.
[0088] In one embodiment, the step of assigning multiple target work groups for decoding to the multiple compressed texture blocks in the video card shader is performed when an image identification of the target image is not present in the offline decoding data set, and when an image identification of the target image is present in the offline decoding data set, target texture data of the target image has been loaded from the offline decoding data set.
[0089] In another embodiment, the processing unit 802 is further used to cause the rendering pipeline to detect a decoding capability of the video card driven central processing unit for the target compressed texture resource, and if it is detected that the video card driven central processing unit has the decoding capability, to perform a step of obtaining an offline decoded data set of the target game in the rendering pipeline, and if it is detected that the video card driven central processing unit does not have the decoding capability, to obtain target texture data of the target image by calling a game engine central processing unit of the target game to perform a decoding process on each compressed texture block in the target compressed texture resource by the rendering pipeline.
[0090] In another embodiment, the processing unit 802 is further used to obtain target texture data of the target image by calling a video card driving central processing unit to perform a decoding process on each compressed texture block in the target compressed texture resource when all or some of the compressed texture blocks in the target compressed texture resource fail to be decoded by the video card shader.
[0091] In another embodiment, when the video card shader includes M work groups, M is an integer greater than 1, and the processing unit 802 is used to assign multiple target work groups for decoding to the multiple compressed texture blocks in the video card shader, specifically, traverse the multiple compressed texture blocks to determine a current compressed texture block currently being traversed, and if there are any remaining unassigned work groups among the M work groups in the video card shader, assign one target work group for decoding to the current compressed texture block in the remaining work group, and if all of the M work groups have compressed texture blocks assigned to them, the processing unit 802 is used to assign one target work group for decoding to the current compressed texture block in the M work groups.
[0092] In another embodiment, the decoding process performed by the video card shader is packaged separately in a single software tool packet, so that when any image's compressed texture resources need to be decoded, the software tool packet is called to perform the decoding process.
[0093] According to another embodiment of the present application, each or all of the units in the image processing device as shown in FIG. 8 may be integrated into one or several other units, or some (several) of the units may be further divided into several units with smaller functions, which can achieve the same operation and do not affect the realization of the technical effect of the embodiment of the present application. The above units are divided based on logic functions, and in actual applications, the function of one unit may be realized by several units, or the function of several units may be realized by one unit. In another embodiment of the present application, based on the fact that the image processing device can include other units, in actual applications, these functions may be realized by other units in cooperation, and may be realized by several units in cooperation.
[0094] According to another embodiment of the present application, an image processing method according to the embodiment of the present application can be realized by running a computer program (including program code) capable of executing each step of the corresponding method as shown in Fig. 2 on a general-purpose computing device, such as a computer, including a processing element and a storage element, such as a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), etc. The image processing device device as shown in Fig. 8 can be constructed and the image processing method according to the embodiment of the present application can be realized. The computer program may be recorded in a computer-readable recording medium, for example, and is loaded into the computing device by the computer-readable recording medium and operated therein.
[0095] The embodiment of the present application introduces a video card shader, in which multiple independent work groups are pre-arranged in the video card shader, and the target compressed texture resource obtained by adopting a compressed texture format to encode a target image includes multiple compressed texture blocks. After obtaining the target compressed texture resource of the target image, the video card shader can assign each compressed texture block to one target work group for decoding, and assign each compressed texture block to the corresponding target work group. Then, the video card shader can perform decoding processing on the compressed texture block received according to the compressed texture format by calling each target work group in parallel. This processing method can not only realize the decoding processing of the target compressed texture resource, but also realize the parallel decoding of each compressed texture block, thereby effectively improving the decoding efficiency. In addition, by realizing the entire decoding process by the video card shader, the entire decoding process does not require the involvement of the central processing unit, thus effectively saving the resources of the central processing unit, thereby effectively reducing the resource occupation rate of the central processing unit and improving the operating efficiency of the computer device.
[0096] Based on the description of the above method embodiment and device embodiment, the embodiment of the present application further provides a computer device. As shown in Fig. 9, the computer device includes at least a video card 901, an input interface 902, an output interface 903 and a computer storage medium 904, and the video card 901, the input interface 902, the output interface 903 and the computer storage medium 904 in the computer device can be connected by a bus or other manner. Here, the video card 901 can further include a graphic processor 9011, a video card shader 9012, a decoding pipeline 9013 and a rendering pipeline 9014. The computer storage medium 904 may be stored in the memory of the computer device, the computer storage medium 904 is used to store a computer program, the computer program includes program instructions, and the video card 901 is used to execute the program instructions stored in the computer storage medium 904. In addition, the computer device further includes a central processing unit 905 (CPU), which is the calculation core and control core of the computer device, and is suitable for implementing one or more instructions, specifically for implementing a process of a corresponding method or a corresponding function by loading and executing one or more instructions.
[0097] In one embodiment, the video card 901 described in the embodiment of the present application is used to perform a series of image processing, specifically including: obtaining a target compressed texture resource of a target image, the target compressed texture resource being obtained by adopting a compressed texture format to encode the target image, the target compressed texture resource including a plurality of compressed texture blocks; allocating a plurality of target work groups for decoding to the plurality of compressed texture blocks in a video card shader, and further allocating each compressed texture block to a corresponding target work group; invoking each target work group in the video card shader in parallel, and performing a decoding process on the received compressed texture blocks based on the compressed texture format, thereby obtaining target texture data of the target image, and the target texture data including decoded data corresponding to each compressed texture block.
[0098] The embodiment of the present application further provides a computer storage medium (Memory), which is a storage device in a computer device and is used to store programs and data. As can be understood, the computer storage medium here can include a built-in storage medium in the computer device, and of course can also include an extended storage medium supported by the computer device. The computer storage medium provides a storage space, which stores an operating system of the computer device. And the storage space further stores one or more instructions suitable for being loaded and executed by the video card 901, which may be one or more computer programs (including program code). For illustrative purposes, the computer storage medium here may be a high-speed RAM memory, a non-volatile memory, such as at least one magnetic disk memory, and preferably at least one computer storage medium remote from the processor.
[0099] In one embodiment, one or more instructions stored in a computer storage medium can be loaded and executed by the video card 901, thereby implementing corresponding steps of the image processing method embodiments as shown in Figure 2, Figure 4 or Figure 6b above. In a specific implementation, the one or more instructions in the computer storage medium are loaded by the video card 901, and the image processing method provided by the embodiment of the present application is executed.
[0100] Illustratively, according to one aspect of the present application, there is further provided a computer program product or computer program, the computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer readable storage medium, a processor of a computer device reading the computer instructions from the computer readable storage medium, and the processor executing the computer instructions, so that the computer device performs the method provided from various selectable manners of the embodiment of the image processing method as shown in Fig. 2, Fig. 4 or Fig. 6b above.
[0101] And it should be understood that the above disclosure is merely a preferred embodiment of the present application, and of course, it cannot be used to limit the scope of the claims of the present application, and therefore, equivalent modifications made according to the scope of the claims of the present application still belong to the scope of the present application.
Claims
1. 1. An image processing method executed by a computer device, comprising: obtaining a target compressed texture resource of a target image, the target compressed texture resource being obtained by encoding the target image using a compressed texture format, the target compressed texture resource including a plurality of compressed texture blocks; assigning a plurality of destination work groups for decoding to the plurality of compressed texture blocks in a video card shader, and assigning each compressed texture block to a corresponding destination work group; and invoking each target work group in the video card shader in parallel to perform a decoding process on the received compressed texture blocks based on the compressed texture format to obtain target texture data of the target image, the target texture data including decoded data corresponding to each of the compressed texture blocks; The video card shader is installed on the computer device, the computer device further includes a rendering pipeline and a decoding pipeline, the video card shader operates according to a process defined in the decoding pipeline; The method further includes the step of: obtaining the target image by performing image rendering by the rendering pipeline based on the target texture data; the rendering pipeline having a virtual first video card address space and the decoding pipeline having a virtual second video card address space; An image processing method, wherein the first video card address space and the second video card address space have a mapping relationship with the same texture storage space.
2. The method according to claim 1, characterized in that a workgroup is a sub-processing unit having a decoding processing capability in the video card shader, each workgroup is independent of each other, and any one of the target workgroups adopts an algorithm instruction for decoding the target compressed texture resource to perform decoding processing on the received compressed texture block.
3. The method further comprises: After performing a decoding process on the compressed texture block received by any one of the target workgroups in the video card shader, the decoding data obtained by decoding the any one of the target workgroups is cached in the texture memory space according to a mapping relationship between the second video card address space and the texture memory space; The step of obtaining the target image by the rendering pipeline performing image rendering based on the target texture data includes:
2. The method of claim 1, further comprising: after all of the decoded data corresponding to each compressed texture block is stored in the texture memory space, the rendering pipeline performs image rendering based on the target texture data in the texture memory space based on a mapping relationship between the first video card address space and the texture memory space to obtain the target image.
4. The method further comprises: the rendering pipeline allocating the texture storage space for the target image in a local space of the computing device; the rendering pipeline mapping the first video card address space to the texture storage space and transmitting a spatial identification of the texture storage space to the decoding pipeline; 2. The method of claim 1, further comprising the step of: said decoding pipeline mapping said second video card address space to said texture storage space based on said spatial identification.
5. The step of obtaining a target compressed texture resource of the target image includes the rendering pipeline loading a target compressed texture resource of the target image from a local space of the computing device; The method further comprises: the rendering pipeline allocating the target compressed texture resource to the decoding pipeline; 2. The method of claim 1, further comprising: after the decoding pipeline successfully receives the target compressed texture resource, the decoding pipeline notifies the video card shader to perform the step of allocating a plurality of target work groups for decoding to the plurality of compressed texture blocks in the video card shader by sending a decode notification to the video card shader.
6. the computer device has a video card decoding mode, the video card decoding mode being a mode in which decoding is performed by the video card shader; The method further comprises: the rendering pipeline detects a mode state of the video card decoding mode, the mode state including a mode on state or a mode off state; If the mode state is the mode on state, the rendering pipeline allocates the target compressed texture resource to the decoding pipeline; 6. The method of claim 5, further comprising, if the mode state is the mode off state, the rendering pipeline calling a video card driving central processing unit to perform a decoding process on each compressed texture block in the target compressed texture resource to obtain target texture data for the target image.
7. the target image is a target game screen image in a target game, The method further comprises: the rendering pipeline obtaining an offline decoded dataset of the target game, the offline decoded dataset including image identifications of game screen images of one or more frames of the target game and texture data associated with each image identification, the texture data associated with any one of the image identifications being obtained by offline decoding a compressed texture resource of a game screen image indicated by any one of the image identifications; 7. The method of claim 6, further comprising: based on an image identification of the target image, the rendering pipeline performs an identification hit process on the image identification in the offline decoded data set; if the hit is not successful, the rendering pipeline performs a step of detecting a mode state of the video card decoding mode; if the hit is successful, the rendering pipeline loads texture data associated with the hit image identification from the offline decoded data set, and sets the loaded texture data as target texture data for the target image.
8. 8. The method of claim 7, wherein the step of allocating multiple target work groups for decoding to the multiple compressed texture blocks in the video card shader is performed if an image identification of the target image is not present in the offline decoding data set, and if an image identification of the target image is present in the offline decoding data set, target texture data for the target image was loaded from the offline decoding data set.
9. The method further comprises:
8. The method of claim 7, further comprising: detecting a decoding capability of the video card driven central processing unit for the target compressed texture resource, and if it is detected that the video card driven central processing unit has the decoding capability, performing a step of obtaining an offline decoded data set of the target game in the rendering pipeline; and if it is detected that the video card driven central processing unit does not have the decoding capability, performing a step of obtaining target texture data of the target image in the rendering pipeline by calling a game engine central processing unit of the target game to perform a decoding process on each compressed texture block in the target compressed texture resource.
10. The method further comprises:
2. The method of claim 1, further comprising the step of, when the video card shader fails to decode all or some of the compressed texture blocks in the target compressed texture resource, calling a video card driving central processing unit to perform a decoding process on each compressed texture block in the target compressed texture resource to obtain target texture data for the target image.
11. the video card shader includes M work groups, where M is an integer greater than 1; The step of allocating a plurality of target work groups for decoding to the plurality of compressed texture blocks in the video card shader comprises:
2. The method of claim 1, further comprising: traversing the plurality of compressed texture blocks to determine a current compressed texture block currently being traversed; if there are any remaining unassigned workgroups among M workgroups in the video card shader, assigning one target workgroup for decoding to the current compressed texture block in the remaining workgroup; and if all of the workgroups among the M workgroups have compressed texture blocks assigned to them, assigning one target workgroup for decoding to the current compressed texture block in the M workgroups.
12. The method of claim 1, characterized in that the decoding process performed by the video card shader is individually packaged in one software tool packet, so that when a compressed texture resource of any one of the images needs to be decoded, the decoding process is performed by calling the software tool packet.
13. An image processing device disposed in a computing device, an acquisition unit used to acquire a target compressed texture resource of a target image, the target compressed texture resource being obtained by encoding the target image using a compressed texture format, the target compressed texture resource including a plurality of compressed texture blocks; a processing unit adapted to assign a plurality of destination workgroups for decoding to the plurality of compressed texture blocks in a video card shader and to assign each compressed texture block to a corresponding destination workgroup; The processing unit is further used to obtain target texture data of the target image by invoking each target work group in the video card shader in parallel and performing a decoding process on the received compressed texture blocks according to the compressed texture format, the target texture data including decoded data corresponding to each of the compressed texture blocks; The video card shader is installed on the computer device, the computer device further includes a rendering pipeline and a decoding pipeline, the video card shader operates according to a process defined in the decoding pipeline; the rendering pipeline performs image rendering based on the target texture data to obtain the target image; the rendering pipeline having a virtual first video card address space and the decoding pipeline having a virtual second video card address space; 13. An image processing device according to claim 12, wherein said first video card address space and said second video card address space have a mapping relationship with the same texture storage space.
14. A computer apparatus comprising an input interface and an output interface, further comprising a video card and a computer storage medium, the video card comprising at least a video card shader, one or more instructions stored on the computer storage medium, and the one or more instructions being loaded by the video card, the computer apparatus being suitable for performing an image processing method according to any one of claims 1 to 12.
15. A computer program which, when executed, is used to carry out the image processing method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Real time on-chip texture decompression using shader processors
US20120320067A1
