Image colorization processing methods and related equipment

Abstract

This application provides an image colorization processing method and related apparatus. The method includes: retrieving an image to be processed from a central processing unit (CPU) via an open graphics library; performing graphical processing on the image to obtain a graphical processing result; retrieving parameter data of a deep learning model from the CPU via the open graphics library; configuring the deep learning model using the parameter data; and retrieving the graphical processing result from the open graphics library and applying it to the configured deep learning model for colorization rendering, thereby sending the colorization rendering result to a display device for display. This eliminates the need to send the colorization rendering result back to the CPU, allowing it to be directly sent to the display device for display, reducing data transmission time and resource consumption during data transmission, and improving the efficiency of colorization rendering.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image coloring processing method and related equipment. Background Technology

[0002] Some terminal boards have limited hardware resources and generally lack dedicated hardware for deep learning models. Therefore, deep learning models are typically housed in the graphics processing unit (GPU). When processing images using a deep learning model, the image sent from the CPU (central processing unit) is rendered and colored on the GPU using the deep learning model. The output then needs to be sent back to the CPU. The CPU then sends the output to the display device for display.

[0003] However, the existing coloring processing method requires the CPU to send the data back, which increases the transmission workload and affects work efficiency. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an image coloring processing method and related equipment that can solve or partially solve the above-mentioned technical problems.

[0005] To achieve the above objectives, the first aspect of this application provides an image colorization processing method, comprising:

[0006] The image to be processed is retrieved from the central processing unit by an open graphics library, and the image to be processed is graphically processed to obtain the graphical processing result.

[0007] The parameter data of the deep learning model in the central processing unit is retrieved through the open graphics library, and the deep learning model is configured using the parameter data;

[0008] The graphical processing results are retrieved from the open graphics library and used in the configured deep learning model for color rendering. The resulting color rendering results are then sent to the display device for display.

[0009] Based on the same inventive concept, a second aspect of this application provides an image coloring processing apparatus, comprising:

[0010] The graphical processing module is configured to retrieve the image to be processed through an open graphics library, perform graphical processing on the image to be processed, and obtain the graphical processing result.

[0011] The parameter configuration module is configured to retrieve parameter data of the deep learning model in the central processing unit through the open graphics library, and to configure the deep learning model using the parameter data;

[0012] The shading and rendering module is configured to retrieve the graphical processing results from the open graphics library and perform shading and rendering processing on the configured deep learning model, and then send the shading and rendering results to the display device for display.

[0013] Based on the same inventive concept, a third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0014] Based on the same inventive concept, a fourth aspect of this application provides a non-transitory computer-readable storage medium that stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described in the first aspect.

[0015] Based on the same inventive concept, the fifth aspect of this application provides a computer program product, including computer program instructions, wherein when the computer program instructions are run on a computer, the computer causes the computer to perform the method as described in the first aspect.

[0016] As can be seen from the above, the image coloring processing method and related equipment provided in this application can directly retrieve the image to be processed from the central processing unit using an open graphics library, perform graphical processing on it, then retrieve the parameter data required by the deep learning model using the open graphics library, configure the deep learning model according to the parameter data, and after configuration, perform coloring rendering processing on the graphical processing result using the configured deep learning model; finally, the coloring rendering result is directly sent to the display device for display. In this way, the coloring rendering process can be completed through the cooperation of the open graphics library and the deep learning model. This eliminates the need to send the obtained coloring rendering result back to the central processing unit; it can be directly sent to the display device for display, reducing data transmission time and resource space occupation during data transmission, and improving the efficiency of coloring rendering. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a structural diagram illustrating an application scenario according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of an image coloring processing method according to an embodiment of this application;

[0020] Figure 3A This is a simplified flowchart of the image coloring processing method according to an embodiment of this application;

[0021] Figure 3B This is a simplified flowchart of the texture binding process according to an embodiment of this application;

[0022] Figure 3C This is a logic processing diagram of the color rendering process in an embodiment of this application;

[0023] Figure 4 This is a structural block diagram of the image coloring processing method according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] The terms used in this application are explained below:

[0028] OpenGL (Open Graphics Library) is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics.

[0029] Central Processing Unit (CPU): As the core of a computer system for computation and control, it is the final execution unit for information processing and program execution.

[0030] Graphics Processing Unit (GPU), also known as display core, visual processor, or display chip, is a microprocessor specifically designed for performing image and graphics-related calculations on personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones).

[0031] Deep learning models are data models that utilize neural networks and include multiple layers of algorithmic neurons.

[0032] A vertex shader is a set of instructions that are executed when a vertex is rendered.

[0033] Fragment shader, also known as pixel shader, can color multiple primitives separately.

[0034] TBO: Texture Buffer Objects.

[0035] FBO: Frame Buffer Objects.

[0036] EBO: Element Buffer Objects.

[0037] VBO: Vertex Buffer Objects.

[0038] VAO: Vertex Array Objects.

[0039] refer to Figure 1 This is a schematic diagram illustrating an application scenario of the image coloring processing method provided in this application embodiment. The application scenario includes a CPU, GPU, and display device, connected via a wired or wireless connection.

[0040] The CPU stores the image to be processed, which requires shading. The GPU, equipped with OpenGL, retrieves the image from the CPU via OpenGL and performs graphical processing on it, yielding the result. The GPU then uses OpenGL to retrieve the necessary parameter data for the deep learning model from the CPU and configures the model accordingly. The graphical processing result is then input into the configured deep learning model for shading and rendering. The entire shading and rendering process occurs within the GPU, utilizing OpenGL for data retrieval in conjunction with the deep learning network. This allows the GPU to directly send the shading and rendering result to the display device without needing to send it back to the CPU, simplifying the transmission process, reducing resource consumption during data transfer, and improving the efficiency of shading and rendering.

[0041] The following is combined with Figure 1 The above application scenarios are used to describe the image coloring processing method according to exemplary embodiments of this application. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect. On the contrary, the embodiments of this application can be applied to any applicable scenario.

[0042] Based on the above description, the embodiments of this application propose an image coloring processing method applied to a graphics processing unit (GPU), such as... Figure 2 As shown, it includes:

[0043] Step 201: Retrieve the image to be processed from the central processing unit through an open graphics library, perform graphical processing on the image to be processed, and obtain the graphical processing result.

[0044] In practice, the image to be processed can be one frame or multiple frames. The OpenGL library is pre-built into the GPU, so that when the GPU performs image shading processing, it can be retrieved from the CPU through OpenGL and then processed in the GPU.

[0045] In some embodiments, the graphics processing is performed using a vertex shader; step 201 includes:

[0046] Step 2011: Use a vertex shader to obtain vertex data of the image to be processed, and convert the vertex data into image data, which is a vector image.

[0047] Step 2012: Use a vertex shader to rasterize the graph data to obtain rasterized data, and use the rasterized data as the result of the graphing process.

[0048] In practice, when OpenGL performs shading, it generates shader programs through compilation and linking processes. These shader programs include vertex shader programs and fragment shader programs. Step 201 involves using the vertex shader to complete the image processing.

[0049] Specifically, the vertex shader processes the vertex data of the image to be processed, and then converts the vertex data into raster data through a primitive assembly process. Since the raster data is a vector image, it needs to be rasterized into rasterized data. This allows the image to be processed to be divided into multiple rasterized data sets. This facilitates segment shading processing for each rasterized data set, simplifying subsequent segment shading processes.

[0050] Step 202: Retrieve parameter data of the deep learning model in the central processing unit through the open graphics library, and configure the deep learning model using the parameter data.

[0051] In some embodiments, step 202 includes:

[0052] Step 2021, executed using the fragment shader.

[0053] Step 2022: Retrieve parameter data of the deep learning model from the central processing unit through the open graphics library.

[0054] Step 2023: Use the open graphics library to retrieve texture data, and use the texture data to bind textures to the parameter data to obtain a parameter texture object.

[0055] Step 2024: Configure the deep learning model according to the parameter texture object to obtain the configured deep learning model.

[0056] In practice, the deep learning model is set up in the fragment shader as a network model composed of multiple layers of neurons, which are connected by parameters (i.e., weight coefficients). This parameter data is pre-stored in the CPU, and OpenGL needs to retrieve it from the CPU. The GPU uses Texture Buffer Objects (TBOs). Since this parameter data cannot be directly configured in the deep learning model within the fragment shader, it needs to be bound to textures using TBOs to form parameter texture objects. These objects can then be configured within the deep learning model. Once all parameter texture objects are configured, the configured deep learning model is obtained.

[0057] The above approach allows us to leverage OpenGL in conjunction with a deep learning model to complete the process of texture binding and configuration of parameter data, thereby ensuring that the configured deep learning model can better perform subsequent fragment shading processing.

[0058] In some embodiments, the process of retrieving texture data in step 2023 includes: retrieving texture data from a texture buffer object using a texture extraction function in the Open Graphics Library.

[0059] In practice, OpenGL requires using texture extraction functions (e.g., TextureFetch) to access the texture buffer in the GPU and retrieve its texture data. The texture in the texture buffer is one-dimensional texture data derived from a Texture Buffer Object (TBO). The TBO allows the fragment shader to access a memory table managed by the texture buffer object. The corresponding texture extraction function (i.e., the sampler) for the buffer texture is of type samplerBuffer.

[0060] Step 203: The graphical processing result is retrieved from the open graphics library and used to perform shading and rendering processing on the configured deep learning model. The resulting shading and rendering result is then sent to the display device for display.

[0061] In some embodiments, the deep learning model is configured in a fragment shader.

[0062] In practice, after configuring the deep learning model according to step 202 above, the shading and rendering process can be achieved by combining the function of the fragment shader.

[0063] In some embodiments, step 203 is performed using the fragment shader, which passes the graphical processing result to the fragment shader in sample2D format, including:

[0064] Step 2031: Obtain the rasterized data of each graphic element in the graphical processing result.

[0065] In the process described in steps 2011 to 2012, the vertex shader can obtain graph data and rasterized data of each graph data. In this way, the fragment shader can retrieve these rasterized data from the vertex shader, or the vertex shader can process these rasterized data and then pass them to the fragment shader.

[0066] Step 2032: Use the open graphics library to retrieve texture data, and use the texture data to bind textures to the rasterized data to obtain a primitive texture object.

[0067] In practice, since rasterization cannot be directly processed in the deep learning model of the fragment shader, it is necessary to use the texture buffer object (TBO) to bind textures to these rasterized data, forming primitive texture objects corresponding to each primitive data.

[0068] Step 2033: Input the primitive texture object into the configured deep learning model for shading and rendering processing, and send the shading and rendering result to the display device for display.

[0069] In some embodiments, the deep learning model is a convolutional neural network model, which includes at least one convolutional layer.

[0070] Step 2033 includes:

[0071] Step 20331: Input the primitive texture object into the convolutional neural network model, and use at least one convolutional layer to perform convolutional coloring processing on the primitive texture object to obtain the primitive coloring result output.

[0072] In specific implementation, the convolutional neural network model also includes an input layer and an output layer. In the fragment shader, OpenGL is used to retrieve primitive texture objects, which are then input into the input layer of the convolutional neural network model. The input layer passes the primitive texture data to at least one convolutional layer for layer-by-layer convolutional coloring processing. Then, the last convolutional layer sends the final convolution result to the output layer, which outputs the result to obtain the primitive coloring result corresponding to each primitive data.

[0073] However, since these primitive coloring results are scattered and cannot be integrated, the following steps are required for vector integration.

[0074] Step 20332: Determine the vector relationship corresponding to the graph data in the vertex shader.

[0075] In practice, OpenGL is used to retrieve the vector relationships of each graph data element obtained during the vertex shading process from the vertex shader. These vector relationships contain the positional relationships of each graph data element.

[0076] In some embodiments, step 20332 includes:

[0077] Execute using the vertex shader:

[0078] Step 203321: Obtain vertex data and graph data.

[0079] Step 203322: Determine the vertex buffer object based on the vertex data.

[0080] Step 203323: Determine the index buffer object based on the graph data.

[0081] Step 203324: Perform data combination processing on the vertex buffer object and the index buffer object to obtain a vertex array object (i.e., VAO, Vertex Buffer Objects).

[0082] Step 203325: Perform data activation processing on the vertex data object to obtain the vector relationship.

[0083] Among them, steps 203322 and 203323 can be executed in parallel, sequentially, or step 203323 can be executed first and then step 203322. The specific choice can be made according to actual needs, and no specific limitation is made here.

[0084] The above scheme can effectively determine the relationships between various graph data elements based on vertex buffer objects and index buffer objects, obtain vertex array objects, and after activating the data of the vertex array objects, obtain vector relationships representing the positional relationships of each graph data element. This facilitates the subsequent arrangement of the primitive coloring results based on the vector relationships, thereby ensuring the subsequent arrangement effect.

[0085] Step 20333: Arrange the primitive coloring results according to the vector relationship to obtain the coloring result.

[0086] In practice, after arranging the coloring results of each primitive according to the vector relationship, a coloring data matrix can be obtained, and this coloring data matrix is ​​used as the coloring result.

[0087] Step 20334: Render the coloring result to obtain a color rendering result and send it to the display device for display.

[0088] In some embodiments, step 20334 includes:

[0089] Step 203341: Determine the surface view window corresponding to the open graphics library.

[0090] For example, for Android devices, if rendering is required in a window using OpenGL, the corresponding surface view window provided is Android glsurfaceview.

[0091] Step 203342: Use the open graphics library to call the rendering function, and use the rendering function to render the shading result in the surface view window to obtain the shading rendering result.

[0092] Step 203343: Send the color rendering result to the display device for display.

[0093] In practice, after the vertex shader and fragment shader perform the shading process, the resulting shading results need to be rendered in the surface view window in order to obtain the final shading rendering result that the display device can directly use for display.

[0094] The technical solution described in the above embodiments enables the direct retrieval of the image to be processed from the central processing unit (CPU) using an open graphics library (OPL), followed by graphical processing. Then, the OPL retrieves the parameter data required by the deep learning model, configures the deep learning model according to the parameter data, and performs color rendering on the graphical processing result using the configured deep learning model. Finally, the color rendering result is directly sent to the display device for display. In this way, the color rendering process can be completed through the cooperation of the OPL and the deep learning model. This eliminates the need to send the color rendering result back to the CPU; it can be directly sent to the display device for display, reducing data transmission time and resource consumption during data transmission, and improving the efficiency of color rendering.

[0095] The implementation process of the image coloring and rendering method is described below with a specific embodiment.

[0096] Specifically, such as Figure 3A As shown, it mainly consists of two processes:

[0097] First, the primitive configuration process using the vertex shader is as follows:

[0098] The vertex shader processes the vertex data of the image and then converts it into primitive data through a primitive configuration process. Since the primitive data is vector data, it needs to be rasterized into rasterized data. This allows the image to be processed to be broken down into multiple rasterized data sets.

[0099] Second, the fragment shader is used to color and render the rasterized data in the primitives.

[0100] Before shading and rendering, texturing is required using TBO, specifically as follows: Figure 3B As shown.

[0101] Using OpenGL, texture data 0 and texture data 1 are retrieved. Texture data 0 is used to bind textures to the rasterized data in the primitive data to obtain a primitive texture object (i.e., Figure 3C In the FBO processing, texture data 1 is used to bind the parameter data (e.g., weight coefficients) to obtain the parameter texture object (i.e., Figure 3C(TBO processing procedure).

[0102] Then, the primitive texture object and the parametric texture object are colored in the fragment shader using a deep learning model.

[0103] like Figure 3C As shown, in the vertex shader:

[0104] Vertex data and primitive data store the position and color information of image pixels, respectively. The VBO (Vertex Buffer Object) is determined based on the vertex data, and the EBO (Primitive Buffer Object) is determined based on the primitive data; both are stored in the GPU.

[0105] Then, the VBO and EBO are combined to obtain the vertex array object VAO, which makes it easier to manage the VBO and EBO.

[0106] In the fragment shader:

[0107] The primitive texture object is obtained by binding the primitive data to a texture object using a frame buffer object (FBO). The parameter texture object is obtained by binding the parameter data to a texture buffer object (TBO).

[0108] Then, the deep learning model set in the fragment shader is used to configure the texture object according to the parameters to perform color processing on the primitive texture object, and the color processing result is rendered through the window provided by Android glSurfaceView.

[0109] Finally, the rendered result is played on a display device.

[0110] In summary, this method combines deep learning models with OpenGL to perform graphics-related operations on the GPU. The processed data is then directly rendered to the screen for playback based on different shaders, eliminating the need to send the data back to the GPU. This saves on data transmission and is highly suitable for scenarios where deep learning computations are directly applied to the screen for playback, such as video, image enhancement, and super-resolution.

[0111] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0112] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0113] Based on the same inventive concept, corresponding to any of the above-described embodiments, this application also provides an image coloring processing apparatus.

[0114] refer to Figure 4 The device includes:

[0115] The graphical processing module 41 is configured to retrieve the image to be processed through an open graphics library, perform graphical processing on the image to be processed, and obtain a graphical processing result.

[0116] The parameter configuration module 42 is configured to retrieve parameter data of the deep learning model in the central processing unit through the open graphics library, and configure the deep learning model using the parameter data;

[0117] The shading and rendering module 43 is configured to retrieve the graphical processing result from the open graphics library and perform shading and rendering processing on the configured deep learning model, and then send the shading and rendering result to the display device for display.

[0118] In some embodiments, the graphics processing module 41 is configured to perform the graphics processing process through a vertex shader, including:

[0119] The vertex data of the image to be processed is obtained using a vertex shader, and the vertex data is converted into image data, which is a vector image. The image data is then rasterized using a vertex shader to obtain rasterized data, which is used as the result of the graphics processing.

[0120] In some embodiments, the deep learning model is configured in a fragment shader.

[0121] In some embodiments, the parameter configuration module 42 includes:

[0122] The parameter retrieval unit is configured to retrieve parameter data of a deep learning model from the central processing unit using the fragment shader through the open graphics library;

[0123] The parameter texture processing unit is configured to retrieve texture data using the open graphics library, and to bind texture processing to the parameter data using the texture data to obtain a parameter texture object.

[0124] The model configuration unit is configured to configure the deep learning model according to the parameter texture object to obtain the configured deep learning model.

[0125] In some embodiments, the parametric texture processing unit is further configured to:

[0126] The texture data in the texture buffer object is retrieved using the texture extraction function in the Open Graphics Library.

[0127] In some embodiments, the shading rendering module 43 includes:

[0128] The primitive acquisition unit is configured to acquire rasterized data of each primitive data element in the graphical processing result using the fragment shader;

[0129] The primitive texture processing unit is configured to retrieve texture data using the open graphics library, and use the texture data to perform texture binding processing on the rasterized data to obtain a primitive texture object.

[0130] The shading and rendering unit is configured to input the primitive texture object into the configured deep learning model for shading and rendering processing, and send the shading and rendering result to the display device for display.

[0131] In some embodiments, the deep learning model is a convolutional neural network model, which includes at least one convolutional layer;

[0132] The shading rendering unit is also configured as follows:

[0133] The primitive texture object is input into a convolutional neural network model, and at least one convolutional layer is used to perform convolutional coloring processing on the primitive texture object to obtain a primitive coloring result output; the vector relationship corresponding to the primitive data in the vertex shader is determined; the primitive coloring result is arranged according to the vector relationship to obtain the coloring result; the coloring result is rendered to obtain a color rendering result and sent to a display device for display.

[0134] In some embodiments, the shading rendering unit is further configured to:

[0135] The vertex shader is used to perform the following: acquire vertex data and graph data; determine a vertex buffer object based on the vertex data; determine an index buffer object based on the graph data; perform data combination processing on the vertex buffer object and the index buffer object to obtain a vertex array object; and perform data activation processing on the vertex data object to obtain the vector relationship.

[0136] In some embodiments, the shading rendering unit is further configured to:

[0137] Determine the surface view window corresponding to the open graphics library; call the rendering function using the open graphics library, and render the shading result in the surface view window using the rendering function to obtain the shading rendering result; send the shading rendering result to the display device for display.

[0138] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0139] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0140] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the above embodiments.

[0141] Figure 5 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0142] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0143] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0144] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0145] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0146] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0147] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0148] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0149] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0150] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0151] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0152] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to perform the method. Corresponding to the execution entity for each step in each embodiment of the method, the processor performing the corresponding step may belong to the corresponding execution entity.

[0153] The computer program products of the above embodiments are used to cause a processor to execute the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0154] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0155] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0156] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0157] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. An image coloring processing method, applied to a graphics processor, wherein the graphics processor has an open image library built-in, and the graphics processor includes a fragment shader; The method includes: The image to be processed is retrieved from the central processing unit by an open graphics library, and the image to be processed is graphically processed to obtain the graphical processing result. The parameter data of the deep learning model in the central processing unit is retrieved through the open graphics library, and the deep learning model is configured using the parameter data. The deep learning model is a convolutional neural network model, which includes at least one convolutional layer. The graphical processing result is retrieved from the open graphics library and used in the configured deep learning model for color rendering. The resulting color rendering result is then sent to the display device for display. Configuring the deep learning model in a fragment shader, and retrieving parameter data of the deep learning model from the central processing unit via the open graphics library, and configuring the deep learning model using the parameter data, includes: The fragment shader executes: The parameter data of the deep learning model is retrieved from the central processing unit through the open graphics library; The open graphics library is used to retrieve texture data, and the texture data is used to bind the parameter data to a texture buffer object (TBO) to obtain a parameter texture object. The deep learning model is configured according to the parameter texture object to obtain the configured deep learning model; The step of retrieving the graphical processing result from the open graphics library and inputting it into the configured deep learning model for color rendering, and then sending the resulting color rendering result to the display device for display, includes: Execute using the fragment shader: Obtain the rasterized data of each image element in the graphical processing result; The open graphics library is used to retrieve texture data, and the rasterized data is bound to a texture through a frame buffer object (FBO) to obtain a primitive texture object. The primitive texture object is input into a convolutional neural network model, and at least one convolutional layer is used to perform convolutional coloring processing on the primitive texture object to obtain the primitive coloring result output. Determine the vector relationships corresponding to the graph data in the vertex shader; The coloring results of the primitives are arranged according to the vector relationship to obtain the coloring result; The coloring result is rendered to obtain a color rendering result, which is then sent to a display device for display.

2. The method according to claim 1, wherein, The graphical processing is performed using a vertex shader; The step of performing graphical processing on the image to be processed to obtain the graphical processing result includes: The vertex data of the image to be processed is obtained using a vertex shader, and the vertex data is converted into image data, which is a vector image. The graph data is rasterized using a vertex shader to obtain rasterized data, which is then used as the result of the graphing process.

3. The method according to claim 1 or 2, wherein, The process of retrieving texture data using the open graphics library includes: The texture data in the texture buffer object is retrieved using the texture extraction function in the Open Graphics Library.

4. The method according to claim 1, wherein, Determining the vector relationships corresponding to the graph data in the vertex shader includes: Execute using the vertex shader: Obtain vertex data and graph data; Determine the vertex buffer object based on the vertex data; The index buffer object is determined based on the graph data; The vertex buffer object and the index buffer object are combined to obtain a vertex data object; The vector relationship is obtained by performing data activation processing on the vertex data object.

5. The method according to claim 1, wherein, The step of rendering the coloring result to obtain a color rendering result and sending it to a display device for display includes: Determine the surface view window corresponding to the open graphics library; The rendering function is called using the open graphics library, and the rendering function is used to render the shading result in the surface view window to obtain the shading rendering result. The color rendering result is sent to the display device for display.

6. An image coloring processing apparatus, disposed in a graphics processor, the graphics processor having an built-in open image library, the graphics processor including a fragment shader; The device includes: The graphical processing module is configured to retrieve the image to be processed through an open graphics library, perform graphical processing on the image to be processed, and obtain the graphical processing result. The parameter configuration module is configured to retrieve parameter data of a deep learning model in the central processing unit through the open graphics library, and configure the deep learning model using the parameter data. The deep learning model is configured in a fragment shader, and the deep learning model is a convolutional neural network model, which includes at least one convolutional layer. The shading and rendering module is configured to retrieve the graphical processing results from the open graphics library and perform shading and rendering processing on the configured deep learning model, and then send the shading and rendering results to the display device for display. The parameter configuration module includes: The parameter retrieval unit is configured to retrieve parameter data of a deep learning model from the central processing unit using the fragment shader through the open graphics library; The parameter texture processing unit is configured to retrieve texture data using the open graphics library, and use the texture data to perform texture binding processing on the parameter data through a texture buffer object (TBO) to obtain a parameter texture object. The model configuration unit is configured to configure the deep learning model according to the parameter texture object to obtain the configured deep learning model; The shading and rendering module includes: The primitive acquisition unit is configured to acquire rasterized data of each primitive data element in the graphical processing result using the fragment shader; The primitive texture processing unit is configured to retrieve texture data using the open graphics library, and use the texture data to bind texture processing to the rasterized data through a frame buffer object (FBO) to obtain a primitive texture object. The shading rendering unit is configured to input the primitive texture object into a convolutional neural network model, perform convolutional shading processing on the primitive texture object using at least one convolutional layer to obtain a primitive shading result output; determine the vector relationship corresponding to the primitive data in the vertex shader; arrange the primitive shading result according to the vector relationship to obtain a shading result; render the shading result to obtain a shading rendering result and send it to a display device for display.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method described in any one of claims 1 to 5.

9. A computer program product comprising computer program instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-5.

Images