Chip system, image processing method and electronic equipment

Abstract

The embodiment of the invention provides a chip system, an image processing method and electronic equipment, and is applied to the technical field of chip image processing. The chip system comprises an image processor, a neural network processor and a scheduling circuit. Wherein the image processor is used for outputting a first rendered image according to the image subtask. And the scheduling circuit is used for instructing the neural network processor to acquire the first rendered image according to the scheduling sub-task. And the neural network processor is used for performing neural network algorithm processing on the first rendered image according to the operation subtask and outputting an operation result. And the scheduling circuit is also used for indicating the image processor to obtain an operation result according to the scheduling sub-task. And the image processor is also used for carrying out image processing on the operation result according to the image subtask to obtain a second rendered image. According to the embodiment of the invention, image processing is carried out based on multi-IP linkage so as to improve the imaging quality, the power consumption and the processing complexity of a chip system are reduced, and the method can be suitable for different chip architectures.

Description

Technical Field

[0001] The present application relates to the technical field of chip image processing, and in particular to a chip system, an image processing method and an electronic device. Background Art

[0002] The chip system includes an image processor. The image task is performed by the image processor to render and generate the image that needs to be imaged and displayed. At present, in some high-power chip architectures (such as desktops based on the X86 architecture), in order to pursue better imaging quality, complex image rendering algorithms are often used to process the image in the image processor. However, complex image rendering algorithm processing will greatly increase the power consumption of the image processor. One way to improve it is to modify the design inside the image processor to add a neural network processing circuit inside the image processor. A rendered image with higher imaging quality is obtained by performing image rendering and AI algorithm processing in a closed loop inside the image processor. Although this method can reduce power consumption, it requires changes to the IP core of the image processor and increases the area of ​​the image processor. At the same time, the neural network processing circuit in the image processor cannot be shared with other multimedia services.

[0003] However, in medium and low power chip architectures (such as mobile terminals based on ARM architecture), there are extremely high requirements for chip device area and power consumption, and it is often necessary to reduce power consumption by reducing the complexity and processing volume of the image rendering algorithm. Therefore, it is difficult to improve the image quality by the above-mentioned method of increasing the complexity of image rendering algorithm processing or by the above-mentioned method of adding neural network processing circuits in medium and low power chip architectures. With the technological updates and iterations of electronic products, users' demand for the image quality of displayed images is also increasing. How to obtain a rendered image with higher image quality in chip architectures with different power consumption and different hardware designs is a major problem. Summary of the invention

[0004] The embodiments of the present application provide a chip system, an image processing method and an electronic device, which reduce the power consumption of the chip system on the basis of improving the imaging quality of the rendered image and can be applied to different chip architectures.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a chip system is provided, which includes an image processor, a neural network processor, and a scheduling circuit. The image processor is used to output a first rendered image according to an image subtask. The scheduling circuit is used to instruct the neural network processor to obtain the first rendered image according to the scheduling subtask. The neural network processor is used to perform neural network algorithm processing on the first rendered image and output the operation result according to the operation subtask. The scheduling circuit is also used to instruct the image processor to obtain the operation result according to the scheduling subtask. The image processor is also used to perform image processing on the operation result according to the image subtask to obtain a second rendered image.

[0007] In an embodiment of the present application, a scheduling circuit is used to instruct the image processor and the neural network processor to perform multi-IP linkage processing to generate a rendered image with better imaging parameters. The specific operation of the multi-IP linkage processing is: the image processor executes an image subtask to render the imaged image to obtain a first rendered image. The neural network processor performs a neural network algorithm on the first rendered image to obtain a calculation result to complete the calculation operation of parameter optimization. The image processor then obtains a second rendered image based on the calculation result. One or more imaging parameters of the second rendered image are better than the first rendered image. In this processing method: First, the present application uses a neural network processor to assist in improving the imaging quality of the rendered image based on a neural network processing algorithm. Compared with the operation of using a more complex image rendering algorithm in the image processor to improve the imaging quality, the embodiment of the present application can greatly reduce the complexity of image processing and improve the imaging quality. Secondly, the neural network processor and the image processor are independent devices, and the data path interaction and control signal interaction between the two are quite complex. And the software control logic of the two is relatively simple. The embodiment of the present application can realize multi-IP linkage image processing between the neural network processor and the image processor in a simple control manner by setting a hardware scheduling circuit. Finally, this solution does not require hardware redesign of each chip in the chip system, and can be applied to chip architectures with different power consumption and different designs.

[0008] In one possible implementation, the image processor is also used to: when outputting the first rendered image, output a first trigger signal to the scheduling circuit according to the image subtask. The scheduling circuit is specifically used to: in response to the first trigger signal, output a second trigger signal to the neural network processor according to the scheduling subtask; the second trigger signal is used to instruct the neural network processor to obtain the first rendered image. The image processor is also used to: when outputting the operation result, output a third trigger signal to the scheduling circuit according to the image subtask. The scheduling circuit is specifically used to: in response to the third trigger signal, output a fourth trigger signal to the image processor according to the scheduling subtask; the fourth trigger signal is used to instruct the image processor to obtain the operation result. In an embodiment of the present application, the scheduling of subtasks with different execution orders in multi-IP linkage processing can be completed by a few trigger signals. The scheduling method is simple and fast.

[0009] In a possible implementation, the chip system also includes a storage circuit. Among them: the image processor is specifically used to: write the first rendered image into the storage circuit; in response to the fourth trigger signal, obtain the operation result from the storage circuit. The neural network processor is specifically used to: respond to the second trigger signal, read the first rendered image from the first buffer according to the operation subtask; write the operation result to the storage circuit. In an embodiment of the present application, the first rendered image and the operation result interacted by the neural network processor and the image processor are stored and transferred through the storage circuit, so that multi-IP linkage processing can be carried out more orderly.

[0010] In one possible implementation, the image processor is also used to: generate an initial rendered image according to an image subtask. Multiple first rendered images are obtained based on the initial rendered image; the multiple first rendered images correspond to different imaging parts of the initial rendered image. In an embodiment of the present application, when the image processor and the neural network processor perform image processing in conjunction, there may be a situation where the processing delay of one party is large, causing the other party to wait for feedback. At this time, a complete imaging frame can be split into multiple first rendered images. By using the image processor and the neural network processor to perform multi-IP linkage processing operations on multiple first rendered images in a pipeline manner, the processing performance of the image processor and the neural network processor can be effectively utilized, and the processing delay of each complete frame can be reduced.

[0011] In one possible implementation, the operation result is an image processing coefficient. The image processor is also used to: obtain at least one pre-processed image according to the first rendered image. The first rendered image and at least one pre-processed image are processed according to the image processing coefficient to obtain a second rendered image. In the embodiment of the present application, it is necessary to perform operations on the relevant image processing coefficients through a neural network processor. The neural network processor has a small amount of data operations, and the computing power requirements of the neural network processor are low. The neural network processor can quickly output the image processing coefficient as the operation result.

[0012] In a possible implementation, the operation result is a processed image. The image processor is also used to obtain at least one preprocessed image according to the first rendered image. The neural network processor is specifically used to perform neural network algorithm processing on the first rendered image according to at least one preprocessed image to obtain a processed image. In an embodiment of the present application, the image processor can perform image processing to generate one or more preprocessed images. The preprocessed image is used to assist in optimizing image parameters. When the neural network processor inputs the first rendered image, at least one preprocessed image is also input. The neural network processor performs neural network algorithm processing on at least one preprocessed image and the first rendered image to obtain a processed image. The processed image is an image after the imaging parameters are optimized. The neural network processor outputs the processed image as a result of the operation. In an embodiment of the present application, the neural network processor can directly obtain a processed image after the imaging parameters are optimized by performing neural network operation, and then output the processed image as a result of the operation to the image processor. The image processor performs post-processing of the relevant rendered image according to the processed image to obtain a second rendered image. In this implementation, the computing power requirements for the neural network processor will be slightly higher. At the same time, the processing time of the neural network processor will also be longer. However, the processing capacity of the image processor will be greatly reduced.

[0013] In a possible implementation, the chip system also includes a processor. The processor is used to: output an image subtask, an operation subtask, and a scheduling subtask. At least one of the image subtask, the operation subtask, or the scheduling subtask is provided with a heterogeneous synchronization signal. The heterogeneous synchronization signal is used to indicate the execution dependency between the image subtask and the scheduling subtask, or the heterogeneous synchronization signal is used to indicate the execution dependency between the operation subtask and the scheduling subtask. In an embodiment of the present application, when encapsulating each subtask, the corresponding heterogeneous synchronization signal is inserted according to the processing order of the subtask, and the execution dependency between different subtasks is formed through the heterogeneous synchronization signal. Under this implementation, there is no need to make too many software-level changes to the image processor, the neural network processor, etc., so that the image processor, the neural network processor, etc. can directly schedule the execution of the corresponding subtasks according to the execution dependency.

[0014] In a possible implementation, the processor is also used to: generate an image rendering task group in response to a rendering command input by the application; the image rendering task group includes an image subtask, an operation subtask, and a scheduling subtask. In an embodiment of the present application, the application manufacturer has coordinated the design of the rendering command. At this time, the issued rendering command has adaptively set parameters of multiple related subtasks that the IP hardware needs to perform on its own. Then, according to the rendering command, a corresponding rendering image task group can be directly generated, and the rendering image task group already includes an image subtask, an operation subtask, and a scheduling subtask.

[0015] In one possible implementation, the processor is also used to: generate an image rendering task group in response to a rendering command input by the application; the image rendering task group includes image subtasks. Operation subtasks and scheduling subtasks are set in the image rendering task group. In an embodiment of the present application, the application manufacturer did not change the rendering command. At this time, the rendering command does not include parameters related to multiple subtasks for image processing by multi-IP linkage. Because the rendering command has not been changed, the image rendering task group generated by the processor according to the rendering command only includes image subtasks related to image rendering, and does not include subtasks related to the neural network processor and the scheduling circuit. It is necessary to set the operation subtasks and scheduling subtasks by the processor.

[0016] In some examples, when setting a computing subtask in the image rendering task group, the processor is specifically used to perform at least one of the following: modifying at least one image subtask into a computing subtask, or adding a new computing subtask to the image rendering task group. In the embodiment of the present application, some image subtasks originally executed by the image processor can be modified into computing subtasks, and some computing subtasks can also be added.

[0017] In some examples, the processor is further used to: add a new image subtask to the image rendering task group. In the embodiment of the present application, in some application scenarios, some new image subtasks need to be added to achieve linkage processing of multiple IPs.

[0018] In some possible implementations, different imaging parameters of the rendered image may be optimized according to different neural network algorithm processing.

[0019] Exemplarily, the second rendered image satisfies at least one of the following conditions: a resolution of the second rendered image is greater than a resolution of the first rendered image, or a frame rate of the second rendered image is greater than a frame rate of the first rendered image.

[0020] In a second aspect, an embodiment of the present application also provides an image processing method, which is based on a chip system. The chip system includes an image processor, a neural network processor and a scheduling circuit. The method includes: the image processor outputs a first rendered image according to an image subtask. The scheduling circuit instructs the neural network processor to obtain the first rendered image according to the scheduling subtask. The neural network processor performs a neural network algorithm on the first rendered image according to the operation subtask and outputs the operation result. The scheduling circuit also instructs the image processor to obtain the operation result according to the scheduling subtask. The image processor also performs image processing on the operation result according to the image subtask to obtain a second rendered image.

[0021] In a possible implementation, the method further includes: when outputting the first rendered image, the image processor outputs a first trigger signal to the scheduling circuit according to the image subtask. The scheduling circuit responds to the first trigger signal and outputs a second trigger signal to the neural network processor according to the scheduling subtask; the second trigger signal is used to instruct the neural network processor to obtain the first rendered image. When outputting the operation result, the image processor outputs a third trigger signal to the scheduling circuit according to the image subtask. The scheduling circuit responds to the third trigger signal and outputs a fourth trigger signal to the image processor according to the scheduling subtask; the fourth trigger signal is used to instruct the image processor to obtain the operation result.

[0022] In a possible implementation, the method further includes: an image processor generating an initial rendered image according to an image subtask; obtaining a plurality of first rendered images according to the initial rendered image; and the plurality of first rendered images corresponding to different imaging parts of the initial rendered image.

[0023] In a possible implementation, the operation result is an image processing coefficient. The method further includes: the image processor obtains at least one pre-processed image according to the first rendered image. The above-mentioned outputting the second rendered image according to the operation result includes: the image processor processes the first rendered image and the at least one pre-processed image according to the image processing coefficient to obtain the second rendered image.

[0024] In a possible implementation, the operation result is a processed image. The method further includes: an image processor obtains at least one pre-processed image according to the first rendered image. The above-mentioned performing a neural network algorithm on the first rendered image to obtain the operation result includes: a neural network processor performs a neural network algorithm on the first rendered image according to at least one pre-processed image to obtain a processed image.

[0025] In a possible implementation, the method further includes: outputting an image subtask, a computing subtask, and a scheduling subtask; at least one of the image subtask, the computing subtask, or the scheduling subtask is provided with a heterogeneous synchronization signal. The heterogeneous synchronization signal is used to indicate the execution dependency relationship between the image subtask and the scheduling subtask, or the heterogeneous synchronization signal is used to indicate the execution dependency relationship between the computing subtask and the scheduling subtask.

[0026] In a possible implementation, the method further includes: generating an image rendering task group in response to a rendering command input by the application; the image rendering task group includes an image subtask, a calculation subtask, and a scheduling subtask.

[0027] In a possible implementation, the method further includes: generating an image rendering task group in response to a rendering command input by the application; the image rendering task group includes an image subtask, and setting a computing subtask and a scheduling subtask in the image rendering task group.

[0028] In a possible implementation, the above-mentioned setting of the operation subtask in the image rendering task group includes: performing at least one of the following operations: modifying at least one image subtask into an operation subtask, or adding a new operation subtask to the image rendering task group.

[0029] In a possible implementation, the method further includes: inserting a new image subtask into the image rendering task group.

[0030] In a possible implementation manner, the second rendered image satisfies at least one of the following: a resolution of the second rendered image is greater than a resolution of the first rendered image, or a frame rate of the second rendered image is greater than a frame rate of the first rendered image.

[0031] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a circuit board and a chip system as described in the first aspect above; the chip system is arranged on the circuit board.

[0032] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which includes instructions; when the instructions are executed on an electronic device as described in the third aspect above, the electronic device executes the image processing method as described in the second aspect above.

[0033] Regarding the technical principles and beneficial effects of the second, third and fourth aspects, please refer to the relevant description of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of the structure of a first chip system provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of the structure of another first chip system provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of the structure of a second chip system provided in an embodiment of the present application Figure 1 ;

[0038] Figure 5 A schematic diagram of another second chip system provided in an embodiment of the present application Figure 2 ;

[0039] Figure 6 A schematic diagram of the structure of another second chip system provided in the embodiment of the present application Figure 3 ;

[0040] Figure 7 A schematic diagram of the structure of another second chip system provided in the embodiment of the present application Figure 4 ;

[0041] Figure 8 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 1 ;

[0042] Fig. 9 Another image processing method provided in the present invention is shown in FIG. Figure 2 ;

[0043] Fig.10 A time delay schematic diagram of an image processing method without splitting imaging frames provided in an embodiment of the present application;

[0044] Fig.11 A time delay schematic diagram of an image processing method for splitting imaging frames provided in an embodiment of the present application;

[0045] Fig.12 A schematic diagram of an inter-chip processing flow for a neural network algorithm processing provided in an embodiment of the present application;

[0046] Fig.13 A schematic diagram of another image processing method provided in the embodiment of the present application Figure 3 ;

[0047] Fig.14 A schematic diagram of a software architecture of a processor provided in an embodiment of the present application;

[0048] Fig.15 A schematic diagram of another image processing method provided in the embodiment of the present application Figure 4 ;

[0049] Fig.16 A schematic diagram of another image processing method provided in the embodiment of the present application Figure 5 ;

[0050] Fig.17 A schematic diagram of software scheduling of a second image processor provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] It should be noted that the terms "first", "second", etc. involved in the embodiments of the present application are only used to distinguish features of the same type and cannot be understood as indicating relative importance, quantity, order, etc.

[0052] The terms "exemplary" or "for example" and the like in the embodiments of the present application are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the related concepts in a specific way.

[0053] The terms "coupling" and "connection" involved in the embodiments of the present application should be understood in a broad sense. For example, they may refer to a direct physical connection, or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0054] First, some basic concepts involved in this application are explained:

[0055] Electronic design automation (EDA) refers to the use of computer-aided design (CAD) software to complete the functional design, synthesis, verification, physical design (including layout, routing, layout, design rule checking, etc.) of integrated circuit chips.

[0056] Intellectual property (IP) cores play a very important role in EDA. They can be understood as pre-designed circuit function modules in integrated circuits. With the development of complex semiconductors and communication technologies, the research and development of integrated circuits has also been gradually carried out. The rapid improvement of chip integration has enabled more IP cores to be integrated on a single chip, thus forming a chip system. Chip systems have different forms of chip architectures to be applied in different application scenarios. In high-power application scenarios, large-area and high-power chip architectures can be designed, such as chip systems based on the X86 architecture. Because high-power chip architectures have higher processing performance, they also have larger chip areas and higher processing power consumption, and are usually used in computers, high-performance data processing equipment, etc. In medium- and low-power application scenarios, medium- and low-power chip architectures can be designed, such as chip systems based on advanced RISC machines (ARM) architectures. Because medium- and low-power chip architectures have smaller chip areas and lower processing power consumption, they are usually used in mobile and portable data processing devices, such as mobile phones and tablets. However, in some application scenarios, some computer products will also use chip systems based on the ARM architecture. Take the chip system as a system on chip (SoC) as an example: SoC may integrate a central processing unit (CPU), a graphics processing unit (GPU), a neural-network processing unit (NPU), a digital signal processing unit (DSP), a microcontroller unit (MCU), a memory (memory), an analog to digital converter (ADC), a digital to analog converter (DAC), a universal asynchronous receiver / transmitter (UART), and software and hardware modules that implement special customized functions, etc., homogeneous or heterogeneous IP cores. Therefore, a chip system may include multiple different IP cores. Data communication and exchange between different IP cores has become one of the key technologies in SoC chip design.

[0057] Image rendering is the process of converting three-dimensional light energy transfer processing into a two-dimensional rendered image. Image rendering is usually achieved by using an image processor in a chip system. In the process of image rendering, scenes and entities are represented in three-dimensional form, which is closer to the real world and is easy to manipulate and transform. The display devices for rendering images are mostly two-dimensional rasterized displays and dot-matrix printers. Image rendering can be regarded as rasterization. A raster display can be regarded as a pixel matrix. Any graphic displayed on a raster display is actually a collection of pixels with one or more colors and grayscales. Rendering is one of the most important technologies in computer graphics. At present, image rendering has important applications in various fields such as game entertainment, film and television animation, architectural design, education and medical care, digital twin modeling, and artificial intelligence interaction. With the development of science and technology, people have higher requirements for the imaging quality of rendered images. For example, in terms of game imaging, rendered images are required to have functions such as ray tracing (RT), deep learning super sampling (DLSS), artificial intelligence (AI), and Reflex low latency. Compared with rasterization, ray tracing makes the rendering more realistic and can achieve more lighting effects. In order to pursue better imaging quality, engineers use software and hardware to continuously optimize various imaging parameters of the rendered image (such as resolution, frame rate and special effects). Taking resolution as an example, from low to high, it can be divided into low resolution (LR), high resolution (HR) and super resolution (SR).

[0058] The present application embodiment provides an electronic device, such as Figure 1 As shown, the electronic device 1000 includes a circuit board (not shown in the figure) and a chip system 100. The chip system 100 is arranged on the circuit board. Figure 1 As shown, the chip system 100 includes an image processor 10. The image processor 10 is used to generate a rendering image for display imaging according to an image task.

[0059] In some possible implementations, the chip system 100 is a first chip system based on a high power consumption chip architecture.

[0060] In some examples, in order to improve the imaging quality of the rendered image of the first chip system, such as Figure 2As shown, the first chip system 100A includes a first image processor 10A. A more complex image rendering algorithm is used in the first image processor 10A to obtain a rendered image with higher imaging quality. However, in the embodiment of the present application, the use of a more complex image rendering algorithm will greatly increase the power consumption of the first image processor 10A. Therefore, this solution is not suitable for chip systems 100 with medium and low power consumption.

[0061] In some examples, such as Figure 3 As shown, the first image processor 10A includes a first image processing circuit 11A and a neural network processing circuit 12A. In the first image processor 10A, a rendered image is generated by the first image processing circuit 11A, and the rendered image is processed by an artificial intelligence (AI) algorithm through the neural network processing circuit 12A to obtain a rendered image with higher imaging quality. In an embodiment of the present application, a neural network processing circuit 12A for implementing an AI algorithm is additionally designed and integrated in the first image processor 10A. To realize image rendering and AI algorithm processing inside the first image processor 10A. Compared to Figure 2 In the embodiment, a more complex image rendering algorithm is adopted, and the imaging quality of the rendered image is improved by AI algorithm processing, which can save the image rendering load to reduce the computing power consumption of the device, and the imaging has a good anti-aliasing effect, etc. At the same time, the rendering and AI processing are closed-loop inside the first image processor 10A, and its task scheduling overhead is small and the data path is simple. However, in this way, the IP core of the first image processor 10A needs to be redesigned, and with the introduction of the neural network processing circuit 12A, the area of ​​the first image processor 10A will increase. Therefore, although this solution can reduce the power consumption of the first image processor 10A. However, it is not suitable for medium and low power chip architectures. In medium and low power chip architectures, there are also extremely high requirements for the device area of ​​the chip. And this solution requires the modification and design of the IP core of the first image processor 10A, which increases the difficulty of using the solution and cannot be applied in chip systems 100 based on other IP cores.

[0062] In some possible implementations, the chip system 100 may be a second chip system 100B that implements image rendering and AI algorithm processing based on multi-IP collaboration. In this case, the second chip system 100B may be a high-power chip architecture or a medium-to-low-power chip architecture. Figure 4As shown, the second chip system 100B includes a second image processor 10B, a neural network processor 20B and a scheduling circuit 30B. Among them: the second image processor 10B is used to: output a first rendered image according to an image subtask. The scheduling circuit 30B is used to: instruct the neural network processor 20B to obtain the first rendered image according to the scheduling subtask. The neural network processor 20B is used to: perform neural network algorithm processing on the first rendered image according to the operation subtask and output the operation result. The scheduling circuit 30B is also used to: instruct the second image processor 10B to obtain the operation result according to the scheduling subtask. The second image processor 10B is also used to: perform image processing on the operation result according to the image subtask to obtain the second rendered image.

[0063] In the traditional chip architecture, the image processor 10 and the neural network processor 20B are independent IP cores, and in the usual chip design, the image processor 10 and the neural network processor 20B are mainly used as auxiliary devices to assist in image data processing and data calculation processing. The two chips have poor logical control capabilities. In order to achieve the linkage image processing of the two, the amount of data and control signals that need to be interacted are also quite complex. Therefore, it is difficult to achieve collaborative image data processing between the two IPs. In this application, Figure 4In the illustrated embodiment, the second image processor 10B generates a first rendered image according to the image subtask. After the second image processor 10B outputs the first rendered image, the scheduling circuit 30B instructs the neural network processor 20B to obtain the first rendered image. After the neural network processor 20B obtains the first rendered image, it performs AI algorithm processing (such as neural network algorithm processing) on ​​the first rendered image to obtain the operation result. At this time, the scheduling circuit 30B instructs the second image processor 10B to obtain the operation result. The second image processor 10B generates a final rendered image, i.e., a second rendered image, according to the operation result. The imaging quality of the second rendered image is higher than that of the first rendered image. By setting the scheduling circuit 30B, the execution of tasks between multiple IPs (including the second image processor 10B and the neural network processor 20B) is scheduled in a hardware manner. In this embodiment, there is no need to change the IP core design of the second image processor 10B and the neural network processor 20B, and it can be implemented compatibly under different chip architectures (such as X86 architecture or ARM architecture). Compared with using more complex rendering algorithms to improve the image quality, AI algorithm processing based on the existing neural network processor 20B in the chip architecture can greatly reduce the power consumption of the second chip system 100B on the basis of improving the image quality of the rendered image, so that the solution can be applied under chip architectures with different power consumption. At the same time, the hardware-based scheduling circuit 30B controls the execution order of the second image processor 10B executing the image subtask and the neural network processor 20B executing the calculation subtask. The scheduling method is simple, and the second image processor 10B and the neural network processor 20B only need to exchange the first rendered image and the calculation result, avoiding the interaction of complex data signals and control signals between multiple IPs.

[0064] In some possible implementations, such as Figure 4 As shown, the electronic device 1000 further includes a display 200. The display 200 is used to display the second rendered image.

[0065] Exemplarily, the second chip system 100B may be an X86 architecture or an ARM architecture. Exemplarily, the second chip system 100B may be a very large scale integration circuit (VLSI), such as a system on chip (SoC), a network on chip (NoC), etc.

[0066] Exemplarily, the second image processor 10B may be a GPU, and the neural network processor 20B may be an NPU.

[0067] For example, the electronic device 1000 may be a mobile device or a terminal device. For example, the electronic device 1000 may be a mobile phone, a tablet computer, a smart watch, a television, a vehicle control device, a smart home device, etc.

[0068] In some possible implementations, the scheduling circuit 30B may instruct the second image processor 10B and the neural network processor 20B to perform related task operations by interacting with the second image processor 10B and the neural network processor 20B to execute task-related trigger signals.

[0069] For example, Figure 5 As shown, the second image processor 10B is also used to: when outputting the first rendered image, output a first trigger signal to the scheduling circuit 30B according to the image subtask. The scheduling circuit 30B is specifically used to: in response to the first trigger signal, output a second trigger signal to the neural network processor 20B according to the scheduling subtask. The second trigger signal is used to instruct the neural network processor 20B to obtain the first rendered image. The second image processor 10B is also used to: when outputting the operation result, output a third trigger signal to the scheduling circuit 30B according to the image subtask. The scheduling circuit 30B is specifically used to: in response to the third trigger signal, output a fourth trigger signal to the second image processor 10B according to the scheduling subtask. The fourth trigger signal is used to instruct the second image processor 10B to obtain the operation result. In an embodiment of the present application, after the second image processor 10B generates the relevant rendered image according to the image subtask, it outputs the first rendered image, and then outputs the first trigger signal to the scheduling circuit 30B. In response to the received first trigger signal, the scheduling circuit 30B outputs a second trigger signal to the neural network processor 20B. The neural network processor 20B responds to the second trigger signal, receives the first rendered image output by the second image processor 10B, and performs AI algorithm processing (i.e., neural network algorithm processing) on ​​the first rendered image to obtain the operation result. While outputting the operation result, the neural network processor 20B outputs a third trigger signal to the scheduling circuit 30B. The scheduling circuit 30B responds to the third trigger signal and outputs a fourth trigger signal to the second image processor 10B. The second image processor 10B responds to the fourth trigger signal, obtains the operation result, and obtains the second rendered image according to the operation result. In an embodiment of the present application, only a few trigger signals are required to complete the scheduling of the second image processor 10B and the neural network processor 20B to realize multi-IP linkage processing. The various trigger signals involved in this embodiment may be notification signals or indication signals, which are used to control or notify other IP cores to perform the next step of operation. This embodiment does not limit its implementation form.

[0070] In some possible implementations, the first rendered image or the calculation result may be stored in a cache inside the second image processor 10B or the neural network processor 20B.

[0071] In some possible implementations, in order to facilitate orderly linkage processing between the second image processor 10B and the neural network processor 20B, a general or dedicated storage circuit can be provided in the second chip system 100B to store the first rendered image and the operation result in the storage circuit, so as to facilitate the transfer and exchange of the first rendered image and the operation result between the second image processor 10B and the neural network processor 20B.

[0072] For example, Figure 6 As shown, the second chip system 100B also includes a storage circuit 40B. The second image processor 10B outputs the first rendered image to the storage circuit 40B, and outputs a first trigger signal to the scheduling circuit 30B. The scheduling circuit 30B outputs a second trigger signal to the neural network processor 30B according to the first trigger signal. After the neural network processor 30B inputs the second trigger signal, in response to the second trigger signal, the first rendered image is obtained from the storage circuit 40B, and the first rendered image is processed by a neural network operation to obtain an operation result; then the operation result is stored in the storage circuit 40B, and a third trigger signal is output to the scheduling circuit 30B. The scheduling circuit 30B outputs a fourth trigger signal to the second image processor 10B according to the third trigger signal. When the second image processor 10B inputs the fourth trigger signal, in response to the fourth trigger signal, the operation result is obtained from the storage circuit 40B, and then the second rendered image is obtained according to the operation result.

[0073] Exemplarily, the storage circuit 40B may be an on-chip system cache (SC). In some examples, the first rendered image and the operation result may be stored in different storage areas of the same storage circuit 40B. In some examples, the first rendered image and the operation result may be stored in different storage circuits 40B.

[0074] In some possible implementations, at least one of the multiple imaging parameters of the second rendered image is better than the imaging parameters of the first rendered image. Depending on the imaging parameters, different neural network algorithm processing can be performed in the neural network processor 20B to optimize the corresponding imaging parameters. In some examples, AI super-resolution algorithm processing can be performed by the neural network processor 20B so that the resolution of the second rendered image is greater than the resolution of the first rendered image. In some examples, AI interpolation algorithm processing can be performed by the neural network processor 20B so that the frame rate of the second rendered image is greater than the frame rate of the first rendered image. It should be understood that super-resolution or interpolation are two application scenarios of the present technical solution, and actual applications are not limited to this.

[0075] In some possible implementations, the neural network processor 20B may process the image using a neural network algorithm to obtain a calculation result as an image processing coefficient. The second image processor 10B processes the first rendered image according to the image processing coefficient to obtain a second rendered image.

[0076] Exemplarily, the second image processor 10B is also used to: obtain at least one preprocessed image based on the first rendered image. The first rendered image and at least one preprocessed image are processed according to the image processing coefficient to obtain the second rendered image. In an embodiment of the present application, the second image processor 10B can perform image processing to generate one or more preprocessed images while generating the first rendered image. The preprocessed image is used to assist in optimizing image parameters. After the neural network processor 20B outputs the image processing coefficient, the second image processor 10B can process the first rendered image and at least one preprocessed image according to the image processing coefficient to obtain the second rendered image. In an embodiment of the present application, it is necessary to perform calculations on the relevant image processing coefficients through the neural network processor 20B. The amount of data calculation of the neural network processor 20B is small, and the computing power requirements of the neural network processor 20B are low, and the neural network processor 20B can quickly output the image processing coefficient as the calculation result.

[0077] In some possible implementations, the neural network processor 20B can process the image through the neural network algorithm to obtain a processed image that has been processed by the neural network algorithm to optimize imaging parameters, and output the processed image as a calculation result. The second image processor 10B performs some image rendering post-processing operations on the processed image to obtain a second rendered image.

[0078] Exemplarily, the second image processor 10B is also used to obtain at least one preprocessed image according to the first rendered image. The neural network processor 20B is specifically used to perform neural network algorithm processing on the first rendered image according to at least one preprocessed image to obtain a processed image. In an embodiment of the present application, the second image processor 10B can also perform image processing to generate one or more preprocessed images while generating the first rendered image. The preprocessed image is used to assist in optimizing image parameters. When the neural network processor 20B inputs the first rendered image, at least one preprocessed image is also input. The neural network processor 20B performs neural network algorithm processing on at least one preprocessed image and the first rendered image to obtain a processed image. The processed image is an image after the imaging parameters are optimized. The neural network processor 20B outputs the processed image as a calculation result. In an embodiment of the present application, the neural network processor 20B can directly obtain a processed image after the imaging parameters are optimized by performing a neural network operation, and then output the processed image as a calculation result to the second image processor 10B. The second image processor 10B performs post-processing of the relevant rendered image according to the processed image to obtain a second rendered image. In an embodiment of the present application, the computing power requirements for the neural network processor 20B will be slightly higher. At the same time, the processing time of the neural network processor 20B will be longer, but the processing capacity of the second image processor 10B will be reduced.

[0079] In some possible implementations, each first rendered image is a complete imaging frame image. In an embodiment of the present application, the second image processor 10B generates an initial rendered image according to an image subtask, and each initial rendered image corresponds to a complete imaging frame image. Each initial rendered image is output as a first rendered image. The second image processor 10B and the neural network processor 20B process the first rendered image based on multi-IP linkage to obtain a second rendered image after the imaging parameters are optimized. In this implementation, for each first rendered image, it is necessary to first undergo a transit process from the second image processor 10B to the neural network processor 20B, and then undergo a transit process from the neural network processor 20B to the second image processor 10B. Therefore, in the process of this transit processing, limited by the processing performance of the neural network processor 20B, the second image processor 10B may have a waiting delay for the neural network processor 20B to output the calculation result; or, limited by the processing performance of the second image processor 10B, the neural network processor 20B may have a waiting delay for the second image processor 10B to output the next first rendered image or a waiting delay for the output of the first rendered image corresponding to the next initial rendered image.

[0080] In some possible implementations, multiple first rendered images correspond to a complete imaging frame image. Exemplarily, the second image processor 10B is also used to: generate an initial rendering image according to an image subtask. Multiple first rendered images are obtained according to the initial rendering image; multiple first rendered images correspond to different imaging parts of the initial rendering image. In an embodiment of the present application, the second image processor 10B generates a complete imaging frame image (i.e., an initial rendering image) according to the image subtask. Then the second image processor 10B splits each initial rendering image into multiple sub-rendering images, i.e., multiple first rendering images. Then multiple first rendering images are output. The neural network processor 20B performs a neural network algorithm on each first rendering image in turn and outputs the corresponding operation results. The second image processor 10B receives the operation results corresponding to each first rendering image, and obtains the corresponding second rendering image according to the operation results. Finally, multiple second rendering images corresponding to multiple first rendering images are spliced ​​to obtain a final complete frame image. In some examples, there is no same imaging part between multiple first rendering images corresponding to the same initial rendering image. In some examples, there may be partially the same imaging part between multiple first rendering images corresponding to the same initial rendering image.

[0081] In some possible implementations, such as Figure 7 As shown, the second chip system 100B also includes a processor 50B. An application (APP) is running on the processor 50B. When the application needs to display a rendered image, a rendering command is issued. The processor 50B is used to: output an image subtask, a calculation subtask, and a scheduling subtask in response to a rendering command issued by the application. Among them, at least one of the image subtask, the calculation subtask, or the scheduling subtask is provided with a heterogeneous synchronization signal; the heterogeneous synchronization signal is used to indicate the execution dependency between the image subtask and the scheduling subtask, or the heterogeneous synchronization signal is used to indicate the execution dependency between the calculation subtask and the scheduling subtask. In an embodiment of the present application, multiple subtasks with execution dependencies are output by the processor 50B. In the actual image processing process, the second image processor 10B, the neural network processor 20B, and the scheduling circuit 30B can contact the relevant dependencies of the subtasks to be processed according to the received trigger signal to execute and process the corresponding subtasks.

[0082] In some possible implementations, the processor 50B generates image subtasks, computing subtasks, and scheduling subtasks in different ways depending on the rendering commands issued by the application:

[0083] In some examples, the application manufacturer has coordinated the design of the rendering command. At this time, the rendering command already contains parameters related to multiple subtasks of image processing for multi-IP linkage. In this case: the processor 50B is also used to: generate an image rendering task group in response to the rendering command input by the application; the image rendering task group includes an image subtask, a calculation subtask and a scheduling subtask. In an embodiment of the present application, the application manufacturer can be linked with the hardware manufacturer. When designing the application, the application manufacturer makes design changes to the rendering command issued by the application. The issued rendering command has been adaptively set with parameters of related subtasks that multiple IP hardware needs to perform on its own. Then, according to the rendering command, the corresponding rendering image task group can be directly generated, and the rendering image task group already includes image subtasks, calculation subtasks and scheduling subtasks. At this time, it is only necessary to insert heterogeneous synchronization signals in the corresponding subtasks through the SDK software program. After outputting the image subtask, calculation subtask and scheduling subtask, multiple IPs are linked to realize image processing based on the corresponding subtasks and their heterogeneous synchronization signals. In this solution, the processor 50B does not need to modify or generate some subtasks, thereby reducing the processing workload of the processor 50B.

[0084] In some examples, the application manufacturer did not change the rendering command. In this case, the rendering command does not include parameters related to multiple subtasks for image processing linked by multiple IPs. In this case: the processor 50B is also used to: generate an image rendering task group in response to the rendering command input by the application; the image rendering task group includes image subtasks. Operation subtasks and scheduling subtasks are set in the image rendering task group. In an embodiment of the present application, because the rendering command has not been changed, the image rendering task group generated by the processor 50B according to the rendering command only includes image subtasks related to image rendering, and does not include subtasks related to the neural network processor 20B and the scheduling circuit 30B. It is necessary to set the operation subtasks and scheduling subtasks through the processor 50B.

[0085] In some possible implementations, the application may be a game application, a model design application, a drawing software application, a video editing application, etc.

[0086] In some examples, when the application is running, the second chip system 100B can automatically execute the contents recorded in the above embodiments. In some examples, when the application is running, in response to the user's input, the second chip system 100B starts to execute the contents recorded in the above embodiments. For example, a control switch is provided on the electronic device 1000, and the functions recorded in the above embodiments are turned on or off by the control switch. Alternatively, a human-computer interaction module (such as a display screen 200) is provided on the electronic device 1000, and the functions recorded in the above embodiments are turned on or off by the user's control operation on the human-computer interaction module.

[0087] Based on the above Figure 4 , Figure 5 , Figure 6 and Figure 7 The second chip system 100B of the structure can perform the following steps: Figure 8 The image processing method shown includes the following steps S200-S600:

[0088] S200 : The second image processor 10B outputs a first rendered image.

[0089] In the embodiments of the present application, Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, the second image processor 10B performs image rendering processing according to the image subtask to obtain a first rendered image.

[0090] Exemplarily, when the second image processor 10B outputs the first rendered image, it outputs a first trigger signal to the scheduling circuit 30B according to the image subtask. The first trigger signal is used to instruct the neural network processor 20B to execute the subsequent corresponding scheduling subtask.

[0091] S300, instructing the neural network processor 20B to obtain a first rendered image.

[0092] In some possible implementations, such as Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, the second image processor 10B performs preliminary image rendering and image processing to obtain a first rendered image. Then, the scheduling circuit 30B instructs the neural network processor 20B to obtain the first rendered image according to the scheduling subtask.

[0093] Exemplarily, the scheduling circuit 30B responds to the first trigger signal and outputs a second trigger signal to the neural network processor 20B according to the scheduling subtask; the second trigger signal is used to instruct the neural network processor 20B to obtain the first rendered image.

[0094] S400 , the neural network processor 20B performs neural network algorithm processing on the first rendered image to obtain a calculation result.

[0095] In some possible implementations, such as Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, the neural network processor 20B responds to the instruction of the scheduling circuit 30B to perform neural network algorithm processing on the first rendered image obtained by the preliminary rendering processing of the second image processor 10B, and obtains the operation result through the neural network algorithm processing.

[0096] Exemplarily, the neural network processor 20B obtains the first rendered image according to the second trigger signal, and performs neural network algorithm processing on the first rendered image to obtain a calculation result. When the calculation result is output, the second image processor 10B outputs a third trigger signal to the scheduling circuit 30B according to the image subtask. The third trigger signal is used to instruct the scheduling circuit 30B to execute subsequent related scheduling subtasks.

[0097] S500 , instruct the second image processor 10B to obtain a calculation result.

[0098] In some possible implementations, the scheduling circuit 30B instructs the second image processor 10B to obtain the operation result according to the scheduling subtask.

[0099] Exemplarily, the scheduling circuit 30B responds to the third trigger signal and outputs a fourth trigger signal to the second image processor 10B according to the scheduling subtask. The fourth trigger signal is used to instruct the second image processor 10B to obtain the operation result.

[0100] S600: The second image processor 10B performs image processing on the calculation result to obtain a second rendered image.

[0101] In some possible implementations, the second image processor 10B performs image processing on the operation result according to the image subtask to obtain a second rendered image. The imaging parameters of the second rendered image are better than the imaging parameters of the first rendered image.

[0102] Exemplarily, the second image processor 10B executes the relevant image subtask in response to the fourth trigger signal to perform image processing on the operation result to obtain the second rendered image.

[0103] In the embodiment of the present application, by Figure 8The image processing method shown can realize the linkage between the second image processor 10B and the neural network processor 20B to optimize the imaging parameters of the rendered image. In this processing process, the use of the neural network processor 20B for neural network algorithm processing can improve the imaging parameters of the rendered image, and compared with the method of using a more complex image rendering algorithm in the second image processor 10B to improve the imaging parameters of the rendered image, this solution can reduce the power consumption of the second chip system 100B. At the same time, because the second image processor 10B and the neural network processor 20B are independent of each other, the direct linkage processing between the two requires quite complex data path interaction and control signal interaction on the control link, and the embodiment of the present application schedules the linkage processing between the second image processor 10B and the neural network processor 20B in a hardware scheduling manner by setting a scheduling circuit 30B. In this implementation, there is no need to make complex hardware or software modifications to multiple IPs. Only a few simple trigger signals are needed to control the second image processor 10B and the neural network processor 20B to perform the corresponding subtasks in the agreed order, so as to optimize the imaging parameters of the rendered image through multi-IP linkage.

[0104] In some possible implementations, such as Figure 8 In step S200 of the illustrated embodiment, each first rendered image is a complete imaging frame image.

[0105] In some possible implementations, such as Figure 8 In step S200 of the embodiment shown in FIG. 1 , the plurality of first rendered images correspond to a complete imaging frame image. At this time, step S200 further includes the following steps: Fig. 9 Sub-operations of steps S210 to S220 are shown as follows:

[0106] S210: Generate an initial rendering image according to the image subtask.

[0107] In some possible implementations, the second image processor 10B generates an initial rendering image according to the image subtask. The initial rendering image is a complete imaging frame image.

[0108] S220: Obtain a plurality of first rendered images according to the initial rendered image.

[0109] In the embodiment of the present application, the multiple first rendered images correspond to different imaging parts of the initial rendered image.

[0110] In such Fig. 9 In the embodiment shown, for each first rendered image, the following steps can be performed: Figure 8The operations of step S300 to step S500 are shown. In step S600, a second rendered image corresponding to a complete imaging frame image is obtained according to a plurality of first rendered images corresponding to a complete imaging frame image and the calculation results thereof.

[0111] In the embodiments of the present application, Fig.10 As shown in FIG. 1 , when the first rendered image is a complete imaging frame image, the time delay for optimizing each imaging frame image is shown. Fig.10 It can be seen that first, the second image processor 10B performs the compilation process. Then, the second image processor 10B performs the initial rendering process of step S200 to obtain the first rendered image. Then, the neural network processor 20B performs the neural network algorithm processing of step S400 to obtain the operation result. Then, the second image processor 10B performs the subsequent rendering processing operation of step S600, and obtains the second rendered image after parameter optimization according to the operation result. At this time, the processing delay of each complete imaging frame image is about 19.5ms. Fig.11 As shown, Fig. 9 The process flow diagram corresponding to the embodiment shown in FIG. Fig. 9 In the embodiment shown, when the initial rendering image of the complete imaging frame image is split into multiple first rendering images, the operations of step S300 to step S500 are sequentially performed on the multiple first rendering images. Then, in step S600, a second rendering image is obtained based on the multiple first rendering images corresponding to the same initial rendering image and the operation result. Taking the example of splitting an initial rendering image into four first rendering images, in the example Fig.11 In the schematic diagram shown, the four first rendered images are processed in a pipeline manner, which reduces the processing delay by 2ms and improves the delay benefit by about 10.3%. Fig. 9 and Fig.11 In the embodiment shown, the processing performance of the second image processor 10B and the neural network processor 20B can be more reasonably utilized to reduce the waiting delay for waiting processing as much as possible.

[0112] In some possible implementations, the image processing method further includes: the second image processor 10B obtains at least one pre-processed image according to the first rendered image. In some examples, the at least one pre-processed image may be a pre-processed image obtained according to the first rendered image in step S100. In some examples, the at least one pre-processed image may be Fig.10 and Fig.11In the illustrated embodiment, the second image processor 10B performs some image processing operations in parallel when the neural network processor 20B performs step S400 to obtain an image. In some examples, at least one pre-processed image may be a pre-processed image obtained by directly processing the first rendered image. In some examples, at least one pre-processed image may be a pre-processed image obtained by processing the initial rendered image and then splitting the pre-processed initial rendered image according to the distribution relationship of the multiple first rendered images on the initial rendered image.

[0113] In some possible implementations, such as Fig.10 and Fig.11 As shown, when the neural network processor 20B performs the operation of step S400, the second image processor 10B may also perform some image processing operations in parallel, and process the first rendered image or the initial rendered image or at least one pre-processed image through the image processing operation to obtain some adjusted images, etc. For example, the image processing operation may be binning image processing.

[0114] In some possible implementations, in step S400, different calculation results are obtained according to different neural network algorithms.

[0115] In some examples, the operation result is an image processing coefficient. At this time, the specific operation of step S600 is: processing the first rendered image and at least one pre-processed image according to the image processing coefficient to obtain a second rendered image. In an embodiment of the present application, the operation result obtained by the neural network algorithm processing performed in step S400 is an image processing coefficient. Under this implementation, the data processing amount of the neural network processor 20B is small, and the computing power requirements and processing delay requirements of the neural network processor 20B are not high. In step S600, the second image processor 10B performs image processing on the obtained first rendered image and at least one pre-processed image according to the image processing coefficient. For example, when the image processing coefficient is a fusion coefficient, the first rendered image and at least one pre-processed image are fused based on the image processing coefficient to obtain a second rendered image with super resolution.

[0116] In some examples, the operation result is a processed image. The processed image is an image after the imaging parameters are optimized after the neural network processor 20B performs neural network algorithm processing in step S400. At this time, the specific operation of step S400 is: the neural network processor 20B performs neural network algorithm processing on the first rendered image according to at least one pre-processed image to obtain a processed image. The specific operation of step S600 is: the second image processor 10B performs post-rendering processing on the processed image as the operation result to obtain a second rendered image. In an embodiment of the present application, the neural network processor 20B directly generates an image after parameter optimization in the operation of step S400, and then the second image processor 10B performs post-rendering processing on the generated processed image to obtain the final second rendered image. Under this implementation, the data processing amount of the second image processor 10B is reduced.

[0117] In the embodiment of the present application, according to the actual application, the performance difference and processing delay difference between the second image processor 10B and the neural network processor 20B, etc., any one of the above-mentioned forms of operation results can be selected as the actual implementation plan.

[0118] For example, Fig.12 As shown, the second image processor 10B generates an initial rendering image through an image subtask. The initial rendering image is processed to obtain a first rendering image and at least one pre-processed image. The neural network processor 20B processes the first rendering image using a neural network algorithm to obtain a calculation result of an image processing coefficient. The second image processor 10B obtains a second rendering image based on the initial rendering image (or the first rendering image), the image processing coefficient (i.e., the calculation result) and at least one pre-processed image. In some examples, the second image processor 10B may also use previous historical frame images and historical frame data in the process of obtaining the first rendering image and / or obtaining at least one pre-processed image. The second image processor 10B may also use previous historical frame images in the process of obtaining the second rendering image.

[0119] Exemplarily, at least one pre-processed image may be an image directly processed by the first rendered image, or may be an image indirectly obtained by the first rendered image. In some examples, at least one pre-processed image may include an upsampling image, which is an upsampled large image without offset obtained by upsampling the first rendered image (or the corresponding initial rendered image) based on the Jitter offset coordinates. In some examples, at least one pre-processed image may include a Warp image, which is an image obtained by performing a Warp operation on the preceding complete frame image corresponding to the first rendered image based on the motion vector (MV) and depth information.

[0120] In some possible implementations, at least one of the following parameters may also be used in the processing of step S200, step S400 or step S600: Jitter offset coordinates of the current frame, color data of the current frame, color data of the historical frame (or previous frame), motion vector, depth information and neural network filter (NNF) parameters, etc.

[0121] The above description of the neural network algorithm processing is only an example. In actual applications, depending on the neural network algorithm adopted, a corresponding type of parameter image can be generated as at least one pre-processed image. These pre-processed images can be used in the neural network processor 20B, and can also be used in the second image processor 10B. It is also possible to add some other image processing parameters to assist in the processing when performing image processing. The embodiment of the present application does not limit the neural network algorithm adopted. In theory, any algorithm for optimizing imaging parameters can be adaptively applied to the scheme recorded in the embodiment of the present application.

[0122] In some possible implementations, such as Fig.13 As shown, the image processing method further includes the operation of step S100:

[0123] S100, output image subtask, calculation subtask and scheduling subtask.

[0124] In the embodiments of the present application, Figure 7 As shown, the processor 50B outputs an image subtask, a computing subtask, and a scheduling subtask respectively. At least one of the image subtask, computing subtask, or scheduling subtask is provided with a heterogeneous synchronization signal; the heterogeneous synchronization signal is used to indicate the execution dependency between the image subtask and the scheduling subtask, or the heterogeneous synchronization signal is used to indicate the execution dependency between the computing subtask and the scheduling subtask.

[0125] For example, Fig.14As shown, an application and a driver are running in the processor 50B. In the driver, there are call programs corresponding to different IP cores, such as an image call program, an operation call program, a heterogeneous call program, and a memory allocation program. The application issues a rendering command. According to the rendering command, the application programming interface (API) required for the relevant task processing is called to call the driver. According to the called API interface, the driver encapsulates and sends the image subtask to the second image processor 10B through the image call program, encapsulates and sends the operation subtask to the neural network processor 20B through the operation call program, and encapsulates and sends the scheduling subtask to the scheduling circuit 30B through the heterogeneous call program. In this process, when it is necessary to assist in the storage of data such as the first rendered image and the operation result based on the storage circuit 40B, a storage subtask can also be sent to the storage circuit 40B through the memory allocation program to allocate corresponding storage space to different IP hardware.

[0126] Exemplarily, the API interface may include GLES service, Vulkan service, etc.

[0127] For example, taking the optimized image parameter as resolution as an example, the driver may be a super resolution SDK program. A deep learning super sampling (DLSS) driver may be designed as a super resolution SDK.

[0128] In some examples, a heterogeneous synchronization signal is provided in the image subtask, and the heterogeneous synchronization signal can be used to instruct the second image processor 10B to output a first trigger signal when executing the output of the first rendered image, and trigger the scheduling circuit 30B to perform the relevant scheduling subtask (for example, output the second trigger signal) through the first trigger signal. Alternatively, the heterogeneous synchronization signal is used to instruct the second image processor 10B to perform the relevant image subtask when receiving the fourth trigger signal to output the second rendered image according to the operation result. In some examples, a heterogeneous synchronization signal is provided in the scheduling subtask, and the heterogeneous synchronization signal is used to instruct the scheduling circuit 30B to perform the relevant scheduling subtask to output the second trigger signal to the neural network processor 20B when the first trigger signal is received. The heterogeneous synchronization signal can also be used to instruct the scheduling circuit 30B to perform the relevant scheduling subtask to output the fourth trigger signal to the second image processor 10B when the third trigger signal is received. In some examples, a heterogeneous synchronization signal is set in the operation subtask, which can be used to instruct the neural network processor 20B to execute related operation subtasks (for example, performing AI algorithm processing on the first rendered image) when receiving the third trigger signal, or to instruct the neural network processor 20B to output a third trigger signal to the scheduling circuit 30B when outputting the operation result.

[0129] In some examples, a software development kit (SDK) may be provided in the processor 50B. The SDK generates an API logic diagram according to the API interface called by the rendering command, confirms the processing execution dependency of different subtasks according to the API logic diagram, and inserts heterogeneous synchronization information into the corresponding subtask according to the determined processing execution dependency.

[0130] In some possible implementations, the processor 50B generates image subtasks, computing subtasks, and scheduling subtasks in different ways depending on the rendering commands issued by the application:

[0131] In some examples, the application manufacturer has coordinated the design of the rendering command. At this time, the rendering command already contains parameters related to multiple subtasks of image processing for multi-IP linkage. At this time, step S100 includes the following: Fig.15 The sub-operations of step S110 to step S130 shown are:

[0132] S110: Generate an image rendering task group according to the rendering command.

[0133] In the embodiment of the present application, since the application manufacturer has coordinated the design of the rendering command, the processor 50B can directly call the relevant API interface according to the rendering command to encapsulate and generate the corresponding image subtasks, calculation subtasks, scheduling subtasks and storage subtasks, etc.

[0134] S120: inserting a heterogeneous synchronization signal into the image rendering task.

[0135] In an embodiment of the present application, after an image rendering task group including subtasks such as image subtasks, calculation subtasks, scheduling subtasks and storage subtasks is generated according to a rendering command, the image rendering task group does not include execution dependencies between subtasks executed by different IPs, and heterogeneous synchronization signals need to be inserted into the corresponding subtasks to construct execution dependencies between subtasks executed by different IPs.

[0136] S130, output image subtask, calculation subtask, scheduling subtask and storage subtask.

[0137] In this application Fig.15 In the illustrated embodiment, the technical principles and technical effects of the embodiment in which the application manufacturer has coordinated the design of the rendering commands can be referred to the relevant description in the aforementioned hardware embodiment, and will not be repeated here.

[0138] In some examples, the application manufacturer does not modify the rendering command. In this case, the rendering command does not include parameters related to multiple subtasks of image processing for multi-IP linkage. In this case, step S100 includes the following: Fig.16 The sub-operations of step S110' to step S140' are shown as follows:

[0139] S110', generating an image rendering task group according to the rendering command.

[0140] In the embodiment of the present application, because the application manufacturer has not adapted the rendering command, the processor 50B generates an image rendering task group in response to the rendering command input by the application, but the image rendering task group includes image subtasks, but does not include operation subtasks and scheduling subtasks.

[0141] S120', setting a computing subtask and a scheduling subtask in the image rendering task group.

[0142] Exemplarily, setting a calculation subtask in the image rendering task group includes at least one of the following operations: modifying at least one image subtask into a calculation subtask, or adding a new calculation subtask to the image rendering task group. In an embodiment of the present application, because the second image processor 10B and the neural network processor 20B are required to perform multi-IP linkage. Therefore, some image subtasks that originally required the second image processor 10B to execute may be assigned to the neural network processor 20B for execution. At this time, some image subtasks can be modified and encapsulated into calculation subtasks. At the same time, compared with traditional image rendering processing, neural network algorithm processing is added. Therefore, it is also necessary to add some new calculation subtasks to instruct the neural network processor 20B to perform corresponding neural network algorithm processing, as well as related coordinated scheduling operations, etc.

[0143] Exemplarily, in order to ensure fast and easy coordinated processing between the second image processor 10B and the neural network processor 20B, some new scheduling subtasks need to be added to instruct the scheduling circuit 30B to perform corresponding scheduling operations.

[0144] S130', inserting a heterogeneous synchronization signal into the image rendering task.

[0145] S140', output image subtask, calculation subtask, scheduling subtask and storage subtask.

[0146] For the related description of step S130' and step S140', reference may be made to the related description of the aforementioned step S120 and step S130, which will not be repeated here.

[0147] In this application Fig.16In the illustrated embodiment, because the rendering command has not been changed, the image rendering task group generated by the processor 50B according to the rendering command only includes image subtasks related to image rendering, and does not include subtasks related to the neural network processor 20B and the scheduling circuit 30B. It is necessary to set the operation subtask and the scheduling subtask through the processor 50B. Under this embodiment, there is no need to change the application, and the application can be seamless, that is, different applications can run normally and follow the rendering commands, and the SDK intercepts the API interface of the relevant image subtask, and performs the encapsulation and modification to the operation subtask, the insertion of heterogeneous synchronization signals, etc. This application is as follows Fig.16 The illustrated embodiment can be adapted to different applications.

[0148] Exemplarily, the SDK may intercept the API interface in the following ways: field identification, view point identification, etc.

[0149] For example, the application is a game application, and the optimized image parameter is resolution. The driver in the processor 50B is a super-resolution SDK. After the super-resolution SDK starts working, it intercepts the API interface (such as GLES service and Vulkan service) called by the game application through rendering instructions. Generate a logical graph of the API interface. Determine the execution order between different subtasks according to the logical graph, set operation subtasks in the corresponding processing flow, and then insert heterogeneous synchronization signals in the corresponding subtasks to generate execution dependencies between different subtasks.

[0150] In some possible implementations, a hardware trigger circuit may be provided in the second image processor 10B to implement the generation of the first trigger signal in step S200 and to implement the start of the execution of the related image subtask of generating the second rendered image in response to the fourth trigger signal in step S600.

[0151] In some possible implementations, the generation of the first trigger signal in step S200 can be implemented by software in the second image processor 10B, and the start of the execution of the related image subtask for generating the second rendered image in response to the fourth trigger signal in step S600 can be implemented by software control. Exemplarily, the driver in the processor 50B encapsulates the recording command list (CmdList) and constructs a function entry (entry) according to the rendering command issued by the application, and encapsulates the related application programming interface (application programming interface, API) required for the execution of the image subtask through the function entry. The execution dependency is inserted into the related image subtask in the entry buffer (entry buffer) in the form of heterogeneous synchronization information, for example, in the form of a waiting execution signal HTS_wait_instruction and a notification execution signal HTS_notify_instruction. Fig.17As shown, the driver development kit (DDK) uploads the service to the second image processor 10B. The task processing queue program (QueueM) of the second image processor 10B reads back the entry buffer to process the synchronization relationship between different entries. When the second image processor 10B executes a certain image subtask (for example, outputting a first rendered image), the dependency on outputting the first trigger signal is released, and the first trigger signal is a notification type signal, namely HTS_notify_instruction. At this time, the second image processor 10B sends the first trigger signal to the scheduling circuit 30 through the internal trigger notification signal driver (Unique SignalEngine). The first trigger signal can be sent in the form of an event identifier (EventID). Similarly, the scheduling circuit 30B and the neural network processor 30B also have related heterogeneous synchronization signals. In response to the first trigger signal, the scheduling circuit 30B executes subsequent scheduling subtasks and schedules the neural network processor 30B to perform related calculation subtasks. Then, after the neural network processor 30B completes the execution of the relevant operation subtask, the scheduling circuit 30B can actively send a fourth trigger signal to the second image processor 10B, and the fourth trigger signal is also sent in the form of an event identifier. Inside the second image processor 10B, an event queue (event MBX) is set according to the entry buffer. In the event queue, a corresponding reference count (refcnt) is set for the enrty of each image subtask. When the second image processor 10B can execute subsequent image subtasks (for example, it can start to execute the operation of generating the second rendering image in step S600) after processing other matters, the event queue is read back through the built-in trigger waiting driver (wait engine). When the fourth trigger signal is received, the corresponding HTS_wait_instruction dependency is released, and the event reference count value of the corresponding image subtask is 0, that is, EventID Refcnt! = 0. When the waiting driver reads back the corresponding reference count value of 0, it means that the dependency of the subsequent image subtask is released, and the subsequent image subtask can be executed, for example, the image subtask generating the second rendering image in step S600 is executed. APB is a data transmission bus module. IRQ is an interrupt signal sending module. The count control module is used to adjust the technology in event MBX, etc.

[0152] about Fig.13 , Fig.14 , Fig.15 , Fig.16 and Fig.17The technical principle and technical effect of performing the operations of step S200 to step S600 after performing the operation of step S100 in the embodiment shown in the figure can be referred to in the foregoing Figure 8 , Fig. 9 , Fig.10 , Fig.11 and Fig.12 The relevant descriptions in the embodiments of the present invention will not be repeated here.

[0153] The embodiment of the present application provides a chip system, an image processing method and an electronic device. First, the chip system includes a hardware scheduling circuit, an image processor and a neural network processor. The scheduling circuit is used to instruct the image processor and the neural network processor to perform multi-IP linkage processing to generate a rendered image with better imaging parameters. The specific operation of the multi-IP linkage processing is: the image processor performs an image subtask to render the imaged image to obtain a first rendered image. The neural network processor performs a neural network algorithm on the first rendered image to obtain a calculation result to complete the calculation operation of parameter optimization. Finally, the image processor obtains a second rendered image based on the calculation result. One or more imaging parameters of the second rendered image are better than the first rendered image. First, the neural network processor and the image processor are independent devices, and the data path interaction and control signal interaction between the two are quite complex. And the software control logic of the two is relatively simple. The embodiment of the present application can realize multi-IP linkage image processing between the neural network processor and the image processor in a simple control manner by setting a hardware scheduling circuit. Secondly, the present application uses a neural network processor to assist in improving the imaging quality of the rendered image based on a neural network processing algorithm. Compared with the operation of using a more complex image rendering algorithm in the image processor to improve the imaging quality, the embodiment of the present application can greatly reduce the complexity of image processing and improve the imaging quality. Second, the embodiment of the present application can split a complete imaging frame into multiple first rendered images. By performing multi-IP linkage processing operations on multiple first rendered images in a pipeline manner by the image processor and the neural network processor, the processing performance of the image processor and the neural network processor can be effectively utilized, and the processing delay of each complete frame can be reduced. Third, the chip system also includes a processor. The processor outputs image subtasks, scheduling subtasks, and operation subtasks according to the rendering commands issued by the application. In the embodiment of the present application, a corresponding SDK can be set in the processor, and the application can call the relevant image processing API interface through conventional rendering commands without modifying the issued rendering commands. The set SDK intercepts the relevant API interface, modifies some image subtasks, and adds some new image subtasks, operation subtasks, and scheduling subtasks. By inserting corresponding heterogeneous synchronization signals according to the processing order of the subtasks when encapsulating each subtask, and forming execution dependency relationships between different subtasks through heterogeneous synchronization signals, the embodiment of this solution can be adapted to different applications.

[0154] The present application also provides a computer-readable storage medium, which includes instructions; when the instructions are in the electronic device (for example, including Figure 4 , Figure 5 , Figure 6 and Figure 7 When the electronic device is operated on an electronic device having the structure shown in FIG. 1 , the electronic device executes the image processing method described in the above embodiment (for example, Figure 8 , Fig. 9 , Fig.13 , Fig.15 and Fig.16 The image processing method described herein).

[0155] The processor involved in the embodiments of the present application may be a chip. For example, it may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0156] The memory involved in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0157] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0158] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0161] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one device or distributed on multiple devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0162] In addition, each functional module in each embodiment of the present application may be integrated into one device, or each module may exist physically separately, or two or more modules may be integrated into one device.

[0163] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing a computer program instruction on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may contain one or more servers, data centers and other data storage devices that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0164] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A chip system, characterized in that: It includes an image processor, a neural network processor and a scheduling circuit; wherein: The image processor is used to: output a first rendered image according to the image subtask; The scheduling circuit is used to: instruct the neural network processor to obtain the first rendering image according to the scheduling subtask; The neural network processor is used to: perform neural network algorithm processing on the first rendered image according to the operation subtask and output the operation result; The scheduling circuit is further used to: instruct the image processor to obtain the operation result according to the scheduling subtask; The image processor is further configured to: perform image processing on the operation result according to the image subtask to obtain a second rendered image.

2. The chip system according to claim 1, characterized in that: The image processor is further configured to: when outputting the first rendered image, output a first trigger signal to the scheduling circuit according to the image subtask; The scheduling circuit is specifically used to: in response to the first trigger signal, output a second trigger signal to the neural network processor according to the scheduling subtask; the second trigger signal is used to instruct the neural network processor to obtain the first rendering image; The image processor is further configured to: when outputting the operation result, output a third trigger signal to the scheduling circuit according to the image subtask; The scheduling circuit is specifically used to: in response to the third trigger signal, output a fourth trigger signal to the image processor according to the scheduling subtask; the fourth trigger signal is used to instruct the image processor to obtain the operation result.

3. The chip system according to claim 2, characterized in that: The chip system also includes a storage circuit; wherein: The image processor is specifically configured to: write the first rendered image into the storage circuit; and obtain the operation result from the storage circuit in response to the fourth trigger signal; The neural network processor is specifically used to: respond to the second trigger signal, read the first rendered image from the first buffer according to the operation subtask; and write the operation result into the storage circuit.

4. The chip system according to any one of claims 1 to 3, characterized in that: The image processor is also used for: According to the image subtask, generating an initial rendering image; A plurality of first rendering images are obtained according to the initial rendering image; the plurality of first rendering images correspond to different imaging parts of the initial rendering image.

5. The chip system according to any one of claims 1 to 4, characterized in that: The operation result is an image processing coefficient; the image processor is also used for: Obtain at least one pre-processed image according to the first rendered image; The first rendered image and the at least one pre-processed image are processed according to the image processing coefficient to obtain the second rendered image.

6. The chip system according to any one of claims 1 to 4, characterized in that: The operation result is a processed image; The image processor is further used to: obtain at least one pre-processed image according to the first rendered image; The neural network processor is specifically used to: perform neural network algorithm processing on the first rendered image according to the at least one pre-processed image to obtain the processed image.

7. The chip system according to any one of claims 1 to 6, characterized in that: The chip system also includes a processor; the processor is used for: Output the image subtask, the operation subtask and the scheduling subtask; at least one of the image subtask, the operation subtask or the scheduling subtask is provided with a heterogeneous synchronization signal; the heterogeneous synchronization signal is used to indicate the execution dependency between the image subtask and the scheduling subtask, or the heterogeneous synchronization signal is used to indicate the execution dependency between the operation subtask and the scheduling subtask.

8. The chip system according to claim 7, characterized in that The processor is further configured to: In response to a rendering command input by an application, an image rendering task group is generated; the image rendering task group includes the image subtask, the calculation subtask and the scheduling subtask.

9. The chip system according to claim 7, characterized in that: The processor is further configured to: In response to a rendering command input by an application, an image rendering task group is generated; the image rendering task group includes the image subtask; The calculation subtask and the scheduling subtask are set in the image rendering task group.

10. The chip system according to claim 9, characterized in that: The processor is specifically configured to perform at least one of the following: At least one of the image subtasks is modified into the computing subtask, or a new computing subtask is added to the image rendering task group.

11. The chip system according to claim 9 or 10, characterized in that: The processor is further configured to: Adding a new image subtask to the image rendering task group.

12. The chip system according to any one of claims 1 to 11, characterized in that: The second rendered image satisfies at least one of the following: A resolution of the second rendered image is greater than a resolution of the first rendered image, or a frame rate of the second rendered image is greater than a frame rate of the first rendered image.

13. An image processing method, characterized in that: Based on a chip system; the chip system includes an image processor, a neural network processor and a scheduling circuit; the method includes: The image processor outputs a first rendered image according to the image subtask; The scheduling circuit instructs the neural network processor to obtain the first rendered image according to the scheduling subtask; The neural network processor performs neural network algorithm processing on the first rendered image according to the operation subtask and outputs the operation result; The scheduling circuit further instructs the image processor to obtain the operation result according to the scheduling subtask; The image processor also performs image processing on the operation result according to the image subtask to obtain a second rendered image.

14. The image processing method according to claim 13, characterized in that: The method further comprises: When outputting the first rendered image, the image processor outputs a first trigger signal to the scheduling circuit according to the image subtask; The scheduling circuit outputs a second trigger signal to the neural network processor in response to the first trigger signal according to the scheduling subtask; the second trigger signal is used to instruct the neural network processor to obtain the first rendering image; When outputting the operation result, the image processor outputs a third trigger signal to the scheduling circuit according to the image subtask; The scheduling circuit responds to the third trigger signal and outputs a fourth trigger signal to the image processor according to the scheduling subtask; the fourth trigger signal is used to instruct the image processor to obtain the operation result.

15. The image processing method according to claim 13 or 14, characterized in that: The method further comprises: The image processor generates an initial rendering image according to the image subtask; obtains a plurality of first rendering images according to the initial rendering image; and the plurality of first rendering images correspond to different imaging parts of the initial rendering image.

16. The image processing method according to any one of claims 13 to 15, characterized in that: The operation result is an image processing coefficient; The method further includes: obtaining, by the image processor, at least one pre-processed image according to the first rendered image; Outputting the second rendered image according to the operation result includes: the image processor processes the first rendered image and the at least one pre-processed image according to the image processing coefficient to obtain the second rendered image.

17. The image processing method according to any one of claims 13 to 15, characterized in that: The operation result is a processed image; The method further includes: obtaining, by the image processor, at least one pre-processed image according to the first rendered image; The performing neural network algorithm processing on the first rendered image to obtain a calculation result includes: the neural network processor performing neural network algorithm processing on the first rendered image according to the at least one pre-processed image to obtain the processed image.

18. The image processing method according to any one of claims 13 to 17, characterized in that: The method further comprises: Output the image subtask, the operation subtask and the scheduling subtask; at least one of the image subtask, the operation subtask or the scheduling subtask is provided with a heterogeneous synchronization signal; the heterogeneous synchronization signal is used to indicate the execution dependency between the image subtask and the scheduling subtask, or the heterogeneous synchronization signal is used to indicate the execution dependency between the operation subtask and the scheduling subtask.

19. The image processing method according to claim 18, characterized in that: The method further comprises: In response to a rendering command input by an application, an image rendering task group is generated; the image rendering task group includes the image subtask, the calculation subtask and the scheduling subtask.

20. The image processing method according to claim 18, characterized in that: The method further comprises: In response to a rendering command input by an application, an image rendering task group is generated; the image rendering task group includes the image subtask; The calculation subtask and the scheduling subtask are set in the image rendering task group.

21. The image processing method according to claim 20, characterized in that: The step of setting the computing subtask in the image rendering task group includes: Perform at least one of the following operations: modify at least one of the image subtasks to be the computing subtask, or add a new computing subtask to the image rendering task group.

22. The image processing method according to claim 20 or 21, characterized in that: The method further comprises: Insert the new image subtask into the image rendering task group.

23. The image processing method according to any one of claims 13 to 22, characterized in that: The second rendered image satisfies at least one of the following: A resolution of the second rendered image is greater than a resolution of the first rendered image, or a frame rate of the second rendered image is greater than a frame rate of the first rendered image.

24. An electronic device, characterized in that: It comprises a circuit board and a chip system as described in any one of claims 1 to 12; the chip system is arranged on the circuit board.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions; when the instructions are executed on the electronic device as claimed in claim 24, the electronic device executes the image processing method as claimed in any one of claims 13-23.

Images