Graph optimization method and device, equipment and storage medium

Abstract

According to the graph optimization method and device, the equipment and the storage medium, high-flexibility graph optimization is realized, and the performance overhead of graph optimization is reduced. The method comprises the following steps: acquiring a to-be-optimized image; performing image optimization processing on the to-be-optimized image through the joint network to obtain a target optimized image; wherein the joint network comprises a target enhanced network deployed in a script layer and a target operator deployed in a primary layer; the target enhancement network is used for carrying out image optimization operation on an input image; the target operator is used for carrying out image processing on a plurality of image blocks of the input image in parallel.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to an image optimization method, apparatus, device, and storage medium. Background Art

[0002] An image enhancement network is a deep learning-based network mainly used to improve the visual quality of low-resolution or low-quality images, which can meet different image optimization requirements of users in different application scenarios and improve the user experience.

[0003] However, the above method is not flexible, resulting in a relatively large performance overhead for image optimization. Summary of the Invention

[0004] This application provides an image optimization method, apparatus, device, and storage medium, which realizes highly flexible image optimization and reduces the performance overhead of image optimization.

[0005] In a first aspect, an image optimization method is provided, which is applied to an electronic device. The method includes: obtaining an image to be optimized; performing image optimization processing on the image to be optimized through a joint network to obtain a target optimized image; where the joint network includes a target enhancement network deployed on a script layer and a target operator deployed on a native layer; the target enhancement network is used to perform image optimization operations on the input image; the target operator is used to perform image processing on multiple image blocks of the input image in parallel.

[0006] In this application, an electronic device can obtain an image to be optimized and perform image optimization processing on the image to be optimized through a joint network to obtain a target optimized image; where the joint network includes a target enhancement network deployed on a script layer and a target operator deployed on a native layer; the target enhancement network is used to perform image optimization operations on the input image; the target operator is used to perform image processing on multiple image blocks of the input image in parallel. Compared with the related art method of integrating the target operator into the image enhancement network, in the embodiment of this application, the target operator is implemented on the native layer, which can avoid allocating independent memory for each operator on the script layer, effectively reduce the memory occupation of the joint network during the image optimization process, and during the process of the target operator performing image processing on multiple image blocks of the input image in parallel, in addition to reducing the time consumption of the joint network for image optimization, it can also batch load the data of the above multiple image blocks from memory to avoid frequent access to memory, so as to reduce the power consumption overhead of the joint network. In other words, highly flexible image optimization is realized through the above joint network, and the performance overhead of image optimization is reduced.

[0007] In a second aspect, a graph optimization device is provided, which is applied to an electronic device. The graph optimization device includes: an acquisition module and an optimization module. The acquisition module is configured to acquire an image to be optimized. The optimization module is configured to perform image optimization processing on the image to be optimized through a joint network to obtain a target optimized image. The joint network includes a target enhancement network deployed in the script layer and a target operator deployed in the native layer. The target enhancement network is configured to perform image optimization operations on the input image. The target operator is configured to perform image processing on multiple image blocks of the input image in parallel.

[0008] In a third aspect, an electronic device is provided, including a processor, which is coupled to a memory and can be used to execute instructions in the memory to implement the method in any possible implementation manner of the first aspect above. Optionally, the electronic device further includes a memory. Optionally, the electronic device further includes a communication interface, and the processor is coupled to the communication interface.

[0009] In a fourth aspect, a processor is provided, including: an input circuit, an output circuit, and a processing circuit. The processing circuit is configured to receive a signal through the input circuit and transmit the signal through the output circuit, so that the processor executes the method in any possible implementation manner of the first aspect above.

[0010] In a specific implementation process, the above-mentioned processor may be a chip, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be transistors, gate circuits, flip-flops, and various logic circuits, etc. The input signal received by the input circuit may be received and input by, for example, but not limited to, a receiver. The signal output by the output circuit may be output to, for example, but not limited to, a transmitter and transmitted by the transmitter, and the input circuit and the output circuit may be the same circuit, which serves as the input circuit and the output circuit at different times respectively. The embodiments of the present application do not limit the specific implementation manners of the processor and various circuits.

[0011] In a fifth aspect, a processing device is provided, including a processor and a memory. The processor is configured to read instructions stored in the memory and can receive a signal through a receiver and transmit the signal through a transmitter to execute the method in any possible implementation manner of the first aspect above.

[0012] Optionally, there is one or more processors, and there is one or more memories.

[0013] Optionally, the memory may be integrated with the processor, or the memory is separately provided from the processor.

[0014] In the specific implementation process, the memory can be a non-transitory memory, such as a read only memory (ROM), which can be integrated with the processor on the same chip or can be separately arranged on different chips. The embodiments of the present application do not limit the type of the memory and the arrangement manner of the memory and the processor.

[0015] It should be understood that related data interaction processes such as sending indication information can be a process of outputting indication information from the processor, and receiving capability information can be a process of the processor receiving input capability information. Specifically, the processed output data can be output to the transmitter, and the input data received by the processor can come from the receiver. Among them, the transmitter and the receiver can be collectively referred to as the transceiver.

[0016] The processing device in the above fifth aspect can be a chip. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0017] In a sixth aspect, there is provided a computer program product, which includes a computer program (which can also be referred to as code or instruction). When the computer program is run, it causes the computer to execute the method in any one of the possible implementation manners in the above first aspect.

[0018] In a seventh aspect, there is provided a computer-readable storage medium, which stores a computer program (which can also be referred to as code or instruction). When it runs on a computer, it causes the computer to execute the method in any one of the possible implementation manners in the above first aspect. Description of the Drawings

[0019] Figure 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of the system architecture of an electronic device provided by an embodiment of the present application;

[0021] Figure 3 is a schematic flowchart of a graph optimization method provided by an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of the network structure provided by an embodiment of the present application;

[0024] Figure 6 is a schematic flowchart of the first specific example of the graph optimization method provided by an embodiment of the present application;

[0025] Figure 7 is a schematic flowchart of the second specific example of the graph optimization method provided by an embodiment of the present application;

[0026] Figure 8 is a schematic flowchart of the third specific example of the graph optimization method provided by an embodiment of the present application;

[0027] Figure 9 is a schematic block diagram of a graph optimization device provided by an embodiment of the present application;

[0028] Figure 10 is a schematic block diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0029] Next, the technical solutions in the present application will be described with reference to the accompanying drawings.

[0030] For the convenience of clearly describing the technical solutions of the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish identical items or similar items with basically the same functions and effects. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and terms such as "first" and "second" do not necessarily mean different.

[0031] It should be noted that in the present application, words such as "exemplarily" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplarily" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplarily" or "for example" aims to present relevant concepts in a specific manner.

[0032] In addition, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item)" or its similar expression below refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, and c can represent: a, or b, or c, or a and b, or a and c, or b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0033] To make the objectives and technical solutions of this application clearer and more intuitive, the following will, in conjunction with the accompanying drawings and embodiments, elaborate in detail on the graph optimization method, apparatus, device, and storage medium provided by this application. It should be understood that the specific embodiments described herein are merely used to explain this application and are not used to limit this application.

[0034] Figure 1 It is a schematic diagram of application scenario 100 provided by this application. As Figure 1 shown, an image enhancement network 102 is deployed in the electronic device 101. The electronic device 101 can input the image to be optimized into the image enhancement network 102 to obtain an optimized image. Among them, the image enhancement network 102 is a deep learning model that can be used to automatically optimize image quality, such as denoising, super-resolution, color correction, low-light enhancement, etc. It is usually composed of multiple neural network layers. The neural network layer is a logical module composed of operators, and the operator is the basic computing unit for performing specific mathematical operations (such as convolution, pooling, matrix multiplication). In other words, the above image to be optimized is the original input image processed by the above image enhancement network 102, which is usually an image with certain quality problems or requires feature enhancement. The optimized image is a high-quality image obtained by the electronic device 101 through the optimization process of the above image enhancement network 102. The improvement direction of the optimization process of the above image enhancement network 102 for the above image to be optimized corresponds to the problems of the image to be optimized.

[0035] However, the above graph optimization method is not flexible, resulting in a large problem of graph optimization performance overhead. The image enhancement network in the electronic device is usually deployed in a quantized manner using its original network structure without considering the design of the electronic device, which may lead to problems such as long operation time, relatively high memory and power consumption during the graph optimization process through this image enhancement network.

[0036] The embodiments of the present application provide an image optimization method, apparatus, device, storage medium. An electronic device can obtain an image to be optimized and perform image optimization processing on the image to be optimized through a joint network to obtain a target optimized image. Among them, the joint network includes a target enhancement network deployed on the script layer and a target operator deployed on the native layer. The target enhancement network is used to perform image optimization operations on the input image. The target operator is used to perform image processing on multiple image blocks of the input image in parallel. Compared with the related technology of integrating the target operator into the image enhancement network, in the embodiments of the present application, the target operator is implemented on the native layer, which can avoid allocating independent memory for each operator on the script layer, effectively reduce the memory occupation of the joint network during the graph optimization process. Moreover, during the process of the target operator performing image processing on multiple image blocks of the input image in parallel, in addition to reducing the time consumption of the joint network for graph optimization, it can also batch load the data of the above multiple image blocks from the memory to avoid frequent access to the memory, so as to reduce the power consumption overhead of the joint network. In other words, high-flexibility graph optimization is achieved through the above joint network, and the performance overhead of graph optimization is reduced.

[0037] The electronic device involved in the embodiments of the present application can be a mobile phone, a watch, a laptop computer, a palm computer, a mobile internet device (MID), a personal computer (PC), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in self-driving, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a personal digital assistant (PDA), etc. The embodiments of the present application do not limit this.

[0038] Exemplarily, Figure 2 It is a schematic diagram of the system architecture of an electronic device provided by the embodiments of the present application.

[0039] As Figure 2 shown, the electronic device includes a processor 210, a transceiver 220, and a display unit 270. Among them, the display unit 270 may include a display screen.

[0040] Optionally, the electronic device may further include a memory 230. The processor 210, the transceiver 220, and the memory 230 may communicate with each other through an internal connection path to transfer data. The memory 230 is used to store computer programs, and the processor 210 is used to call and run the computer programs from the memory 230. The above-mentioned processor 210 and the memory 230 may be integrated into a processing device, but more commonly they are independent components. The processor 210 is used to execute the program code stored in the memory 230 to implement the above functions. Specifically, in implementation, the memory 230 may also be integrated in the processor 210, or be independent of the processor 210.

[0041] In addition, to make the functions of the electronic device more complete, the electronic device may further include one or more of an input unit 260, a sensor 201, etc.

[0042] Optionally, the above-mentioned electronic device may further include a power supply 250 for supplying power to various devices or circuits in the electronic device.

[0043] It can be understood that Figure 2 The operations and / or functions of the various modules in the shown electronic device are respectively for implementing the corresponding processes in the following method embodiments. For details, reference may be made to the descriptions in the following method embodiments. To avoid repetition, the detailed descriptions are appropriately omitted here.

[0044] It can be understood that Figure 2 The processor 210 in the shown electronic device may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0045] A memory may also be provided in the processor 210 for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory may hold instructions or data that the processor 210 has just used or recycled. If the processor 210 needs to use the instruction or data again, it can be directly called from the said memory. This avoids repeated accesses and reduces the waiting time of the processor 210, thus improving the efficiency of the system.

[0046] It can be understood that Figure 2The power supply 250 shown is used to supply power to the processor 210, the memory 230, the display unit 270, the input unit 260, the transceiver 220, etc. The transceiver 220 can provide solutions for wireless communications applied to electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The transceiver 220 can be one or more devices integrating at least one communication processing module. The display unit 270 is used to display images, videos, etc. The display unit 270 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-led, a Micro-Led, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. The memory 230 can be used to store computer-executable program codes, and the executable program codes include instructions. The memory 230 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc. The data storage area can store data created during the use of the electronic device, etc. In addition, the memory 230 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 210 executes various functional applications and data processing of the electronic device by running the instructions stored in the memory 230 and / or the instructions stored in the memory provided in the processor.

[0047] Figure 3 It is a schematic flowchart of a graph optimization method 300 provided by an embodiment of the present application. AsFigure 3 As shown, the method 300 may include the following steps:

[0048] S301, obtain the image to be optimized.

[0049] The image to be optimized is the original input image processed by the above-mentioned joint network, which is usually an image with certain quality problems or requiring feature enhancement.

[0050] In some embodiments, the electronic device may obtain the image to be optimized when the camera application is turned on.

[0051] Such as the image displayed in real time on the interface of the electronic device when the camera is powered on.

[0052] In some embodiments, the electronic device may obtain the image to be optimized from the stored images.

[0053] For example, the user may perform a selection operation on at least one image according to their own needs, and the electronic device may obtain the image to be optimized in response to the user's selection operation on the image.

[0054] In some embodiments, the image to be optimized may also be obtained from other devices, such as other devices that require image optimization, or a server that stores image data, etc.

[0055] S302, perform image optimization processing on the image to be optimized through the joint network to obtain a target optimized image; wherein, the above-mentioned joint network includes a target enhancement network deployed on the script layer and a target operator deployed on the native layer; the target enhancement network is used to perform image optimization operations on the input image; the target operator is used to perform image processing on multiple image blocks of the input image in parallel.

[0056] The target optimized image is a high-quality image after the electronic device performs optimization processing, and the improvement direction of this optimization processing corresponds to the problems of the image to be optimized.

[0057] In some embodiments, during the process of the electronic device performing image optimization processing on the image to be optimized through the joint network, the target enhancement network and the target operator may cooperate to achieve the optimization processing of the image to be optimized to obtain the above-mentioned target optimized image.

[0058] Figure 4 It is a schematic diagram of an application scenario 400 provided by this application. As Figure 4 shown, the application scenario 400 includes an electronic device 401, and the electronic device 401 includes a joint network 402. The electronic device 401 may perform image processing on the image to be optimized through the joint network 402 to obtain a target optimized image. As Figure 4As shown in the figure, the joint network 402 includes a target enhancement network 421 and a target operator 422. During the process of image optimization by the electronic device 401 through the joint network 402, the joint network 402 can collaboratively process the image to be optimized through the target enhancement network 421 and the target operator 422, and output the target optimized image.

[0059] In some embodiments, during the process of implementing image optimization through the joint network 402, the order of image processing performed by different target operators 422 and the target enhancement network 421 can be different.

[0060] In a possible implementation manner, the electronic device can first perform image processing on multiple image blocks of the image to be optimized in parallel through the target operator 422 in the joint network 402 to obtain an optimized image after the target operator optimization, such as a first optimized image, and can input the first optimized image into the target enhancement network 421 in the joint network 402 to perform image optimization processing on the first optimized image through the target enhancement network 421, and output the above-mentioned target optimized image.

[0061] In a possible implementation manner, it is also possible to first input the above-mentioned image to be optimized into the target enhancement network 421 in the joint network 402 to perform image optimization processing on the image to be optimized through the target enhancement network 421, and output an optimized image after the target enhancement network processing, such as a second optimized image, and perform image processing on multiple image blocks of the second optimized image in parallel through the target operator 422 in the joint network 402 to obtain the target optimized image.

[0062] Optionally, it is also possible to input the image to be optimized into the target enhancement network 421 in the joint network 402 to perform a first image optimization process on the image to be optimized through the target enhancement network 421, and output a third optimized image, and perform image processing on multiple image blocks of the third optimized image in parallel through the target operator 422 in the joint network 402 to obtain a fourth optimized image, and then input the fourth optimized image into the target enhancement network 421 in the joint network 402 to perform a second image optimization process on the fourth optimized image through the target enhancement network 421, and output the target optimization network.

[0063] The script layer is a code layer written in the corresponding language, mainly responsible for high-level logic control, model definition, and rapid experimentation. For example, the Python script layer is a code layer written in the Python language, mainly responsible for high-level logic control, model definition, and rapid experimentation. In deep learning frameworks (such as PyTorch, TensorFlow), users define the network structure, data preprocessing, and training process through the Python interface.

[0064] In some embodiments, the target enhancement network of the script layer is used to perform image optimization operations on the input image. The network structure of the target enhancement network is a computational graph formed by connecting multiple operators in a specific manner. It can gradually extract and transform features from the input image through this network structure to complete its corresponding image processing tasks.

[0065] The Native layer is a high-performance computing layer implemented using low-level languages such as C++ and CUDA, and is mainly responsible for efficiently executing computationally intensive tasks.

[0066] In some embodiments, the target operator of the Native layer is used to perform image processing on multiple image patches of the input image in parallel. That is, the target operator is migrated to the Native for implementation and accelerated and optimized through SIMD neon.

[0067] An image patch is a local area composed of multiple pixels, usually a small rectangular block. A pixel is the smallest unit of an image, representing the color / brightness value of a single point (such as RGB or grayscale value).

[0068] In a possible implementation, the electronic device can simultaneously perform the same image processing on multiple image patches of the input image through the target operator.

[0069] In another possible implementation, the electronic device can also simultaneously perform the same image processing on multiple pixels of the input image through the target operator.

[0070] Exemplarily, operations such as addition, subtraction, and filtering can be simultaneously performed on each pixel through the target operator to effectively reduce the processing time.

[0071] In some embodiments, during the process of performing image processing on multiple image patches of the input image in parallel through the target operator of the Native layer, the data of the above-mentioned multiple image patches (or pixels) can be loaded in batches from the memory, avoiding frequent access to the memory and reducing the power consumption overhead of the joint network. And during the process of graph optimization through the joint network, since a part of the image processing is implemented by the target enhancement network and another part of the image processing is implemented by the target operator of the Native layer, the target enhancement network can at least not include the target operator, reducing the need to allocate independent memory for each operator in the target enhancement network, resulting in excessive memory occupancy.

[0072] In this application, an electronic device can obtain an image to be optimized, and perform image optimization processing on the image to be optimized through a joint network to obtain a target optimized image. Among them, the joint network includes a target enhancement network deployed at the script layer and a target operator deployed at the native layer. The target enhancement network is used to perform image optimization operations on the input image. The target operator is used to perform image processing on multiple image blocks of the input image in parallel. Compared with the related art method of integrating the target operator into the image enhancement network, in the embodiment of this application, the target operator is implemented at the native layer, which can avoid allocating independent memory to each operator at the script layer, effectively reducing the memory occupation of the joint network during the graph optimization process. Moreover, during the process of the target operator performing image processing on multiple image blocks of the input image in parallel, in addition to reducing the time-consuming of the joint network for graph optimization, it can also batch load the data of the above multiple image blocks from memory, avoiding frequent access to memory to reduce the power consumption overhead of the joint network. In other words, high-flexibility graph optimization is achieved through the above joint network, reducing the performance overhead of graph optimization.

[0073] In some embodiments, the above target enhancement network and target operator are networks and operators that are determined through testing according to the needs of graph optimization and can cooperate to achieve the required graph optimization requirements.

[0074] In some embodiments, the above target enhancement network is obtained by removing the target operator from the original image enhancement network.

[0075] Exemplarily, if the above target operator is the pixel rearrangement downsampling operator PixelUnshuffle, the target enhancement network can be obtained by removing the pixel rearrangement downsampling operator PixelUnshuffle from the original image enhancement network. For details, see Figure 5 .

[0076] Figure 5 This is a schematic diagram of the network structure provided by this application. As Figure 5 shown, the left side is the network structure of the original image enhancement network. There are a pixel rearrangement downsampling operator PixelUnshuffle and other operators in the network structure of the original image enhancement network.

[0077] As Figure 5 shown, in the case of inputting the image to be optimized into the original image enhancement network, graph optimization can be sequentially performed through the pixel rearrangement downsampling operator PixelUnshuffle and other operators to output the target image to be optimized. Among them, the pixel rearrangement downsampling operator PixelUnshuffle is used to rearrange the image to be optimized from the spatial dimension to the channel dimension to obtain the downsampled image to be optimized, and the other operators are used to process the downsampled image to be optimized to obtain the target optimized image.

[0078] As Figure 5 shown, the function of the pixel rearrangement downsampling operator PixelUnshuffle can be achieved by combining multiple basic operators such as the reshape operator, the transpose operator, the unsqueeze operator, and the squeeze operator.

[0079] Among them, the reshape operator is used to split the spatial dimension and expand the channels, the transpose operator is used to adjust the dimension order to merge the channels, the unsqueeze operator is used to increase the dimension in preprocessing, and the squeeze operator is used to remove the redundant dimension at the output.

[0080] As Figure 5 shown, by removing the pixel rearrangement downsampling operator PixelUnshuffle in the network structure, the network on the right side of the figure, that is, the target enhancement network, can be obtained. It can be seen that the target enhancement network can perform other image optimization operations except the pixel rearrangement downsampling operator PixelUnshuffle through other operators, so that the electronic device can implement the pixel rearrangement downsampling operator PixelUnshuffle at the native layer. In other words, the electronic device collaborates with the target enhancement network in the script layer (such as Python) and the target operator deployed at the native layer to achieve graph optimization.

[0081] It should be understood that except for the pixel rearrangement downsampling operator PixelUnshuffle shown above, the above target enhancement network can also be obtained by removing operators at other positions in the network structure of the original image enhancement network, and the present application does not limit this.

[0082] It should also be understood that the network structure of the target enhancement network shown above is only exemplary, and the present application can also optimize the target enhancement network according to the usage requirements, so that there are other differences from the network structure of the original image enhancement network after removing the target operator, and the present application does not limit this.

[0083] In a possible case, the target operator can be a preprocessing operator, so that the electronic device can perform image processing on the target operator and the target enhancement network in the joint network in sequence to obtain a target optimized image.

[0084] Next, taking the target operator as the pixel rearrangement downsampling operator PixelUnshuffle as an example, the graph optimization method provided by the present application will be described in detail.

[0085] Figure 6It is a schematic flowchart of an image optimization method 600 provided by an embodiment of the present application. As Figure 6 shown, the method 600 may include the following steps:

[0086] S601, obtain the image to be optimized.

[0087] The image to be optimized is the original input image processed by the above-mentioned joint network, which is usually an image with certain quality problems or requiring feature enhancement.

[0088] In some embodiments, the electronic device may obtain the image to be optimized when the camera application is turned on.

[0089] Such as the image displayed in real time on the electronic device interface when the camera is turned on.

[0090] In some embodiments, the electronic device may obtain the image to be optimized from the stored images.

[0091] For example, the user can perform a selection operation on at least one image according to their own needs, and the electronic device can obtain the image to be optimized in response to the user's selection operation on the image.

[0092] In some embodiments, the image to be optimized may also be obtained from other devices.

[0093] S602, through the pixel rearrangement downsampling operator, parallelly rearrange multiple image blocks of the image to be optimized from the spatial dimension to the channel dimension respectively to obtain the downsampled image to be optimized.

[0094] The Native layer is a high-performance computing layer implemented using low-level languages (such as C++ and CUDA), mainly responsible for efficiently executing computationally intensive tasks.

[0095] In some embodiments, the pixel rearrangement downsampling operator of the Native layer is used to perform image processing on multiple image blocks of the input image in parallel. That is, the pixel rearrangement downsampling operator is migrated to the Native layer for implementation and accelerated optimization through SIMD neon to effectively reduce the processing time.

[0096] An image block is a local area composed of multiple pixels, usually a small rectangular block. A pixel is the smallest unit of an image, representing the color / brightness value of a single point (such as RGB or grayscale value).

[0097] In a possible implementation manner, the electronic device may simultaneously perform the same image processing on multiple image blocks of the input image through the pixel rearrangement downsampling operator.

[0098] In another possible implementation, the electronic device can also perform the same image processing on multiple pixels of the input image through a pixel rearrangement downsampling operator.

[0099] In some embodiments, during the process of parallel image processing of multiple image blocks of the input image through the pixel rearrangement downsampling operator of the native layer Native, the data of the above-mentioned multiple image blocks (or pixels) can be batch-loaded from the memory, avoiding frequent access to the memory and reducing the power consumption overhead of the joint network.

[0100] S603, perform an image optimization operation on the downsampled image to be optimized through the above-mentioned target enhancement network to obtain a target optimized image.

[0101] The target optimized image is a high-quality image after the optimization process of the electronic device, and the improvement direction of this optimization process corresponds to the problems of the above-mentioned image to be optimized.

[0102] For example, the Python script layer is a code layer written in the Python language, mainly responsible for high-level logic control, model definition, and rapid experimentation. In deep learning frameworks (such as PyTorch, TensorFlow), users define the network structure, data preprocessing, and training process through the Python interface.

[0103] In some embodiments, the target enhancement network of the Python script layer is used to perform an image optimization operation on the input image. The network structure of this target enhancement network is a computational graph formed by connecting multiple operators in a specific manner, and it can gradually extract and transform features from the input image through this network structure to complete its corresponding image processing tasks.

[0104] During the process of graph optimization through the joint network, since a part of the image processing is implemented by the target enhancement network and another part of the image processing is implemented by the pixel rearrangement downsampling operator of the native layer Native, the target enhancement network can at least not include the pixel rearrangement downsampling operator, reducing the need to allocate independent memory for each operator in the target enhancement network, resulting in excessive memory occupation.

[0105] In a possible implementation, the target operator can be a post-processing operator, so that the electronic device can perform image processing through the target enhancement network and the target operator in the joint network in sequence to obtain a target optimized image.

[0106] Taking the target operator as the super-resolution upsampling operator PixelShuffle as an example, the graph optimization method provided in this application will be described in detail below.

[0107] Figure 7It is a schematic flowchart of an image optimization method 700 provided by an embodiment of the present application. As Figure 7 shown, the method 700 may include the following steps:

[0108] S701, Obtain the image to be optimized.

[0109] The image to be optimized is the original input image processed by the above-mentioned joint network, which is usually an image with certain quality problems or requiring feature enhancement.

[0110] In some embodiments, the electronic device may obtain the image to be optimized when the camera application is turned on.

[0111] Such as the image displayed in real time on the interface of the electronic device when the camera is turned on.

[0112] In some embodiments, the electronic device may obtain the image to be optimized from the stored images.

[0113] For example, the user can select at least one image according to their own needs, and the electronic device can obtain the image to be optimized in response to the user's selection operation on the image.

[0114] In some embodiments, the image to be optimized may also be obtained from other devices.

[0115] S702, Perform an image optimization operation on the image to be optimized through the above-mentioned target enhancement network to obtain a downsampled image to be optimized.

[0116] For example, the Python script layer is a code layer written in the Python language, mainly responsible for high-level logic control, model definition, and rapid experimentation. In deep learning frameworks (such as PyTorch, TensorFlow), users define the network structure, data preprocessing, and training process through the Python interface.

[0117] In some embodiments, the target enhancement network of the Python script layer is used to perform an image optimization operation on the input image. The network structure of the target enhancement network is a computational graph formed by connecting multiple operators in a specific manner, and it can gradually extract and transform features from the input image through this network structure to complete its corresponding image processing tasks.

[0118] S703, Through the super-resolution upsampling operator, parallelly rearrange multiple image blocks of the above-mentioned downsampled image to be optimized from the channel dimension to the spatial dimension to obtain the target optimized image.

[0119] The target optimized image is a high-quality image after the electronic device's optimization process, and the improvement direction of this optimization process corresponds to the problems of the above-mentioned image to be optimized.

[0120] The Native layer is a high-performance computing layer implemented using low-level languages such as C++ and CUDA, mainly responsible for efficiently executing compute-intensive tasks.

[0121] In some embodiments, the super-resolution upsampling operator of the Native layer is used to perform image processing on multiple image patches of the input image in parallel. That is, the target operator is migrated to the Native layer for implementation and accelerated and optimized through SIMD neon to effectively reduce the processing time.

[0122] An image patch is a local area composed of multiple pixels, usually a small rectangular block. A pixel is the smallest constituent unit of an image, representing the color / brightness value of a single point (such as RGB or grayscale value).

[0123] In one possible implementation, the electronic device can perform the same image processing on multiple image patches of the input image simultaneously through the super-resolution upsampling operator.

[0124] In another possible implementation, the electronic device can also perform the same image processing on multiple pixels of the input image simultaneously through the super-resolution upsampling operator.

[0125] In some embodiments, during the process of performing image processing on multiple image patches of the input image in parallel through the super-resolution upsampling operator of the Native layer, the data of the above-mentioned multiple image patches (or pixels) can be loaded in batches from the memory, avoiding frequent access to the memory and reducing the power consumption overhead of the joint network. And during the process of graph optimization through the joint network, since a part of the image processing is implemented by the target enhancement network and another part of the image processing is implemented by the super-resolution upsampling operator of the Native layer, the target enhancement network can at least not include the target operator, reducing the need to allocate independent memory for each operator in the target enhancement network, resulting in excessive memory occupation.

[0126] It should be understood that the cooperation between the above-mentioned target operator and the above-mentioned target enhancement network to achieve the optimization processing of the image to be optimized is merely exemplary, and the present application does not limit this.

[0127] Optionally, the target operator can be an operator other than those for preprocessing and postprocessing, so that the electronic device can perform image processing on the input image through the target enhancement network, the target operator, and the target enhancement network in the joint network in sequence to obtain the target optimized image.

[0128] In some embodiments, the above-mentioned target operator can also be an operator with a performance overhead ratio greater than a threshold in the original image enhancement network, so that the electronic device can achieve graph optimization through the target enhancement network deployed in the script layer and the target operator deployed in the Native layer. For specific details, see the followingFigure 8 。

[0129] Figure 8 is a schematic flowchart of an image optimization method 800 provided by an embodiment of the present application. As Figure 8 shown, the method 800 may include the following steps:

[0130] S801, Quantize and deploy the original image enhancement network on an electronic device.

[0131] In some embodiments, the quantization deployment of the original image enhancement network may be performed by a quantization bit width w8a16 (weights INT8, activation values FP16). When performing quantization deployment through W8A16, it may specifically include steps such as quantization-aware training (QAT), model conversion, and model compilation optimization.

[0132] Among them, QAT is used to simulate the quantization process during the training phase, enabling the original image enhancement network to adapt to low-precision calculations and reducing the precision loss after deployment. Model conversion is used to convert the trained QAT original image enhancement network into a low-precision format (such as W8A16). Model compilation optimization is used to generate an efficient inference engine for the target hardware (such as GPU / NPU), fuse operators, and optimize the memory layout.

[0133] S802, During the process of optimizing the graph through the original image enhancement network, determine the operators in the original image enhancement network whose performance overhead ratio is greater than a threshold.

[0134] In some embodiments, the performance overhead of the above-mentioned quantized original image enhancement network may be analyzed through a side performance analysis tool.

[0135] Exemplarily, if the performance overhead of the pixel unshuffle 4x downsampling operator in the above-mentioned original image enhancement network is greater than 50%, the target operator may be determined as the pixel unshuffle operator.

[0136] S803, Remove the operators in the original image enhancement network whose performance overhead ratio is greater than the threshold from the original image enhancement network to obtain a target enhancement network.

[0137] Exemplarily, if the target operator is the pixel unshuffle operator, this operator may be removed from the original image enhancement network. For specific details, refer to the above Figure 5 , To avoid repetition, it will not be elaborated here.

[0138] S804. Deploy the target enhanced network for quantization on the electronic device.

[0139] S805. Migrate the operators in the original image enhancement network whose performance overhead ratio is greater than the threshold to the native layer for implementation.

[0140] S806. Use the target enhanced network and the operators in the original image enhancement network whose performance overhead ratio is greater than the threshold to perform image optimization on the image to be optimized, and obtain the target optimized image.

[0141] The image to be optimized is the original input image processed by the above joint network, and it is usually an image with certain quality problems or requiring feature enhancement.

[0142] In some embodiments, the electronic device can obtain the image to be optimized when the camera application is turned on.

[0143] Such as the image displayed in real time on the interface of the electronic device when the camera is turned on.

[0144] In some embodiments, the electronic device can obtain the image to be optimized from the stored images.

[0145] For example, the user can perform a selection operation on at least one image according to their own needs, and the electronic device can obtain the image to be optimized in response to the user's selection operation on the image.

[0146] In some embodiments, the image to be optimized can also be obtained from other devices.

[0147] The target optimized image is a high-quality image after the optimization process of the electronic device, and the improvement direction of this optimization process corresponds to the problems of the above image to be optimized.

[0148] In some embodiments, during the process of realizing image optimization through the joint network, the execution order of different target operators and the target enhanced network for image processing can be different.

[0149] In a possible implementation manner, the electronic device can first perform image processing on multiple image blocks of the image to be optimized in parallel through the target operators in the joint network 402 to obtain an optimized image after the target operator optimization, such as the first optimized image, and can input the first optimized image into the target enhanced network in the joint network 402 to perform image optimization processing on the first optimized image through the target enhanced network, and output the above target optimized image.

[0150] In a possible implementation, the above-mentioned image to be optimized can also be preferentially input into the target enhancement network in the joint network to perform image optimization processing on the image to be optimized through the target enhancement network, output the optimized image processed by the target enhancement network, such as the second optimized image, and perform image processing on multiple image patches of the second optimized image in parallel through the target operator in the joint network to obtain the target optimized image.

[0151] Optionally, the image to be optimized can also be input into the target enhancement network in the joint network to perform first image optimization processing on the image to be optimized through the target enhancement network, output the third optimized image, and perform image processing on multiple image patches of the third optimized image in parallel through the target operator in the joint network to obtain the fourth optimized image, and then input the fourth optimized image into the target enhancement network in the joint network to perform second image optimization processing on the fourth optimized image through the target enhancement network to output the target optimization network.

[0152] For example, the script layer Python is the code layer written in the Python language, mainly responsible for high-level logic control, model definition, and rapid experimentation. In deep learning frameworks (such as PyTorch, TensorFlow), users define the network structure, data preprocessing, and training process through the Python interface.

[0153] In some embodiments, the target enhancement network of the script layer Python is used to perform image optimization operations on the input image. The network structure of the target enhancement network is a computational graph formed by connecting multiple operators in a specific manner, and it can gradually extract and transform features from the input image through this network structure to complete its corresponding image processing tasks.

[0154] The native layer Native is a high-performance computing layer implemented using low-level languages (such as C++, CUDA), mainly responsible for efficiently executing computationally intensive tasks.

[0155] In some embodiments, the target operator of the native layer Native is used to perform image processing on multiple image patches of the input image in parallel. That is, the target operator is migrated to the native layer Native for implementation and accelerated optimization through SIMD neon.

[0156] An image patch is a local area composed of multiple pixels, usually a small rectangular block. A pixel is the smallest component unit of an image, representing the color / brightness value of a single point (such as RGB or grayscale value).

[0157] In a possible implementation, the electronic device can perform the same image processing on multiple image patches of the input image simultaneously through the target operator.

[0158] In another possible implementation, the electronic device can also perform the same image processing on multiple pixels of the input image through a target operator.

[0159] Exemplarily, operations such as addition, subtraction, and filtering can be simultaneously performed on each pixel through the target operator to effectively reduce the processing time consumption.

[0160] In some embodiments, during the process of parallelly performing image processing on multiple image blocks of the input image through the target operator in the native layer (Native), the data of the above-mentioned multiple image blocks (or pixels) can be batch-loaded from the memory, avoiding frequent access to the memory and reducing the power consumption overhead of the joint network. And during the process of graph optimization through the joint network, since a part of the image processing is implemented by the target enhancement network and another part of the image processing is implemented by the target operator in the native layer (Native), the target enhancement network can at least not include the target operator, reducing the need to allocate independent memory for each operator in the target enhancement network, resulting in excessive memory occupation.

[0161] Table 1 shows the performance comparison between the original image enhancement network and the joint network when the target operator is the pixel rearrangement downsampling operator.

[0162] Table 1

[0163]

[0164] As shown in Table 1, in the case of performing graph optimization on the same image to be optimized, the processing time consumption for each image block of the image to be optimized by the original image enhancement network is 5.5 ms, and the total time consumption is 150 ms. While in the case of performing graph optimization on the same image to be optimized by the joint network, the processing time consumption for each image block of the image by the target enhancement network (i.e., the original image enhancement network after removing the target operator) in the joint network is 2.0, and the total time consumption is 59 ms. The target operator in the joint network, such as Pixelunshuffle, can process multiple image blocks of the image in parallel, making the processing time consumption for each image block only 0.08 ms, and the total time consumption is 3.844 ms. It can be seen that the graph optimization time consumption of the joint network is significantly reduced compared to that of the original image enhancement network.

[0165] In some embodiments, the above-mentioned target enhancement network can be run through a neural network processor (NPU), and the target operator can be run through a central processing unit (CPU).

[0166] The NPU is designed for matrix operations (such as convolution and matrix multiplication), supports large-scale parallel computing, is suitable for processing dense computing layers in neural networks (such as Conv and ReLU), and is optimized for AI computing. The energy consumption per unit computing power is much lower than that of the CPU (suitable for continuous operation on mobile devices). Running the target enhancement network on the NPU can not only improve the graph optimization efficiency of the combined network but also reduce power consumption. The CPU is good at processing complex logics such as conditional branches and loops and is suitable for running operators related to data preprocessing (such as image chunking and normalization). For operators related to post-processing (such as non-maximum suppression and result fusion), that is, the CPU processes the pre- and post-stages of graph optimization, enabling the NPU to focus on computing the core network and reducing data transfer.

[0167] In some embodiments, the electronic device can also perform image processing on multiple image chunks of the input image of the target operator in parallel through multiple threads in the CPU.

[0168] Exemplarily, when the above target operator is the pixel rearrangement downsampling operator PixelUnshuffle, the multi-threaded pixel rearrangement downsampling operator PixelUnshuffle runs on the CPU to utilize the multiple cores of the CPU to perform different image processing operations on different image chunks of the image in parallel.

[0169] In some embodiments, the NPU runs a first thread and a second thread. The first thread is used to perform a first image optimization operation on the input image of the above target enhancement network, and the second thread is used to perform a second image optimization operation on the input image of the target enhancement network.

[0170] In a possible case, the first thread and the second thread can be parallel. The two threads can process different input data respectively. For example, the first thread processes the odd frames of the image, and the second thread synchronously processes the even frames of the image.

[0171] In another possible case, the first thread and the second thread can also be serial. One thread processes the feedforward calculation, and the other thread processes the post-processing or the preprocessing of the next frame.

[0172] It should be understood that the above various embodiments can also be coupled to each other, and this application does not make any limitation in this regard. And the magnitudes of the sequence numbers of the above various processes do not mean the order of execution is prior or subsequent. The execution order of each process should be determined by its function and internal logic and should not constitute any limitation to the implementation process of the embodiments of this application.

[0173] In the above text, in combination with Figures 1 to 8 , the graph optimization method of the embodiments of this application is described in detail. Next, in combination with Figure 9 and Figure 10 , the graph optimization device of the embodiments of this application will be described in detail.

[0174] Figure 9 Shown is a graph optimization device 900 provided by an embodiment of the present application. The graph optimization device 900 includes: an acquisition module 901 and an optimization module 902. Among them, the acquisition module 901 is used to: acquire an image to be optimized; the optimization module 902 is used to: perform image optimization processing on the above-mentioned image to be optimized through a joint network to obtain a target optimized image; wherein, the joint network includes a target enhancement network deployed on the script layer and a target operator deployed on the native layer; the target enhancement network is used to perform image optimization operations on the input image; the target operator is used to perform image processing on multiple image blocks of the input image in parallel.

[0175] Optionally, the above-mentioned target enhancement network is obtained by removing the target operator from the original image enhancement network.

[0176] Optionally, the target operator is an operator in the original image enhancement network whose performance overhead ratio is greater than a threshold.

[0177] Optionally, the above-mentioned target operator includes a pixel rearrangement downsampling operator, and the optimization module 902 is used to: through the pixel rearrangement downsampling operator, parallelly rearrange multiple image blocks of the image to be optimized from the spatial dimension to the channel dimension respectively to obtain a downsampled image to be optimized; perform image optimization operations on the downsampled image to be optimized through the above-mentioned target enhancement network to obtain the above-mentioned target optimized image.

[0178] Optionally, the above-mentioned target operator includes a super-resolution upsampling operator, and the optimization module 902 is used to: perform image optimization operations on the image to be optimized through the target enhancement network to obtain a downsampled image to be optimized; through the super-resolution upsampling operator, parallelly rearrange multiple image blocks of the above-mentioned downsampled image to be optimized from the channel dimension to the spatial dimension respectively to obtain the above-mentioned target optimized image.

[0179] Optionally, the above-mentioned target enhancement network runs through a neural network processor NPU, and the target operator runs through a central processing unit.

[0180] Optionally, the optimization module 902 is used to: through multiple threads in the central processing unit, parallelly perform image processing on multiple image blocks of the image input to the target operator.

[0181] Optionally, the neural network processor runs a first thread and a second thread. The first thread is used to perform a first image optimization operation on the image input to the target enhancement network, and the second thread is used to perform a second image optimization operation on the image input to the target enhancement network.

[0182] It should be understood that the graph optimization device 900 herein is embodied in the form of a functional module. The term "module" herein may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group of processors, etc.) for executing one or more software or firmware programs, and a memory, a combined logic circuit, and / or other suitable components supporting the described functions. In an alternative example, those skilled in the art can understand that the graph optimization device 900 may specifically be the electronic device in the above embodiments, or the functions of the electronic device in the above embodiments may be integrated in the graph optimization device 900. The graph optimization device 900 may be used to execute each process and / or step corresponding to the electronic device in the above method embodiments. To avoid repetition, it will not be elaborated herein. The above graph optimization device 900 has the function of implementing the corresponding steps executed by the electronic device in the above method; the above function may be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In the embodiments of the present application, Figure 9 the graph optimization device 900 in

[0183] Figure 10 may also be a chip or a chip system, for example: a system on chip (SoC).

[0184] Figure 1000 of an electronic device provided in an embodiment of the present application is shown. The electronic device 1000 includes: a processor 1001, a memory 1002, a communication interface 1003, and a bus 1004. Among them, the memory 1002 is used to store instructions, and the processor 1001 is used to execute the instructions stored in the memory 1002. The processor 1001, the memory 1002, and the communication interface 1003 are communicatively connected to each other through the bus 1004.

[0185] It should be understood that the electronic device 1000 can specifically be the electronic device in the above embodiments, or the functions of the electronic device in the above embodiments can be integrated in the electronic device 1000, and the electronic device 1000 can be used to execute the respective steps and / or processes corresponding to the electronic device in the above method embodiments. Optionally, the memory 1002 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1001. A part of the memory 1002 may also include a non-volatile random access memory. For example, the memory 1002 may also store information about the device type. The processor 1001 can be used to execute the instructions stored in the memory, and when the processor executes the instructions, the processor 1001 can execute the respective steps and / or processes corresponding to the electronic device in the above method embodiments. It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In the implementation process, the respective steps of the above method can be completed by the integrated logic circuit in the hardware of the processor or the instructions in the form of software. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware processor, or executed and completed by a combination of the hardware and software modules in the processor. The software module may be located in a mature storage medium in the art such as a random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. This storage medium is located in the memory, and the processor executes the instructions in the memory and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways.For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms. The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks or optical discs and other various media that can store program codes. The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A graph optimization method, characterized in that, Applied to an electronic device, the method includes: Obtain an image to be optimized; Perform image optimization processing on the image to be optimized through a joint network to obtain a target optimized image; Wherein, the joint network includes a target enhancement network deployed in the script layer and a target operator deployed in the native layer; the target enhancement network is used to perform image optimization operations on the input image; the target operator is used to perform image processing on multiple image blocks of the input image in parallel.

2. The method according to claim 1, characterized in that, The target enhancement network is obtained by removing the target operator from the original image enhancement network.

3. The method according to claim 2, wherein The target operator is an operator in the original image enhancement network with a performance overhead ratio greater than a threshold.

4. The method according to claim 1, wherein The target operator includes a pixel rearrangement downsampling operator. The performing image optimization processing on the image to be optimized through the joint network to obtain a target optimized image includes: Through the pixel rearrangement downsampling operator, respectively rearrange multiple image blocks of the image to be optimized from the spatial dimension to the channel dimension in parallel to obtain a downsampled image to be optimized; Perform image optimization operations on the downsampled image to be optimized through the target enhancement network to obtain the target optimized image.

5. The method according to claim 1, wherein The target operator includes a super-resolution upsampling operator. The performing image optimization processing on the image to be optimized through the joint network to obtain a target optimized image includes: Perform image optimization operations on the image to be optimized through the target enhancement network to obtain a downsampled image to be optimized; Through the super-resolution upsampling operator, respectively rearrange multiple image blocks of the downsampled image to be optimized from the channel dimension to the spatial dimension in parallel to obtain the target optimized image.

6. The method according to claim 1, characterized in that, The target enhancement network is run by a neural network processor, and the target operator is run by a central processing unit.

7. The method according to claim 6, characterized in that, The method further includes: Through multiple threads in the central processing unit, perform image processing on multiple image blocks of the image input to the target operator in parallel.

8. The method according to claim 6, characterized in that, The neural network processor runs a first thread and a second thread. The first thread is used to perform a first image optimization operation on the image input to the target enhancement network, and the second thread is used to perform a second image optimization operation on the image input to the target enhancement network.

9. A graph optimization device, characterized in that, Applied to an electronic device, the graph optimization module includes: An acquisition module, configured to obtain an image to be optimized; An optimization module, configured to perform image optimization processing on the image to be optimized through a joint network to obtain a target optimized image; Wherein, the joint network includes a target enhancement network deployed in the script layer and a target operator deployed in the native layer; the target enhancement network is used to perform image optimization operations on the input image; the target operator is used to perform image processing on multiple image blocks of the input image in parallel.

10. An electronic device, characterized in that, Includes a processor and a memory. The memory is used to store code instructions; the processor is used to run the code instructions to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 8.

Images