Image processing method and chip

Abstract

Embodiments of the present application provide an image processing method. The method comprises: an image signal processor in a chip processing data received from a camera to generate a target image; the image signal processor writing a first patch in the target image into a system cache in the chip; a neural network processor of the chip reading the first patch from the system cache; the neural network processor processing the first patch based on a neural network model to obtain a first output patch; and the neural network processor writing the first output patch into the system cache. The processing includes at least one of color interpolation or high dynamic range (HDR) processing. The technical solution provided by the present application saves the time-consuming of image processing and saves the system power consumption.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an image processing method and chip. Background Technology

[0002] Currently, in terminals with photo and video recording capabilities, such as mobile devices and tablets, the image signal processor (ISP) and neural network processing unit (NPU) within the terminal perform pixel-level image processing on captured photos, significantly improving image quality. However, due to the increasingly higher resolution of photos, the amount of data processed by pixel-level image processing is large, and the NPU reads images from memory for processing, resulting in significant processing time. For example, processing a captured photo using the NPU on a mobile terminal takes 2-3 seconds. If the NPU performs the same processing on every frame of a video, it may lead to frame drops or pixelated images; that is, the pixel-level image processing function performs poorly in video scenarios. Summary of the Invention

[0003] This application provides an image processing method and apparatus that allows the use of system cache to store data during image processing, thereby saving image processing time and system power consumption.

[0004] A first aspect of this application provides an image processing method, comprising: a first processor in an image processing device writing a first slice of a target image into a system cache; a neural network processor of the image processing device reading the first slice of the target image from the system cache; the neural network processor processing the first slice to obtain a first output slice; and the neural network processor writing the first output slice into the system cache.

[0005] In this image processing method, a large target image is divided into smaller pieces for processing, which allows the system cache to store the data during the image processing process, thereby saving image processing time and system power consumption and improving image processing efficiency.

[0006] In one embodiment of the first aspect of this application, the method further includes generating the target image by the first processor before the neural network processor of the image processing device reads a first slice of the target image from the system cache.

[0007] In one embodiment of the first aspect of this application, the method further includes, after the neural network processor writes the first output slice into the system cache, the first processor reads the first output slice from the system cache and obtains an output image based on the first output slice.

[0008] In one embodiment of the first aspect of this application, the first processor includes an image signal processor. This application improves image processing efficiency and enhances image or video quality by causing the image signal processor and neural network processor to execute the image processing method. Optionally, the image processing device is a chip or chipset. Optionally, the image processing device includes the system cache.

[0009] In one embodiment of the first aspect of this application, the system cache includes an input area and an output area. The first processor writes a first slice of the target image into the system cache, and the first processor writes the first slice of the target image into the input area. The neural network processor writes the first output slice into the system cache, and the neural network processor writes the first output slice into the output area. This application, by setting an input area and an output area in the system cache, enables simultaneous processing of input slices and output slices, further improving the efficiency of image processing.

[0010] In one embodiment of the first aspect of this application, the input area includes a first buffer and a second buffer. The method of the first processor writing a first slice of the target image into the input area of ​​the system buffer includes the first processor writing the first slice into the first buffer. The method further includes, simultaneously, the first processor writing a second slice into the second buffer while the neural network processor reads the first slice from the input area, wherein the second slice is different from the first slice. This application, by setting a first buffer and a second buffer in the input area, enables simultaneous writing of one slice by the first processor and reading of another slice by the neural network processor, further improving the efficiency of image processing.

[0011] In one embodiment of the first aspect of this application, the output area includes a third buffer and a fourth buffer. The method of the neural network processor writing the first output fragment into the output area of ​​the system buffer includes the neural network processor writing the first output fragment into the third buffer. The method further includes, simultaneously with the neural network processor writing the first output fragment into the third buffer, the first processor reading a third output fragment corresponding to the third fragment from the fourth buffer, wherein the third fragment is different from the first fragment. The third fragment is, for example, a fragment in the target image that is adjacent to the first fragment and processed before the first fragment. This application, by setting a third buffer and a fourth buffer in the output area, enables simultaneous writing of output fragments by the neural network processor and reading of output fragments by the first processor, further improving the efficiency of image processing.

[0012] In one embodiment of the first aspect of this application, the neural network processor processes the first slice by preprocessing the first slice to obtain first intermediate image data, reading pre-stored second intermediate image data, the second intermediate image data being obtained by performing the preprocessing on a portion of a third slice in the target image, and the neural network processor performing convolution calculation on the concatenated data of the second intermediate image data and the first intermediate image data. This application saves computational resources and further improves image processing efficiency by enabling the neural network processor to pre-store intermediate processing data from a portion of the previous slice during the slice processing.

[0013] In one embodiment of the first aspect of this application, the method further includes the neural network processor reading a portion of a third slice from the target image from the second cache, and the neural network processor processing the first slice including performing convolution calculations on the concatenated data of the portion of the third slice and the first slice to obtain intermediate image data, and cropping the intermediate image data. This application saves computational resources by using lossy overlapping row calculations during the slice processing, further improving image processing efficiency while ensuring image quality.

[0014] In one embodiment of the first aspect of this application, the method further includes, after the first processor writes a first slice of the target image into a system cache, the first processor notifies the neural network processor to read the slice from the system cache; and after the neural network processor writes the first output slice into the system cache, the neural network processor notifies the first processor to read the output slice from the system cache. This application further improves image processing efficiency by enabling the first processor and the neural network processor to collaboratively execute the image processing method.

[0015] A second aspect of this application provides an image processing apparatus, including a first processor and a neural network processor. The first processor is configured to write a first slice of a target image into a system cache; the neural network processor is configured to read the first slice of the target image from the system cache, process the first slice to obtain a first output slice, and write the first output slice into the system cache.

[0016] In one embodiment of the second aspect of this application, the first processor is further configured to generate the target image before writing a first slice of the target image into the system cache.

[0017] In one embodiment of the second aspect of this application, the first processor is further configured to read the first output slice from the system cache and obtain an output image based on the first output slice.

[0018] In one embodiment of the second aspect of this application, the first processor includes an image signal processor.

[0019] In one embodiment of the second aspect of this application, the image processing device further includes a system cache.

[0020] In one embodiment of the second aspect of this application, the system cache further includes an input area and an output area, the first processor is further configured to write the first slice into the input area; the neural network processor is further configured to write the first output slice into the output area.

[0021] In one embodiment of the second aspect of this application, the input area includes a first buffer area and a second buffer area; the first processor is further configured to: write the first slice into the first buffer area, and while the neural network processor reads the first slice from the input area, write the second slice of the target image into the second buffer area, wherein the second slice is different from the first slice.

[0022] In one embodiment of the second aspect of this application, the output area includes a third buffer and a fourth buffer; the neural network processor is further configured to write the first output slice into the third buffer; the first processor is further configured to: while the neural network processor writes the first output slice into the third buffer, read a third output slice corresponding to the third slice in the target image from the fourth buffer, wherein the third slice is different from the first slice.

[0023] In one embodiment of the second aspect of this application, the neural network processor is further configured to: preprocess the first slice to obtain first intermediate image data, read pre-stored second intermediate image data, the second intermediate image data being obtained by performing the preprocessing on a portion of the third slice in the target image, and perform convolution calculation on the spliced ​​data of the second intermediate image data and the first intermediate image data.

[0024] In one embodiment of the second aspect of this application, the neural network processor is further configured to: read a portion of a third slice from the target image from the second buffer, perform processing including convolution calculation on the spliced ​​data of the portion of the third slice and the first slice to obtain intermediate image data, and crop the intermediate image data.

[0025] A third aspect of this application provides an image processing apparatus, including a first writing unit deployed in a first processor of the image processing apparatus for writing a first slice of a target image into a system cache; and the following units deployed in a neural network processor of the image processing apparatus: a first reading unit for reading the first slice of the target image from the system cache; a processing unit for processing the first slice to obtain a first output slice; and a second writing unit for writing the first output slice into the system cache.

[0026] In one embodiment of the third aspect of this application, the first writing unit is further configured to generate the target image before writing the first fragment of the target image into the system cache.

[0027] In one embodiment of the third aspect of this application, the image processing device further includes a second reading unit deployed in a first processor of the image processing device, configured to read the first output slice from the system cache and obtain an output image based on the first output slice after the neural network processor writes the first output slice into the system cache.

[0028] In one embodiment of the third aspect of this application, the first processor includes an image signal processor.

[0029] In one embodiment of the third aspect of this application, the system cache includes an input area and an output area, wherein the first writing unit is further configured to write a first slice of the target image into the input area; and the second writing unit is further configured to write the first output slice into the output area.

[0030] In one embodiment of the third aspect of this application, the input area includes a first buffer area and a second buffer area, and the first writing unit is further configured to write the first slice into the first buffer area. The image processing device further includes a third writing unit deployed on the first processor, configured to write a second slice of the target image into the second buffer area while the neural network processor reads the first slice from the input area, wherein the second slice is different from the first slice.

[0031] In one embodiment of the third aspect of this application, the output area includes a third buffer and a fourth buffer, and the second writing unit is further configured to write the first output fragment into the third buffer. The image processing device further includes a third reading unit deployed in the first processor, configured to read a third output fragment corresponding to the third fragment in the target image from the fourth buffer while the neural network processor writes the first output fragment into the third buffer, wherein the third fragment is different from the first fragment.

[0032] In one embodiment of the third aspect of this application, the processing unit includes: a calculation subunit for preprocessing the first slice to obtain first intermediate image data; a reading subunit for reading pre-stored second intermediate image data, wherein the second intermediate image data is obtained by performing the preprocessing on a portion of the third slice; and a convolution calculation subunit for performing convolution calculation on the spliced ​​data of the second intermediate image data and the first intermediate image data.

[0033] In one embodiment of the third aspect of this application, the image processing device further includes a fourth reading unit deployed in the neural network processor, configured to read a portion of the third slice from the second cache before the neural network processor reads the first slice from the first cache. The processing unit includes a processing subunit configured to process the spliced ​​data of the portion of the third slice and the first slice to obtain intermediate image data, and a cropping subunit configured to crop the intermediate image data.

[0034] In one embodiment of the third aspect of this application, the image processing device further includes: a first notification unit deployed on the first processor, configured to notify the neural network processor to read a slice from the system cache after the first processor writes a first slice of the target image into the system cache; and a second notification unit deployed on the neural network processor, configured to notify the first processor to read an output slice from the system cache after the neural network processor writes the first output slice into the system cache.

[0035] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed in an image processing device, causes the image processing device to perform the method described in the first aspect of this application.

[0036] The fifth aspect of this application provides an image processing apparatus, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect of this application.

[0037] The sixth aspect of this application provides a computer program product that, when run in an image processing device, causes the image processing device to perform the method described in the first aspect of this application.

[0038] In this embodiment, by dividing image data into multiple segments for separate processing, intermediate data during the Artificial Intelligence (AI) processing can be stored in the system cache, significantly reducing AI processing time and system power consumption. In video scenarios, this ensures real-time video quality and improves image quality. After segmenting the image data, this embodiment also provides an overlapping data pre-storage method and a lossy overlap processing method to overlap the image segments, thereby reducing the additional computational cost to the NPU caused by image data segmentation and ensuring NPU performance. Furthermore, by enabling the ISP and NPU to work collaboratively, the efficiency of AI processing is further improved, saving system power consumption. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the mobile terminal structure; Figure 2 This is a schematic diagram of the CNN structure; Figure 3 This is a schematic diagram illustrating the process of convolving image data using a convolution kernel. Figure 4 This is a schematic diagram illustrating the process of AI processing of images. Figure 5This is a schematic diagram illustrating the AI ​​processing of images in an embodiment of this application; Figure 6 A flowchart illustrating an image processing method provided in an embodiment of this application; Figure 7 This is a schematic diagram of segmenting image 51; Figure 8 A schematic diagram of the structure of the SC provided in the embodiments of this application; Figure 9 This is a schematic diagram illustrating the process of NPU processing slice 511; Figure 10 A schematic diagram illustrating the sequential processing of fragments 511, 512, and 513 by the ISP and NPU; Figure 11 A flowchart illustrating a method for processing fragments provided in this application embodiment; Figure 12 This is a schematic diagram illustrating the process of the NPU processing slice 512; Figure 13 This is a schematic diagram illustrating the process of NPU processing slice 513; Figure 14 A flowchart illustrating another method for processing image segments provided in this application embodiment; Figure 15 This is a schematic diagram illustrating the process of the NPU processing slice 512; Figure 16 This is a schematic diagram of a method for performing lossy processing on fragments, provided in an embodiment of this application. Figure 17 This is a schematic diagram of an image processing device provided in an embodiment of this application. Detailed Implementation

[0040] The image signal processing method and apparatus provided in this application can be applied to electronic devices, such as mobile terminals, mobile stations (MS), user equipment (UE), and other mobile devices, as well as fixed devices such as landline telephones, desktop computers, and video surveillance equipment. These electronic devices are image acquisition and processing devices with image signal acquisition and processing functions. They may also selectively have wireless connectivity to provide users with voice and / or data connectivity via handheld devices or other processing devices connected to a wireless modem. For example, the electronic device can be a mobile phone (or "cellular" phone), a computer with a mobile terminal, a portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile device, or a wearable device (such as a smartwatch, smart bracelet), tablet computer, personal computer (PC), personal digital assistant (PDA), point of sale (POS), etc. In this application embodiment, the following description uses a mobile terminal as an example of an electronic device.

[0041] Figure 1 This is a schematic diagram of the structure of mobile terminal 100. Mobile terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 151, a wireless communication module 152, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180A, an accelerometer sensor 180B, a proximity sensor 180G, a fingerprint sensor 180H, and a touch sensor 180K. Of course, the mobile terminal 100 may also include other sensors, such as temperature sensors, pressure sensors, distance sensors, magnetic sensors, ambient light sensors, barometric pressure sensors, bone conduction sensors, etc. Figure 1 Not shown in the image.

[0042] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile terminal 100. In other embodiments of this application, the mobile terminal 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0043] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), central processing unit (CPU), video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, processor 110 may be a chip or chipset. For example, the application processor may be the CPU.

[0044] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a system cache (SC), which is an internal cache of the processor 110, such as an on-chip cache, and features high bandwidth and low latency. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated memory accesses outside the processor 110, reduces the processor 110's latency, and thus improves system efficiency. Figure 1 The processor 110 schematically shows the CPU, ISP, NPU and SC, which can be connected via a bus.

[0045] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, mobile terminal 100 may include one or N displays screens 194, where N is a positive integer greater than 1. Display screen 194 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces (GUIs). For example, display screen 194 can display photos, videos, web pages, or documents, etc.

[0046] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the mobile terminal 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, photos, videos, and other files can be saved on the external storage card.

[0047] Internal memory 121, also called main memory, can be used to store executable program code, including instructions. Processor 110 executes various functional applications and data processing of mobile terminal 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area can store the operating system, application code (such as camera applications, WeChat applications, etc.), etc. The data storage area can store data created during the use of mobile terminal 100 (such as images and videos captured by camera applications, etc.).

[0048] In addition, the internal memory 121 may include random access memory (RAM), such as double data rate synchronous dynamic random access memory (DDR memory), and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0049] The functions of sensor module 180 are described below.

[0050] The gyroscope sensor 180A can be used to determine the motion attitude of the mobile terminal 100. In some embodiments, the gyroscope sensor 180A can determine the angular velocity of the mobile terminal 100 about three axes (i.e., the x, y, and z axes). That is, the gyroscope sensor 180A can be used to detect the current motion state of the mobile terminal 100, such as whether it is shaking or stationary.

[0051] Accelerometer 180B can detect the magnitude of acceleration of mobile terminal 100 in various directions (generally three axes). That is, gyroscope sensor 180A can be used to detect the current motion state of mobile terminal 100, such as shaking or being stationary.

[0052] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The mobile terminal emits infrared light outward through the LED. The mobile terminal uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that an object is near the mobile terminal. When insufficient reflected light is detected, the mobile terminal can determine that no object is near the mobile terminal.

[0053] The gyroscope sensor 180A (or accelerometer 180B) can send the detected motion state information (such as angular velocity) to the processor 110. The processor 110 determines whether the current state is handheld or tripod based on the motion state information (for example, if the angular velocity is not 0, it means that the mobile terminal 100 is in handheld state).

[0054] The fingerprint sensor 180H is used to collect fingerprints. The mobile terminal 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0055] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K is used to detect touch operations applied to or near it. The touch sensor can then transmit the detected touch operation to the application processor to determine the type of touch event.

[0056] For example, the display screen 194 of the mobile terminal 100 displays a main interface, which includes icons for multiple applications (such as a camera application, a WeChat application, etc.). The user taps the camera application icon on the main interface using the touch sensor 180K, triggering the processor 110 to launch the camera application and open the camera 193. The display screen 194 then displays the camera application's interface, such as a viewfinder.

[0057] The wireless communication function of the mobile terminal 100 can be implemented through antenna 1, antenna 2, mobile communication module 151, wireless communication module 152, modem processor, and baseband processor.

[0058] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile terminal 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0059] The mobile communication module 151 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the mobile terminal 100. The mobile communication module 151 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 151 can receive electromagnetic waves via the antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to the modem processor for demodulation. The mobile communication module 151 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 151 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 151 and at least some modules of the processor 110 may be housed in the same device. In this embodiment, the mobile communication module 151 can also be used to interact with other terminal devices, i.e., send audio output requests to other terminal devices, or the mobile communication module 151 can be used to receive audio output requests and encapsulate the received audio output requests into a message of a specified format.

[0060] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 151 or other functional modules.

[0061] The wireless communication module 152 can provide solutions for wireless communication applications on the mobile terminal 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 152 can be one or more devices integrating at least one communication processing module. The wireless communication module 152 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 152 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0062] In addition, the mobile terminal 100 can implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor. Examples include music playback and recording. The mobile terminal 100 can receive key input 190, generating key signal inputs related to user settings and function control. The mobile terminal 100 can use a motor 191 to generate vibration alerts (such as vibration alerts for incoming calls). The indicator 192 in the mobile terminal 100 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. The SIM card interface 195 in the mobile terminal 100 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the mobile terminal 100.

[0063] The mobile terminal 100 implements display functions through a GPU, a display screen 194, and a processor 110. The GPU is a microprocessor for image processing, connected to the display screen 394 and the application processor, used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. Optionally, the GPU may be located within the processor 110 or be a separate device located outside the processor 110.

[0064] Mobile terminal 100 can perform shooting functions through an ISP, NPU, camera 193, video codec, GPU, display 194, and application processor. Camera 193 may include a lens assembly and a photosensitive element. For example, when mobile terminal 100 takes a picture, it opens the shutter, and light is transmitted through the lens to the camera's photosensitive element. The photosensitive element converts the light signal into an electrical signal and transmits the electrical signal to the ISP for processing. The ISP processes the data received from camera 193, including processing the data with the aid of the NPU to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for further processing. The DSP converts the digital image signal into a standard RGB, YUV, or other format image signal, which can be displayed on the display 194 after processing by the GPU and application processor. In some embodiments, mobile terminal 100 may include one or N cameras 393, where N is a positive integer greater than 1. The video codec is used to compress or decompress digital video. Mobile terminal 100 may support one or more video codecs. In this way, the mobile terminal 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0065] Specifically, the processing of image signals by the ISP can include the following aspects: 1. Correction and compensation: Defective pixel correction (DPC), black level compensation (BLC), lens distortion correction (LSC), geometric correction for distortion, stretching, and offset, gamma correction, and corrections related to perspective principles, etc. 2. Denoising and Image Enhancement: Temporal and spatial filtering, hierarchical compensation filtering, removal of various noises, sharpening, suppression of ringing effects and banding artifacts, edge enhancement, brightness enhancement, and contrast enhancement; 3. Color and format conversion: color interpolation Demosaic (raw->RGB), color space conversion RGB->YUV or YCbCr or YPbPr, tone mapping, chroma adjustment, color correction, saturation adjustment, scaling, rotation, etc. 4. Adaptive processing: automatic white balance, automatic exposure, automatic focus, flicker detection, etc.; 5. Visual recognition (face and pose recognition) and image processing in extreme environments, including vibration, rapid movement, low light, and excessive brightness. The processing involved generally includes deblurring, point spread function estimation, brightness compensation, motion detection, motion capture, image stabilization, and high dynamic range (HDR) image processing.

[0066] It is understood that the ISP involved in the embodiments of this application may include one or a group of chips, that is, it may be integrated or standalone. For example, Figure 1 The ISP is located in processor 110. Alternatively, the ISP included in mobile terminal 100 may be an integrated ISP chip integrated into another computing processor.

[0067] The NPU is suitable for performing AI processing on captured images based on a neural network (NN). In this embodiment, the NPU can, for example, perform image processing such as color interpolation (Demosasic) or HDR on the image signal, thereby significantly improving image quality. The NPU can operate independently of... Figure 1 The processor 110 shown in the figure can be configured or integrated into it. Figure 1 The processor 110 shown is an example. The NPU can also be a type distinct from... Figure 1The NPU is a special processing unit that operates on the CPU or application processor shown. For example, the NPU can be a CPU running an operating system, or it can be other types of computing devices, such as a dedicated hardware acceleration processor. In this embodiment, the NPU is described as independent of the CPU or application processor.

[0068] The neural network mentioned here is a network structure that mimics the behavioral characteristics of animal neural networks for information processing, also known simply as an artificial neural network (ANN). The neural network can include at least one of several types of neural networks, such as convolutional neural networks (CNN), deep neural networks (DNN), and recurrent neural networks (RNN). The neural network structure consists of a large number of interconnected nodes (or neurons), and it processes information by learning and training on input information based on a specific computational model. A neural network includes an input layer, hidden layers, and an output layer. The input layer receives the input signal, the output layer outputs the computational results of the neural network, and the hidden layers are responsible for learning, training, and other computational processes; they are the network's memory units. The memory function of the hidden layers is represented by a weight matrix, typically with each neuron corresponding to a weight coefficient.

[0069] Since CNNs are more suitable for processing image signals, this application embodiment will use CNNs as an example to describe the NPU's image processing. Figure 2 This is a schematic diagram of the CNN structure. Figure 2 As shown, the CNN includes an input layer 21, at least one convolutional layer 22, and an output layer 23. The input layer 21 is used to preprocess the raw image data, such as mean removal or normalization. Each convolutional layer 22 includes at least one convolutional kernel, also known as a feature extraction filter. This kernel is equivalent to a neuron in the neural network, corresponding to specific features of the image (such as edge features or embossing features). After performing convolution calculations on the image using this kernel, a feature image of the specific features of the image can be output.

[0070] Figure 3 This is a schematic diagram illustrating the process of convolving image data using a convolution kernel. For example... Figure 3 As shown, the input image 31 is the matrix data indicated by the white box in the left rectangle, which is the input. Figure 2Image data is input to convolutional layer 22 from input layer 21, where each element in the matrix data represents a pixel of the corresponding image. The convolutional kernel 33 is any convolutional kernel in convolutional layer 22, and the feature image 34 is the image data output from convolutional layer 22. The convolutional kernel 33 is a two-dimensional matrix with equal rows and columns (shown as a 3×3 matrix in the figure). During convolution, the convolutional kernel 33 slides across the two-dimensional data (i.e., pixel values) of the image and performs inner product calculations on the overlapping local data in the image. The length of each slide of the convolutional kernel is called the stride; for example, if the convolutional kernel slides one pixel at a time, the stride is 1.

[0071] In some cases, such as to ensure that the size of the feature image output by the convolutional layer is the same as the size of the input image, it is necessary to pad the input image before performing the convolution calculation. Typically, padding is applied to the four sides of the input image. The total number of rows and columns to be padded is equal to the side length of the convolution kernel minus 1. For example, in... Figure 3 In this example, the convolution kernel has a side length of 3 (i.e., the number of pixels), meaning it needs to fill 3-1=2 rows and 2 columns, that is, one row on each of the four sides of the image. The values ​​of the elements in these filled rows or columns are, for example, 0. Figure 3 As shown, in order to output a 6-row × 8-column feature image 34 after convolving the 6-row × 8-column input image 31 with the convolution kernel shown in the figure, firstly, a row of padding pixels 32 is added to each of the four sides of the input image 31 according to the size of the convolution kernel 33 (e.g., ...). Figure 3 (As shown in the medium gray squares), and a convolution kernel is used to perform convolution calculations on the padded data. Specifically, as... Figure 3 As shown, convolution kernel 33 first interacts with... Figure 3 The top-left nine pixels of the padded image on the left are multiplied to obtain the value of the top-left pixel of the feature image 34. After this calculation, the convolution kernel 33 moves horizontally one pixel to the right in the padded image to calculate the value of the second pixel in the first row of the feature image 34. The convolution kernel 33 slides sequentially in the padded image until the value of each pixel in the feature image 34 is calculated. The following description will take the padded input image of the convolution layer as an example. It can be understood that the technical solution of this application embodiment is not limited to performing convolution calculation after padded image during image processing. For example, if the output image is not required to be the same size as the input image, then image padded is not necessary.

[0072] CNNs typically include activation layers and pooling layers between adjacent convolutional layers. Figure 2(Not shown in the image), the activation layer, for example, uses a rectified linear unit (ReLU) to non-linearly map the output of the convolutional layer, and the pooling layer is used to compress the amount of data and parameters. The output layer 23 is, for example, a fully connected layer, to weight and fuse feature images generated based on at least one convolutional kernel included in the nearest convolutional layer and then output them.

[0073] Figure 4 This is a schematic diagram illustrating the AI ​​processing of images in related technologies. For example... Figure 4 As shown, after receiving image data 40 from the photosensitive element, the ISP processes the image data 40 to obtain image data 41. Then, the ISP stores the image data 41 in the main memory (DDR memory is used as an example). The NPU reads the DDR memory to retrieve the image data 41. Then, the NPU performs AI processing on the image data 41 based on a neural network (e.g., CNN) model to generate image data 42, and stores the image data 42 in the DDR memory. Thus, the ISP or DSP reads the image data 42 from the DDR memory. In this related technology, in order to perform AI processing on the image data by the NPU, both the ISP and the NPU need to frequently access (read or write) the DDR memory, resulting in long AI processing times and high system power consumption. Therefore, the following embodiment proposes an improved technical solution to reduce time consumption and power consumption.

[0074] Figure 5 This is a schematic diagram illustrating the AI ​​processing of images in an embodiment of this application. In this embodiment, since the SC (Search Engine Controller) is typically located within the chip, its storage space is usually small and cannot store all the image data of a single frame of a photo or video. To facilitate data interaction between the ISP (Internet Service Provider) and NPU (Numerous Processing Unit) during AI processing of image data, the image data is divided into smaller data segments for separate AI processing, thereby avoiding frequent access to main memory, such as DDR memory. Specifically, as shown... Figure 5As shown, image data 50 may be, for example, image data of a photograph taken by a camera, or image data of a frame from a video captured by a camera. After receiving image data 50 from the photosensitive element, the ISP processes the image data 50 to obtain image 51. Then, based on a predetermined segmentation rule, the ISP sequentially reads a predetermined number of rows from image 51 and stores them in the SC, thereby sequentially storing, for example, three segments 511, 512, and 513 of image 51 into the SC. After storing segment 511 into the SC, the ISP may notify the NPU to read the segmented data, so that the NPU can read segment 511 from the SC and perform AI processing on segment 511 to generate an output segment 521 corresponding to segment 511, and store the output segment 521 into the SC. After storing the output segment 521 into the SC, the NPU may notify the ISP to read the AI-processed segment, so that the ISP can read the output segment 521 from the SC. By processing slices 512 and 513 sequentially using the same method, the ISP can finally obtain output slices 521, 522, and 523. Then, the ISP stitches together output slices 521, 522, and 523 to obtain the complete AI-processed output image 52 corresponding to image 51. In the above embodiment, one slice corresponds to one or more rows in the image data; in reality, it can also correspond to one or more columns or any part of the image data. This embodiment only uses one or more rows of image data as an example and is not intended to limit the scope.

[0075] In this embodiment, by dividing image data into multiple slices for separate processing, data during AI processing can be stored using SC, significantly reducing AI processing time and system power consumption. In video scenarios, this ensures real-time video quality and improves image quality. After segmenting the image data, to ensure output correctness, the NPU typically needs to consider overlap processing between slices. This embodiment also provides an overlap data pre-storage method and a lossy overlap processing method to perform overlap processing, thereby reducing the additional computational cost of the NPU caused by image data segmentation and ensuring NPU performance. Furthermore, by enabling the ISP and NPU to work collaboratively, the efficiency of AI processing is further improved, saving system power consumption. The image processing method provided by this embodiment will be described in detail below.

[0076] Figure 6 This is a flowchart of an image processing method provided in an embodiment of this application. Figure 6 The diagram shows that the process is jointly performed by the ISP and NPU. Figure 6 The method shown is not limited to being executed by the ISP and NPU. For example, Figure 6The method shown can be executed by the NPU and other processors in the image processing device, such as a CPU or DSP. Figure 6 This introduction will only use the collaborative processing of ISP and NPU as an example.

[0077] First, in step S602, the ISP writes the image segments to the SC. (See reference) Figure 5 The image is, for example, Figure 5 In image 51, the ISP can sequentially read each row of pixels from the first row of pixels in image 51, and write that row of pixels into the SC after reading it. The ISP stops reading pixels from image 51 after reading and writing a predetermined number of rows of pixels into the SC. That is, the predetermined number of rows of pixels constitutes the first slice 511 of image 51. The predetermined number can be a preset number determined according to the system design, or it can be the result of negotiation between the ISP and the NPU relative to the current image or video. Typically, the predetermined number is determined based on the available space of the SC and the network processing architecture requirements of the NPU. For example, the predetermined number can be set to be greater than or equal to the size of the CNN convolution kernel, and the predetermined number is set such that the slice size is smaller than the available space of the SC. Figure 7 This is a schematic diagram illustrating the segmentation of image 51. (For example...) Figure 7 As shown, assuming image 51 comprises 17 rows × 20 columns of pixels, the ISP can first read... Figure 7 The first 6 rows of pixels are written into the SC as fragment 511, which can be read sequentially by the ISP in subsequent processing. Figure 7 The middle 6 rows of pixels are written into SC as fragment 512, and the last 5 rows of pixels are read and written into SC as fragment 513.

[0078] In one implementation, the SC includes an input area and an output area. The input area stores slices to be processed by the NPU, and the output area stores output slices after NPU processing. Thus, the ISP writes the slices into the input area of ​​the SC. By setting input and output areas in the SC, after writing one slice, and after the NPU generates an output slice based on that slice, the ISP can immediately write the next slice into the input area, without waiting for the NPU to write the output slice into the SC and for the ISP to read the output slice from the SC before writing the next slice, further saving AI processing time.

[0079] In one implementation, as in Figure 8The schematic diagram of the SC provided in the embodiment of this application shows that the SC includes an input area 81 and an output area 82. The input area 81 includes a buffer 1 and a buffer 2, and the output area 82 includes a buffer 3 and a buffer 4. By configuring the SC in this way, it can be used via PingPone caching. Specifically, for example, the ISP writes the segments 511 cut from image 51 to... Figure 8 In buffer 1, after the write operation, the ISP can continuously write the second slice (i.e., slice 512) from image 51 into buffer 2 without needing to write slice 512 after the NPU has finished processing slice 511. While the ISP is writing slice 512 into buffer 2, the NPU can simultaneously read slice 511 from buffer 1 for processing. Similarly, when the NPU writes output slices (e.g., ...) into buffer 4... Figure 5 While output fragment 522 is being written, the ISP can simultaneously read output fragment 521 written to buffer 3 by the NPU before being written to output fragment 522. Thus, Figure 8 The structure of SC shown further reduces the processing time of AI.

[0080] In step S604, the ISP notifies the NPU to read the fragment from the SC. For example... Figure 6 As shown, this step is illustrated with a dashed arrow to indicate that it is optional. Any other implementation that allows the NPU to know that the ISP has completed writing, besides the method of the ISP notifying the NPU, is within the scope of protection of the embodiments of this application. For example, the ISP can lock the write buffer when writing to the SC, and the NPU can determine whether the ISP has completed writing by polling whether the buffer is unlocked.

[0081] By notifying the NPU after writing fragments to the SC via the ISP, system processing efficiency can be improved and system power consumption can be saved. For example, the ISP can notify the NPU to read fragments from the SC through inter-core service communication. When the ISP writes, for example, fragment 511 to... Figure 8 When the ISP is in buffer 1, it can notify the NPU to read the fragment from buffer 1. Alternatively, the ISP can indicate that it has written the fragment to the SC by modifying the value of a variable stored in the SC. After the NPU determines that the variable value has been modified, it can determine that the ISP has written the fragment and then execute the subsequent step S606.

[0082] In step S606, the NPU reads the fragments from the SC. For example, as... Figure 5 and Figure 8As shown, the ISP writes fragment 511 into buffer 1, so that the NPU can read fragment 511 from buffer 1. In one embodiment, the NPU can read fragment 511 from buffer 1 based on a notification from the ISP.

[0083] In step S608, the NPU processes the image slices to obtain output slices. The processing of image slices by the NPU is basically the same as the processing of the entire image by the NPU. Figure 9 This is a schematic diagram illustrating the process of the NPU processing slice 511. Assume the NPU is based on... Figure 2 The CNN model shown processes slices, assuming the CNN model contains only one convolutional layer. Figure 9 As shown, after the NPU reads fragment 511 from the SC, it first performs operation ①, using... Figure 2 The input layer 21 preprocesses the slice 511, and then performs operation ②, adding fill pixels to the top and sides of the preprocessed slice 511. Figure 9 (Shown in small gray boxes) to ensure the final processing result is consistent with the result of processing the entire image 51. Next, the NPU performs operation ③, using a 3×3 convolution kernel to perform convolution calculations on the padded data. After the convolution calculation, the NPU inputs the calculation results of the convolution layer into... Figure 2 The output layer 23 in the middle can output 5×20 output segments 521.

[0084] During the aforementioned operations ① to ③, the NPU can read a portion of the data from buffer 511 from buffer 1 according to the order in which the convolution kernel slides within the fragment 511, and perform operations ① to ③ on that portion of data. By sequentially performing operations ① to ③ on each portion of data in fragment 511, the output fragment 521 is finally obtained. That is, the NPU processes the fragments while reading them. Optionally, after processing fragment 511, the NPU can notify the ISP. Based on this notification, the ISP knows that buffer 1 is currently writable. Therefore, after completing the writing of fragment 512 to buffer 2, the ISP can continue writing fragment 513 to buffer 1.

[0085] When the NPU processes each slice starting from the second slice of the image, in order to ensure that the final processing result is consistent with or substantially consistent with the result of processing the entire image 51, in addition to performing convolution calculations on the current slice, it is also necessary to perform convolution calculations on the data corresponding to the pixels of several rows at the bottom of the previous slice together, that is, to perform overlap processing between slices. The following will refer to... Figures 11-16 The diagram details the NPU's processing of fragments, including overlapping operations.

[0086] In step S610, the NPU writes the output fragments to the SC. (See reference...) Figure 5 After generating output segment 521, the NPU can write this output segment 521 into the SC. Figure 6 In the case of SC shown, the NPU can write output fragments into, for example, buffer 3.

[0087] In step S612, the NPU notifies the ISP to read the output fragments from the SC. Similar to step S604, this step is optional. By having the NPU notify the ISP after writing the output fragments to the SC, the system processing efficiency can be improved and system power consumption can be saved.

[0088] In step S614, the ISP reads the output fragments from the SC. (See reference...) Figure 5 After the NPU writes output fragment 521 to the SC, the ISP can read output fragment 521 from the SC. Since the ISP and NPU process fragments 512 and 513 sequentially, the ISP can read output fragments 522 and 523 sequentially from the SC. Then, the ISP can obtain the processed output image 52 by splicing output fragments 521, 522, and 523, and can send this output image 52, for example, to the DSP for subsequent processing. Alternatively, the ISP can perform the subsequent processing itself; this embodiment is not limited to this.

[0089] Figure 10 A schematic diagram illustrating the sequential processing of fragments 511, 512, and 513 by the ISP and NPU. Figure 10 In this diagram, each horizontal row corresponds to a buffer, and the operations within that row represent operations performed on the corresponding buffer. These operations include write and read operations. Each vertical column corresponds to a time period, and the operations within that column represent operations that occurred within that time period. Figure 10 Although each operation is represented by a square, this does not mean that the time periods are strictly equal. The length of each time period is determined based on the negotiation between the ISP and the NPU. Figure 10 The white boxes indicate ISP write operations, the gray boxes indicate NPU read operations, the light-shaded boxes indicate NPU write operations, and the dark-shaded boxes indicate ISP read operations. The numbers in each box indicate... Figure 5 The various fragments in the process.

[0090] like Figure 10 As shown, firstly, during time period t1, the ISP reads a predetermined number of rows from image 51 and writes them into buffer 1 after reading, thereby writing fragment 511 into buffer 1. After completing this write, the ISP notifies the NPU to read the fragment from buffer 1, thus ending time period t1 and entering time period t2.

[0091] During time period t2, the NPU sequentially reads partial data (e.g., data corresponding to the convolution kernel size) from buffer 1 based on the ISP's notification. After reading each partial data, it processes that partial data to obtain the output data and sequentially writes the output data into buffer 3 until all data of slice 511 has been read, and all output data is written into buffer 3, i.e., output slice 521 is written into buffer 3. After writing output slice 521, the NPU can notify the ISP to read buffer 3. While the NPU is performing the above operations, the ISP continues to read a predetermined number of rows from image 51, and after reading each row, writes the data of that row into buffer 2, thereby writing slice 512 into buffer 2. After writing slice 512, the ISP can notify the NPU to read buffer 2. After the NPU notifies the ISP to read buffer 3 and the ISP notifies the NPU to read buffer 2, time period t3 begins.

[0092] During time interval t3, to perform overlap processing, the NPU first reads several rows of data from the end of slice 511 from buffer 1 as overlap data for overlap processing, and writes the processed data into buffer 4. After processing these rows of pixels, slice 512 is read from buffer 2 for further processing, and finally, output slice 522 is written into buffer 4. It can be understood that, for overlap processing, it is not necessarily necessary to read only part of the data from slice 511 from buffer 1; alternatively, it could read some intermediate image data pre-stored by the NPU after processing slice 511, which will be discussed below. Figure 14 and Figure 15 Detailed Description. After writing output fragment 522, the NPU notifies the ISP to read buffer 4. During the NPU's writing of output fragment 522, after processing several rows of data in fragment 511, the NPU notifies the ISP to write to buffer 1. The ISP then continues to read a predetermined number of rows from image 51 and writes them to buffer 1, thus writing fragment 513. Afterward, the ISP reads output fragment 521 from buffer 4. After reading output fragment 521, the ISP notifies the NPU to write to buffer 3, thus entering time period t4.

[0093] During time period t4, similarly, the NPU first reads a portion of the data from fragment 512 in buffer 2 for overlay processing, then reads from buffer 1 and writes the final output fragment 523 into buffer 3. Simultaneously, the ISP reads output fragment 522 from buffer 4. After writing fragment 523, the NPU notifies the ISP to read from buffer 3, thus entering time period t5. During time period t5, the ISP reads output fragment 523 from buffer 3. Having read output fragments 521, 522, and 523 respectively, the ISP can synthesize the output image 52 based on these three output fragments.

[0094] The overlapping processing method for image segments provided in this application will be described below through various implementation methods. Figure 11 This is a flowchart of a method for processing fragments provided in an embodiment of this application. The method is executed by an NPU. Figure 12 This is a schematic diagram illustrating the NPU's processing of slice 512. Figure 13 This is a schematic diagram illustrating the NPU's processing of fragment 513. The following text will combine... Figures 11-13 Describe it.

[0095] like Figure 11 As shown, in step S111, the overlapping row image data corresponding to the previous slice and the image data corresponding to the current slice are obtained. The image data is matrix data including pixel values.

[0096] Assuming NPU is based on Figure 2 The CNN model shown processes slices, and it is assumed that the CNN model includes one convolutional layer. Figure 12 As shown on the left, the NPU in... Figure 9 After processing segment 511, in order to ensure that the final output image is consistent with the output image of the entire image 51, it is necessary to first read the two lines of data at the bottom of segment 511. Figure 12 (Shown in small shaded boxes) as overlapping rows for overlapping processing, and then the data in slice 512 is read. The number of overlapping rows H is equal to the difference between the side length of the convolution kernel and the stride. For example, in this example, the side length of the convolution kernel is 3 and the stride is 1, then the number of overlapping rows H = 3 - 1 = 2.

[0097] In CNN models that include more than one convolutional layer, for example, a CNN model that sequentially includes an input layer, a first convolutional layer, an activation layer, a second convolutional layer, and an output layer, the NPU can... Figure 9 During the processing of slice 511, overlapping row data in the intermediate image data output by the first convolutional layer can be pre-stored in SC or DDR memory. After the first convolutional layer completes the processing of slice 512 and outputs intermediate image data, the NPU can read the pre-stored overlapping row data and use the overlapping row data and the intermediate image data corresponding to slice 512 as input activation layer data.

[0098] In step S112, the acquired image data is preprocessed. (Still referring to...) Figure 12 As shown on the left, after the NPU reads the overlapping rows of fragment 511 and fragment 512, it inputs... Figure 2 The input layer 21 is used to preprocess the read data. This is performed after the first convolutional layer processing is complete. Figure 11In the case of the method shown, the input data is preprocessed by the activation layer.

[0099] It is understandable that, depending on the CNN network structure, it is possible that a layer for preprocessing image data may not be included in the CNN model. Therefore, this step is as follows: Figure 11 The dashed lines in the diagram represent optional steps, meaning that the image data preprocessing process may not be included in the process of processing image data through the CNN model.

[0100] In step S113, padding pixels are added to the preprocessed image data. Similarly, to ensure the final processing result is the same as the result of processing the entire image 51, as follows... Figure 12 As shown in the middle section, the NPU adds padding pixels to both sides of the preprocessed data. As mentioned above, depending on the requirements for the output image, adding padding pixels to the image data is not always necessary; therefore, this step is optional.

[0101] In step S114, convolution calculations are performed on the image data. After the NPU performs the padding, it inputs the padded data into the convolutional layer, whereby the convolutional layer performs convolution calculations on the padded data using a 3×3 convolution kernel. Figure 12 As shown in the right-hand side, after performing convolution calculations, the NPU inputs the calculation results of the convolutional layer into... Figure 2 The output layer 23 is used to output 6×20 output fragments 522 from the output layer 23.

[0102] In the case described above where the second convolutional layer performs convolution calculations on the intermediate image data of slice 522, the NPU inputs the padded data into the second convolutional layer for convolution calculations and finally outputs a 6×20 output slice 522.

[0103] refer to Figure 13 When the NPU processes fragment 513, the processing procedure is the same as... Figure 12 The processes shown are basically the same, the difference being that, for example... Figure 13 As shown in the middle section, since slice 513 includes data from the bottom of image 51, the NPU adds padding pixels to the sides and bottom of the two overlapping rows and the preprocessed data of slice 513. The NPU performs the following... Figure 13 After the processing shown, the final output is 6×20 output fragments, 523.

[0104] The image obtained by stitching together the output segments 521, 522, and 523 is consistent with the image output from processing image 51 directly input into the same CNN model. Figure 12 and Figure 13In the segmentation processing method shown, compared to processing the complete image 51, the NPU needs to read 4 more rows of overlapping data from the SC, and the input layer 21 performs two repeated calculations on the overlapping rows, resulting in a large computational loss.

[0105] Figure 14 This is a flowchart illustrating another method for processing image segments according to an embodiment of this application. This method is executed by an NPU. Figure 15 This is a schematic diagram illustrating the NPU's processing of slice 512. The following will combine... Figure 14 and Figure 15 Describe it.

[0106] like Figure 14 As shown, firstly, in step S141, the image data corresponding to the current slice is obtained. Assume the NPU is based on... Figure 2 The CNN model shown processes slices, and it is assumed that this CNN model includes one convolutional layer. (Reference) Figure 15 On the left side, after the NPU starts processing slice 512, slice 512 is read from the SC.

[0107] In step S142, the acquired image data is preprocessed. (See reference...) Figure 15 On the left side, the NPU passes through Figure 2 The input layer 21 in the middle preprocesses the data in the fragment 512.

[0108] In step S143, the pre-stored overlapping row data corresponding to the previous slice is read. In this embodiment, during the processing of slice 511, after inputting slice 511 into the input layer 21 for preprocessing, the NPU pre-stores the bottom two rows of data from the output data of the input layer 21 as the overlapping row data corresponding to slice 511. The NPU can pre-store this overlapping row data in SC or DDR memory. When the NPU processes slice 512, it can read the pre-stored overlapping row data from the corresponding address.

[0109] In step S144, padding pixels are added to the overlapping row data and the preprocessed data of the current slice. (See reference) Figure 15 The middle part, and Figure 12 Similarly, to ensure consistency in the calculation results, padding pixels are added to both sides of the data concatenated from the overlapping rows and the preprocessed data. Figure 11 Similarly, in the method shown, this step is optional.

[0110] In step S145, convolution is performed on the padded data. This step can be referred to the above description. Figure 11 The description of step S114 will not be repeated here.

[0111] exist Figure 14 In the overlapping processing method shown, the NPU pre-stores the overlapping row data in the preprocessed data after preprocessing the previous slice. Therefore, the NPU does not need to repeatedly read the overlapping rows of the previous slice from the SC, nor does it need to repeatedly preprocess the overlapping rows in the slice. This reduces the computational overhead caused by overlapping processing and improves system efficiency.

[0112] The above reference Figures 11-15 The described piecewise overlap processing is a lossless process where the result is consistent with the result of processing the complete image. However, in practical applications, a large number of overlapping rows need to be processed (e.g., 92 rows). This means that compared to processing the complete image, the convolutional layer needs to perform additional convolution calculations on the overlapping rows, resulting in a higher computational cost for the NPU. Therefore, this application provides a lossy overlap processing method. This method allows for overlap processing with a specified number of overlapping rows that is less than the exact number of overlapping rows, thereby reducing the computational cost of the NPU while maintaining image quality.

[0113] Figure 16 This is a schematic diagram illustrating a method for lossy processing of fragments provided in an embodiment of this application. Figure 16 As shown, the processing of slices 511, 512, and 513 is described as an example. The numbers within each rectangle represent the number of pixel rows in the image data corresponding to that rectangle. Assuming the convolution kernel has a side length of 5 pixels and a stride of 1 in this embodiment, then, as described above, to ensure the output image size is the same after convolution calculation, a total of 5-1=4 rows and 4 columns of pixels need to be filled. Furthermore, the number of overlapping rows used to ensure lossless calculation should be 5-1=4 rows. In lossy calculation in this embodiment, the number of overlapping rows can be specified as, for example, 2.

[0114] Specifically, see reference Figure 16 The NPU performs operations ④, ⑤, and ⑥ on fragments 511, 512, and 513 respectively, and finally obtains output fragments 521, 522, and 523. The processing of fragments 511, 512, and 513 will be described below.

[0115] After the ISP writes fragment 511 to the SC, the NPU performs operation ④, which reads fragment 511 from the SC and... Figure 2 The input layer 21 of the CNN model shown preprocesses slice 511 and pads each side of the preprocessed image data with two rows of pixels. Since slice 511 is the first slice segmented from image 51, overlapping processing of this slice is not required. Afterwards, the NPU performs operation ⑤, through... Figure 2 The convolutional layer shown uses, for example, a 5×5 convolutional kernel ( Figure 16 (As shown in the square marked 5) Convolution calculation is performed on the above-mentioned padded image data. For example... Figure 16 As shown, the padded image data corresponding to slice 511 has a total of 6 + 4 = 10 rows. Therefore, after convolution with a 5×5 kernel, the output will include image data with 10 - (5 - 1) = 6 rows of pixels. After the above convolution calculation, the NPU performs operation ⑥, cropping one row of pixels from the bottom edge of the image data output from the convolutional layer (in... Figure 16 (shown in a shaded dashed box), and through Figure 2 The output layer 23 processes the remaining 5 rows of image data after cropping, finally obtaining an output slice 521, which includes 5 rows of pixels. The number of cropped rows is determined based on the number of overlapping rows. For example, the number of cropped rows is half the number of overlapping rows. In this example, the specified number of overlapping rows is 2, therefore, the number of cropped rows is 2 / 2 = 1. Typically, the specified number of overlapping rows can be set to an even number. If the specified number of overlapping rows is not even, additional rules can be set for cropping the output image to ensure that the size of the output image is the same as the size of the input image. This embodiment does not limit this.

[0116] After the ISP writes fragment 512 to the SC, the NPU performs operation ④, which is similar to the above. Figure 11 Similarly, the NPU first reads two overlapping rows of slice 511 from the SC (shown as shaded solid boxes in the figure), then reads slice 512. After preprocessing both the overlapping rows and slice 512, two overlapping rows of pixels are added to each of the four sides of the preprocessed image data. Next, the NPU performs operation ⑤, performing convolution calculations on the 6+2+4=12 rows of image data using a 5×5 convolution kernel, thereby outputting 8 rows of image data. After the convolution calculation, the NPU performs operation ⑥, cropping one row of pixels from the top and bottom edges of the output 8 rows of image data, and performing output layer processing on the cropped 6 rows of image data, finally obtaining an output slice 522 containing 6 rows of pixels.

[0117] After the ISP writes slice 513 to the SC, the NPU performs the same processing on slice 513 as it does on slice 512. The difference is that in operation ⑥, since slice 513 is the last slice cut from image 51, only the top edge row of pixels is cropped from the 7 rows of image data output from the convolutional layer, thereby obtaining output slice 523 including 6 rows of pixels.

[0118] Will pass Figure 16The output segments 521, 522, and 523 obtained by the method shown are stitched together to obtain output image 52. Data verification shows that the image quality of output image 52 is not significantly lower than the output image corresponding to image 51 obtained through the lossless computation described above, while the computational cost of the NPU is significantly reduced. For example, in practice, each segment of the image requires reading 92 additional overlapping lines and performing extra computational processing (including preprocessing, convolution calculations, etc.). By using... Figure 16 The processing method shown allows the number of overlapping rows to be specified as 32. The NPU only needs to read and process 32 more overlapping rows for each shard, which effectively reduces the additional computational cost of the NPU caused by overlapping rows.

[0119] Figure 17 An image processing device 1700 is provided as an embodiment of this application, the image processing device 1700 being used to implement the above-described reference. Figures 5-16 The image processing method includes a first writing unit 171 deployed in a first processor of an image processing device, for writing a first slice of a target image into a system cache; and the following units deployed in a neural network processor of the image processing device: a first reading unit 172, for reading the first slice of the target image from the system cache; a processing unit 173, for processing the first slice to obtain a first output slice; and a second writing unit 174, for writing the first output slice into the system cache.

[0120] In one embodiment of this application, the first writing unit 171 is further configured to generate the target image before writing the first fragment of the target image into the system cache.

[0121] In one embodiment of this application, the image processing device 1700 further includes a second reading unit 175 deployed in the first processor of the image processing device, for reading the first output slice from the system cache and obtaining an output image based on the first output slice after the neural network processor writes the first output slice into the system cache.

[0122] In one embodiment of this application, the first processor includes an image signal processor.

[0123] In one embodiment of this application, the system cache includes an input area and an output area, wherein the first writing unit 171 is further configured to write a first slice of the target image into the input area; and the second writing unit 174 is further configured to write the first output slice into the output area.

[0124] In one embodiment of this application, the input area includes a first cache area and a second cache area. The first writing unit 171 is further configured to write the first slice into the first cache area. The image processing device further includes a third writing unit 176 deployed on the first processor, configured to write the second slice into the second cache area while the neural network processor reads the first slice from the input area, wherein the second slice is different from the first slice.

[0125] In one embodiment of this application, the output area includes a third buffer and a fourth buffer. The second writing unit 174 is further configured to write the first output fragment into the third buffer. The image processing device further includes a third reading unit 177 deployed in the first processor, configured to read a third output fragment corresponding to the third fragment in the target image from the fourth buffer while the neural network processor writes the first output fragment into the third buffer, wherein the third fragment is different from the first fragment.

[0126] In one embodiment, the processing unit 173 includes a calculation subunit 1731 for preprocessing the first slice to obtain first intermediate image data, a reading subunit 1732 for reading pre-stored second intermediate image data, wherein the second intermediate image data is obtained by performing the preprocessing on a portion of the third slice in the target image, and a convolution calculation subunit 1733 for performing convolution calculation on the spliced ​​data of the second intermediate image data and the first intermediate image data.

[0127] In one embodiment, the image processing device 1700 further includes a fourth reading unit 178 deployed in the neural network processor, for reading a portion of the third slice from the second cache before the neural network processor reads the first slice from the first cache. The processing unit 173 includes a processing subunit 1734 for processing the spliced ​​data of the portion of the third slice and the first slice to obtain intermediate image data, and a cropping subunit 1735 for cropping the intermediate image data.

[0128] The various units in the image processing device 1700 can be implemented through software, hardware, firmware, or any combination thereof, and are deployed, for example, in... Figure 1 In the processor 110 (e.g., ISP or NPU).

[0129] This application, in another aspect, provides an image processing apparatus for implementing the above-mentioned reference. Figures 5-16The image processing method includes a first processor and a neural network processor. The first processor is used to write a first slice of a target image into a system cache. The neural network processor is used to read the first slice of the target image from the system cache, process the first slice to obtain a first output slice, and write the first output slice into the system cache.

[0130] In one implementation, the first processor is further configured to generate the target image before writing a first slice of the target image into a system cache. The first processor is, for example, a... Figure 1 The CPU or ISP in processor 110.

[0131] In one embodiment, the first processor is further configured to read the first output slice from the system cache and obtain an output image based on the first output slice.

[0132] In one embodiment, the first processor includes an image signal processor.

[0133] In one embodiment, the image processing device further includes a system cache.

[0134] In one embodiment, the system cache further includes an input area and an output area, and the first processor is further configured to write the first slice into the input area; the neural network processor is further configured to write the first output slice into the output area.

[0135] In one embodiment, the input area includes a first buffer and a second buffer; the first processor is further configured to: write the first slice into the first buffer, and while the neural network processor reads the first slice from the input area, write the second slice of the target image into the second buffer, wherein the second slice is different from the first slice.

[0136] In one embodiment, the output area includes a third buffer and a fourth buffer; the neural network processor is further configured to write the first output slice into the third buffer; the first processor is further configured to: while the neural network processor writes the first output slice into the third buffer, read a third output slice corresponding to the third slice in the target image from the fourth buffer, wherein the third slice is different from the first slice.

[0137] In one embodiment, the neural network processor is further configured to: preprocess the first slice to obtain first intermediate image data, read pre-stored second intermediate image data, the second intermediate image data being obtained by preprocessing a portion of the third slice, and perform convolution calculation on the concatenated data of the second intermediate image data and the first intermediate image data.

[0138] In one embodiment, the neural network processor is further configured to: read a portion of the third slice from the second cache, perform convolution calculations on the concatenated data of the portion of the third slice and the first slice to obtain intermediate image data, and crop the intermediate image data.

[0139] This application, in another aspect, provides a computer-readable storage medium having a computer program stored thereon, which, when executed in an image processing device, causes the image processing device to perform the above-mentioned [reference]. Figures 5-16 The image processing method described above.

[0140] This application, in another aspect, provides an image processing device, including a memory and a processor. The memory stores executable code. The memory may be a main memory as previously described. The processor may refer to a previous processor 110, including an ISP and an NPU, for executing the executable code and implementing the above-described... Figures 5-16 The image processing method described above.

[0141] This application, in another aspect, provides a computer program product that, when run in an image processing device, causes the image processing device to perform the functions described above. Figures 5-16 The image processing method described above.

[0142] In this embodiment, by dividing image data into multiple slices for separate processing, intermediate data during AI processing can be stored using SC (System-Side Memory), significantly reducing AI processing time and system power consumption. In video scenarios, this ensures real-time video quality and improves image quality. After segmenting the image data, to ensure output correctness, the NPU typically needs to consider overlap processing between slices. This embodiment also provides an overlap data pre-storage method and a lossy overlap processing method to perform overlap processing, thereby reducing the additional computational cost to the NPU caused by image data segmentation and ensuring NPU performance. Furthermore, by enabling the ISP (Internet Service Provider) and NPU to work collaboratively, the efficiency of AI processing is further improved, and system power consumption is saved.

[0143] It should be understood that the descriptions such as "first" and "second" in this article are merely for the sake of simplicity in description and to distinguish similar concepts, and do not have any other limiting function.

[0144] Those skilled in the art will clearly understand that the descriptions of the various embodiments provided in this application can be referenced to each other. For the sake of convenience and brevity, for example, the functions and execution steps of the various devices and equipment provided in the embodiments of this application can be referred to the relevant descriptions of the method embodiments of this application. The method embodiments and the device embodiments can also be referenced to each other.

[0145] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs all or part of the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random-access memory (RAM), magnetic disk, or optical disk.

[0146] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0147] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways without exceeding the scope of this application. For example, the embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Those skilled in the art can understand and implement this without any creative effort.

[0148] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: The image signal processor in the chip processes the data received from the camera and generates the target image; The image signal processor writes the first slice of the target image into the on-chip cache of the chip; The neural network processor of the chip reads the first slice from the on-chip cache; The neural network processor processes the first slice based on a neural network model to obtain a first output slice. The processing of the first slice includes at least one of color interpolation or high dynamic range (HDR) processing. The neural network processor writes the first output slice into the on-chip cache; The neural network model includes multiple convolutional layers, and the on-chip cache is further used to store intermediate image data output by the first convolutional layer among the multiple convolutional layers.

2. The method according to claim 1, characterized in that, Also includes: After the neural network processor writes the first output slice into the on-chip cache, the image signal processor reads the first output slice from the on-chip cache and obtains the output image based on the first output slice.

3. The method according to claim 1, characterized in that, The on-chip cache includes an input area and an output area; wherein... The image signal processor writing the first slice of the target image into the on-chip buffer includes: the image signal processor writing the first slice into the input area; The neural network processor writing the first output fragment into the on-chip cache includes: the neural network processor writing the first output fragment into the output area.

4. The method according to claim 3, characterized in that, The input area includes a first buffer area and a second buffer area; The image signal processor writing the first slice into the input area includes: the image signal processor writing the first slice into the first buffer area; The method further includes: while the neural network processor reads the first slice from the input area, the image signal processor writes a second slice from the target image into the second buffer area, wherein the second slice is different from the first slice.

5. The method according to any one of claims 1-4, characterized in that, The neural network processor processes the first slice based on the neural network model, including: the neural network processor preprocesses the first slice to obtain first intermediate image data; the neural network processor reads pre-stored second intermediate image data, the second intermediate image data is obtained by performing the preprocessing on a portion of the third slice in the target image; and the neural network processor performs convolution calculation on the concatenated data of the second intermediate image data and the first intermediate image data.

6. The method according to any one of claims 1-4, characterized in that, The neural network processor processes the first slice based on a neural network model, including: the neural network processor performs convolution calculations on a portion of the third slice in the target image and the spliced ​​data of the first slice to obtain intermediate image data; the neural network processor then crops the intermediate image data to obtain the first output slice.

7. A chip, characterized in that, include: On-chip cache, image signal processor, and neural network processor; The image signal processor is used to process data received from the camera, generate a target image, and write a first slice of the target image into the on-chip buffer; The neural network processor is configured to read a first slice of the target image from the on-chip cache, process the first slice based on the neural network model to obtain a first output slice, and write the first output slice into the on-chip cache, wherein the processing of the first slice includes at least one of color interpolation or high dynamic range (HDR) processing. The neural network model includes multiple convolutional layers, and the on-chip cache is further used to store intermediate image data output by the first convolutional layer among the multiple convolutional layers.

8. The chip according to claim 7, wherein the image signal processor is further configured to: after the neural network processor writes the first output slice into the on-chip cache, read the first output slice from the on-chip cache, and obtain an output image based on the first output slice.

9. The chip according to claim 7, characterized in that, The on-chip cache includes an input area and an output area. The image signal processor is further configured to write the first slice into the input area; the neural network processor is further configured to write the first output slice into the output area.

10. The chip according to claim 9, characterized in that, The input area includes a first buffer area and a second buffer area; the image signal processor is further configured to: write the first slice into the first buffer area, and while the neural network processor reads the first slice from the input area, write the second slice of the target image into the second buffer area, wherein the second slice is different from the first slice.

11. The chip according to any one of claims 7-10, characterized in that, The neural network processor is further configured to: preprocess the first slice to obtain first intermediate image data, read pre-stored second intermediate image data, the second intermediate image data being obtained by preprocessing a portion of the third slice in the target image, and perform convolution calculation on the concatenated data of the second intermediate image data and the first intermediate image data.

12. The chip according to any one of claims 7-10, characterized in that, The neural network processor is further configured to: read a portion of a third slice from the target image from the second cache area of ​​the on-chip cache, perform processing including convolution calculation on the concatenated data of the portion of the third slice and the first slice to obtain intermediate image data, and crop the intermediate image data to obtain the first output slice.

13. The chip according to any one of claims 7-10, characterized in that, Also includes: Central processing unit (CPU), graphics processing unit (GPU), baseband processor, and video codec.

Images