Electronic device and image processing method of electronic device

Abstract

The embodiment of the present application provides a system on chip and an image processing method of the system on chip, the system on chip comprising: an image signal processor ISP, for receiving image data from an image sensor, and performing a third image signal processing on the image data to obtain a first image signal; an artificial intelligence AI processor, for performing a first image signal processing on the first image signal to obtain a second image signal, wherein the AI ​​processor comprises a dedicated neural processor; a memory, coupled to the AI ​​processor and the ISP, for transferring an image block in the first image signal between the ISP and the AI ​​processor, wherein the image block is a local image signal in a frame of the image signal in the first image signal; wherein the ISP stores the image block in the memory. This solution can flexibly combine the image processing of the ISP with the AI ​​processor, which is conducive to improving the image processing results.

Description

[0001] This application is a divisional application. The application number of the original application is 202080011168.6, and the original application date is September 16, 2020. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The embodiments of the present application relate to the field of electronic technology, and in particular, to an electronic device and an image processing method of the electronic device. Background Art

[0003] With the advancement of electronic science and technology, smart terminals are integrated with more and more functions. Thanks to the development of image processing technology, more and more users like to use smart terminal devices to take photos, record videos and make video calls.

[0004] Due to the limitation of the computing power of the algorithm of the image signal processor (ISP) in the smart terminal, in order to improve the image processing effect, the industry has proposed a method of combining the traditional image processing algorithm with artificial intelligence (AI) technology. For example, the AI ​​processor is set at the back end of the ISP as a supplementary correction to the ISP image processing results, that is, AI post-processing. In the specific implementation, the ISP stores the processed image in the off-chip memory, and the AI ​​processor reads the image stored by the ISP from the off-chip memory, and further corrects the image based on the ISP image processing to generate the final image. In this solution, due to the bottleneck of the traditional ISP's capabilities, the original image data collected by the image sensor loses information after being processed by the ISP, thereby reducing the image processing effect. Although AI post-processing technology can improve the image processing results output by the ISP to a certain extent, the effect is still not ideal. Therefore, the existing technology has failed to fully solve the problem of poor processing effect of the traditional ISP. Summary of the invention

[0005] The electronic device and the image processing method of the electronic device provided in the present application can improve the image processing effect. To achieve the above purpose, the present application adopts the following technical solution.

[0006] In a first aspect, an embodiment of the present application provides an electronic device, comprising: an artificial intelligence (AI) processor, for performing a first image signal processing on a first image signal to obtain a second image signal, wherein the first image signal is obtained based on image data output by an image sensor; and an image signal processor ISP, for performing a second image signal processing on the second image signal to obtain an image processing result.

[0007] By having the ISP perform the remaining image processing after the AI ​​processor has processed the image data, the processing power of the AI ​​processor can be fully utilized in the entire image signal processing flow, so that the AI ​​processor can replace the traditional ISP to perform part of the processing process, thereby improving the quality of the image processing results output by the ISP.

[0008] Based on the first aspect, in a possible implementation manner, the ISP is further used to: receive the image data from the image sensor, and perform third image signal processing on the image data to obtain the first image signal.

[0009] This implementation method can set the first image processing process executed by the AI ​​processor between multiple image processing processes executed by the ISP. The AI ​​processor can replace the ISP to execute a part of the processing process in the middle of the image processing process to achieve a preset effect, thereby improving the flexibility of the combination of the AI ​​processor and the ISP, thereby improving the image processing effect.

[0010] Based on the first aspect, in a possible implementation, the third image signal processing includes multiple processing processes, and in two adjacent processing processes of the multiple processing processes, the first processing process is used to generate the third image signal, and the second processing process is used to process the fourth image signal; the AI ​​processor is further used to perform the fourth image signal processing on the third image signal to obtain the fourth image signal. This solution further improves the flexibility of using the AI ​​processor to perform image processing processes.

[0011] Based on the first aspect, in a possible implementation manner, the first image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction or gamma correction.

[0012] Based on the first aspect, in a possible implementation method, the second image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, color difference correction, gamma correction, color difference correction or RGB to YUV domain conversion.

[0013] Based on the first aspect, in a possible implementation manner, the third image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction or demosaicing.

[0014] Based on the first aspect, in a possible implementation manner, the fourth image signal processing includes at least one of the following processing processes: black level correction, shadow correction, white balance correction, demosaicing or chromatic aberration correction.

[0015] Based on the first aspect, in a possible implementation manner, the electronic device further includes: a memory, coupled to the AI ​​processor and the ISP, and configured to transfer a first image unit in any image signal between the AI ​​processor and the ISP.

[0016] Based on the first aspect, in a possible implementation manner, the memory, the AI ​​processor and the ISP are located in a system on chip in the electronic device; the memory includes an on-chip random access memory RAM.

[0017] Based on the first aspect, in a possible implementation manner, the first image unit includes any one of the following: a single-frame image or an image block in a single-frame image.

[0018] Based on the first aspect, in a possible implementation, the electronic device further includes: an off-chip memory located outside the on-chip system, for transferring a second image unit in any image signal between the AI ​​processor and the ISP, wherein the second image unit includes multiple frames of images.

[0019] Based on the first aspect, in a possible implementation manner, the electronic device further includes: a controller, configured to trigger the AI ​​processor to perform the first image signal processing, and control the ISP to perform the second image signal processing.

[0020] Based on the first aspect, in a possible implementation, the AI ​​processor and the ISP transmit interrupt signals via an electronic circuit connection. Optionally, the electronic circuit connection includes connection via an interrupt controller. By setting an electronic circuit connection between the AI ​​processor and the ISP to realize interrupt signal transmission, it is not necessary to forward the signal through other processors such as the CPU, which can increase the signal transmission speed. In some real-time video playback, the image output delay can be reduced, which is conducive to improving the user experience.

[0021] Optionally, the electronic device further includes the image sensor.

[0022] In a second aspect, an embodiment of the present application provides an image processing method for an electronic device, the image processing method comprising: controlling an artificial intelligence AI processor to perform a first image signal processing on a first image signal to obtain a second image signal, wherein the first image signal is obtained based on image data output by an image sensor; controlling an image signal processor ISP to perform a second image signal processing on the second image signal to obtain an image processing result.

[0023] Based on the second aspect, in a possible implementation method, before controlling the artificial intelligence AI processor to perform the first image signal processing on the first image signal, it also includes: controlling the ISP to receive the image data from the image sensor, and performing the third image signal processing on the image data to obtain the first image signal.

[0024] Based on the second aspect, in a possible implementation, the third image signal processing includes multiple processing processes, and in two adjacent processing processes of the multiple processing processes, the former processing process is used to generate a third image signal, and the latter processing process is used to process a fourth image signal; the method also includes: controlling the AI ​​processor to perform fourth image signal processing on the third image signal to obtain the fourth image signal.

[0025] Based on the second aspect, in a possible implementation manner, the first image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction or gamma correction.

[0026] Based on the second aspect, in a possible implementation method, the second image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, color difference correction, gamma correction, color difference correction or RGB to YUV domain conversion.

[0027] Based on the second aspect, in a possible implementation manner, the third image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction or demosaicing.

[0028] Based on the second aspect, in a possible implementation manner, the fourth image signal processing includes at least one of the following processing processes: black level correction, shadow correction, white balance correction, demosaicing or chromatic aberration correction.

[0029] In a third aspect, an embodiment of the present application provides an image processing device, which includes: an AI processing module, used to perform first image signal processing on a first image signal to obtain a second image signal, wherein the first image signal is obtained based on image data output by an image sensor; and an image signal processing module, used to perform second image signal processing on the second image signal to obtain an image processing result.

[0030] Based on the third aspect, in a possible implementation manner, the image signal processing module is further used to: receive the image data from the image sensor, and perform third image signal processing on the image data to obtain the first image signal.

[0031] Based on the third aspect, in a possible implementation method, the third image signal processing includes multiple processing processes, and in two adjacent processing processes of the multiple processing processes, the former processing process is used to generate a third image signal, and the latter processing process is used to process a fourth image signal; the AI ​​processing module is also used to perform fourth image signal processing on the third image signal to obtain the fourth image signal.

[0032] Based on the third aspect, in a possible implementation manner, the first image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction or gamma correction.

[0033] Based on the third aspect, in a possible implementation method, the second image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, color difference correction, gamma correction, color difference correction or RGB to YUV domain conversion.

[0034] Based on the third aspect, in a possible implementation manner, the third image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction or demosaicing.

[0035] Based on the third aspect, in a possible implementation manner, the fourth image signal processing includes at least one of the following processing processes: black level correction, shadow correction, white balance correction, demosaicing or chromatic aberration correction.

[0036] In a fourth aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a memory and at least one processor, the memory being used to store a computer program, the at least one processor being configured to call all or part of the computer program stored in the memory to execute the method described in the second aspect above. The at least one processor comprises the AI ​​processor and the ISP. Optionally, the electronic device also comprises the image sensor.

[0037] In a fifth aspect, an embodiment of the present application provides a system on chip, the system on chip comprising at least one processor and an interface circuit, the interface circuit being used to obtain a computer program from outside the chip system; the computer program is used to implement the method described in the second aspect when executed by the at least one processor. The at least one processor comprises the AI ​​processor and the ISP.

[0038] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, it is used to implement the method as described in the second aspect. The at least one processor includes the AI ​​processor and the ISP.

[0039] In a seventh aspect, an embodiment of the present application provides a computer program product, which is used to implement the method described in the second aspect when the computer program product is executed by at least one processor. The at least one processor includes the AI ​​processor and the ISP.

[0040] It should be understood that the second to seventh aspects of the present application are consistent with the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 is a hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0043] Figure 2 is a schematic flow chart of an image processing method provided in an embodiment of the present application;

[0044] Figure 3 is another hardware structure diagram of the electronic device provided in the embodiment of the present application;

[0045] Figure 4 is another hardware structure diagram of the electronic device provided in the embodiment of the present application;

[0046] Figure 5 is another schematic flow chart of the image processing method provided in the embodiment of the present application;

[0047] Figure 6 is another schematic flow chart of the image processing method provided in the embodiment of the present application;

[0048] Figure 7 is another hardware structure diagram of the electronic device provided in the embodiment of the present application;

[0049] Figure 8 is another hardware structure diagram of the electronic device provided in the embodiment of the present application;

[0050] Fig. 9 It is a schematic diagram of the software structure of the image processing device involved in the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The words "first", "second" and similar words mentioned herein do not indicate any order, quantity or importance, but are only used to distinguish different parts. Similarly, words such as "one" or "an" do not indicate quantity limitation, but indicate the existence of at least one. Words such as "coupled" are not limited to direct physical or mechanical connection, but may include electrical connection, whether direct or indirect, which is equivalent to communication in a broad sense.

[0053] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In the description of the embodiments of the present application, unless otherwise stated, the meaning of "multiple" refers to two or more. For example, multiple processors refers to two or more processors.

[0054] The electronic device provided in the embodiment of the present application may be an electronic device or a module, chip, chipset, circuit board or component integrated in an electronic device. The electronic device may be a user equipment (UE), such as various types of devices such as a mobile phone, a tablet computer, a smart screen or an image capture device. The electronic device may be provided with a camera device, which may also be referred to as an image sensor, for collecting image data. The electronic device may also be installed with various software applications such as camera applications, video call applications or online video capture applications for driving the camera device to collect images, and the user may start the above-mentioned various applications to use the camera device to take photos or videos. In addition, the user may also perform various personalized settings for image beautification through such applications. Taking the video call application as an example, the user may choose to automatically adjust the screen (such as the facial avatar presented, or the background picture presented) during the video call (such as "one-key beautification"). When the user starts the above-mentioned various applications or starts the above-mentioned various applications and selects image beautification, the image processing service supported by the above-mentioned various applications in the electronic device can trigger the electronic device to process the image data collected by the camera device, so as to present the processed image on the screen of the electronic device to achieve the effect of image beautification. The above-mentioned image beautification may include, but is not limited to: increasing the brightness of the image part or the entire frame, changing the display color of the image, smoothing the facial object presented in the image, adjusting the saturation of the picture, adjusting the exposure of the picture, adjusting the vividness of the picture, adjusting the highlight of the picture, adjusting the contrast of the picture, adjusting the sharpness of the picture or adjusting the clarity of the picture. The image processing described in the embodiment of the present application may include, but is not limited to: noise removal, black level correction, shadow correction, white balance correction, de-mosaicing, color difference correction, gamma correction or red, green and blue (RGB) to YUV (YCrCb) domain, so as to achieve the above-mentioned image beautification effect. Based on the electronic device described in the embodiment of the present application, in a specific application scenario, when user A starts the above-mentioned image processing service, when user A and user B have a video call, the image presented on the screen of the electronic device used by user A, and the image of user A presented on the screen of the electronic device used by user B, may be images processed by the electronic device described in the embodiment of the present application, and the processed image will continue to be presented until user A and user B terminate the video call or user A turns off the image processing service.

[0055] Based on the application scenarios described above, please continue to refer to Figure 1, which shows a hardware structure diagram of the electronic device provided in an embodiment of the present application. The electronic device 100 may be, for example, a chip or a chipset or a circuit board equipped with a chip or a chipset or an electronic device including the circuit board, but it is not used to limit the embodiment. The specific electronic device is as described above and is omitted here. The chip or chipset or the circuit board equipped with the chip or chipset can work under the necessary software driver. The electronic device 100 includes one or more processors, such as ISP102 and AI processor 101. Optionally, the one or more processors may be integrated in one or more chips, and the one or more chips may be regarded as a chipset. When one or more processors are integrated in the same chip, the chip is also called a system on a chip (System on aChip, SOC). In addition to the one or more processors, the electronic device 100 also includes one or more other necessary components, such as a memory 103. In one possible implementation, the memory 103 may be located in the same system on chip in the electronic device 100 with the AI ​​processor 101 and ISP102, that is, the memory 103 is integrated in the above Figure 1 In the SOC shown in FIG. 1 , the memory 103 may include an on-chip random access memory (RAM).

[0056] In an embodiment of the present application, the AI ​​processor 101 may include a dedicated neural processor such as a neural network processor (Neural-network Processing Unit, NPU), including but not limited to a convolutional neural network processor, a tensor processor or a neural processing engine. The AI ​​processor can be used as a component alone or integrated into other digital logic devices, which include but are not limited to: CPU (Central Processing Unit), GPU (Graphics Processing Unit) or DSP (Digital Signal Processing). Exemplarily, the CPU, GPU and DSP are all processors in the system on chip. The AI ​​processor 101 can perform one or more image processing operations, and the one or more image processing operations may include but are not limited to: noise elimination, black level correction, shadow correction, white balance correction, de-mosaicing, chromatic aberration correction or gamma correction. The AI ​​processor 101 can run one or more image processing models, each of which is used to perform a specific image processing operation. For example, the image processing model for noise elimination is used to perform the image processing operation of noise elimination, and the image processing model for de-mosaicing is used to perform the image processing operation of de-mosaicing. Each image processing model can be obtained by training the neural network using training samples using a traditional neural network training method, which is not described in detail in the present embodiment. ISP102 can set up multiple hardware modules or run necessary software programs to process images or communicate with AI processor 101. Among them, ISP102 and AI processor 101 can communicate through hardware direct connection (for example, Figure 3 , Figure 4 and Figure 7 The relevant description in the embodiment shown in the figure) can also communicate by forwarding signals through the controller, wherein the relevant description of the communication between ISP102 and AI processor 101 by forwarding signals through the controller is specifically referred to Figure 8 The relevant description of the controller 104 is shown in .

[0057] In an embodiment of the present application, the image data obtained from the camera device 105 may undergo multiple image processing processes to generate a final image processing result, and the multiple image processing processes may include but are not limited to: noise removal, black level correction, shadow correction, white balance correction, de-mosaicing, chromatic aberration correction, gamma correction or RGB to YUV domain. The AI ​​processor 101 may perform one or more of the above-mentioned image processing processes, that is, corresponding to one or more of the above-mentioned image processing operations, and the ISP102 may also perform one or more of the above-mentioned image processing processes. Among them, the AI ​​processor 101 may perform different image processing processes with the ISP102. In addition, the AI ​​processor 101 and the ISP102 may also perform the same image processing process, such as performing further enhancement processing, which is not limited in this embodiment. When the AI ​​processor 101 and the ISP102 perform the same image processing process, the image processing performed by the AI ​​processor 101 may serve as an enhancement or supplement to the image processing process. For example, when the AI ​​processor 101 and ISP 102 perform the noise elimination process at the same time, ISP 102 is used to perform the primary denoising, and the AI ​​processor 101 is used to perform the secondary denoising based on the primary denoising of ISP 102. Therefore, the entire image processing flow includes multiple processing processes, which are assigned as tasks to the AI ​​processor 101 and ISP 102, which is equivalent to the AI ​​processor 101 replacing the traditional ISP to perform part of the processing process before ISP 102 completes all the processing processes, and the final processing result is output by ISP 102.

[0058] In the electronic device 100 shown in the embodiment of the present application, after the AI ​​processor 101 performs one or more image processing processes on the image data, the ISP 102 performs the remaining image processing processes. The AI ​​processor 101 can be used to replace the ISP 102 to perform part of the processing, thereby avoiding the loss of information of the original image data caused by insufficient processing power of the ISP 102 during the image processing process, thereby improving the image processing effect. In addition, in some other implementations, when some image processing processes in the traditional ISP cannot achieve the preset effect, the image processing process performed by the AI ​​processor 101 can be set between the multiple image processing processes performed by the ISP 102 (such as in the embodiment Figure 4-Figure 7 ) is used to replace ISP to perform the image processing process to achieve the preset effect, thereby improving the flexibility of the combination of AI processor and ISP.

[0059] The following describes in detail the process of combining the AI ​​processor 101 and the ISP 102 to perform image processing and the hardware structure of the electronic device corresponding to each image processing process. Figure 2 , which shows a schematic flow chart of an image processing method 200 provided in an embodiment of the present application. The image processing method 200 is applied to Figure 1or Figure 3 The electronic device 100 shown. The image processing method 200 includes the following image processing steps: Step 201, the camera device 105 provides the collected image data to the ISP 102. Step 202, the ISP 102 processes the image data to generate an image signal A. Step 203, the ISP 102 provides the image signal A to the AI ​​processor 101. Step 204, the AI ​​processor 101 performs image processing on the image signal A to generate an image signal B. Step 205, the AI ​​processor 101 provides the image signal B to the ISP 102. Step 206, the ISP 102 performs image processing on the image signal B to obtain an image processing result.

[0060] based on Figure 2 Please refer to the image processing steps shown in Figure 3 , which shows another structural schematic diagram of the electronic device 100 provided in an embodiment of the present application. Figure 3 In the electronic device 100 shown, ISP102 may include multiple cascaded image processing modules, and the multiple cascaded image processing modules include image processing module 01, image processing module 02, image processing module 03...image processing module N and image processing module N+1. Each image processing module may include multiple logic devices or circuits to perform specific image processing functions. For example, image processing module 01 is used to perform image processing for black level correction, image processing module 02 is used to perform image processing for shadow correction, image processing module 03 is used to perform image processing for shadow correction..., and image processing module N+1 is used to perform RGB to YUV processing. Based on the processing requirements for the image, any one of the above-mentioned multiple cascaded image processing modules may be provided with an output port and an input port, the output port being used to send an image signal A to the AI ​​processor 101, and the input port being used to obtain an image signal B from the AI ​​processor 101, Figure 3 The image processing module 02 is schematically shown to be provided with an output port V po1 The image processing module 03 is provided with an input port V pi1 .exist Figure 3 In the embodiment, ISP 102 is connected to camera 105 to obtain image data from camera 105. Electronic device 100 is also provided with an on-chip RAM, which is integrated with ISP 102 and AI processor 101 in a chip in electronic device 100, and the on-chip RAM is used to store Figure 2 The image signal A and the image signal B shown. In addition, the on-chip RAM is also used to store the intermediate data generated during the operation of the AI ​​processor 101 and the weight data of each network node in the neural network run by the AI ​​processor 101. Figure 3 In order to increase the signal transmission speed, the output port Vpo1 , the input port V of the image processing module 03 pi1 and the input port V of the AI ​​processor 101 ai1 , output port V ao1 Electronic circuits can be used to connect to the on-chip RAM.

[0061] Furthermore, in a possible implementation, the AI ​​processor 101 and the image processing module 02 in the ISP 102 transmit an interrupt signal Z1 through an electronic circuit connection L1, and the interrupt signal Z1 is used to instruct the image processing module 02 to store the image signal A in the on-chip RAM; the AI ​​processor 101 and the image processing module 03 in the ISP 102 transmit an interrupt signal Z2 through an electronic circuit connection L2, and the interrupt signal Z2 is used to instruct the AI ​​processor 101 to store the image signal B in the on-chip RAM. Specifically, Figure 3 The AI ​​processor shown may include a task scheduler 1011 and multiple computing units 1012. Each of the components in the task scheduler 1011 and the multiple computing units 1012 may include multiple logic devices or circuits. The task scheduler 1011 may be configured to receive an input signal through the input port V ai2 The output terminal Vp of the image processing module 02 o2 The above electronic circuit is connected to L1 through the output port V of the AI ​​processor 101. ao2 The electronic circuit connection L2 is realized with the input terminal Vpi2 of the image processing module 03. In addition, each of the plurality of computing units can be connected via the input port V ai1 The electronic circuit is connected to the on-chip RAM to read the image signal A from the on-chip RAM; through the output port V ao1 Make electronic circuit connection with the above-mentioned on-chip RAM to write image signal B into the on-chip RAM.

[0062] In a specific scenario, ISP102 obtains image data from the camera device 105, and the image data is processed by the image processing module 01 and the image processing module 02 in sequence to generate image signal A and store it in the on-chip RAM. After the image processing module 02 stores the image signal A in the on-chip RAM, it sends an interrupt signal Z1 to the AI ​​processor 101 through the electronic circuit connection L1. In response to the interrupt signal Z1, the AI ​​processor 101 obtains the image signal A from the on-chip RAM. The AI ​​processor 101 performs de-mosaic processing on the image signal A to generate an image signal B, and stores the image signal B in the on-chip RAM. After the AI ​​processor 101 stores the image signal B in the on-chip RAM, it sends the above interrupt signal Z2 to the image processing module 03. In response to the interrupt signal Z2, the image processing module 03 reads the image signal B from the on-chip RAM, and the image processing module 03..., the image processing module N and the image processing module N+1 in the ISP102 perform chromatic aberration correction,...Gamma correction and RGB to YUV domain processing in sequence to generate the final image processing result. It should be noted that more image processing modules may be included before the image processing module 01, so that the ISP 102 performs more image processing processes on the image data.

[0063] In one embodiment, the electronic circuit connection between the AI ​​processor 101 and the ISP 102 is also called a physical connection or an interrupt connection. The AI ​​processor 101 and the ISP 102 implement the sending and receiving of interrupt signals through this connection, so that the interrupt signal does not need to be forwarded by other processors such as the CPU, and the CPU does not need to participate in related control. The transmission speed of the interrupt signal can be increased. In some real-time video playback, the image output delay can be reduced, which is beneficial to improving the user experience. Specifically, the interrupt connection includes an interrupt signal processing hardware circuit for implementing the interrupt signal sending and receiving functions and a connecting line for transmitting the signal to realize the sending and receiving of the interrupt signal. The interrupt signal processing hardware circuit includes but is not limited to the traditional interrupt controller circuit. Regarding the specific implementation scheme of the interrupt signal processing hardware circuit, reference can be made to the relevant description of the interrupt controller in the prior art, which will not be repeated here.

[0064] exist Figure 2 and Figure 3In the illustrated embodiment, the image processing process performed by the AI ​​processor 101 is arranged between the multiple image processing processes performed by the ISP 102 to replace or supplement certain intermediate image processing processes performed by the ISP 102. In some other possible implementations, the AI ​​processor 101 may directly obtain image data from the camera device 105 and perform the front-end image processing process. In this implementation, the AI ​​processor 101 may replace certain image processing modules at the front end of the ISP 102 and perform the corresponding image processing process. At this time, the AI ​​processor 101 may directly communicate with the image processing module behind the ISP 102. The hardware structure of this implementation is referred to in Figure 4 , please refer to Figure 4 , which shows another hardware structure diagram provided in an embodiment of the present application.

[0065] exist Figure 4 The structure of ISP102 and the structure of AI processor 101 are similar. Figure 3 The structure of the ISP 102 shown is the same as that of the AI ​​processor 101. Figure 3 The relevant description of the embodiment shown is not repeated here. Figure 3 The difference between the embodiment shown is that in this embodiment, the AI ​​processor 101 skips the image processing module 01 and the image processing module 02 and communicates with the image processing module 03. Specifically, the input port V pi1 and the output port V of the AI ​​processor 101 ao1 The AI ​​processor 101 and the image processing module 03 in the ISP 102 can be connected to each other by electronic circuits. The interrupt signal Z3 is transmitted between the AI ​​processor 101 and the image processing module 03 in the ISP 102 via the electronic circuit connection L3. The interrupt signal Z3 is used to instruct the AI ​​processor 101 to store the image signal C in the on-chip RAM. Figure 4 In the hardware structure shown, in a specific scenario, the AI ​​processor 101 obtains image data from the camera device 105, and then generates an image signal C after removing noise from the image data and stores it in the on-chip RAM. After the AI ​​processor 101 stores the image signal C in the on-chip RAM, it sends an interrupt signal Z3 to the image processing module 03 through the electronic circuit connection L3. The image processing module 03 obtains the image signal C from the on-chip RAM in response to the interrupt signal Z3. The image signal C is processed by the image processing module 03 in ISP102..., the image processing module N, and the image processing module N+1, and then the image processing process such as black level correction, shadow correction, white balance correction...RGB to YUV domain is performed in sequence, and the final image processing result is generated.

[0066] Please continue to refer to Figure 5, which shows another flow chart of the image processing method 500 provided in the embodiment of the present application. The image processing method 500 is applied to Figure 1 or Figure 4 The electronic device 100 shown. The image processing method 500 includes the following image processing steps: Step 501, the camera device 105 provides the collected image data to the AI ​​processor 101. Step 502, the AI ​​processor 101 processes the image data to generate an image signal C. Step 503, the AI ​​processor 101 provides the image signal C to the ISP 102. Step 504, the ISP 102 processes the image signal C to obtain an image processing result.

[0067] In the embodiment of the present application, in the process of performing multiple processing processes on image data to obtain image processing results, the AI ​​processor 101 can perform multiple continuous image processing processes to process image data or image signals, such as Figure 2-Figure 5 In addition, in some other implementations, the AI ​​processor 101 may also perform multiple discontinuous image processing processes to process the image data or image signal. Figure 6 , which shows another flow chart of the image processing method 600 provided in the embodiment of the present application. The image processing method 600 is applied to Figure 1 or Figure 7 The electronic device 100 shown. The image processing method 600 includes the following image processing steps: Step 601, the camera device 105 provides the collected image data to the ISP 102. Step 602, the ISP 102 processes the image data to generate an image signal D. Step 603, the ISP 102 provides the image signal D to the AI ​​processor 101. Step 604, the AI ​​processor 101 processes the image signal D to generate an image signal E. Step 605, the AI ​​processor 101 provides the image signal E to the ISP 102. Step 606, the ISP 102 processes the image signal E to generate an image signal F. Step 607, the ISP 102 provides the image signal F to the AI ​​processor 101. Step 608, the AI ​​processor 101 processes the image signal F to generate an image signal G. Step 609, the AI ​​processor 101 provides the image signal G to the ISP 102. Step 610, the ISP 102 processes the image signal G to obtain an image processing result.

[0068] based on Figure 6 Please refer to the image processing steps shown in Figure 7 , which shows another structural schematic diagram of the electronic device 100 provided in an embodiment of the present application. Figure 7 In the embodiment, the electronic device includes an AI processor 101, an ISP 102, and an on-chip RAM. Figure 3The ISP102 shown is the same as Figure 7 The ISP 102 in the electronic device shown in the figure also includes a cascaded image processing module 01, an image processing module 02, an image processing module 03 ... an image processing module N and an image processing module N+1, wherein the structure and function of each module are similar to Figure 3 The structures and functions of the modules in the ISP 102 shown are the same and will not be described in detail here. Figure 7 The internal structure of the AI ​​processor 101 is similar to Figure 3 The internal structure of the AI ​​processor 101 shown is the same as that of the AI ​​processor 101, which will not be described in detail here. The on-chip RAM is used to store image signal D, image signal F, image signal F and image signal G. Figure 3 , Figure 4 The difference between the electronic device shown in FIG. 1 and FIG. 2 is that two of the plurality of cascaded image processing modules are provided with output ports, and the two image processing modules are provided with output ports, specifically as shown in FIG. Figure 7 The output end of the image processing module 02 and the output end of the image processing module 03 are respectively provided with output ports V po1 and output port V po2 Input ports V are provided at the input end of the image processing module 03 and the input end of the image processing module N. pi1 and input port V pi2 The output ports of the image processing module 02 and the image processing module 03 are used to output Figure 6 The image signal D and the image signal F shown in FIG. 1; the input port of the image processing module 03 and the input port of the image processing module N are respectively used to input the image signal D and the image signal F shown in FIG. Figure 6 The image signal E and the image signal G are shown. Figure 7 In the image processing module 02, the output port V po1 and the output port V of the image processing module 03 po2 The input port V of the image processing module 03 is connected to the on-chip RAM by electronic circuits. pi1 and the input port V of the image processing module N pi2 The input port V of the AI ​​processor 101 is connected to the on-chip RAM by electronic circuits; ai1 and output port V ao1It can also be connected to the on-chip RAM by electronic circuits. In addition, the AI ​​processor 101 and the image processing module 02 in the ISP102 transmit an interrupt signal Z4 through an electronic circuit connection L4, and the interrupt signal Z4 is used to instruct the image processing module 02 to store the image signal D in the on-chip RAM; the AI ​​processor 101 and the image processing module 03 in the ISP102 transmit interrupt signals Z5 and Z6 through an electronic circuit connection L5, and the interrupt signal Z5 is used to instruct the AI ​​processor 101 to store the image signal E in the on-chip RAM, and the interrupt signal Z6 is used to instruct the image processing module 03 to store the image signal F in the on-chip RAM; the AI ​​processor 101 and the image processing module N in the ISP102 transmit an interrupt signal Z7 through an electronic circuit connection L6, and the interrupt signal Z7 is used to instruct the AI ​​processor 101 to store the image signal G in the on-chip RAM.

[0069] above Figure 3-Figure 7 The image processing methods shown in the embodiments of the present application and the hardware structure of the electronic device corresponding to each image processing method are schematically shown. It should be noted that the ISP102 in the electronic device shown in the embodiments of the present application may also include more output ports and input ports, so that images processed by more processing flows are transmitted between the AI ​​processor 101 and the ISP102, so that the AI ​​processor 101 can perform more image processing flows at intervals. In other words, the AI ​​processor 101 and the ISP102 can perform processing alternately, so that both parties complete the image processing process together to obtain the processing result, thereby replacing the image processing process of the traditional ISP.

[0070] based on Figure 3 , Figure 4 and Figure 7 In a possible implementation of the electronic device, the ISP 102 and the AI ​​processor 101 are in the form of image blocks. Figure 3 , Figure 4 or Figure 7 The image block may be a local image signal in one frame of image signal (for example, one frame of image signal has 1280 rows of pixels, and the on-chip RAM stores an image signal formed by 320 rows of pixels). In a specific implementation, each image processing module in ISP102 performs image processing in units of one row of pixels, and ISP102 stores the processed image signal to the on-chip RAM row by row. When ISP102 completes storing the image signal in the on-chip RAM (for example, the number of rows of stored image signals reaches a preset threshold, the size of the written image signal reaches a preset threshold, or the last address in the storage address allocated to ISP101 stores an image signal), the image signal is stopped from being transmitted to the on-chip RAM, and the image signal is transmitted to the on-chip RAM by the following method: Figure 3The electronic circuit connection L1 shown sends an interrupt signal Z1 to the AI ​​processor. The AI ​​processor 101 can store the image signal to the on-chip RAM in the same storage method as the ISP 102, and after the image signal is stored in the on-chip RAM, the AI ​​processor 101 can store the image signal to the on-chip RAM in the same manner as the ISP 102. Figure 3 The electronic circuit connection L2 shown sends an interrupt signal Z2 to the AI ​​processor. Further, the on-chip RAM can be logically divided into a first storage area and a second storage area, and the ISP102 stores the image signal in the form of an image block in the first storage area, and the AI ​​processor 101 stores the image signal in the form of an image block in the second storage area. In this way, the image processing by the ISP102 and the image processing by the AI ​​processor 101 can be performed in parallel, reducing the waiting time of the AI ​​processor 101 and the ISP102, and improving the transmission rate of the image signal.

[0071] In addition, when ISP102 sends a video to the Figure 3 , Figure 4 or Figure 7 When storing in the on-chip RAM shown in FIG. 1 , ISP 102 may also send an indication signal to AI processor 101 indicating that the current image block is the starting position of a frame of image. Specifically, as Figure 3 As shown, the image processing module 02 in the ISP 102 has an electronic circuit connection L7 with the AI ​​processor 101, and sends an interrupt signal Z8 to the AI ​​processor 101. The interrupt signal Z8 is used to indicate that the image signal A is the starting position of a frame of image.

[0072] In a possible implementation of the embodiment of the present application, the electronic device further includes an off-chip memory 106, such as Figure 8As shown. The off-chip memory 106 can store multiple frames of images, and the multiple frames of images can be the previous frame of image, the previous two frames of image, or the previous multiple frames of image before the current image. Since the off-chip memory 106 has a larger storage space, it can replace the on-chip RAM to store larger units of image data. Among them, the image signal stored in the off-chip memory 106 can be an image signal processed by the AI ​​processor 101, or an image signal provided to the AI ​​processor 101 for processing by the AI ​​processor 101. When the AI ​​processor 101 processes the current image signal, it can also obtain the image information of the previous frame of image signal or the previous frames of image signal of the current image signal from the off-chip memory 106, and then process the current image signal based on the image information of the previous frame of image signal or the previous frames of image signal. In addition, the AI ​​processor 101 can also store the processed image signal in the off-chip memory 106. The off-chip memory 106 may include a random access memory (RAM), which may include a volatile memory (such as SRAM, DRAM, DDR (double data rate SDRAM, Double Data Rate SDRAM) or SDRAM, etc.) and a non-volatile memory. In addition, the off-chip memory 106 may store an executable program of the image processing model running in the AI ​​processor 101, and the AI ​​processor runs the image processing model by loading the executable program.

[0073] Exemplarily, the memory 103, such as an on-chip RAM, can be used to store a single frame image or an image block in a single frame image. The off-chip memory 106 is used to store multiple frames of images. Figure 2 , Figure 5 or Figure 7 In the method shown, the image signal transmitted between the AI ​​processor 101 and the AI ​​processor 101 is divided into unit sizes, and the larger unit image information is transmitted through the off-chip memory 106 to make up for the lack of on-chip RAM space and effectively utilize the faster transmission speed of the on-chip RAM to achieve performance optimization.

[0074] In the embodiment of the present application, the electronic device 100 may further include a controller 104, such as Figure 8 As shown. The controller 104 may be an integrated controller located on the same chip as the AI ​​processor 101, ISP 102, and memory 103. The controller 104 runs necessary software programs or software plug-ins to drive the controller 104 to control the operation of the ISP 102, the operation of the AI ​​processor 101, and the communication between the ISP 102 and the AI ​​processor 101, thereby achieving the following. Figure 2 , Figure 5 or Figure 6The image processing method shown is used to replace the electronic circuit connection between the AI ​​processor 101 and the ISP 102 as a communication medium. In a specific implementation, the controller 104 can be various digital logic devices or circuits, including but not limited to: CPU, GPU, microcontroller, microprocessor or DSP. In addition, the controller 104 can also be set separately from the AI ​​processor 101, ISP10 and memory 10, which is not limited in this embodiment. Further, the controller 104 can also be integrated with the AI ​​processor 101 in the same logic operation device (such as CPU), and the functions performed by the controller 104 and the AI ​​processor 101 described in the embodiment of the present application are implemented by the same logic operation device. In practice, the controller 104 can determine whether to use ISP102 in combination with the AI ​​processor 101 to process the image data based on various information indicated by the image data collected by the camera device 105 (such as light intensity information, white balance information or mosaic information, etc.), and determine the image processing model selected when the ISP102 is selected to process the image data in combination with the AI ​​processor 101. Taking light intensity information as an example, the controller 104 will be used to control the above components and process the image in detail. Based on the light intensity information indicated by the image data, the controller 104 compares the information with the preset image light intensity threshold. Based on the comparison result, when it is determined that the image light intensity threshold cannot be reached after the image light intensity is processed by ISP102, the AI ​​processor 101 can be used to process the light intensity of the image data or image signal. In addition, the AI ​​processor 101 can run a variety of image processing models for light intensity processing. The controller determines which image processing model to use based on the difference between the light intensity information indicated by the image data and the image light intensity threshold. After determining the selected image processing model, the controller 104 can send the storage address information of the algorithm program, parameters or instructions for storing the image processing model in the above-mentioned memory 103 or the off-chip memory 106 to the AI ​​processor 101, so that various computing units in the AI ​​processor 101 obtain the algorithm program, parameters or instructions from the storage address to run the image processing model and perform light intensity processing on the image data or image signal. When multiple image processing models are required to process image data or image signals, the controller may also send sequence information or priority information indicating the execution order of the multiple image processing models to the AI ​​processor 101 .It should be noted that, in the embodiment of the present application, the controller 104 can detect various information indicated by the image data in real time or periodically. When it is determined based on the detection results that the currently running image detection model is not applicable, the selected image processing model can be replaced in time (for example, when the ambient light intensity changes from strong to weak, the image processing model used to process the light intensity is replaced), and then the storage address information of the executable program of the replaced image processing model is sent to the AI ​​processor 101, so that the AI ​​processor 101 runs the replaced image processing model when performing image processing in the next image processing cycle. Therefore, the electronic device described in the embodiment of the present application can dynamically adjust the image processing model used based on changes in the external environment or changes in the collected image data, so that when the user uses the electronic device described in the embodiment of the present application to change the scene (for example, from outdoor to indoor or from a strong light area to a weak light area), the collected image can be processed in a targeted manner, thereby improving the image processing effect and improving the user experience.

[0075] In the embodiment of the present application, the image data usually has a preset clock cycle T from the start of processing to the generation of the final image processing result. The AI ​​processor has different durations in the clock cycle T when executing the image processing process based on the different image processing models being run or the different image processing processes being executed. For example, the clock cycle T from the start of image data processing to the generation of the final image processing result is 33.3ms, and the duration of the AI ​​processor performing image processing is 18ms, that is, the AI ​​processor is idle for half of the clock cycle T. Based on this, in order to improve the utilization rate of the AI ​​processor and improve the operating efficiency of the electronic device, in a possible implementation, the controller 104 can also determine the running time of the image processing model used. When it is determined that the running time of the image processing model within the clock cycle T is less than the preset threshold, the controller 104 can also allocate the idle time of the AI ​​processor to other AI services. The other AI services may include, but are not limited to: biometrics (such as facial recognition or fingerprint recognition) services, and services for adding special effects to images (such as adding objects to images). At this time, the executable program or parameters of the AI ​​model for executing the other AI service may also be stored in the above-mentioned off-chip memory 106. Further, when the storage capacity of the memory 103 is large enough, the executable program or parameters of the AI ​​model for the other AI service may also be stored in the memory 103.

[0076] In other possible implementations, the controller 104 may include multiple independent controllers, each of which may be a digital logic device (for example, including but not limited to: GPU or DSP). The multiple independent controllers include an ISP controller for controlling the operation of each component in ISP102 and an AI controller for controlling the operation of each component in AI processor 101. At this time, the AI ​​controller may be integrated inside the AI ​​processor 101. In this implementation, the ISP controller and the AI ​​controller may transmit information in the form of inter-core communication. For example, after the ISP controller determines the selected image processing model based on the image data, before the image processing starts, various configuration information may be sent to the AI ​​controller, and the configuration information may include but is not limited to: the address information of the executable program of the image processing model in the memory 103 or the off-chip memory 106, or the priority information of each image processing model in the multiple image processing models. In addition, the information transmission between the AI ​​processor 101 and ISP102 may also be achieved through communication between the ISP controller and the AI ​​controller. For example, when no electronic circuit connection is set between the AI ​​processor 101 and the ISP 102 to transmit an interrupt signal, the ISP 102 may store the image signal in the on-chip RAM and then notify the ISP controller, the ISP controller sends information indicating that the image signal is stored in the on-chip RAM to the AI ​​controller, and the AI ​​controller controls the computing unit in the AI ​​processor 101 to read the image signal from the on-chip RAM for image processing; the computing unit in the AI ​​processor 101 stores the image signal in the on-chip RAM and then notifies the AI ​​controller, the AI ​​controller sends information indicating that the image signal is stored in the on-chip RAM to the ISP controller, and the ISP controller ISP102 reads the image signal from the on-chip RAM for processing.

[0077] In this embodiment, the electronic device 100 may also include a communication unit (not shown in the figure), which includes but is not limited to a short-range communication unit or a cellular communication unit. Among them, the short-range communication unit exchanges information with a terminal for accessing the Internet located outside the mobile terminal by running a short-range wireless communication protocol. The short-range wireless communication protocol may include but is not limited to: various protocols supported by radio frequency identification technology, Bluetooth communication technology protocols, or infrared communication protocols. The cellular communication unit accesses the Internet through the wireless access network by running a cellular wireless communication protocol to enable the mobile communication unit to exchange information with servers that support various applications on the Internet. The communication unit can be integrated into the same SOC as the AI ​​processor 101 and ISP102 described in the above embodiments, or can be separately set. In addition, the electronic device 100 may also selectively include a bus, an input / output port I / O, or a storage controller. The storage controller is used to control the memory 103 and the off-chip memory 106. Among them, the bus, the input / output port I / O, and the storage controller can be integrated into the same SOC as the above-mentioned ISP102 and AI processor 101. It should be understood that in practical applications, the electronic device 100 may include Figure 1 or Figure 8 The embodiments of the present application are not limited to more or fewer components shown.

[0078] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0079] In this embodiment, one or more processors can be divided into functional modules according to the above method examples. For example, different processors can be divided according to different functions, or two or more processors with two or more functions can be integrated into one processor module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0080] In the case of dividing each functional module into corresponding functional modules, Fig. 9 FIG. 9 is a schematic diagram showing a possible image processing device 900 involved in the above embodiment, and the above-mentioned device can be further expanded. Fig. 9As shown, the image processing device 900 may include: an AI processing module 901 and an image signal processing module 902. The AI ​​processing module 901 is used to perform a first image signal processing on a first image signal to obtain a second image signal, wherein the first image signal is obtained based on image data output by an image sensor; and the image signal processing module 902 is used to perform a second image signal processing on the second image signal to obtain an image processing result.

[0081] In a possible implementation manner, the image signal processing module 902 is further configured to: receive the image data from the image sensor, and perform third image signal processing on the image data to obtain the first image signal.

[0082] In one possible implementation, the third image signal processing includes multiple processing processes, in which two adjacent processing processes are used to generate a third image signal, and a subsequent processing process is used to process a fourth image signal; the AI ​​processing module 901 is also used to perform fourth image signal processing on the third image signal to obtain the fourth image signal.

[0083] In a possible implementation manner, the first image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction or gamma correction.

[0084] In a possible implementation, the second image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, color difference correction, gamma correction, color difference correction or RGB to YUV domain conversion.

[0085] In a possible implementation manner, the third image signal processing includes at least one of the following processing procedures: noise elimination, black level correction, shadow correction, white balance correction, or demosaicing.

[0086] In a possible implementation manner, the fourth image signal processing includes at least one of the following processing processes: black level correction, shadow correction, white balance correction, demosaicing or chromatic aberration correction.

[0087] The image processing device 900 provided in this embodiment is used to execute the image processing method executed by the electronic device 100, and can achieve the same effect as the above-mentioned implementation method or device. Fig. 9 The corresponding modules can be implemented in software, hardware or a combination of both. For example, each module can be implemented in software, corresponding to Figure 1The corresponding processor corresponding to the module is used to drive the corresponding processor to work. Alternatively, each module may include a corresponding processor and corresponding driver software, that is, it is implemented in combination with software or hardware. Therefore, the image processing device 900 can be considered to logically include Figure 1 , Figure 3 , Figure 4 , Figure 7 or Figure 8 In the device shown, each module at least includes a driver software program of corresponding function, which is not elaborated in this embodiment.

[0088] Exemplarily, the image processing device 900 may include at least one processor and a memory, and specifically refer to Figure 1 . Among them, at least one processor can call all or part of the computer program stored in the memory to control and manage the actions of the electronic device 100, for example, it can be used to support the electronic device 100 to execute the steps executed by the above modules. The memory can be used to support the electronic device 100 to execute stored program codes and data, etc. At least one processor can implement or execute various exemplary multiple logic modules described in conjunction with the contents disclosed in this application, which can be a combination of one or more microprocessors that implement computing functions, for example, including but not limited to Figure 1 The AI ​​processor 101 and the image signal processor 102 are shown. In addition, at least one processor may also include other programmable logic devices, transistor logic devices, or discrete hardware components. The memory described in this embodiment may include but is not limited to Figure 8 Off-chip memory 106 or memory 103 is shown.

[0089] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0090] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0091] Among them, the computer-readable storage medium or computer program product provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0092] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0093] In addition, each functional unit in each embodiment of the present application may be integrated into one product, or each unit may exist physically separately, or two or more units may be integrated into one product. Fig. 9 , if the above modules are implemented in the form of software functional units and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0094] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A system on a chip, integrated in the same chip, characterized in that: include: An image signal processor ISP, configured to receive image data from an image sensor and perform a third image signal processing on the image data to obtain a first image signal; an artificial intelligence (AI) processor, configured to perform first image signal processing on the first image signal to obtain a second image signal, wherein the AI ​​processor comprises a dedicated neural processor, and the first image signal processing comprises at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction, or gamma correction; A memory, coupled to the AI ​​processor and the ISP, for transferring an image block in the first image signal between the ISP and the AI ​​processor, wherein the image block is a local image signal in a frame of the first image signal; wherein the ISP stores the image block in the memory.

2. The system on chip according to claim 1, characterized in that: The dedicated neural processor includes a neural processing engine.

3. The system on chip according to claim 1, characterized in that: The memory is also used to store intermediate data generated during the operation of the AI ​​processor.

4. The system on chip according to claim 1, characterized in that: The memory is also used to store weight data of each network node in the neural network run by the AI ​​processor.

5. The system on chip according to any one of claims 1 to 4, characterized in that: The memory includes an on-chip random access memory RAM.

6. The system on chip according to any one of claims 1 to 4, characterized in that: The AI ​​processor and the ISP are connected via an electronic circuit to transmit an interrupt signal.

7. The system on chip according to any one of claims 1 to 4, characterized in that: Also includes: A controller is used to control the operation of the ISP and the AI ​​processor.

8. The system on chip according to claim 7, characterized in that: The controller comprises: a central processing unit CPU.

9. The system on chip according to any one of claims 1 to 4, characterized in that: The AI ​​processor runs one or more image processing models to perform the first image signal processing.

10. The system on chip according to claim 9, characterized in that: The AI ​​processor runs the image processing model by loading the executable program stored in the off-chip memory.

11. The system on chip according to any one of claims 1 to 4, characterized in that: The third image signal processing includes at least one of the following processing procedures: noise elimination, black level correction, shadow correction, white balance correction or demosaicing.

12. The system on chip according to any one of claims 1 to 4, characterized in that: Also includes: Communication unit.

13. The system on chip according to claim 12, characterized in that: The communication unit includes: a short-range communication unit or a cellular communication unit.

14. The system on chip according to any one of claims 1 to 4, characterized in that: Also includes: Graphics processing unit GPU.

15. An image processing method, characterized in that: The method comprises: An image signal processor ISP in the system on chip receives image data from the image sensor, and performs a third image signal processing on the image data to obtain a first image signal; The memory in the system on chip transfers an image block in the first image signal between the ISP and the artificial intelligence (AI) processor in the system on chip, wherein the image block is a local image signal in a frame of the image signal in the first image signal; The AI ​​processor performs first image signal processing on the first image signal to obtain a second image signal, wherein the AI ​​processor includes a dedicated neural processor, and the first image signal processing includes at least one of the following processing processes: noise elimination, black level correction, shadow correction, white balance correction, demosaicing, color difference correction, or gamma correction; The ISP is stored in the memory in the form of the image block.

Images