Image processing apparatus, image processing method, and electronic device

Abstract

The present application provides an image processing apparatus, an image processing method, and an electronic device, applied to the technical field of electronics, and for use in reducing power consumption and computing power requirements when an artificial intelligence (AI) processor and an image signal processor collaboratively process images. The image processing apparatus comprises an AI processor, an image signal processor, and a memory. The AI processor may acquire a second image signal, and perform first image processing on the second image signal to obtain a third image signal. The second image signal is an image signal of a region of interest in a first image signal obtained on the basis of image data of an image sensor. Then, the memory transfers the third image signal between the image signal processor and the AI processor. Finally, the image signal processor can perform second image processing comprising a fusion process on the first image signal by means of the third image signal to obtain a fourth image signal. On this basis, the power consumption and computing power requirements of the AI processor can be reduced.

Description

Image processing device, image processing method and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 8, 2023, with application number 202311682875.X and application name “An image processing device, image processing method and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to an image processing device, an image processing method, and an electronic device. Background Art

[0003] With the advancement of image processing technology, more and more users are using smart devices for taking photos, recording videos, and making video calls. However, due to the limited computing power of the image signal processor (ISP) within smart devices, existing smart devices suffer from poor image processing performance. To improve image processing, related technologies have proposed an image processing method that combines the ISP with artificial intelligence (AI) technology. The AI ​​processor can be located at the back end of the ISP to supplement and correct the ISP's image processing results. In a specific implementation, the ISP stores the processed image in memory, and the AI ​​processor reads the ISP-stored image from the memory and performs further image correction based on the ISP-processed image to generate the final image. However, due to bottlenecks in the capabilities of traditional ISPs, the raw image data captured by the image sensor suffers from information loss after ISP processing, which in turn reduces image processing performance and further increases reliance on the AI ​​processor. While the AI ​​processor can improve the image processing results output by the ISP to a certain extent, it significantly increases computing power and computing power requirements. Therefore, a solution to this problem is urgently needed. Summary of the Invention

[0004] Embodiments of the present application provide an image processing device, an image processing method, and an electronic device to reduce power consumption and computing power requirements when an AI processor and an image signal processor collaboratively process images.

[0005] In order to solve the above problems, the technical solutions provided in the embodiments of the present application are as follows.

[0006] In a first aspect, an image processing device is provided. The image processing device includes an artificial intelligence processor, an image signal processor, and a memory. When performing image processing, the artificial intelligence processor can acquire a second image signal and perform a first image processing on the second image signal to obtain a third image signal. The second image signal is an image signal of a region of interest in the first image signal obtained based on image data from an image sensor. The memory is coupled to the image signal processor and the artificial intelligence processor and can transfer the third image signal between the image signal processor and the artificial intelligence processor. The image signal processor can then perform a second image processing on the first image signal using the third image signal to obtain a fourth image signal. The second image processing includes fusion processing of the first image signal and the third image signal.

[0007] Through the above method, the artificial intelligence processor only needs to enhance the image signal of the area of ​​interest in the first image signal without processing the entire first image signal, thereby improving the image quality while reducing the computing power and power consumption requirements of the artificial intelligence processor hardware.

[0008] In a possible implementation, the first image processing includes at least one of demosaicing or noise reduction. Through the above approach, signal quality enhancement of the region of interest can be achieved.

[0009] In one possible implementation, the second image processing further includes at least one of noise removal, black level correction, shading correction, white balance correction, demosaicing, gamma correction, chromatic aberration correction, or color space conversion. Through the above approach, the image signal processor can perform secondary enhancement processing on the fused image signal after fusing the first image signal with the third image signal to further improve image quality.

[0010] In one possible implementation, the memory is further used to transfer the first image signal between the image signal processor and the artificial intelligence processor. When the artificial intelligence processor acquires the second image signal, it can perform region-of-interest extraction on the first image signal to obtain the second image signal. In this manner, the artificial intelligence processor can perform region-of-interest extraction. This allows the computing power of the artificial intelligence processor to be fully utilized, improving processing efficiency.

[0011] In one possible implementation, when the artificial intelligence processor acquires the second image signal, it can first perform a third image processing on the image data through the image signal processor to obtain the first image signal, and then perform a region of interest extraction on the first image signal to obtain the second image signal. The second image signal is then transferred between the image signal processor and the artificial intelligence processor via a memory. Finally, the artificial intelligence processor acquires the second image signal from the memory. In this manner, the image signal processor can first perform a region of interest extraction on the first image signal before transferring it to the artificial intelligence processor via the memory for processing, reducing the amount of data that needs to be transferred to the memory, thereby reducing the power consumption generated by the memory's read and write operations and the power consumption of the artificial intelligence processor.

[0012] In one possible implementation, the image signal processor may further perform a third image processing on the image data to obtain the first image signal. The third image processing may include at least one of noise reduction, black level correction, shading correction, white balance correction, demosaicing, chromatic aberration correction, or gamma correction. This approach improves the quality of the first image signal, thereby improving the quality of the second image signal obtained after performing region of interest extraction on the first image signal.

[0013] In one possible implementation, the AI ​​processor performs the first image processing after receiving a first interrupt signal from the image signal processor. The image signal processor performs the second image processing after receiving a second interrupt signal from the AI ​​processor. This allows the AI ​​processor and the image signal processor to directly transmit interrupt signals, which can increase signal transmission speed and, in certain real-time video playback scenarios, reduce image output latency, improving user experience.

[0014] In one possible implementation, the image processing device is a chip, in which the artificial intelligence processor, image signal processor, and memory are integrated.

[0015] In a possible implementation, the third image signal is stored in a memory in the form of an image block and transferred between the image signal processor and the artificial intelligence processor.

[0016] In a second aspect, an embodiment of the present application provides an image processing method. When the image processing method is executed, a second image signal can be obtained by an artificial intelligence processor, and the second image signal can be subjected to a first image processing to obtain a third image signal. The second image signal is an image signal of an area of ​​interest in the first image signal obtained based on image data of the image sensor. Then, the memory transfers the third image signal between the image signal processor and the artificial intelligence processor. Finally, the image signal processor performs a second image processing on the first image signal using the third image signal to obtain a fourth image signal. The second image processing includes a fusion process of the first image signal and the third image signal.

[0017] In a possible implementation, the first image processing includes at least one of demosaicing and noise reduction.

[0018] In a possible implementation, the second image processing further includes: at least one of noise elimination, black level correction, shadow correction, white balance correction, demosaicing, gamma correction, chromatic aberration correction, or color space conversion.

[0019] In one possible implementation, when the artificial intelligence processor acquires the second image signal, the memory may first transfer the first image signal between the image signal processor and the artificial intelligence processor. The artificial intelligence processor then performs region of interest extraction on the first image signal to obtain the second image signal.

[0020] In one possible implementation, when the artificial intelligence processor acquires the second image signal, it may first perform region-of-interest extraction on the first image signal using the image signal processor to obtain the second image signal. The second image signal is then transferred between the image signal processor and the artificial intelligence processor via a memory. Finally, the artificial intelligence processor retrieves the second image signal from the memory.

[0021] In one possible implementation, the image signal processor may further perform a third image processing on the image data to obtain the first image signal. The third image processing may include at least one of noise removal, black level correction, shading correction, white balance correction, demosaicing, chromatic aberration correction, or gamma correction.

[0022] In one possible implementation, the artificial intelligence processor performs the first image processing after receiving a first interrupt signal sent by the image signal processor, and performs the second image processing after receiving a second interrupt signal sent by the artificial intelligence processor.

[0023] In a possible implementation, the third image signal is stored in a memory in the form of an image block and transferred between the image signal processor and the artificial intelligence processor.

[0024] In one possible implementation, the above method is applied to an image processing device; the image processing device is a chip, and an artificial intelligence processor, an image signal processor, and a memory are integrated in the chip.

[0025] In a third aspect, embodiments of the present application provide a computer-readable storage medium having computer program instructions stored therein. When the computer program instructions are executed by an artificial intelligence processor and an image signal processor, the method of any possible implementation of the second aspect described above is implemented.

[0026] In a fourth aspect, embodiments of the present application provide a computer program product. When executed by an artificial intelligence processor and an image signal processor, the computer program product implements the method in any possible implementation of the second aspect.

[0027] In a fifth aspect, an electronic device is provided, comprising an image sensor and an image processing device according to any possible implementation of the first aspect above, connected to the image sensor.

[0028] The technical effects brought about by the above-mentioned second to fifth aspects and possible implementation methods can be found in the description of the technical effects brought about by the above-mentioned first aspect and possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a schematic structural diagram of an electronic device;

[0030] FIG2 is a schematic diagram of the structure of an image signal processor;

[0031] FIG3 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

[0032] FIG4 is a flow chart of an image processing method provided in an embodiment of the present application;

[0033] FIG5 is a schematic structural diagram of another image processing device provided in an embodiment of the present application;

[0034] FIG6 is a schematic structural diagram of another image processing device provided in an embodiment of the present application;

[0035] FIG7 is a schematic flow chart of another image processing method provided in an embodiment of the present application;

[0036] FIG8 is a schematic diagram of the structure of a neural network model provided in an embodiment of the present application;

[0037] FIG9 is a schematic structural diagram of another image processing device provided in an embodiment of the present application;

[0038] FIG10 is a schematic structural diagram of another image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0040] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit differences. At the same time, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for ease of understanding. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more. For example, multiple processors refers to two or more processors.

[0041] When describing some embodiments, the terms "coupled" and "connected," and their derivatives, may be used. For example, when describing some embodiments, the term "connected" may be used to indicate that two or more components are in direct physical contact or point contact with each other. For another example, when describing some embodiments, the term "coupled" may be used to indicate that two or more components are in direct physical contact or electrical contact, or it may mean that two or more components are not in direct contact with each other but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the contents herein.

[0042] In addition, for ease of understanding, the technical terms involved in the embodiments of the present application are first introduced below.

[0043] Region of Interest (ROI) extraction: In image processing, a region of interest (ROI) refers to a specific object or target within a specified area, including features such as the object's outline, edges, and texture. ROI extraction extracts the image signal of the region of interest from image data. Common ROI extraction methods vary depending on the application scenario and requirements, but include the following.

[0044] 1. Threshold segmentation method

[0045] Threshold segmentation separates an image into its corresponding object and background. This method sets the grayscale of pixels with grayscale values ​​less than a fixed value to 1, while those with grayscale values ​​greater than a fixed value are set to 0. Pixels with grayscale values ​​between these two fixed values ​​are then judged based on the specific application. This allows the object in the image to be separated from the background, thereby extracting the region of interest.

[0046] 2. Edge detection method

[0047] Edge detection is the process of detecting the outline and edge locations of an object in an image. Common edge detection methods include the Sobel operator, the Canny operator, and the Laplacian operator. These operators can detect boundaries in an image and thus extract regions of interest.

[0048] 3. Region Growing Method

[0049] Region growing is a method that starts from a specific point in an image and gradually grows the target object. This method requires specifying a reference point for growth and a growth rule. The growth rule can be based on indicators such as pixel grayscale similarity and spatial proximity to determine whether a pixel should be included in the ROI. This method can automatically grow and extract the ROI without knowing the target region's shape in advance.

[0050] 4. Template matching based method

[0051] For certain objects with specific shapes, ROI can be extracted by searching for the corresponding template in the image. This method requires preparing a template that matches the shape of the object in advance, and by searching for similar areas in the image, the object can be extracted from the background.

[0052] The present application is described in detail below with reference to the accompanying drawings and embodiments:

[0053] Image signal processors are currently widely used in smart terminal devices, such as smartphones. With people's pursuit of mobile phone shooting performance, improving performance to obtain better imaging effects is the current research focus. The main function of the image signal processor is to perform image processing on the signal output by the front-end image sensor, which largely determines the image quality of photos and videos. With the popularization of smartphones and the improvement of shooting quality, more and more users use their mobile phones to shoot videos, and video quality has become one of the core competitiveness of various mobile phone manufacturers. However, the image signal processors based on traditional algorithms have already encountered bottlenecks. However, AI image processing algorithms have broken through the bottlenecks of traditional processing algorithms and achieved better image processing effects. Therefore, many image processing methods choose to improve image processing effects through collaborative processing of image signal processors and AI processors (including dedicated neural network processors). Based on the above background, as shown in Figure 1, an embodiment of the present application provides an electronic device 10, including an image processing device 100. The image processing device 100 includes multiple processors, such as an AI processor 120 and an image signal processor 110. Optionally, the multiple processors described above may be integrated into one or more chips, which may be considered a chipset. When the multiple processors described above are integrated into the same chip, the chip is also called a system on a chip (SOC). In addition to the multiple processors described above, the image processing device 100 also includes one or more other necessary components, such as a memory 130. In one possible implementation, the memory 130 may be located in the same SoC in the image processing device 100 as the AI ​​processor 120 and the image signal processor 110, that is, the memory 130 is integrated into the image processing device 100 shown in FIG1 above. In this case, the memory 130 may include on-chip random access memory (RAM). In one possible implementation, the image processing device 100 may also include a controller 140. The controller 140 may also be located in the same SoC in the image processing device 100 as the AI ​​processor 120 and the image signal processor 110, that is, the memory 130 is integrated into the image processing device 100 shown in FIG1 above. The controller 140 may run necessary software programs or software plug-ins to drive the controller 140 to control the operation of the AI ​​processor 120 and the image signal processor 110 and to control the communication between the AI ​​processor 120 and the image signal processor 110. For example, the controller 140 may be a central processing unit (CPU).

[0054] In some embodiments, the image processing device 100 may be a module, chip, chip system, circuit board or component integrated into the electronic device 10. The electronic device 10 may be a user equipment (UE), such as a mobile phone, a tablet computer, a smart screen, a cloud server with video processing capabilities, or an image capture device (such as a camera, a camcorder) and other types of devices. The electronic device 10 may be provided with a camera, which may also be referred to as an image sensor 200, for collecting image data. The electronic device 10 may also be installed with various software applications for driving the camera to capture images, such as camera applications, video call applications, or online video shooting applications. Users can use the camera to take photos or videos by starting the above-mentioned various applications. In addition, users can also use such applications to make various personalized settings for image beautification. Taking video call applications as an example, users can choose to automatically adjust the screen image (such as the facial avatar presented, or the background image presented) during a video call. When a user launches any of the aforementioned applications, or launches any of the aforementioned applications and selects image beautification, the image processing services supported by the aforementioned applications in the electronic device may trigger the electronic device to process the image data captured by the camera, thereby presenting the processed image on the screen of the electronic device to achieve an image beautification effect. Such image beautification includes, but is not limited to, increasing the brightness of a portion of the image or the entire image, changing the image's display color, smoothing the skin of facial objects presented in the image, adjusting the image's saturation, exposure, vividness, highlights, contrast, sharpness, or clarity, etc.

[0055] As shown in FIG2 , in the image signal processor 110 , the image data received by the image sensor 200 can be processed by multiple image processing modules (such as image processing module 1 to image processing module n in FIG2 ) on the ISP pipeline 111 to perform various image processing operations. Among them, the image processing modules on the ISP pipeline 111 may include one or more of a noise elimination module, a black level correction module, a shadow correction module, a white balance correction module, a demosaicing module, a color difference correction module, a gamma correction module, or a color space conversion module (for example, RGB domain to YUV domain. Wherein, R represents red; G represents green; B represents blue. Y is the brightness component, representing physical linear space brightness; U represents the chrominance component of blue; V represents the chrominance component of red). Among them, the parameters of each module in the image signal processor 110 can be configured through the ISP controller 112. The processing processes in the image signal processor 110 can all be online processing processes, and the processing results can then be stored in the memory for offline processing or output. Alternatively, the processing results of certain processing modules in the image signal processor 110 can be stored in the memory as needed, processed offline, and then fed back to the image signal processor 110.

[0056] In the above embodiment, the AI ​​processor 120 includes but is not limited to a convolutional neural network processor, a tensor processor or a neural processing engine. The AI ​​processor can be used as a component alone or integrated into other digital logic devices, which include but are not limited to: a CPU, a graphics processing unit (GPU) or a digital signal processor (DSP). Exemplarily, the CPU, GPU and DSP are all processors within the system on chip. The AI ​​processor 120 can perform one or more image processing operations, which may include but are not limited to: noise elimination, black level correction, shading correction, white balance correction, demosaicing, chromatic aberration correction or gamma correction. The AI ​​processor 120 can run one or more image processing models, each of which is used to perform a specific image processing operation. For example, the noise elimination image processing model is used to perform the noise elimination image processing operation, and the demosaicing image processing model is used to perform the demosaicing image processing operation. Each image processing model can be obtained by training the neural network using training samples using a traditional neural network training method, and the embodiments of the present application will not be repeated here. The image signal processor 110 may be configured with multiple hardware modules or run necessary software programs to process images or communicate with the AI ​​processor 120. The image signal processor 110 and the AI ​​processor 120 may communicate via a direct hardware connection or by forwarding signals through the controller 140.

[0057] In the image processing device 100 shown in FIG1 , the AI ​​processor 120 can perform some image processing operations on behalf of the image signal processor 110, thereby avoiding the loss of raw image data due to insufficient processing power of the image signal processor 110 during image processing and improving image processing performance. If the memory 130 needs to transfer all image data captured by the image sensor between the image signal processor 110 and the AI ​​processor 120, the power consumption of the entire image processing process will increase. Furthermore, the AI ​​processor 120 needs to fully process each frame of image data, which increases both the power consumption and the computing power required of the AI ​​processor 120. To address the above issues, as shown in FIG3 , an embodiment of the present application provides an image processing device 300, which includes an AI processor 310, an image signal processor 320, and a memory 330. The image signal processor 320 can be connected to the image sensor 400. The image signal processor 320 includes a region of interest extraction module 321, a first image processing unit 322, and a second image processing unit 323. The first image processing unit 322 includes multiple image processing modules (such as image processing modules 1, ..., and n in FIG3 ). These multiple image processing modules may include at least one of a noise elimination module, a black level correction module, a shading correction module, a white balance correction module, a demosaicing module, a chromatic aberration correction module, and a gamma correction module. The second image processing unit 323 includes a fusion module, a detection module, and a subsequent image processing module. These subsequent image processing modules may include at least one of a noise elimination module, a black level correction module, a shading correction module, a white balance correction module, a demosaicing module, a chromatic aberration correction module, a gamma correction module, or a color space conversion module. The region of interest extraction module 321 may perform region of interest extraction processing on the image signal output by any module in the first image processing unit 322 (e.g., image processing module 3) to obtain an image signal of the region of interest (i.e., the ROI signal in FIG3 ). The image signal of the region of interest may then be transferred via the memory 330 to the image enhancement network within the dedicated neural network processor in the AI ​​processor 310 for enhancement processing. Finally, the enhanced image signal can be transferred to the second image processing unit 323 in the image signal processor 320 through the memory 330 for second image processing to obtain an image processing result. The memory 330 can be an on-chip random access memory (RAM). The region of interest extraction module in the AI ​​processor 310 can include multiple logic devices or circuits. The image signal processor 320 and the AI ​​processor 310 can be controlled uniformly by a controller.In addition, the image signal processor 320 and the AI ​​processor 310 can communicate through a direct hardware connection; or, the image signal processor 320 and the AI ​​processor 310 can also communicate through signal forwarding by a controller.

[0058] Based on FIG3 , as shown in FIG4 , the embodiment of the present application further provides an image processing method applied to the above-mentioned image processing apparatus 300 . The specific processing flow of the image processing method is as follows.

[0059] S401: The image sensor sends the acquired image data to the image signal processor.

[0060] S402: The image signal processor performs third image processing on the image data to obtain a first image signal.

[0061] The third image processing may be performed by the first processing unit 322 in the image signal processor 320. The third image processing may include at least one of noise removal, black level correction, shading correction, white balance correction, demosaicing, chromatic aberration correction, and gamma correction. The specific selection may be based on actual needs and will not be detailed in detail in this embodiment of the present application.

[0062] S403: The image signal processor extracts a region of interest from the first image signal to obtain a second image signal.

[0063] In an example, the region of interest extraction module provided in the image signal processor 320 in FIG3 may perform region of interest extraction on the first image signal to obtain the second image signal.

[0064] Exemplarily, assuming that the third image processing only includes a noise elimination processing operation, the region of interest extraction module can perform region of interest extraction on the image signal output by the noise elimination module to obtain a second image signal. Of course, the region of interest extraction module can also choose to perform region of interest extraction processing on the image signal output by any one of the black level correction module, shadow correction module, white balance correction module, demosaicing module, color difference correction module, gamma correction module or color space conversion module in the image signal processor 320 to obtain a second image signal. The embodiment of the present application does not impose specific restrictions on this. In addition, the region of interest extraction method of the region of interest extraction module can choose to adopt any one of the threshold segmentation method, edge detection method, region growing method and template matching-based method, and the embodiment of the present application does not impose specific restrictions on this.

[0065] S404: The image signal processor provides the second image signal to the AI ​​processor.

[0066] The image signal processor 320 may store the second image signal in the first storage area of ​​the memory 330 in the form of an image block. The image block is a partial image signal of a frame of the image signal (for example, a frame of the image signal has 1280 rows of pixels, and the memory 330 stores an image signal formed by 320 rows of pixels). Then, the image signal processor 320 may directly send an instruction signal to the AI ​​processor 310 to instruct the AI ​​processor 310 to obtain the second image signal from the first storage area. Alternatively, the image signal processor 320 may send an instruction signal to the AI ​​processor 310 via the controller to instruct the AI ​​processor 310 to read the second image signal from the first storage area.

[0067] S405: The AI ​​processor acquires a second image signal, and performs the first image processing on the second image signal to obtain a third image signal.

[0068] The AI ​​processor 310 may read the second image signal from the first storage area via an instruction signal sent by the image signal processor 320. Then, the first image processing is performed on the second image signal to obtain a third image signal. Alternatively, the AI ​​processor 310 may read the second image signal from the first storage area via an instruction signal sent by the image signal processor 320 via the controller. Then, the first image processing is performed on the second image signal to obtain a third image signal.

[0069] In one embodiment, the AI ​​processor 310 can read each image block of the second image signal from the first storage area of ​​the memory 330 in the form of image blocks and perform the first image processing on the read image blocks, that is, perform block-based processing. After the image signal processor 320 stores an image block in the first storage area of ​​the memory 330, it sends an instruction signal to the AI ​​processor 310. Upon receiving the instruction signal, the AI ​​processor 310 reads the image block from the first storage area of ​​the memory 330 and performs the first image processing. In a specific implementation, the AI ​​processor 310 performs the first image processing on the image blocks in units of one row of pixels. Furthermore, while the AI ​​processor 310 performs the first image processing on one image block, the image signal processor 320 can continue to store the next image block in the first storage area of ​​the memory 330. Similarly, the first image processing on the second image signal is gradually completed, thereby implementing block-based pipeline processing. This approach can reduce the waiting time of the AI ​​processor 310, thereby improving the processing efficiency of the AI ​​processor 310.

[0070] In the above implementation process, because the second image signal is only the image signal of the area of ​​interest in the first image signal, the AI ​​processor 310 can choose to perform at least one of demosaicing or noise reduction when performing the first image processing on the second image signal. Among them, demosaicing can select any one of the various existing demosaicing algorithms, and the embodiment of the present application does not impose specific restrictions on this. Noise reduction can select any one or more noise reduction methods among time domain noise reduction, spatial domain noise reduction and hybrid noise reduction, or other existing noise reduction methods can be selected, and the embodiment of the present application does not impose specific restrictions on this.

[0071] In this way, the computing power requirements for the AI ​​processor 310 hardware can be reduced, so that the image signal of the region of interest can be enhanced under low-computing hardware conditions. In addition, if the computing power of the AI ​​processor 310 is sufficient, the above method can also reduce the power consumption of the AI ​​processor 310 while meeting user needs.

[0072] Optionally, based on the above implementation, if the computing power of the AI ​​processor 310 meets the requirements, the AI ​​processor 310 may also selectively perform at least one of noise removal, black level correction, shading correction, white balance correction, demosaicing, gamma correction, chromatic aberration correction, or color space conversion when performing the first image processing. In this way, the image signal of the region of interest can be further enhanced, thereby improving the quality of the final image processing result.

[0073] S406: The AI ​​processor provides the third image signal to the image signal processor.

[0074] The AI ​​processor 310 may store the third image signal in the second storage area of ​​the memory 330 in the form of an image block. The AI ​​processor 310 may then directly send an instruction signal to the image signal processor 320 to instruct the image signal processor 320 to obtain the third image signal from the second storage area. Alternatively, the AI ​​processor 310 may send an instruction signal to the image signal processor 320 via a controller to instruct the image signal processor 320 to obtain the third image signal from the second storage area.

[0075] S407: The image signal processor performs second image processing on the first image signal using the third image signal to obtain a fourth image signal.

[0076] The second image processing can be performed by the second image processing unit 323 in the image signal processor 320. The second image processing unit 323 can read the third image signal from the second storage area via an instruction signal sent by the AI ​​processor 310. Then, the second image processing is performed on the first image signal using the third image signal to obtain a fourth image signal. Alternatively, the second image processing unit 323 can read the third image signal from the second storage area via an instruction signal sent by the AI ​​processor via the controller. Then, the second image processing is performed on the first image signal using the third image signal to obtain a fourth image signal.

[0077] In one embodiment, the image signal processor 320 can read each image block of the third image signal from the second storage area of ​​the memory 330 in the form of image blocks and perform the second image processing on the read image blocks, that is, perform block-based processing. After the AI ​​processor 310 stores an image block in the second storage area of ​​the memory 330, it sends an instruction signal to the image signal processor 320. Upon receiving the instruction signal, the image signal processor 320 reads the image block from the second storage area of ​​the memory 330 and performs the second image processing. The image signal processor 320 also performs the second image processing on the image blocks in units of one row of pixels. Furthermore, while the image signal processor 320 performs the second image processing on one image block, the AI ​​processor 310 can continue to store the next image block in the second storage area of ​​the memory 330. Similarly, the second image processing of the third image signal is gradually completed, thereby implementing block-based pipeline processing. In this way, the waiting time of the image signal processor 320 can be reduced, thereby improving the processing efficiency of the image signal processor 320.

[0078] In one embodiment, the second image processing includes a first processing process and a second processing process. The first processing process is used to perform fusion processing on the first image signal and the third image signal. The purpose of the fusion processing is to replace the second image signal in the first image signal with the third image signal to obtain a fifth image signal, that is, to replace the ROI area before processing in the first image signal with the ROI area after processing, thereby achieving ROI area enhancement. The second processing process is used to perform a fourth image processing on the fifth image signal to obtain a fourth image signal. The fourth image signal can be output as a processing result. The first processing process can be executed by a fusion module in the second image processing unit 323. The second processing process can be executed by a subsequent image processing module in the second image processing unit 323. In addition, when executing the first processing process, the edge of the third image signal can be extracted, and the edge smoothing filter of the replaced fifth image signal can be performed using a Gaussian filter to reduce the abruptness after fusion.

[0079] In another embodiment, the second image processing may further include a third processing step performed between the first and second processing steps. The third processing step may be performed by a detection module in the second image processing unit 323. The third processing step may determine the image quality of the fifth image signal. If the image quality meets a reference value, a fifth image signal is output. The fifth image signal is then subjected to a fourth image processing step through the second processing step to obtain a fourth image signal. If the image quality falls below the reference value, the first image signal is output. The first image signal is then subjected to a fourth image processing step through the second processing step to obtain a sixth image signal. Finally, the sixth image signal is output as the image processing result.

[0080] In the above two embodiments, the fourth image processing may include at least one of noise removal, black level correction, shading correction, white balance correction, demosaicing, gamma correction, chromatic aberration correction, or color space conversion. The third image processing may perform a portion of the processes selected from the group consisting of noise removal, black level correction, shading correction, white balance correction, demosaicing, chromatic aberration correction, and gamma correction. On this basis, the fourth image processing may also selectively perform a portion of the processes selected from the group consisting of black level correction, shading correction, white balance correction, demosaicing, gamma correction, chromatic aberration correction, and color space conversion (e.g., RGB to YUV conversion).

[0081] Exemplarily, if the third image processing includes noise elimination, the fourth image processing may include black level correction, shading correction, white balance correction, demosaicing, gamma correction, color difference correction, and color space conversion (for example, RGB domain to YUV domain), etc. If the third image processing includes noise elimination and black level correction, the fourth image processing may include shading correction, white balance correction, demosaicing, gamma correction, color difference correction, and color space conversion (for example, RGB domain to YUV domain), etc. Referring to the above method, the operator can configure the processing operations included in the third image processing and the fourth image processing according to actual needs. The embodiment of the present application does not impose specific restrictions on the processing operations included in the third image processing and the fourth image processing.

[0082] In another embodiment, the third image processing may perform all of the following processing steps: noise removal, black level correction, shading correction, white balance correction, demosaicing, color difference correction, and gamma correction. The fourth image processing may also perform all of the following processing steps: noise removal, black level correction, shading correction, white balance correction, demosaicing, gamma correction, color difference correction, and color space conversion. In this way, the fifth image signal or the first image signal may be subjected to secondary enhancement processing, thereby improving the quality of the final output image processing result.

[0083] In one embodiment, the third processing process can determine the image quality of the fifth image signal by at least one of a mean square error (MSE), a peak signal to noise ratio (PSNR), and a structural similarity (SSIM) between the first image signal and the fifth image signal.

[0084] In one example, the operator can select one of the mean square error, peak signal-to-noise ratio and structural similarity between the first image signal and the fifth image signal as a basis for judging whether the third processing process outputs the fifth image signal or the first image signal according to actual needs. For example, assuming that the structural similarity between the first image signal and the fifth image signal is used as the basis for judging whether the third processing process outputs the fifth image signal or the first image signal, a reference value can be set according to actual needs, such as 95%. If the structural similarity between the first image signal and the fifth image signal reaches 95% (i.e., the structural similarity ≥ 95%), the third processing process outputs the fifth image signal. Otherwise, the third processing process outputs the first image signal. Of course, the above reference value is only an example given in the embodiment of the present application. The specific numerical value of the reference value can be selected according to actual needs, and the embodiment of the present application does not impose specific restrictions on this. In addition, the operator can also refer to the above method to set the corresponding reference value according to the mean square error or peak signal-to-noise ratio between the first image signal and the fifth image signal, which will not be described in detail in the embodiment of the present application.

[0085] In another example, when determining the image quality of the fifth image signal through the third processing step in the second image processing, multiple parameters such as the mean square error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity between the first and fifth image signals can also be used as a basis for determining whether the third processing step outputs the fifth image signal or the first image signal. For example, an operator can construct an image quality calculation equation for the fifth image signal based on the mean square error (MSE), PSNR, and structural similarity weighting parameters between the first and fifth image signals. For example, y = ax1 + bx2 + cx3. Here, a, b, and c represent weighting parameters, and a + b + c = 1; x1 represents the mean square error (MSE) between the first and fifth image signals; x2 represents the PSNR between the first and fifth image signals; and x3 represents the structural similarity between the first and fifth image signals. A computer program is then written based on the aforementioned calculation equation to calculate the image quality and compare the image quality with a set reference value. Finally, if the image quality is lower than the reference value, the third processing step outputs the first image signal. Otherwise, the third processing step outputs the fifth image signal. Of course, the above calculation method is only an example provided in the embodiment of the present application. In specific implementation, the operator can select two or three of the above parameters to construct an image quality calculation method according to actual needs, and write a corresponding computer program based on the constructed calculation method to perform the corresponding calculation process.

[0086] Through the above approach, the image quality of the fifth image signal can be determined through the third processing process before the second processing process is performed. When the image quality reaches the reference value, the second processing process is then performed, thereby avoiding the impact of uncertainty problems that may be generated by the artificial intelligence processor or image smoothness problems that may be caused by the first processing process on image quality, thereby ensuring that the bottom line of the image quality of the output image is controllable. Of course, when determining the image quality of the fifth image signal, the parameters used can also be adjusted according to the actual enhancement processing operation performed by the AI ​​processor 310, and this embodiment of the application does not impose specific limitations on this.

[0087] In one possible implementation, the AI ​​processor 310 and the image signal processor 320 can be connected via an electronic circuit to transmit an interrupt signal. The first image processing is performed after the AI ​​processor 310 receives the first interrupt signal sent by the image signal processor 320. The second image processing is performed after the image signal processor 320 receives the second interrupt signal sent by the AI ​​processor 310. The electronic circuit connection between the AI ​​processor 310 and the image signal processor 320 is also called a physical connection or an interrupt connection. This connection allows the AI ​​processor 310 and the image signal processor 320 to send and receive interrupt signals. This eliminates the need for interrupt signals to be forwarded by other processors, such as the CPU, and eliminates the need for the CPU to participate in related control. This can increase the speed of interrupt signal transmission and, in certain real-time video playback situations, reduce image output latency, thereby improving the user experience. Specifically, the interrupt connection includes an interrupt signal processing hardware circuit for implementing interrupt signal transmission and reception functions and a signal transmission line to enable the transmission and reception of interrupt signals. The interrupt signal processing hardware circuit includes, but is not limited to, a conventional interrupt controller circuit. For the specific implementation of the interrupt signal processing hardware circuit, reference can be made to the relevant description of interrupt controllers in the prior art and will not be elaborated here.

[0088] For example, as shown in FIG5 , the output ports Vpo1 and Vpo2 of the region of interest extraction module 321, the input ports Vpi1 and Vpi2 of the fusion module, and the input ports Vai1, Vai2, and Vao1 of the AI ​​processor in the image signal processor 320 can all be connected to the memory 330 using electronic circuits. The AI ​​processor 310 may include a task scheduler 312 and multiple computing units 311. Each component of the task scheduler 312 and the multiple computing units 311 may include multiple logic devices or circuits. The task scheduler 312 may establish the electronic circuit connection L1 via the input port Vai2 of the AI ​​processor and the output port Vpo2 of the region of interest extraction module 321, and establish the electronic circuit connection L2 via the output port Vao2 of the AI ​​processor and the input port Vpi2 of the fusion module. The image signal processor 320 may send an interrupt signal Z1 (i.e., the first interrupt signal) to the AI ​​processor 310 via the electronic circuit connection L1. The AI ​​processor 310 may send an interrupt signal Z2 (i.e., the aforementioned second interrupt signal) to the image signal processor 320 via the electronic circuit connection L2. For example, when the image signal processor 320 has completed storing the image signal in the form of image blocks in the memory 330 (e.g., when the number of rows of stored image signals reaches a preset threshold, the size of the written image signal reaches a preset threshold, or the last address of the storage addresses allocated to the image signal processor 320 stores an image signal), the AI ​​processor 310 stops transmitting the image signal to the memory 330 and sends an interrupt signal Z1 to the AI ​​processor 310 via the electronic circuit connection L1 shown in FIG. 5 . The AI ​​processor 310 may store the image signal in the memory 330 in the same storage manner as the image signal processor 320, and after storing the image signal in the memory 330, it may send an interrupt signal Z2 to the image signal processor 320 via the electronic circuit connection L2 shown in FIG. 5 . Furthermore, each of the plurality of computing units 311 can be electronically connected to the memory 330 via input port Vai1 to read the image signal output by the region of interest extraction module 321 from the memory 330; or can be electronically connected to the memory 330 via output port Vao1 to write the image signal to the memory 330. The region of interest extraction module 321 is connected to the memory 330 via output port Vpo1 to write the image signal to the memory 330. The output port Vpo3 of the image processing module n is connected to the memory 330 to write the image signal to the memory 330. The fusion module is connected to the memory 330 via input port Vpi1 to read the image signal from the memory 330.

[0089] In one embodiment, as shown in FIG6 , the present application also provides an image processing device 600 . The image processing device 600 includes an AI processor 610 , an image signal processor 620 , and a memory 630 . The image signal processor 620 may be connected to the image sensor 700 . The image signal processor 620 includes a first image processing unit 621 and a second image processing unit 622 . The first image processing unit 621 includes multiple image processing modules (such as image processing modules 1, ..., image processing modules n in FIG6 ). The multiple image processing modules may include at least one of a noise elimination module, a black level correction module, a shading correction module, a white balance correction module, a demosaicing module, a chromatic aberration correction module, or a gamma correction module. The second image processing unit 622 includes a fusion module, a detection module, and a subsequent image processing module. The subsequent image processing modules may include at least one of a noise elimination module, a black level correction module, a shading correction module, a white balance correction module, a demosaicing module, a chromatic aberration correction module, a gamma correction module, or a color space conversion module. When performing image processing, the image data acquired by the image sensor 700 can be transmitted to the first image processing unit 621 in the image signal processor 620. The image signal output by any module in the first image processing unit 621 can be transferred to the region of interest extraction unit in the AI ​​processor 620 through the memory 630 for region of interest extraction to obtain an image signal of the region of interest (i.e., the ROI signal in Figure 6). The ROI signal can be enhanced by the image enhancement unit. Then, the enhanced image signal can be transferred to the second image processing unit 622 in the image signal processor 620 through the memory 630 for second image processing to obtain an image processing result. Among them, the region of interest extraction unit and the image enhancement unit can both include multiple logic devices or circuits.

[0090] Based on FIG6 , as shown in FIG7 , the embodiment of the present application further provides an image processing method applied to the above-mentioned image processing apparatus 600 . The specific processing flow of the image processing method is as follows.

[0091] S701: The image sensor sends the acquired image data to the image signal processor.

[0092] S702: The image signal processor performs third image processing on the image data to obtain a first image signal.

[0093] The third image processing may be performed by the first image processing unit 621 in the image signal processor. The third image processing may include at least one of noise removal, black level correction, shading correction, white balance correction, demosaicing, chromatic aberration correction, or gamma correction. The specific selection may be based on actual needs and will not be described in detail in this embodiment of the present application.

[0094] S703: The image signal processor provides the first image signal to the AI ​​processor.

[0095] The manner in which the image signal processor provides the first image signal to the AI ​​processor may refer to the manner in which the second image signal is transferred in S404 in FIG. 4 , and will not be further described in detail in the embodiment of the present application.

[0096] S704: The AI ​​processor extracts a region of interest from the first image signal to obtain a second image signal.

[0097] Among them, the AI ​​processor can execute a neural network model for region of interest extraction through multiple logic devices or circuits to extract the region of interest from the first image signal, thereby obtaining a second image signal.

[0098] S705: The AI ​​processor performs the first image processing on the second image signal to obtain a third image signal.

[0099] In one embodiment, the specific execution process of the first image processing can refer to the description of S405 above, and the embodiments of the present application will not be repeated here. In addition, because the second image signal is only the image signal of the area of ​​interest in the first image signal, the AI ​​processor can choose to perform at least one of demosaicing or noise reduction when performing the first image processing on the second image signal. Among them, demosaicing can select any one of the various existing demosaicing algorithms, and the embodiments of the present application do not impose specific restrictions on this. Noise reduction can select any one or more noise reduction methods among time domain noise reduction, spatial domain noise reduction and hybrid noise reduction, or other existing noise reduction methods can be selected, and the embodiments of the present application do not impose specific restrictions on this.

[0100] This approach reduces the computing power requirements for the AI ​​processor, allowing image signal enhancement in the region of interest to be performed on low-computing hardware. Furthermore, even if the AI ​​processor has sufficient computing power, the above approach can still reduce the power consumption of the AI ​​processor while still meeting user needs.

[0101] Optionally, based on the above implementation, if the AI ​​processor's computing power meets the requirements, the AI ​​processor may also selectively perform at least one of noise removal, black level correction, shading correction, white balance correction, demosaicing, gamma correction, chromatic aberration correction, or color space conversion when performing the first image processing. In this way, the image signal of the region of interest can be further enhanced, thereby improving the quality of the final image processing result.

[0102] In one embodiment, as shown in FIG8 , an embodiment of the present application provides a neural network model architecture 800. The neural network model architecture 800 includes a region of interest extraction network 810 and an image enhancement network 820. The program corresponding to the region of interest extraction network 810 can be executed by a region of interest extraction unit. The program corresponding to the image enhancement network 820 can be executed by the image enhancement unit. The region of interest extraction network 810 can perform region of interest extraction on an input image signal to obtain an image signal of the region of interest. The image enhancement network 820 can then perform enhancement processing on the image signal of the region of interest to obtain an enhanced image signal. The enhancement processing includes, but is not limited to, demosaicing and noise reduction. To reduce the amount of computation performed by the neural network model, the region of interest extraction network 810 can include two 3x3 convolutional layers, one 3x3 downconvolutional layer, one 5x5 downconvolutional layer, and a gating unit G. The image enhancement network 820 can include one 3x3 deconvolutional layer and one 3x3 convolutional layer. The two convolutional layers in the region of interest extraction network 810 can perform feature extraction on the input image signal to obtain a first processing result. The first processed result is deconvolved through a 3x3 deconvolution layer and then processed through a Sigmoid activation function, outputting either 0 or 1. If the extracted feature is a feature of the region of interest, the output is 1; otherwise, the output is 0. When the output is 1, the image enhancement network can be activated via the gating unit. At this point, the first processed result is deconvolved through a 5x5 deconvolution layer to obtain further extracted features. These features are then calculated sequentially through a 3x3 deconvolution layer and a 3x3 convolution layer in the image enhancement network 820 to obtain an enhanced image signal.

[0103] In the above implementation process, the structure of the region of interest extraction network 810 and the image enhancement network 820 is only an example given in the embodiment of the present application. The operator can adjust it according to actual needs during the specific implementation, and this embodiment of the present application will not be described in detail.

[0104] S706: The AI ​​processor provides the third image signal to the image signal processor.

[0105] The AI ​​processor may store the third image signal in the second storage area of ​​the memory 630 in the form of an image block. The AI ​​processor may then directly send an instruction signal to the image signal processor to instruct the image signal processor to read the third image signal from the second storage area. Alternatively, the AI ​​processor may send an instruction signal to the image signal processor via the controller to instruct the image signal processor to obtain the third image signal from the second storage area.

[0106] S707: The image signal processor performs second image processing on the first image signal using the third image signal to obtain a fourth image signal.

[0107] In one embodiment, the second image processing may be performed by the second image processing unit 622 in the image signal processor 620. The second image processing may also refer to the processing method of the corresponding image block in S407, which will not be described in detail in this embodiment of the present application.

[0108] In one embodiment, the second image processing includes a first processing process and a second processing process. The first processing process is used to perform fusion processing on the first image signal and the third image signal. The purpose of the fusion processing is to replace the second image signal in the first image signal with the third image signal to obtain a fifth image signal, that is, to replace the ROI area before processing in the first image signal with the ROI area after processing, thereby achieving ROI area enhancement. This process can be performed by a fusion module. The second processing process is used to perform a fourth image processing on the fifth image signal to obtain a fourth image signal. The fourth image signal can be output as a processing result. This process can be performed by a subsequent image processing module. In addition, when executing the first processing process, the edge of the third image signal can be extracted, and the edge smoothing filter of the replaced fifth image signal can be performed using a Gaussian filter to reduce the abruptness after fusion.

[0109] In another embodiment, the second image processing may further include a third processing step performed between the first and second processing steps. The third processing step may be performed by a detection module in the second image processing unit 622. The third processing step may determine the image quality of the fifth image signal. If the image quality meets a reference value, a fifth image signal is output. The fifth image signal is then subjected to a fourth image processing step using the second processing step to obtain a fourth image signal. If the image quality falls below the reference value, the first image signal is output. The first image signal is then subjected to a fourth image processing step using the second processing step to obtain a sixth image signal. Finally, the sixth image signal is output as the image processing result.

[0110] In the above implementation process, the processing operations that may be included in the third image processing and the fourth image processing can be referred to the description of the corresponding parts in Figure 4, and the embodiment of the present application will not be repeated here.

[0111] In one embodiment, when the computing power of the AI ​​processor meets the requirements, the above-mentioned S701 to S702 can also be selected to be executed by the AI ​​processor to further improve the efficiency of image processing.

[0112] In one embodiment, the AI ​​processor 610 and the image signal processor 620 can be connected via an electronic circuit to transmit an interrupt signal. The first image processing is performed after the AI ​​processor 610 receives the first interrupt signal sent by the image signal processor 620. The second image processing is performed after the image signal processor 620 receives the second interrupt signal sent by the AI ​​processor 610. The electronic circuit connection between the AI ​​processor 610 and the image signal processor 620 is also called a physical connection or an interrupt connection. The AI ​​processor 610 and the image signal processor 620 use this connection to send and receive interrupt signals, eliminating the need for the interrupt signal to be forwarded by other processors, such as the CPU, and eliminating the need for the CPU to participate in related control. This can increase the speed of interrupt signal transmission and, in certain real-time video playback, reduce image output latency, thereby improving the user experience. Specifically, the interrupt connection includes an interrupt signal processing hardware circuit for implementing the interrupt signal sending and receiving functions and a connecting line for transmitting signals to enable the transmission and reception of interrupt signals. The interrupt signal processing hardware circuit includes, but is not limited to, a traditional interrupt controller circuit. For the specific implementation of the interrupt signal processing hardware circuit, reference can be made to the relevant description of the interrupt controller in the prior art and will not be repeated here.

[0113] For example, as shown in FIG9 , the output ports Vpo1 and Vpo2 of the image processing module n in the image signal processor 620, the input ports Vpi1 and Vpi2 of the fusion module, and the input ports Vai1 and Vai2, and output ports Vao1 and Vao2 of the AI ​​processor 610 can all be connected to the memory using electronic circuits. The AI ​​processor 610 may include a task scheduler 611 and multiple computing units 612. Computing unit 1 among the multiple computing units 612 can function as a region of interest extraction unit to perform region of interest extraction processing. Computing unit 2 among the multiple computing units 612 can function as an image enhancement unit to perform image enhancement processing. Each component of the task scheduler 611 and computing units 612 may include multiple logic devices or circuits. The task scheduler 611 may implement the aforementioned electronic circuit connection L1 via the input port Vai2 of the AI ​​processor 610 and the output port Vpo2 of the image processing module n, and implement the aforementioned electronic circuit connection L2 via the output port Vao2 of the AI ​​processor and the input port Vpi2 of the fusion module. The image signal processor 620 can send an interrupt signal Z1 (i.e., the first interrupt signal described above) to the AI ​​processor 610 via the electronic circuit connection L1. The AI ​​processor 610 can send an interrupt signal Z2 (i.e., the second interrupt signal described above) to the image signal processor 620 via the electronic circuit connection L2. For example, when the image signal processor 620 has completed storing the image signal in the form of image blocks in the memory 630 (e.g., when the number of rows of stored image signals reaches a preset threshold, the size of the written image signal reaches a preset threshold, or the last address of the storage address allocated to the image signal processor 620 stores the image signal), it stops transmitting the image signal to the memory 630 and sends an interrupt signal Z1 to the AI ​​processor 610 via the electronic circuit connection L1 shown in FIG. 9. The AI ​​processor 610 can store the image signal in the memory 630 using the same storage method as the image signal processor 620, and after storing the image signal in the memory 630, it sends an interrupt signal Z2 to the image signal processor 620 via the electronic circuit connection L2 shown in FIG. 9. In addition, the computing unit 1 can be electronically connected to the above-mentioned memory via the input port Vai1 to read the image signal from the memory. The computing unit 2 can be electronically connected to the aforementioned memory via output port Vao1 to write image signals to the memory. The image processing module n can be connected to the memory via output port Vpo1 to write image signals to the memory. The fusion module's input port Vpi1 is connected to the memory to read image signals from the memory.

[0114] Of course, the above scheme is only an example given in the embodiment of the present application. The input port Vai2 of the AI ​​processor 610 can be selectively connected to the output port of any processing module in the noise elimination module, black level correction module, shadow correction module, white balance correction module, demosaicing module, color difference correction module, gamma correction module or color space conversion module to realize the above-mentioned electronic circuit connection L1. The memory can also be connected to the output port of the image processing module corresponding to L1 to store the image signal. The embodiment of the present application does not impose specific restrictions on this.

[0115] In some embodiments, based on the image processing device 300 in FIG. 5 , the image processing device 300 provided in the embodiments of the present application further includes an off-chip memory 340, as shown in FIG. 10 . The off-chip memory 340 can store multiple image frames, which may include the previous frame, two frames, or multiple frames preceding the current image. Because the off-chip memory has a larger storage space, it can replace the memory 330 to store larger units of image data. The off-chip memory 340 can store the image processing results ultimately generated by the embodiments of the present application. Furthermore, the image signal stored in the off-chip memory 340 can be an image signal processed by the AI ​​processor 310, or an image signal provided to the AI ​​processor 310 for processing. While processing the current image signal, the AI ​​processor 310 can also obtain image information from the off-chip memory 340 for the previous frame or frames of the current image signal, and then process the current image signal based on the image information of the previous frame or frames of the current image signal. Furthermore, the AI ​​processor 310 can also store the processed image signal in the off-chip memory 340. The off-chip memory 340 may include random access memory (RAM), which may include volatile memory (such as SRAM, DRAM, DDR (Double Data Rate SDRAM), or SDRAM) and non-volatile memory. In addition, the off-chip memory 340 may store an executable program for the image processing model running in the AI ​​processor. The AI ​​processor 310 runs the image processing model by loading the executable program.

[0116] For example, memory 330 is an on-chip memory that can be used to store a single frame image or image blocks within a single frame image. Off-chip memory 340 is used to store multiple frames of images. Therefore, for the image processing method shown in Figure 4, the image signal transmitted between the image signal processor 320 and the AI ​​processor 310 is divided into unit sizes, and the larger unit image information is transmitted through the off-chip memory 340, which makes up for the lack of on-chip RAM space and effectively utilizes the faster transmission speed of the on-chip RAM to achieve performance optimization.

[0117] In other possible implementations, the controller may include multiple independent controllers, each of which may be a digital logic device (e.g., including but not limited to a GPU or DSP). These independent controllers include an ISP controller 324 for controlling the operation of various components in the image signal processor and an AI controller 313 for controlling the operation of various components in the AI ​​processor (e.g., components corresponding to the computing unit and task scheduler). In this case, the AI ​​controller 313 may be integrated within the AI ​​processor. In this implementation, the ISP controller 324 and the AI ​​controller 313 may transmit information using inter-core communication. For example, after determining the image processing model to be used based on the image data, the ISP controller 324 may send various configuration information to the AI ​​controller 313 before image processing begins. This configuration information may include, but is not limited to, the address of the executable program of the image processing model in memory or in off-chip memory, or the priority level of each of the multiple image processing models. Furthermore, information transmission between the AI ​​processor 310 and the image signal processor 320 may also be achieved through communication between the ISP controller 324 and the AI ​​controller 313. For example, when no electronic circuit connection is provided between the AI ​​processor 310 and the image signal processor 320 for transmitting an interrupt signal, the image signal processor 320 may store the image signal in the memory 330 and then notify the ISP controller 324. The ISP controller 324 sends information indicating that the image signal is stored in the memory 330 to the AI ​​controller. The AI ​​controller 313 controls the computing unit in the AI ​​processor 310 to read the image signal from the memory 330 for image processing. The computing unit in the AI ​​processor stores the image signal in the memory 330 and then notifies the AI ​​controller 313. The AI ​​controller 313 sends information indicating that the image signal is stored in the memory 330 to the ISP controller 324. The ISP controller 324 controls the image signal processor 320 to read the image signal from the memory 330 for processing.

[0118] In one embodiment, the image processing device 300 may further include a communication unit (not shown in the figure), which includes but is not limited to a short-range communication unit or a cellular communication unit. The short-range communication unit exchanges information with a terminal for accessing the Internet located outside the mobile terminal by running a short-range wireless communication protocol. The short-range wireless communication protocol may include but is not limited to: various protocols supported by radio frequency identification technology, Bluetooth communication technology protocols, or infrared communication protocols. The cellular communication unit accesses the Internet through a wireless access network by running a cellular wireless communication protocol, so as to enable the mobile communication unit to exchange information with servers that support various applications on the Internet. The communication unit can be integrated into the same SOC as the AI ​​processor and image signal processor described in the above embodiments, or can be separately provided. In addition, the image processing device may also optionally include a bus, an input / output port I / O, or a storage controller. The storage controller is used to control the memory 330 and the off-chip memory. The bus, input / output port I / O, and storage controller can all be integrated into the same SOC as the above-mentioned image signal processor and AI processor. It should be understood that in actual applications, the image processing device may include more or fewer components, and this embodiment of the present application does not limit this. Of course, the image processing device 600 in Figure 6 can also be configured with an off-chip memory, ISP controller, AI controller, and communication unit in the same manner as described above, and this embodiment of the present application will not be described in detail here.

[0119] Based on the image processing device and image processing method provided by the embodiments of the present application, in a specific application scenario (including but not limited to daylight, dim light, night scenes, portraits, sports, high-speed video recording and other scenarios), when user A and user B conduct a video call, if user A starts the image processing service corresponding to the above-mentioned image processing method, the image presented on the screen of the electronic device used by user A, and the image of user A presented on the screen of the electronic device used by user B, may be images processed by the graphics processing device improved by the embodiments of the present application, and the processed images will continue to be presented until user A and user B terminate the video call or user A turns off the image processing service.

[0120] In one embodiment, this embodiment also provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed on a computer, the computer executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0121] In one implementation, this embodiment further provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0122] Among them, the computer-readable storage medium or computer program product provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0123] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a terminal device. Of course, the processor and storage medium can also exist as discrete components in a network device or a terminal device.

[0124] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the process or function of the embodiment of the present application is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid-state drive (SSD).

[0125] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing device, characterized in that: include: an artificial intelligence processor, configured to acquire a second image signal, and perform the first image processing on the second image signal to obtain a third image signal; wherein the second image signal is an image signal of an area of ​​interest in the first image signal obtained according to image data of the image sensor; An image signal processor, configured to perform a second image processing on the first image signal to obtain a fourth image signal; wherein the second image processing includes a fusion processing of the first image signal and the third image signal; A memory is coupled to the image signal processor and the artificial intelligence processor, and is used to transfer the third image signal between the image signal processor and the artificial intelligence processor.

2. The image processing device according to claim 1, characterized in that The first image processing includes at least one of demosaicing or noise reduction.

3. The image processing device according to claim 1 or 2, characterized in that: The second image processing further includes at least one of noise elimination, black level correction, shadow correction, white balance correction, demosaicing, gamma correction, chromatic aberration correction or color space conversion.

4. The image processing device according to any one of claims 1 to 3, characterized in that: The memory is further used to transfer the first image signal between the image signal processor and the artificial intelligence processor; When performing the operation of acquiring the second image signal, the artificial intelligence processor is specifically used to: Performing region of interest extraction on the first image signal to obtain the second image signal.

5. The image processing device according to any one of claims 1 to 3, characterized in that: The image signal processor is further used to extract a region of interest from the first image signal to obtain the second image signal; The memory is further used to transfer the second image signal between the image signal processor and the artificial intelligence processor; The artificial intelligence processor is specifically used to obtain the second image signal from the memory.

6. The image processing device according to claim 4 or 5, characterized in that: The image signal processor is further used to perform a third image processing on the image data to obtain the first image signal, and the third image processing also includes at least one of noise elimination, black level correction, shadow correction, white balance correction, demosaicing, chromatic aberration correction or gamma correction.

7. The image processing device according to any one of claims 1 to 6, characterized in that: The artificial intelligence processor performs the first image processing after receiving the first interrupt signal sent by the image signal processor; The image signal processor performs the second image processing after receiving the second interrupt signal sent by the artificial intelligence processor.

8. The image processing device according to any one of claims 1 to 7, characterized in that: The image processing device is a chip, and the artificial intelligence processor, the image signal processor and the memory are integrated in the chip.

9. The image processing device according to any one of claims 1 to 8, characterized in that: The third image signal is stored in the memory in the form of an image block and transferred between the image signal processor and the artificial intelligence processor.

10. An image processing method, characterized in that: The method comprises: The artificial intelligence processor acquires a second image signal, and performs the first image processing on the second image signal to obtain a third image signal; wherein the second image signal is an image signal of an area of ​​interest in the first image signal obtained according to image data of the image sensor; The memory transfers the third image signal between the image signal processor and the artificial intelligence processor; The image signal processor performs second image processing on the first image signal through the third image signal to obtain a fourth image signal; wherein the second image processing includes fusion processing of the first image signal and the third image signal.

11. The method according to claim 10, characterized in that The first image processing includes at least one of demosaicing or noise reduction.

12. The method according to claim 10 or 11, characterized in that: The second image processing further includes at least one of noise elimination, black level correction, shadow correction, white balance correction, demosaicing, gamma correction, chromatic aberration correction or color space conversion.

13. The method according to any one of claims 10 to 12, characterized in that: The method further includes: the memory transferring the first image signal between the image signal processor and the artificial intelligence processor; The step of acquiring the second image signal specifically includes: the artificial intelligence processor extracts the region of interest on the first image signal to obtain the second image signal.

14. The method according to any one of claims 10 to 12, characterized in that: The method further comprises: The image signal processor extracts a region of interest from the first image signal to obtain the second image signal; The memory transfers the second image signal between the image signal processor and the artificial intelligence processor; The step of acquiring the second image signal specifically includes: the artificial intelligence processor acquiring the second image signal from the memory.

15. The method according to claim 13 or 14, characterized in that The method further comprises: The image signal processor performs third image processing on the image data to obtain the first image signal, and the third image processing also includes at least one of noise elimination, black level correction, shadow correction, white balance correction, demosaicing, color difference correction or gamma correction.

16. The method according to any one of claims 10 to 15, characterized in that: The artificial intelligence processor performs the first image processing after receiving the first interrupt signal sent by the image signal processor: The image signal processor performs the second image processing after receiving the second interrupt signal sent by the artificial intelligence processor.

17. The method according to any one of claims 10 to 16, characterized in that: The third image signal is stored in the memory in the form of an image block and transferred between the image signal processor and the artificial intelligence processor.

18. The method according to any one of claims 10 to 17, characterized in that: The method is applied to an image processing device; the image processing device is a chip, and the artificial intelligence processor, the image signal processor and the memory are integrated in the chip.

19. A computer-readable storage medium, characterized in that: Computer program instructions are stored; when the computer program instructions are executed by the artificial intelligence processor and the image signal processor, the method described in any one of claims 10-18 is implemented.

20. An electronic device, characterized in that: The invention comprises an image sensor and an image processing device as claimed in any one of claims 1 to 9 connected to the image sensor.

Images