Image processing architecture, image processing method, device and storage medium

Abstract

The embodiment of the present application discloses an image processing architecture, an image processing method, a device and a storage medium, which are applied to an application processing chip, and the image processing architecture includes a multimedia service module, a first hardware abstraction layer module and a first driver module; wherein the multimedia service module is used to obtain image parameters and send the image parameters to the first hardware abstraction layer module; the first hardware abstraction layer module is used to send the image parameters to the first driver module; the first driver module is used to receive the image parameters and send the image parameters to the image processing chip, so that the image processing chip and the application processing chip cooperate to perform image processing. In this way, the function of the image processing chip can be fully utilized, and it has sufficient scalability and flexibility, so that the image processing capability can be better realized and the image quality can be improved. At the same time, the collaborative work and timing problems between the image processing chip and the application processing chip can be better solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing architecture, an image processing method, a device and a storage medium. Background Art

[0002] With the continuous development of computer technology, the use of electronic devices (such as smart phones, PDAs, cameras, etc.) is becoming more and more popular. The vast majority of users use images captured by electronic devices to record their study, work, and life, and users have higher and higher requirements for the quality of captured images.

[0003] In recent years, in order to improve the quality of captured images, although some image processing solutions have been developed, the existing image processing solutions cannot meet certain requirements, resulting in limited image processing capabilities and poor image quality. Summary of the invention

[0004] The present application proposes an image processing architecture, an image processing method, a device and a storage medium, which can not only give full play to the functions of the image processing chip, but also have sufficient scalability and flexibility, so as to better realize the image processing capability and improve the image quality.

[0005] To achieve the above purpose, the technical solution of this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an image processing architecture, which is applied to an application processing chip, and the image processing architecture includes a multimedia service module, a first hardware abstraction layer module and a first driver module; wherein,

[0007] The multimedia service module is used to obtain image parameters and send the image parameters to the first hardware abstraction layer module;

[0008] A first hardware abstraction layer module, used to send image parameters to a first driver module;

[0009] The first driving module is used to receive image parameters and send the image parameters to the image processing chip, so that the image processing chip cooperates with the application processing chip to perform image processing.

[0010] In a second aspect, an embodiment of the present application provides an image processing method, which is applied to an image processing architecture including a multimedia service module, a first hardware abstraction layer module, and a first driver module, and the image processing architecture is applied to an application processing chip, and the method includes:

[0011] Acquire image parameters through the multimedia service module, and send the image parameters to the first hardware abstraction layer module;

[0012] Sending the image parameters to the first driver module through the first hardware abstraction layer module;

[0013] The image parameters are received through the first driving module, and the image parameters are sent to the image processing chip, so that the image processing chip cooperates with the application processing chip to perform image processing.

[0014] In a third aspect, an embodiment of the present application provides a terminal device, the terminal device comprising a memory and a processor; wherein:

[0015] A memory for storing computer programs that can be run on the processor;

[0016] A processor is used to execute the method described in the second aspect when running a computer program.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, it implements the method described in the second aspect.

[0018] An image processing architecture, image processing method, device and storage medium provided in the embodiment of the present application are applied to an application processing chip, and the image processing architecture includes a multimedia service module, a first hardware abstraction layer module and a first driver module; wherein the multimedia service module is used to obtain image parameters and send the image parameters to the first hardware abstraction layer module; the first hardware abstraction layer module is used to send the image parameters to the first driver module; the first driver module is used to receive the image parameters and send the image parameters to the image processing chip, so that the image processing chip and the application processing chip cooperate to perform image processing. In this way, the function of the image processing chip can be fully utilized, and it has sufficient scalability and flexibility, so that the image processing capability can be better realized and the image quality can be improved. At the same time, the collaborative work and timing problems between the image processing chip and the application processing chip can be better solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of the composition structure of an image processing architecture provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the composition structure of another image processing architecture provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of an overall block diagram of an image processing architecture provided in an embodiment of the present application;

[0022] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0023] Figure 5 A detailed flowchart of an image processing method provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0027] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] With the continuous development of computer technology, image processing technology has become increasingly diversified. At present, different types of system-on-chip (SoC) all contain their own image signal processor (ISP) for processing image data. The ISP in the application processing chip is mainly used for some traditional algorithm implementations, such as bad pixel correction, time domain noise reduction, three-dimensional (3D) noise reduction, automatic white balance, auto focus, automatic exposure and other algorithms. However, although these application processing chips have made great improvements in image processing in recent years, their application in image processing is still not sufficient, resulting in poor image processing quality.

[0029] In order to further improve the image quality, image processing chips can be introduced. Then, by utilizing the customized characteristics of the chip, a lot of personalized image data processing can be performed to improve the image processing quality. The solidified algorithm can also increase the speed of image data processing.

[0030] Accordingly, in order to adapt to the dual-chip processing process of the application processing chip and the image processing chip, an embodiment of the present application provides an image processing architecture that cooperates therewith. The image processing architecture is applied to the application processing chip, and the image processing architecture includes a multimedia service module, a first hardware abstraction layer module and a first driver module; wherein the multimedia service module is used to obtain image parameters and send the image parameters to the first hardware abstraction layer module; the first hardware abstraction layer module is used to send the image parameters to the first driver module; the first driver module is used to receive the image parameters and send the image parameters to the image processing chip, so that the image processing chip and the application processing chip cooperate to perform image processing. In this way, the function of the image processing chip can be fully utilized, and it has sufficient scalability and flexibility, so that the image processing capability can be better realized and the image quality can be improved. At the same time, the collaborative work and timing problems between the image processing chip and the application processing chip can be better solved.

[0031] The technical solutions of various embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0032] In one embodiment of the present application, see Figure 1 , which shows a schematic diagram of the composition structure of an image processing architecture provided by an embodiment of the present application. Figure 1 As shown, the image processing architecture 10 may include a multimedia service (Multimedia Service, MMS) module 101, a first hardware abstraction layer module 102 and a first driver module 103; wherein,

[0033] The multimedia service module 101 is used to obtain image parameters and send the image parameters to the first hardware abstraction layer module 102;

[0034] The first hardware abstraction layer module 102 is used to send image parameters to the first driver module 103;

[0035] The first driving module 103 is used to receive image parameters and send the image parameters to the image processing chip 200 so that the image processing chip 200 cooperates with the application processing chip 100 to perform image processing.

[0036] It should be noted that the image processing architecture 10 of the embodiment of the present application is applied to the application processing chip 100, which can enable the image processing chip 200 to cooperate with the application processing chip 100, thereby better realizing the image processing capability.

[0037] It should also be noted that, in the embodiments of the present application, the application processing chip may refer to an application processor (AP), which may also be called a "main chip", which generally has the characteristics of processing multiple tasks and strong processing capabilities; the image processing chip may refer to a chip for image processing added before the application processing chip, which is generally used to process some personalized needs, such as customized effect needs, always on (AON) needs, and so on.

[0038] In some embodiments, Figure 1 Based on the image processing architecture 10 shown, see Figure 2 , the image processing architecture 10 may further include a second hardware abstraction layer module 104; wherein,

[0039] The second hardware abstraction layer module 104 is used to obtain configuration parameters of the application processing chip 100 itself and send the configuration parameters to the multimedia service module 101 .

[0040] It should be noted that, in the embodiment of the present application, the configuration parameters may include scene parameters, gyroscope parameters, frame rate, resolution, debugging binary (Bin) files, etc. In this way, after receiving the configuration parameters of the application processing chip itself, the multimedia service module 101 may also send them to the image processing chip through the multimedia service module 101, the first hardware abstraction layer module 102 and the first driver module 103 in sequence, so that the image processing chip 200 can perform image processing according to these configuration parameters.

[0041] In addition, in an embodiment of the present application, the image parameters may include at least one of the following: configuration parameters sent by the second hardware abstraction layer module, first image parameters customized within the multimedia service module, and second image parameters that need to be adjusted during image processing.

[0042] In a specific embodiment, taking the power-on initialization of the image processing chip as an example, the image parameters at this time are startup parameters, wherein the startup parameters may include the configuration parameters sent by the second hardware abstraction layer module and the first image parameters customized inside the multimedia service module. Specifically,

[0043] The multimedia service module 101 is also used to obtain startup parameters when opening the camera application;

[0044] The first hardware abstraction layer module 102 is further used to receive the startup parameters sent by the multimedia service module 101, and send the startup parameters to the first driver module 103;

[0045] The first driving module 103 is further used to receive startup parameters and send the startup parameters to the image processing chip, so that the image processing chip 200 completes power-on initialization.

[0046] In another specific embodiment, taking parameter adjustment during image processing as an example, the image parameter at this time may be a second image parameter that needs to be adjusted during image processing. Specifically,

[0047] The multimedia service module 101 is further used to obtain the second image parameter;

[0048] The first hardware abstraction layer module 102 is further used to receive the second image parameter sent by the multimedia service module 101, and send the second image parameter to the first driver module 103;

[0049] The first driving module 103 is further configured to receive the second image parameter and send the second image parameter to the image processing chip, so that the image processing chip 200 performs image processing according to the second image parameter.

[0050] It should be noted that the second image parameter obtained by the multimedia service module 101 can be either a custom image parameter within the multimedia service module or a configuration parameter sent from the second hardware abstraction layer module, and the present application embodiment does not impose any limitation. Here, the second image parameter specifically refers to a parameter that needs to be adjusted during the image processing process, for example, the second image parameter can be an exposure parameter, resolution, frame rate, etc.

[0051] Furthermore, in some embodiments, there may be some abnormal monitoring parameters between the application processing chip 100 and the image processing chip 200, such as whether the working state is abnormal, whether the temperature is too high, etc. For the abnormal monitoring parameters, the specific ones are as follows:

[0052] The first driver module 103 is further used to obtain abnormal monitoring parameters of the image processing chip and send the abnormal monitoring parameters to the first hardware abstraction layer module 102;

[0053] The first hardware abstraction layer module 102 is further used to send the abnormal monitoring parameters to the multimedia service module;

[0054] The multimedia service module 101 is further used to receive abnormal monitoring parameters and send the abnormal monitoring parameters to the second hardware abstraction layer module 104, so that the application processing chip 100 performs corresponding processing according to the abnormal monitoring parameters.

[0055] It should be noted that, for the image parameters acquired by the multimedia service module 101, such as the configuration parameters sent by the second hardware abstraction layer module 104, the first image parameters customized inside the multimedia service module 101, and the second image parameters that need to be adjusted during the image processing, etc., the sending order is to pass through the multimedia service module 101, the first hardware abstraction layer module 102 and the first driver module 103 in turn, and then send it to the image processing chip. For some abnormal monitoring parameters, the sending order is the opposite, that is, to pass through the first driver module 103, the first hardware abstraction layer module 102 and the multimedia service module 101 in turn, and then be sent by the multimedia service module 101 to the second hardware abstraction layer module 104, so that the application processing chip can perform corresponding processing according to the abnormal monitoring parameters.

[0056] Furthermore, in some embodiments, Figure 1 Based on the image processing architecture 10 shown, see Figure 2 , the image processing architecture 10 may also include a high-speed processing interface; wherein,

[0057] The image processing chip 200 is used to receive the first image data through the high-speed processing interface, and perform image processing on the first image data according to the configuration parameters sent by the second hardware abstraction layer module 104 to obtain the second image data;

[0058] The application processing chip 100 is used to receive the second image data and the custom image parameters sent by the image processing chip 200 through the high-speed processing interface, and perform image processing on the second image data according to the custom image parameters to obtain target image data; wherein the custom image parameters at least include 3A statistical information.

[0059] It should be noted that, in the embodiment of the present application, 3A statistical information specifically refers to statistical information of 3A parameters; wherein, 3A parameters may include parameters such as auto focus (Auto Focus, AF), auto exposure (Auto Exposure, AE) and auto white balance (Auto White Balance, AWB).

[0060] It should also be noted that in the embodiment of the present application, the high-speed processing interface is used to realize the interactive communication between the image processing chip and the application processing chip, which can reduce the coupling between the image processing chip and the application processing chip. Specifically, the high-speed processing interface can be a mobile industry processor interface (Mobile Industry Processor Interface, MIPI) interface (such as Figure 2In addition, for the MIPI interface, it may include a transmitting interface (MIPI TX) and a receiving interface (MIPI RX), wherein MIPI TX may be used to transmit data, and MIPI RX may be used to receive data.

[0061] Furthermore, the custom image parameters here may include metadata in addition to 3A statistical information, etc. As for the 3A statistical information, since the second hardware abstraction layer module 104 cannot identify it, the embodiment of the present application also needs to send it to the first hardware abstraction layer module 102 for parsing through the multimedia service module 101.

[0062] In some embodiments, the multimedia service module 101 is further configured to receive 3A statistical information sent by the application processing chip through the second hardware abstraction layer module 104, and send the 3A statistical information to the first hardware abstraction layer module 102;

[0063] The first hardware abstraction layer module 102 is further used to receive and parse the 3A statistical information, and send the parsed 3A statistical information to the multimedia service module 101;

[0064] The multimedia service module 101 is further configured to forward the parsed 3A statistical information to the second hardware abstraction layer module 104, so that the application processing chip performs image processing on the second image data according to the parsed 3A statistical information.

[0065] That is to say, on the image processing chip side, the 3A statistical information can be directly sent to the second hardware abstraction layer module 104 on the application processing chip side through the MIPI interface. Then, the second hardware abstraction layer module 104 can send the 3A statistical information to the multimedia service module 101, and then the multimedia service module 101 sends it to the first hardware abstraction layer module 102, and the 3A statistical information is parsed by the first hardware abstraction layer module 102, and then the parsed 3A statistical information is returned to the multimedia service module 101, and the multimedia service module 101 forwards the parsed 3A statistical information to the second hardware abstraction layer module 104, so that the application processing chip performs image processing on the second image data according to the parsed 3A statistical information.

[0066] In addition, for the three modules, namely, the multimedia service module 101, the first hardware abstraction layer module 102 and the first driver module 103, their specific functions are described as follows:

[0067] In some embodiments, for the multimedia service module 101, the multimedia service module 101 is specifically used to provide an application program interface of a user space (UserSpace) for the image processing chip, and data can be transmitted between the application processing chip and the image processing chip through the application program interface.

[0068] Here, the application program interface may at least include the calling interface of the application layer, the framework layer and the hardware abstraction layer (HAL). In other words, the multimedia service module 101 may be designed to enable the image processing chip to open a control path for the UserSpace, and has interfaces such as the Application layer, the Framework layer and the HAL layer for users to call.

[0069] In some embodiments, for the first hardware abstraction layer module 102, the first hardware abstraction layer module 102 is specifically configured to implement a hardware abstraction layer for parameter configuration and control of an image processing chip.

[0070] Here, the first hardware abstraction layer module may at least include: a strategic control module, a scene control module, a debugging control module, a power control module, a message control module, a parameter module, an engineering mode, a configuration module, and a data analysis module. In other words, the first hardware abstraction layer module 102 is mainly used to initialize the image processing chip, configure scene parameters during photo or video processing, control scene switching, and control power.

[0071] In some embodiments, for the first driver module 103, the first driver module 103 is specifically used to enable an inter-core communication function between the application processing chip end and the image processing chip.

[0072] Here, the first driver module 103 also has a customized inter-process communication (IPC) driver, which may include a secure digital input and output (SDIO) interface, a serial peripheral interface (SPI), etc. In addition, it can also realize the boot function of the image processing chip.

[0073] For example, see Figure 3 , which shows a schematic diagram of the overall block diagram of an image processing architecture provided by an embodiment of the present application. Figure 3For the image processing, the software architecture determined by the dedicated image chip design, operator customization and reasonable data path mainly considers the balance between the comprehensive performance of the chip and software, power consumption and effect, so as to realize image processing and achieve better shooting effects.

[0074] Specific as Figure 3 As shown, the overall block diagram can be mainly divided into a hardware part, a kernel part and a user part. Among them, the hardware part may include a camera, an image signal processor (ISP), an image processing chip, a central processing unit (CPU), and a sensor; the kernel part may include a first driver module and a second driver module; the user part may include a camera application (CameraAPP) / Always on (AON) module, a camera service module (Camera Service), a camera device (CameraDevice), a multimedia service module (Multimedia Service, MMS) and a first hardware abstraction layer module and a second hardware abstraction layer module. Here, the MMS module interacts and communicates with the Camera APP / AON module and the second hardware abstraction layer module respectively, and the MMS module also interacts and communicates with the first hardware abstraction layer module, and the first hardware abstraction layer module also interacts and communicates with the first driver module. In addition, it should be noted that the first hardware abstraction layer module and the first driver module are designed for the image processing chip, and the second hardware abstraction layer module and the second driver module are designed for the camera; therefore, the second hardware abstraction layer module can be represented by Camera HAL, and the second driver module can be represented by Camera Driver.

[0075] In the embodiment of the present application, for the image processing chip, it can be regarded as an independent module of the CPU similar to the application processor side. For the MMS module, it is a software framework designed to enable the image processing chip to open a control path for UserSpace, and has interfaces such as Application layer, Framework layer and HAL layer for users to call. For the first hardware abstraction layer module, it is a hardware abstraction layer for realizing the hardware configuration and control of the image processing chip, which is mainly used for initializing the image processing chip, scene parameter configuration during photo or video processing, scene switching control and power supply (Power) control and other functions. For the first driver module, it is a customized IPC driver that can realize the inter-core communication function of the application processing chip and the image processing chip, which can include interfaces such as SDIO and SPI, and can also realize the boot function of the image processing chip.

[0076] In this way, based on Figure 3 As shown in the overall framework, in the embodiment of the present application, the hardware of the image processing chip may include a controller CPU, a high-speed MIPI interface, an ISP and a neural-network processing unit (NPU), etc. After receiving the first image data obtained by the complementary metal oxide semiconductor (CMOS) sensor, the image processing chip first performs pre-processing through the ISP inside the image processing chip, and then provides it to the NPU for customized effect processing; then the image processing chip transmits the processed second image data to the application processing chip through the MIPI interface.

[0077] In the above-mentioned image processing process, the embodiment of the present application may also have temporarily generated intermediate state data, which will be stored in the dynamic random access memory (DRAM). It should be noted that the intermediate state data can be used in the subsequent image processing process. Exemplarily, the intermediate state data can be data temporarily generated during the process of noise reduction by the NPU.

[0078] It should also be noted that the AP mainly runs Android or other platform software. After receiving the first image data transmitted by the image processing chip through the MIPI interface, the AP sends it to the ISP of the application processing chip after receiving it through the MIPI RX interface, and then performs corresponding processing in combination with the current scene parameters. After the ISP of the application processing chip processes the image, it can directly send it to the display, encoding and other modules to achieve camera preview or recording.

[0079] In addition, in the embodiments of the present application, the image processing architecture proposed here includes many modules such as an MMS module, a first hardware abstraction layer module, a first driver module, etc., among which the MMS module can provide a full set of API interfaces for UserSpace, that is, the Application layer, Framework layer and HAL layer can all access them, so that the image processing capabilities of the image processing chip can be better applied to third-party imaging capabilities through the MMS module and the first hardware abstraction layer module, thereby improving the imaging effect of the third-party software.

[0080] This embodiment provides an image processing architecture, which is applied to an application processing chip, and the image processing architecture includes a multimedia service module, a first hardware abstraction layer module and a first driver module; wherein the multimedia service module is used to obtain image parameters and send the image parameters to the first hardware abstraction layer module; the first hardware abstraction layer module is used to send the image parameters to the first driver module; the first driver module is used to receive the image parameters and send the image parameters to the image processing chip, so that the image processing chip and the application processing chip cooperate to perform image processing. In this way, the function of the image processing chip can be fully utilized, and it has sufficient scalability and flexibility, so that the image processing capability can be better realized and the image quality can be improved. At the same time, the collaborative work and timing problems between the image processing chip and the application processing chip can be better solved.

[0081] In another embodiment of the present application, see Figure 4 , which shows a schematic flow chart of an image processing method provided by an embodiment of the present application. Figure 4 As shown, the method may include:

[0082] S401: Acquire image parameters through the multimedia service module, and send the image parameters to the first hardware abstraction layer module.

[0083] S402: Sending image parameters to the first driver module through the first hardware abstraction layer module.

[0084] S403: receiving image parameters through the first driving module, and sending the image parameters to the image processing chip, so that the image processing chip cooperates with the application processing chip to perform image processing.

[0085] It should be noted that the image processing method described in the embodiment of the present application is applied to an image processing architecture including a multimedia service module, a first hardware abstraction layer module and a first driver module, and the image processing architecture is applied to an application processing chip; in this way, the image processing chip and the application processing chip can cooperate with each other to better realize the image processing capability.

[0086] It should also be noted that, for the image processing architecture, it may also include a second hardware abstraction layer module. Specifically, in some embodiments, the method may also include:

[0087] The configuration parameters of the application processing chip itself are obtained through the second hardware abstraction layer module, and the configuration parameters are sent to the multimedia service module.

[0088] It should be noted that, in the embodiment of the present application, the configuration parameters may include scene parameters, gyroscope parameters, frame rate, resolution, debugging binary (Bin) file, etc.

[0089] In this way, after sending the configuration parameters to the multimedia service module through the second hardware abstraction layer module, the multimedia service module can receive the configuration parameters of the application processing chip itself, and then send them to the image processing chip through the multimedia service module, the first hardware abstraction layer module and the first driver module in sequence, so that the image processing chip can perform image processing according to these configuration parameters.

[0090] It should also be noted that in the embodiment of the present application, the image parameters here may include at least one of the following: configuration parameters sent by the second hardware abstraction layer module, first image parameters customized within the multimedia service module, and second image parameters that need to be adjusted during image processing.

[0091] In a specific embodiment, taking the power-on initialization of the image processing chip as an example, the image parameters at this time are startup parameters, wherein the startup parameters may include the configuration parameters sent by the second hardware abstraction layer module and the first image parameters customized inside the multimedia service module. Thus, in some embodiments, the method may also include:

[0092] When opening the camera application, obtaining startup parameters through the multimedia service module, wherein the startup parameters include configuration parameters and first image parameters;

[0093] The startup parameters are sent to the first hardware abstraction layer module through the multimedia service module, and the first hardware abstraction layer module sends the startup parameters to the first driver module;

[0094] The startup parameters are received through the first driving module, and the startup parameters are sent to the image processing chip, so that the image processing chip completes power-on initialization.

[0095] In another specific embodiment, taking parameter adjustment during image processing as an example, the image parameter at this time may be a second image parameter that needs to be adjusted during image processing. Thus, in some embodiments, the method may further include:

[0096] Acquiring a second image parameter through a multimedia service module;

[0097] The second image parameter is sent to the first hardware abstraction layer module through the multimedia service module, and the first hardware abstraction layer module sends the second image parameter to the first driver module;

[0098] The second image parameter is received through the first driving module, and the second image parameter is sent to the image processing chip, so that the image processing chip performs image processing according to the second image parameter.

[0099] It should be noted that in the embodiment of the present application, the second image parameter can be either a custom image parameter within the multimedia service module or a configuration parameter sent from the second hardware abstraction layer module, and no limitation is made here. In addition, here, the second image parameter specifically refers to a parameter that needs to be adjusted during the image processing process. For example, the second image parameter can be an exposure parameter, resolution, frame rate, etc.

[0100] Furthermore, there may be some abnormal monitoring parameters between the application processing chip and the image processing chip, such as whether the working state is abnormal, whether the temperature is too high, etc. Thus, in some embodiments, the method may further include:

[0101] Acquire the abnormal monitoring parameters of the image processing chip through the first driver module, and send the abnormal monitoring parameters to the first hardware abstraction layer module;

[0102] Sending the abnormal monitoring parameters to the multimedia service module through the first hardware abstraction layer module;

[0103] The abnormal monitoring parameters are received through the multimedia service module, and the abnormal monitoring parameters are sent to the second hardware abstraction layer module, so that the application processing chip performs image processing according to the abnormal monitoring parameters.

[0104] It should be noted that, for the image parameters acquired by the multimedia service module, such as the configuration parameters sent by the second hardware abstraction layer module, the first image parameters customized inside the multimedia service module, and the second image parameters that need to be adjusted during the image processing, etc., the sending order is to pass through the multimedia service module, the first hardware abstraction layer module and the first driver module in turn, and then send them to the image processing chip. For some abnormal monitoring parameters, the sending order is the opposite, that is, to pass through the first driver module, the first hardware abstraction layer module and the multimedia service module in turn, and then be sent by the multimedia service module to the second hardware abstraction layer module, so that the application processing chip can perform corresponding processing according to the abnormal monitoring parameters.

[0105] Furthermore, the image processing architecture may also include a high-speed processing interface, that is, certain parameters may be directly sent from the image processing chip to the application processing chip via the high-speed processing interface. In some embodiments, the method may also include:

[0106] The image processing chip receives the first image data through the high-speed processing interface, and performs image processing on the first image data according to the configuration parameters sent by the second hardware abstraction layer module to obtain second image data;

[0107] The image processing chip sends the second image data and the custom image parameters to the application processing chip through the high-speed processing interface;

[0108] The application processing chip performs image processing on the second image data according to the custom image parameters to obtain target image data; wherein the custom image parameters at least include 3A statistical information.

[0109] It should be noted that, in an embodiment of the present application, the high-speed processing interface can be used to realize interactive communication between the image processing chip and the application processing chip, thereby reducing the coupling degree between the image processing chip and the application processing chip. Specifically, the high-speed processing interface can be a MIPI interface. In addition, for the MIPI interface, it can include a transmitting interface (MIPI TX) and a receiving interface (MIPI RX), wherein MIPI TX is used to send data and MIPI RX is used to receive data.

[0110] It should also be noted that, in the embodiment of the present application, for the image processing chip, the hardware of the image processing chip may include an NPU. Therefore, in some embodiments, performing image processing on the first image data according to the configuration parameters sent by the second hardware abstraction layer module to obtain the second image data may include:

[0111] Perform parameter configuration according to the configuration parameters sent by the second hardware abstraction layer module to determine the preset effect;

[0112] The NPU performs image processing with a preset effect on the first image data to obtain second image data.

[0113] Here, the hardware of the image processing chip may also include an ISP. Thus, before effect processing is performed, the first image data may be pre-processed by the ISP to obtain processed first image data; and then the processed first image data may be processed with a preset effect by the NPU to obtain second image data.

[0114] That is to say, after receiving the first image data obtained by the CMOS sensor, the image processing chip is first pre-processed by the ISP inside the image processing chip, and then provided to the NPU for customized effect processing; then the image processing chip transmits the processed second image data to the application processing chip through the MIPI interface.

[0115] It should also be noted that, in the embodiment of the present application, the custom image parameters here may include metadata (Metadata) and the like in addition to 3A statistical information.

[0116] For 3A statistical information, since the second hardware abstraction layer module cannot identify it, the embodiment of the present application also needs to send it to the first hardware abstraction layer module for parsing through the multimedia service module. Specifically, in some embodiments, the method may also include:

[0117] The application processing chip sends the 3A statistical information to the multimedia service module through the second hardware abstraction layer module, and the multimedia service module sends the 3A statistical information to the first hardware abstraction layer module;

[0118] The 3A statistical information is parsed through the first hardware abstraction layer module, and the parsed 3A statistical information is sent to the multimedia service module,

[0119] The parsed 3A statistical information is forwarded to the second hardware abstraction layer module through the multimedia service module, so that the application processing chip performs image processing on the second image data according to the parsed 3A statistical information.

[0120] That is to say, on the image processing chip side, the 3A statistical information can be directly sent to the second hardware abstraction layer module on the application processing chip side through the MIPI interface. Then, the 3A statistical information can be sent to the multimedia service module through the second hardware abstraction layer module, and the multimedia service module forwards it to the first hardware abstraction layer module; the 3A statistical information is then parsed by the first hardware abstraction layer module, and then the parsed 3A statistical information is returned to the multimedia service module, and finally the multimedia service module forwards the parsed 3A statistical information to the second hardware abstraction layer module, so that the application processing chip performs image processing on the second image data according to the parsed 3A statistical information.

[0121] It should also be noted that, in the embodiment of the present application, for the application processing chip, the hardware of the application processing chip may also include an ISP. Therefore, in some embodiments, the method may also include:

[0122] Determine the current scene information;

[0123] The second image data is processed according to the scene information to obtain target image data.

[0124] That is to say, after receiving the second image data through MIPI RX, the application processing chip can send it to the ISP of the application processing chip, and then process it accordingly in combination with the current scene information. After obtaining the target image data, it can also be directly sent to the display, encoding and other modules to achieve camera preview or recording.

[0125] For example, see Figure 5 , which shows a detailed flow chart of an image processing method provided by an embodiment of the present application. Figure 5 As shown, the detailed process may include:

[0126] S501: The user clicks the camera icon to open the camera application.

[0127] S502: The camera service starts the image processing chip through the multimedia service module, the first hardware abstraction layer module and the first driver module, and powers on the image processing chip through the power management module.

[0128] In the embodiments of the present application, Figure 5 A workflow based on an image processing chip is provided. First, a user clicks a camera icon of a terminal device to open a camera application to enter a camera mode. Then, a camera service starts the image processing chip through a multimedia service module, a first hardware abstraction layer module, and a first driver module, and powers on the image processing chip through a power management module, specifically, the image processing chip is powered on and started through a power management integrated circuit (Power Management IC, PMIC).

[0129] S503: After the operating system of the image processing chip is started, the configuration service program sets the working environment according to the configuration parameters set by the application processing chip.

[0130] In the embodiment of the present application, the configuration parameters may include resolution, frame rate, and gyroscope parameters, etc., and the operating system of the image processing chip may be an RT-thread system. In this way, after the operating system of the image processing chip is started, the configuration service program sets the working environment (i.e., scene information) according to the configuration parameters set by the application processing chip, so that image processing can be performed subsequently according to the set scene information.

[0131] S504: After initializing the image processing chip, the camera requests image data.

[0132] S505: The camera service configures the image parameters that need to be adjusted for each frame and provides them to the image processing chip.

[0133] S506: Determine whether to end the request data (stop recording).

[0134] In an embodiment of the present application, after the image processing chip is powered on and initialized through the multimedia service module, the first hardware abstraction layer module and the first driver module, the camera can request image data. At the same time, the camera service will also configure the image parameters that need to be adjusted for each frame and provide them to the image processing chip.

[0135] In addition, for S506, it is determined whether to end the request data (stop recording). If the determination result is no, then S507 to S510 can be executed; if the determination result is yes, then S511 to S515 can be executed.

[0136] S507: The image processing chip receives the configuration parameters of the application processing chip, updates the parameters according to the configuration parameters, and determines the customization effect.

[0137] S508: The image processing chip receives the first image data provided by the MIPI RX interface, processes the first image data with customized effects, and sends the first image data to the application processing chip through the MIPI TX interface.

[0138] S509: The ISP of the application processing chip receives the second image data and the custom image parameters processed by the image processing chip.

[0139] S510: The application processing chip determines the software and hardware algorithm processing that the application processing chip needs to perform according to the set scene information, and transmits the processed target image data back to the camera application.

[0140] In an embodiment of the present application, the customized image parameters may include 3A statistical information, metadata, etc.

[0141] In the embodiment of the present application, when there is no end request data (stop recording), the image processing chip receives the configuration parameters sent by the application processing chip, updates the parameters, and sets the corresponding effects; then the image processing chip sets the effects on the first image data received from the MIPI RX interface, and sends it to the application processing chip through the MIPI TX interface. The ISP in the application processing chip receives the second image data and custom parameters (such as 3A statistical information, metadata) processed by the image processing chip, and determines the software and hardware algorithm processing that the application processing chip needs to do in combination with the scene information set by the user, and sends the processed target image data back to the Application.

[0142] S511: Turn off the ISP, NPU, etc. of the image processing chip.

[0143] S512: The application processing chip turns off the camera.

[0144] S513: The power management module of the application processing chip controls power-off.

[0145] S514: The second hardware abstraction layer module of the application processing chip and the first hardware abstraction layer module of the image processing chip release resources.

[0146] S515: Exit camera mode.

[0147] In an embodiment of the present application, when the request data is ended (recording is stopped), it is necessary to shut down the ISP, NPU, etc. of the image processing chip, turn off the camera by the application processing chip, control the power-off by the power management module of the application processing chip, and release the resources by the second hardware abstraction layer module of the application processing chip and the first hardware abstraction layer module of the image processing chip, so that the terminal device exits the camera mode.

[0148] In summary, in the embodiment of the present application, on the one hand, the imaging solution is solved by chip design, algorithm definition and software integration ideas to realize multiple functional shooting in the RAW domain. The present solution is mainly based on the differentiated framework designed for the image processing chip, that is, it can give full play to the functions of the image processing chip and has sufficient scalability and flexibility; here, the differentiation is mainly to design the image processing chip to realize the interface of the Usespace of the application processing chip, so that the image processing chip only needs to be controlled through several interfaces in the software framework, thereby reducing the coupling degree with the application processing chip; on the other hand, in the process of taking photos with the camera, the preview can still remain uninterrupted and have algorithm processing effects, which can achieve a better zero-delay shooting solution; on the other hand, the hardware of the present solution is based on the design of an AI+ image processing chip. Since the image has more details in the RAW domain, such as noise and details, brightness, etc., the image processing capability can be better exerted based on the RAW domain, so that the imaging capability can be differentiated; on the other hand, the differentiated processing and design of module functions, such as the ISP function pre-position, the flexible use of RAW domain images to improve image quality, and the dynamic expansion of subsequent processing capabilities without changing the existing design of the application processing chip.

[0149] That is to say, in the embodiment of the present application, an image processing chip is designed here, which mainly combines the chip architecture, algorithm and software and other multi-dimensional considerations, and can realize the image processing function in the RAW domain, with super night scene, high dynamic range image (High-Dynamic Range, HDR), and real-time blurring and other functions. Since the image in the RAW domain has rich details, it can achieve better noise reduction and other effects, thereby improving the image quality; in addition, the realization of video HDR based on the image processing chip can also better solve the power consumption problem. At the same time, the software framework designed based on the image processing chip of this solution can not only better solve the collaborative work and timing problems between the image processing chip and the application processing chip, but also take into account the convenient and efficient control of the image processing chip and the expansion of functions.

[0150] This embodiment provides an image processing method, which obtains image parameters through a multimedia service module and sends the image parameters to a first hardware abstraction layer module; sends the image parameters to a first driver module through the first hardware abstraction layer module; receives the image parameters through the first driver module and sends the image parameters to an image processing chip, so that the image processing chip cooperates with the application processing chip to perform image processing. In this way, the function of the image processing chip can be fully utilized, and it has sufficient scalability and flexibility, so that the image processing capability can be better realized and the image quality can be improved. At the same time, the coordination and timing problems between the image processing chip and the application processing chip can be better solved.

[0151] In another embodiment of the present application, this embodiment provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by at least one processor, the steps of the method described in any one of the aforementioned embodiments are implemented.

[0152] It should be noted that the technical solution of this embodiment is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment.

[0153] It should also be noted that the storage medium here may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0154] Based on the above computer readable storage medium, see Figure 6 , which shows a schematic diagram of the structure of a terminal device 60 provided in an embodiment of the present application. Figure 6 As shown, the terminal device 60 may include: a communication interface 601, a memory 602 and a processor 603; each component is coupled together via a bus system 604. It is understood that the bus system 604 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 604 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 6 Various buses are labeled as bus system 604. Among them, the communication interface 601 is used for receiving and sending signals in the process of sending and receiving information between other external network elements;

[0155] A memory 602, used to store a computer program that can be run on the processor 603;

[0156] The processor 603 is configured to execute, when running the computer program:

[0157] Acquire image parameters through the multimedia service module, and send the image parameters to the first hardware abstraction layer module;

[0158] Sending the image parameters to the first driver module through the first hardware abstraction layer module;

[0159] The image parameters are received through the first driving module, and the image parameters are sent to the image processing chip, so that the image processing chip cooperates with the application processing chip to perform image processing.

[0160] It can be understood that the memory 602 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 602 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0161] The processor 603 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 603. The above processor 603 may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 603 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.

[0162] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0163] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0164] Optionally, as another embodiment, the processor 603 is further configured to execute the steps of the method described in any one of the aforementioned embodiments when running the computer program.

[0165] It should be noted that in the embodiment of the present application, the terminal device 60 can be implemented in various forms. For example, the terminal device described in the embodiment of the present application can be a smart phone, a tablet computer, a laptop computer, a PDA, a navigation device, a digital camera, a video camera, etc., and the embodiment of the present application does not specifically limit it.

[0166] In the embodiment of the present application, for the above-mentioned terminal device, the terminal device includes an application processing chip and an image processing chip, and also has the image processing architecture described in the embodiment of the present application. In this way, it can not only give full play to the functions of the image processing chip, but also have sufficient scalability and flexibility, so as to better realize the image processing capability and improve the image quality. At the same time, it can also better solve the collaborative work and timing problems between the image processing chip and the application processing chip.

[0167] It should be noted that, in this application, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0168] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0169] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0170] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0171] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0172] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing architecture, characterized in that: Applied to an application processing chip, and the image processing architecture includes a multimedia service module, a first hardware abstraction layer module and a first driver module; wherein, The multimedia service module is used to obtain image parameters and send the image parameters to the first hardware abstraction layer module; The first hardware abstraction layer module is used to send the image parameters to the first driver module; The first driving module is used to receive the image parameters and send the image parameters to the image processing chip, so that the image processing chip cooperates with the application processing chip to perform image processing; The image processing architecture further includes a second hardware abstraction layer module; the second hardware abstraction layer module is used to obtain configuration parameters of the application processing chip itself and send the configuration parameters to the multimedia service module.

2. The image processing architecture according to claim 1, characterized in that: The image parameters include at least one of the following: the configuration parameters sent by the second hardware abstraction layer module, the first image parameters customized inside the multimedia service module, and the second image parameters that need to be adjusted during the image processing.

3. The image processing architecture according to claim 2, characterized in that: The multimedia service module is further used to obtain startup parameters when opening the camera application, wherein the startup parameters include the configuration parameters and the first image parameters; The first hardware abstraction layer module is further used to receive the startup parameters sent by the multimedia service module, and send the startup parameters to the first driver module; The first driving module is further used to receive the startup parameters and send the startup parameters to the image processing chip, so that the image processing chip completes power-on initialization.

4. The image processing architecture according to claim 1, characterized in that: The first driver module is further used to obtain abnormal monitoring parameters of the image processing chip and send the abnormal monitoring parameters to the first hardware abstraction layer module; The first hardware abstraction layer module is further used to send the abnormal monitoring parameter to the multimedia service module; The multimedia service module is further used to receive the abnormal monitoring parameters and send the abnormal monitoring parameters to the second hardware abstraction layer module, so that the application processing chip performs corresponding processing according to the abnormal monitoring parameters.

5. The image processing architecture according to claim 1, characterized in that: The image processing architecture also includes a high-speed processing interface, which is used to implement interactive communication between the image processing chip and the application processing chip; wherein, The image processing chip is used to receive the first image data through the high-speed processing interface, and perform image processing on the first image data according to the configuration parameters sent by the second hardware abstraction layer module to obtain second image data; The application processing chip is used to receive the second image data and custom image parameters sent by the image processing chip through the high-speed processing interface, and perform image processing on the second image data according to the custom image parameters to obtain target image data; wherein the custom image parameters include at least 3A statistical information.

6. The image processing architecture according to claim 5, characterized in that: The multimedia service module is further used to receive the 3A statistical information sent by the application processing chip through the second hardware abstraction layer module, and send the 3A statistical information to the first hardware abstraction layer module; The first hardware abstraction layer module is further used to receive and parse the 3A statistical information, and send the parsed 3A statistical information to the multimedia service module; The multimedia service module is further configured to forward the parsed 3A statistical information to the second hardware abstraction layer module, so that the application processing chip performs image processing on the second image data according to the parsed 3A statistical information.

7. An image processing method, characterized in that: The method is applied to an image processing architecture including a multimedia service module, a first hardware abstraction layer module and a first driver module, and the image processing architecture is applied to an application processing chip, and the method includes: Acquire image parameters through the multimedia service module, and send the image parameters to the first hardware abstraction layer module; Sending the image parameters to the first driver module through the first hardware abstraction layer module; receiving the image parameters through the first driving module, and sending the image parameters to the image processing chip, so that the image processing chip cooperates with the application processing chip to perform image processing; The method further comprises: The configuration parameters of the application processing chip itself are obtained through the second hardware abstraction layer module, and the configuration parameters are sent to the multimedia service module.

8. A terminal device, characterized in that: The terminal device includes a memory and a processor; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to claim 7 when running the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to claim 7 is implemented.

Images