An image processing method, device and storage medium

Abstract

The application discloses an image processing method, device and storage medium, and the method comprises the following steps: inputting a current frame image into a first statistical module through an MCU to obtain first statistical data; inputting the current frame image into a second statistical module through the MCU to obtain second statistical data; the real-time requirement of the first statistical data is higher than that of the second statistical data; predicting a first image parameter of a next frame image according to the first statistical data through the MCU; acquiring the second statistical data through an AP, and predicting a second image parameter of the next frame image according to the second statistical data; acquiring the second image parameter through the MCU, reconfiguring configuration parameters of the first statistical module according to the first image parameter, and reconfiguring configuration parameters of the second statistical module according to the second image parameter. The first statistical data with high real-time requirement is processed through the MCU, and the second statistical data with low real-time requirement is processed through the AP, so that the timeliness of image information processing can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology field, and particularly relates to an image processing method, device and storage medium. BACKGROUND

[0002] At present, an image signal processing (ISP) module in a camera system directly interacts with an application processing system (AP-System), and 5A control of an image is completed by an AP core, and the 5A control directly affects image quality. Under this system architecture, since the system running by the AP core belongs to a non-real-time operating system and the business load is heavy, there is a situation of response processing delay, thereby affecting the image quality. SUMMARY

[0003] The present application aims to provide an image processing method, device and storage medium.

[0004] The technical scheme of the present application is implemented as follows:

[0005] In a first aspect, an image processing method is provided, applied to a terminal, and an image signal processing module of the terminal includes a first type of statistical module and a second type of statistical module. The method includes:

[0006] inputting, by a microcontroller (MCU), a current frame image collected by a first sensor into the first type of statistical module to obtain first statistical data, and inputting, by the MCU, the current frame image into the second type of statistical module to obtain second statistical data; wherein a real-time requirement of the first statistical data is higher than a real-time requirement of the second statistical data;

[0007] predicting, by the MCU, a first image parameter of a next frame image according to the first statistical data of the current frame image;

[0008] obtaining, by the AP, the second statistical data of the current frame image, and predicting a second image parameter of the next frame image according to the second statistical data of the current frame image;

[0009] obtaining, by the MCU, the second image parameter, and reconfiguring a configuration parameter of the first type of statistical module according to the first image parameter, and reconfiguring a configuration parameter of the second type of statistical module according to the second image parameter.

[0010] In a second aspect, an image processing device is provided, applied to a terminal, and an image signal processing module of the terminal includes a first type of statistical module and a second type of statistical module. The device includes:

[0011] The processing unit is configured to input a current frame image collected by a first sensor into the first statistical module through a micro controller unit (MCU) to obtain first statistical data, and input the current frame image into the second statistical module through the MCU to obtain second statistical data, wherein a real-time requirement of the first statistical data is higher than that of the second statistical data; and predict a first image parameter of a next frame image according to the first statistical data of the current frame image through the MCU, and acquire the second statistical data of the current frame image through the AP, and predict a second image parameter of the next frame image according to the second statistical data of the current frame image.

[0012] The configuration unit is configured to acquire the second image parameter through the MCU, and reconfigure a configuration parameter of the first statistical module according to the first image parameter, and reconfigure a configuration parameter of the second statistical module according to the second image parameter.

[0013] In a third aspect, an image processing device is provided, which includes a processor and a memory configured to store a computer program capable of running on the processor, wherein the processor is configured to execute steps of the method of the first aspect when running the computer program.

[0014] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, wherein the computer program is configured to implement steps of the method of the first aspect when executed by a processor.

[0015] By using the above technical solution, the first statistical data with high real-time requirement is processed through the MCU, and the second statistical data with low real-time requirement is processed through the AP, so that the timeliness of image information processing can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 FIG. 1 is a simplified schematic diagram of an image processing system in an embodiment of the present application;

[0017] Figure 2 FIG. 2 is a first flowchart of an image processing method in an embodiment of the present application;

[0018] Figure 3 FIG. 3 is a software control architecture of a Camera system 5A based on a cross-core framework in an embodiment of the present application;

[0019] Figure 4 FIG. 4 is an algorithm deployment schematic diagram of a Camera system image 5A based on a cross-core framework in an embodiment of the present application;

[0020] Figure 5 FIG. 5 is a second flowchart of an image processing method in an embodiment of the present application;

[0021] Figure 6 Figure 1 is a schematic diagram of a software node structure of a Camera system 5A based on a cross-kernel framework according to an embodiment of the present application;

[0022] Figure 7 Figure 2 is a schematic diagram of a buffer configuration and flow control of a Camera system 5A based on a cross-kernel framework according to an embodiment of the present application;

[0023] Figure 8 Figure 3 is a schematic diagram of an ACK mechanism for releasing buffers between different cores according to an embodiment of the present application;

[0024] Figure 9 Figure 4 is a schematic diagram of a structure of an image processing device according to an embodiment of the present application;

[0025] Figure 10 Figure 5 is a schematic diagram of a structure of an image processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable a person skilled in the art to better understand the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings, which are only used for reference and do not limit the embodiments of the present application.

[0027] The image 5A control according to the present application will be described in detail below:

[0028] The image 5A control includes Auto Exposure (AE), Auto Focus (AF), Auto White Balance (AWB), Anti-Flicker (ATF) and Auto Lens Shading Correction (ALSC).

[0029] The AE is a mechanism for automatically adjusting the exposure and gain according to the intensity of external light to prevent overexposure or underexposure. The parameters to be adjusted by the AE include exposure time, aperture size, ISO sensitivity of the sensor, etc. The requirements of the AE are "fast, accurate and stable", i.e., it should quickly converge to the target, the brightness should be accurate, the convergence process should be smooth, and the convergence should be stable after completion.

[0030] The AF is to calculate the moving direction and size of the lens according to the sharpness (generally the edge information) or the object distance information of the image, and then drive the circuit to move the lens to change the focus position.

[0031] AWB is to restore the white color in different color temperature of ambient light after imaging by some algorithm to the real white color (usually the white color observed by human eye in natural daylight ambient light).

[0032] ATF, CMOS-Sensor exposure line by line, different row of image element exposure time is different. When the power frequency flicker occurs, regular light and dark stripes will be found on the video, and it will scroll when previewing the video, which looks like flickering.

[0033] ALSC, due to the physical properties of the lens itself, the brightness of the image gradually decreases from the center. Because the image light is on the pixel through the lens, the angle of the focus point at the corner is larger than that at the center, causing the corner to lose light. The effect on the image is that the brightness gradually decays from the center of the image to the periphery, and the farther away from the center of the image, the darker the brightness. In order to compensate for the brightness of the periphery, lens shadow correction is needed.

[0034] Before the image processing method of the present application is described, the image processing system of the present application is introduced, as shown in Figure 1 The image processing system includes a camera system Camera-System and an application processing system AP-System, Camera-System includes ISP and MCU, ISP includes first type statistical module and second type statistical module; the first type statistical module in ISP is controlled by MCU to pre-process the current frame image to obtain first statistical data, and the second type statistical module in ISP is controlled by MCU to pre-process the current frame image to obtain second statistical data; MCU processes the first statistical data with high real-time requirement; the second statistical data with low real-time requirement is transmitted to AP(ACPU) through cross-core communication technology, and the second statistical data with low real-time requirement is processed by AP(ACPU); in this way, the timeliness of image information processing can be improved.

[0035] Figure 2 The first flowchart of the image processing method in the embodiment of the present application is applied to a terminal, and the image signal processing ISP module of the terminal includes a first type statistical module and a second type statistical module, as shown in Figure 2 The image processing method can specifically include:

[0036] Step 201: input the current frame image collected by the first sensor into the first type statistical module through the microcontroller MCU to obtain the first statistical data; and input the current frame image into the second type statistical module through the MCU to obtain the second statistical data; wherein the real-time requirement of the first statistical data is higher than that of the second statistical data.

[0037] It should be noted that, since the real-time requirement of the first statistical data is higher than that of the second statistical data, the first type of statistical module used to obtain the first statistical data can include an AE statistical module and an AF statistical module, and the second type of statistical module used to obtain the second statistical data can include an AWB statistical module, an ATF statistical module and an ALSC statistical module.

[0038] Step 202: predicting, by the MCU, a first image parameter of a next frame image according to first statistical data of a current frame image.

[0039] In the embodiments of the present application, the first image parameter of the next frame image predicted by the MCU according to the first statistical data output by the AE statistical module includes an exposure time. The first image parameter of the next frame image predicted by the MCU according to the first statistical data output by the AF statistical module includes a step length.

[0040] In some embodiments, further comprising: in the case that the camera APP is started, transmitting, by the AP, a configuration file required for executing a preset algorithm to the MCU;

[0041] obtaining, by the MCU, a new preset algorithm according to the preset algorithm and the configuration file;

[0042] The predicting, by the MCU, a first image parameter of a next frame image according to first statistical data of a current frame image, comprises:

[0043] obtaining, by the MCU, a first image parameter of a next frame image by taking the first statistical data of the current frame image as an input of the new preset algorithm.

[0044] Here, when the preset algorithm is an AE algorithm, obtaining, by the MCU, a new AE algorithm according to the AE algorithm and the configuration file required for executing the AE algorithm transmitted by the AP, and then obtaining, by the MCU, a first image parameter of a next frame image by taking the first statistical data as an input of the new AE algorithm. When the preset algorithm is an AF algorithm, obtaining, by the MCU, a new AF algorithm according to the AF algorithm and the configuration file required for executing the AF algorithm transmitted by the AP, and then obtaining, by the MCU, a first image parameter of a next frame image by taking the first statistical data as an input of the new AF algorithm.

[0045] Step 203: acquiring, by an application processor AP, second statistical data of a current frame image, and predicting, according to the second statistical data of the current frame image, a second image parameter of a next frame image.

[0046] In the embodiment of the present application, the second image parameter of the next frame of image predicted by the MCU according to the second statistical data output by the AWB statistical module comprises a color temperature value. The second image parameter of the next frame of image predicted by the MCU according to the second statistical data output by the ATF statistical module comprises a fluorescent lamp flicker frequency. The second image parameter of the next frame of image predicted by the MCU according to the second statistical data output by the ALSC statistical module comprises a shadow parameter.

[0047] Step 204: obtaining the second image parameter by the MCU, and reconfiguring the configuration parameters of the first type statistical module and the second type statistical module according to the first image parameter and the second image parameter of the next frame of image.

[0048] In a feasible implementation, the second image parameter of the next frame of image is transmitted to the MCU by the AP, so that the MCU obtains the second image parameter of the next frame of image.

[0049] In the embodiment of the present application, the configuration parameters of the first type statistical module and the second type statistical module are reconfigured by the MCU according to the first image parameter and the second image parameter of the next frame of image, so that the image effect of the next frame of image can be improved.

[0050] In some embodiments, further comprising:

[0051] obtaining the first statistical data of the current frame of image by the AP;

[0052] caching the first statistical data and the second statistical data in a preset storage space by the AP, for debugging analysis.

[0053] Here, when it is determined that the debugging requirement is met, the first statistical data and the second statistical data are obtained from the preset storage space by the AP, and the first statistical data and the second statistical data are used for debugging analysis.

[0054] Here, the execution subject of steps 201 to 204 can be a processor of a terminal.

[0055] By using the above technical solution, the first statistical data with high real-time requirement is processed by the MCU, and the second statistical data with low real-time requirement is processed by the AP, so that the timeliness of image information processing can be improved.

[0056] The following describes the image 5A control as an example in the embodiment of the present application.

[0057] Since the ISP module in the camera system Camera-System directly interacts with the AP-System, the 5A control of the image is completed by the AP core, and the 5A control directly affects the image quality. Under this system architecture, since the AE control and the AF control have high real-time requirements, the corresponding AE algorithm and AF algorithm are deployed in the AP core, and the system running in the AP core belongs to a non-real-time operating system and has heavy business load, so there is a delay in response processing, which further affects the image quality.

[0058] Based on this, an embodiment of the present application gives a cross-core framework-based Camera system image 5A software control architecture, as shown in Figure 3 The following is introduced in the order from bottom to top:

[0059] The Hardware layer (hardware layer) includes AE statistical modules, AF statistical modules, AWB statistical modules, ATF statistical modules, and ALSC statistical modules in the ISP; each statistical module is used to process the current frame image collected by the first sensor to obtain corresponding statistical data. The statistical data has a special physical effect, mainly that the statistical module divides the image into m*n image blocks (Block), calculates the statistical data through hardware, and outputs the statistical data to provide the algorithm unit of the 5A software as the basis for control decision.

[0060] The Firmware layer (firmware layer) manages the control pipeline of the Camera hardware, such as controlling preview, shooting, video recording, mode switching, etc. The AE and AF control and software control are directly controlled by the bottom MCU core to reduce inter-core communication and realize comprehensive control. The MCU interacts with the AP side through the frame buffer and metadata to realize the uplink data interaction.

[0061] The Kernel layer (kernel layer) is the Android software Linux kernel, which loads the driving devices related to the ISP, sensor, etc., and is used to realize the interaction between the Firmware layer and the User layer.

[0062] The User layer realizes the interaction between the Kernel and the User layer by the HW Layer, such as the Message Manager, the ISP service, and other software modules; the Camera HAL layer controls the specific business logic of the Camera.

[0063] Figure 4 The following is introduced in the order from bottom to top: Figure 4As shown, the algorithm and control strategy balance the classification deployment scheme: AE and AF control has higher real-time requirements, and the core processing algorithm is deployed in the lower MCU core control that is closer to the ISP image signal processing hardware system; AWB, ALSC and ATF control has slightly lower real-time requirements and higher computing power requirements, and the core algorithm is deployed in the AP core, saving MCU core resources and fully utilizing the comprehensive processing capability of the AP core.

[0064] On the upper AP side, the Camera HAL is the specific bearer of the Camera service function, and the 5A software is a sub-function module thereof. The AWB prediction algorithm, the ATF prediction algorithm and the ALSC prediction algorithm are deployed, and each prediction algorithm is used to predict a second image parameter of a next frame of image. The AE channel algorithm and the AF channel algorithm are also deployed, and each channel algorithm is used to transmit a configuration file required for executing the AE prediction algorithm and the AF prediction algorithm to the MCU side, and is also used to receive the first statistical data transmitted from the MCU side for debugging analysis.

[0065] On the lower MCU side, the AE prediction algorithm and the AF prediction algorithm are deployed, and each prediction algorithm is used to predict a first image parameter of a next frame of image. It is also used to receive the algorithm results (second image parameters) of the AWB prediction algorithm, the ATF prediction algorithm and the ALSC prediction algorithm respectively transmitted from the AP side. Under the unified frame parameter matching of the lower IQC module, the first image parameter and the second image parameter are used to reconfigure the hardware register parameters related to the sensor.

[0066] Based on the above embodiment, the embodiment of the application considers that the sensor is equipped with a digital overlap-high-dynamic range image (Digital Overlap-High-Dynamic Range, DOL-HDR) technology, and adopts a "quasi-simultaneous" manner to output multiple frames of images with different exposure times. By performing image fusion processing on the multiple frames of images with different exposure times, an HDR image is generated. Based on this, the application further provides an image processing method, Figure 5 A second flowchart of the image processing method in the embodiment of the application is shown in Figure 5 As shown, the image processing method specifically includes:

[0067] Step 501: input N frames of exposure images corresponding to a current frame of image into N second-type statistical modules through the MCU to obtain N second statistical data; 1≤N≤3.

[0068] Step 502: input N frames of exposure images corresponding to a current frame of image into N first-type statistical modules through the MCU to obtain N first statistical data; the real-time requirement of the first statistical data is higher than that of the second statistical data.

[0069] Here, Table 1 is a diagram showing the relationship between the Sensor DOL type and the number of statistical modules and buffers.

[0070]

[0071] In combination with Table 1, when the DOL type of the sensor is QBIN, one frame of image corresponds to one exposure image, the number N of the second type of statistical module is equal to 1, and the number N of the first type of statistical module is equal to 1; when the DOL type of the sensor is 2DOL, one frame of image corresponds to two exposure images, the number N of the second type of statistical module can be 1 or 2, and the number N of the first type of statistical module can be 1 or 2 (N can be set according to specific conditions); when the DOL type of the sensor is 3DOL, one frame of image corresponds to two exposure images, the number N of the second type of statistical module can be 1 or 2 or 3, and the number N of the first type of statistical module can be 1 or 2 or 3 (N can be set according to specific conditions); when the DOL type of the sensor is 4DOL, one frame of image corresponds to two exposure images, the number N of the second type of statistical module can be 1 or 2 or 3, and the number N of the first type of statistical module can be 1 or 2 or 3 (N can be set according to specific conditions).

[0072] It should be noted that according to the Sensor DOL type on the market, regardless of which type, limited by the hardware resource of the Camera ISP, the Camera hardware provides at most 3 statistical modules for each A (example 5A statistical processing) to process long exposure, medium exposure (or medium long exposure, medium short exposure), short exposure, etc. from the image.

[0073] Step 503: predicting, by the MCU, a first image parameter of a next frame of image according to N first statistical data of a current frame of image.

[0074] In the embodiments of the present application, the first image parameter of the next frame of image predicted by the MCU according to the first statistical data output by the AE statistical module includes the exposure time. The first image parameter of the next frame of image predicted by the MCU according to the first statistical data output by the AF statistical module includes the step length.

[0075] In some embodiments, the predicting, by the MCU, a first image parameter of a next frame of image according to N first statistical data of a current frame of image, comprises:

[0076] predicting N second sub-image parameters of the next frame of image according to the N first statistical data of the current frame of image;

[0077] performing fusion processing on the N second sub-image parameters to obtain the first image parameter.

[0078] That is, the MCU predicts the N second sub-image parameters of the next frame image according to the N first statistical data of the current frame image, and then performs fusion processing on the N second sub-image parameters to predict the first image parameter of the next frame image.

[0079] At step 504, the AP obtains the N second statistical data of the current frame image, and predicts the second image parameter of the next frame image according to the N second statistical data of the current frame image.

[0080] In the embodiment, the second image parameter of the next frame image predicted by the MCU according to the second statistical data output by the AWB statistical module includes a color temperature value. The second image parameter of the next frame image predicted by the MCU according to the second statistical data output by the ATF statistical module includes a fluorescent lamp flicker frequency. The second image parameter of the next frame image predicted by the MCU according to the second statistical data output by the ALSC statistical module includes a shadow parameter.

[0081] In some embodiments, the method further comprises: storing, by the MCU, the N second statistical data in N first buffers corresponding to N first buffer addresses, respectively;

[0082] transmitting, by the MCU, the N first buffer addresses to the AP;

[0083] The method further comprises:

[0084] The method further comprises:

[0085] Here, the MCU stores the N second statistical data in the N first buffers according to the N first buffer addresses, and transmits the N first buffer addresses to the AP, so that the AP obtains the N second statistical data stored in the N first buffers according to the N first buffer addresses.

[0086] In a possible implementation, the MCU and the AP are pre-agreed to configure N first buffer addresses for the second statistical module. N is 1 or 2 or 3.

[0087] In another possible implementation, the N first buffer addresses configured for the second statistical module are configured on the AP side and transmitted to the MCU.

[0088] Based on this, in some embodiments, the method further comprises:

[0089] The method further comprises:

[0090] The AP responds to the resource allocation request, configures N first buffer addresses corresponding to the second-type statistical module, and transmits them to the MCU.

[0091] It should be noted that the camera hardware provides up to three statistical modules for each A (e.g., 5A statistical processing) to process long, medium (or medium-long, medium-short), and short exposures from the image. Therefore, for each frame, a 5A statistical module, for example, can be configured with up to three buffers to meet all modes. This eliminates the need to re-apply, load, distribute, and configure buffers when switching between DOL modes. If buffer management is implemented differently for different DOLs, the cross-core interactive AP and MCU can easily cause timing mismatches when dynamically processing different buffer configurations. Given the current relative abundance of DDR memory resources, a balanced approach is to trade "space" for "security and stability."

[0092] In some embodiments, predicting the second image parameter of the next frame image according to N second statistical data of the current frame image by the MCU includes:

[0093] Predicting N first sub-image parameters of the next frame image according to the N second statistical data of the current frame image;

[0094] The N first sub-image parameters are fused to obtain the second image parameters.

[0095] That is, the MCU predicts N first sub-image parameters of the next frame image based on N second statistical data of the current frame image, and then fuses the N first sub-image parameters to predict the second image parameters of the next frame image.

[0096] In some embodiments, further comprising:

[0097] Acquire, through the AP, N first statistical data of the current frame image;

[0098] The N first statistical data and the N second statistical data are cached in a preset storage space through the AP for debugging and analysis.

[0099] In some embodiments, the method further includes: caching the N first statistical data into N second buffers corresponding to N second buffer addresses respectively by the MCU;

[0100] transmitting the N second buffer addresses to the AP;

[0101] The acquiring, through the AP, N first statistical data of the current frame image includes:

[0102] acquire, by the AP, the N first statistic data cached in the N first buffers according to the N second buffer addresses.

[0103] Here, the MCU caches the first statistic data in the N second buffers according to the N second buffer addresses, and transmits the N second buffer addresses to the AP, so that the AP acquires the N first statistic data cached in the N second buffers according to the N second buffer addresses.

[0104] In a possible implementation, the MCU and the AP are pre-agreed to configure N second buffer addresses for the first statistic module. N is 1 or 2 or 3.

[0105] In another possible implementation, the N second buffer addresses configured for the first statistic module are configured on the AP side and transmitted to the MCU.

[0106] Based on this, in some embodiments, further comprising:

[0107] sending, by the MCU, a resource allocation request to the AP;

[0108] configuring, by the AP in response to the resource allocation request, the N second buffer addresses corresponding to the first statistic module and transmitting the N second buffer addresses to the MCU.

[0109] It should be noted that when the camera is shooting, different cameras often work simultaneously, switch back and forth, and various scenes change dynamically. Therefore, the control of the AP side for each statistic data must also be flexible and dynamically created or destroyed, so as to ensure the flexibility of the control strategy and the timely allocation or release of resources.

[0110] Based on this, in some embodiments, the AP at the Camera HAL layer creates a group of nodes for the first sensor, including at least a source node and a first statistic node;

[0111] The acquiring, by the AP, the N first statistic data cached in the N first buffers according to the N second buffer addresses comprises:

[0112] allocating, by a first output port of the source node, the N first buffer addresses to the first statistic node;

[0113] acquiring, by the first statistic node, the N first statistic data cached in the N first buffers according to the N first buffer addresses.

[0114] Here, the source node is configured to distribute the N first buffer addresses to the corresponding first statistical nodes through the first output ports; the first statistical nodes are configured to obtain N second statistical data stored in the N first buffers according to the N first buffer addresses, and predict the second image parameters of the next frame image according to the N second statistical data of the current frame image.

[0115] It should be noted that the number of source nodes is 1, the number of first statistical nodes is consistent with the number of second statistical modules, the number of first output ports of the source nodes is consistent with the number of second statistical modules, and the number of first buffer addresses is consistent with the number of first statistical nodes. For example Figure 6 , when the second statistical module includes an AWB statistical module, an ATF statistical module and an ALSC statistical module, a set of nodes created for the first sensor includes a first statistical node 1, a first statistical node 2 and a first statistical node 3, wherein the source node is connected to the first statistical node 1 through the first output port 1, the source node is connected to the first statistical node 2 through the first output port 2, and the source node is connected to the first statistical node 3 through the first output port 3.

[0116] In combination with Figure 6 , N first buffer addresses 1 are distributed to the first statistical node 1 through the first output port 1 of the source node, N second statistical data 1 stored in N first buffers 1 are obtained according to the N first buffer addresses 1 through the first statistical node 1, and the N second statistical data 1 are input into the AWB prediction algorithm through the first statistical node 1 to predict the second image parameter 1. N first buffer addresses 2 are distributed to the first statistical node 2 through the first output port 2 of the source node, N second statistical data 2 stored in N first buffers 2 are obtained according to the N first buffer addresses 2 through the first statistical node 2, and the N second statistical data 2 are input into the ATF prediction algorithm through the first statistical node 2 to predict the second image parameter 2. N first buffer addresses 3 are distributed to the first statistical node 3 through the first output port 3 of the source node, N second statistical data 3 stored in N first buffers 3 are obtained according to the N first buffer addresses 3 through the first statistical node 3, and the N second statistical data 3 are input into the ALSC prediction algorithm through the first statistical node 3 to predict the second image parameter 3.

[0117] In some embodiments, further comprising: caching the N second statistical data in a preset storage space through the first statistical nodes.

[0118] Here, the N second statistical data are obtained from the preset storage space through the source node, and debugging analysis is performed using the N second statistical data.

[0119] In some embodiments, further comprising: returning the N first buffers to the buffer pool.

[0120] That is, the N first buffers are returned to the buffer pool, avoiding invalid occupation of memory resources.

[0121] Correspondingly, in some embodiments, the set of nodes created by the AP for the first sensor further includes a second statistical node;

[0122] The N first statistical data cached in the N second buffers are obtained by the AP according to the N second buffer addresses, including:

[0123] The N second buffer addresses are distributed to the second statistical node through the second output port of the source node;

[0124] The N first statistical data cached in the N second buffers are obtained by the second statistical node according to the N second buffer addresses.

[0125] Here, the second statistical node is used to obtain the N first statistical data cached in the N second buffers according to the N second buffer addresses, and to predict the first image parameter of the next frame image according to the N first statistical data of the current frame image.

[0126] It should be noted that the number of second statistical nodes is consistent with the number of first statistical modules, and the number of second output ports of the source node is consistent with the number of first statistical modules. For example Figure 6 , when the first statistical module includes an AE statistical module and an AF statistical module, the set of nodes created for the first sensor further includes a second statistical node 1 and a second statistical node 2, wherein the source node is connected to the second statistical node 1 through the second output port 1, and the source node is connected to the second statistical node 2 through the second output port 2.

[0127] In combination with Figure 6 , the N second buffer addresses 1 are distributed to the second statistical node 1 through the second output port 1 of the source node, the N first statistical data 1 cached in the N second buffers 1 are obtained by the second statistical node 1 according to the N second buffer addresses 1, and the N first statistical data 1 are input into the AE prediction algorithm through the second statistical node 1 to predict the first image parameter 1. The N second buffer addresses 2 are distributed to the second statistical node 2 through the second output port 2 of the source node, the N first statistical data 2 cached in the N second buffers 2 are obtained by the second statistical node 2 according to the N second buffer addresses 2, and the N first statistical data 2 are input into the AF prediction algorithm through the second statistical node 2 to predict the first image parameter 2.

[0128] In some embodiments, further comprising: caching the N first statistical data in a preset storage space by the second statistical node.

[0129] Here, the source node obtains N first statistical data from a preset storage space, and performs debugging analysis by using the N first statistical data.

[0130] In some embodiments, further comprising: returning the N second buffers to the buffer pool.

[0131] That is, the N second buffers are returned to the buffer pool, avoiding invalid occupation of memory resources.

[0132] In some embodiments, further comprising: in a case where the second sensor is enabled, creating a group of nodes for the second sensor by the AP.

[0133] That is, a group of nodes is dynamically created for the currently enabled sensor at the AP side every time a sensor is enabled. This ensures the flexibility of the control strategy and the timely allocation of resources.

[0134] In some embodiments, further comprising: in a case where the third sensor is detected to be in an off state, destroying a group of nodes created for the third sensor; wherein the third sensor is the first sensor or the second sensor.

[0135] That is, in a case where the enabled sensor is detected to be in an off state, a group of nodes created for the sensor is dynamically destroyed. This ensures the flexibility of the control strategy and the timely release of resources.

[0136] Step 505: Obtain the second image parameters by the MCU, and reconfigure the configuration parameters of the first type of statistical module according to the first image parameters, and reconfigure the configuration parameters of the second type of statistical module according to the second image parameters.

[0137] In a feasible implementation manner, the second image parameters of the next frame of image are transmitted to the MCU by the AP, so that the MCU obtains the second image parameters of the next frame of image.

[0138] Here, in combination with Figure 6 , the configuration parameters of the AE statistical module are reconfigured by the MCU according to the first image parameter 1, the configuration parameters of the AF statistical module are reconfigured by the MCU according to the first image parameter 2, the configuration parameters of the AWB statistical module are reconfigured by the MCU according to the second image parameter 1, the configuration parameters of the ATF statistical module are reconfigured by the MCU according to the second image parameter 2, and the configuration parameters of the ALSC statistical module are reconfigured by the MCU according to the second image parameter 3, so as to improve the image effect of the next frame of image.

[0139] Figure 7The figure is a schematic diagram of the configuration and flow control of the Camera system 5A Buffer based on the cross-core framework in the embodiments of the present application, as shown in Figure 7 When the mobile phone camera is turned on, the related flow runs as follows:

[0140] First, the APP&FWK layer creates a preview session (Create Previw Session) and sends a stream configuration (Configure Streams) to the Camera HAL. The Camera HAL then sends a connect sensor (Connect Sensor) command to the Firmware layer to execute the connect sensor (Connect Sensor) command, implement the initialization configuration of the image sensor, and return an ACK to indicate that the configuration is successful.

[0141] Second, the APP&FWK layer sets a repeating request (Set Repeating Request) and sends a preview request processing (Process Preview Request-First) to the Camera HAL layer. The Camera HAL then sends a HAL pipeline configuration (Pipeline Congfiguration) to prepare the resources and configuration parameters required for the operation of the ISP hardware pipeline, including at least the allocation of Buffer resources.

[0142] When allocating Buffer resources, it is necessary to consider that at least one exposure image corresponds to one frame of image during subsequent digital overlap DOL switching. The change in the number of exposure images causes the number of one statistical module (for example, an AE statistical module) to change, and the number of the corresponding buffer also changes. Therefore, it is necessary to apply for and allocate the required number of Buffer in advance to avoid errors caused by the synchronization between the AP side and the MCU side not being timely. The Buffer is returned in time after being used to avoid invalid occupation of memory resources.

[0143] According to the Sensor DOL type on the market, regardless of which type, limited by the Camera ISP hardware resource, the Camera hardware provides at most 3 statistical modules for each A (example 5A statistical processing) to process long exposure, medium exposure (or medium long exposure, medium short exposure), short exposure and the like from an image. Therefore, for each frame of image corresponding to the 5A statistical module, for example, at most 3 buffers Buffer are configured, which can meet all modes, and there is no need to consider the reapplication, loading, delivery and configuration of the Buffer when the DOL mode is switched. If different buffers Buffer management is performed for different DOL, the AP and MCU cross-core interaction and dynamic processing of different buffer Buffer configurations are prone to timing mismatch. Considering that the current DDR memory resource is relatively abundant, balanced consideration is taken to use "space" to exchange "safety and stability".

[0144] Based on this, for each frame of image collected by the first sensor, 3 groups of 5A statistical modules (AE, AF, AWB, ATF and ALSC) are set in the ISP, and 5 buffers are allocated for each group of 5A statistical modules. Among them, Figure 7 The sixth Buffer is also shown in the middle for caching metadata MetaData, which is used in the process of predicting the same image parameters of the next frame.

[0145] After receiving the HAL pipeline configuration command, the Firmware executes the HAL pipeline configuration command, and the 5 buffers allocated based on each group of 5A statistical modules are used to cache the statistical data output by the Hardware layer ISP group of 5A statistical modules.

[0146] In combination Figure 7 , if the DOL type of the sensor is 4DOL, that is, corresponding to 4 exposure images, 3 exposure images can be selected to input 3 same statistical modules (for example, any one of the 5A statistical modules), and cached in the 3 buffers allocated for the statistical module. If the DOL type of the sensor is 2DOL, that is, corresponding to 2 exposure images, 2 exposure images can be selected to input 2 same statistical modules (for example, any one of the 5A statistical modules), and cached in 2 of the 3 buffers allocated for the statistical module, so that the remaining buffer is idle.

[0147] Figure 8 The following is a schematic diagram for implementing the release of buffers between different cores based on the AP core and MCU core ACK mechanism in the embodiments of the present application, as shown in Figure 8 , taking the 5A statistical module as an example for description:

[0148] The AP core sends a cluster of related buffers to the underlying MCU core for periodic storage of statistical data for each frame of image flow. The buffers are returned through the upper and lower layer ACK mechanisms to ensure the safe return, release, and flow of the 5A statistical data buffers.

[0149] After the HardWare layer generates various statistical data and stores them in the corresponding buffers, the MCU core firmware packages the AWB / ATF / ALSC buffer addresses with the preview data metadata and sends an Online Request Response ACK command to the Camera HAL. This initiates the Process Preview process (for details, see the 5A software control section of the Node dynamic node). Further algorithm processing is performed and the buffers are returned to the BufferPool.

[0150] The MCU core firmware obtains the AE / AF Buffer and inputs it into the AE / AF prediction algorithm. After processing, the AE / AF Buffer is returned to the Camera HAL through the Online Request Response ACK command for execution. The MCU core firmware then enters the Process Preview process (for details, refer to the 5A software control of the Node dynamic node). No further algorithm processing is required. After meeting the debugging requirements, the buffer is returned to the Buffer Pool.

[0151] To implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application further provides an image processing device, Figure 9 FIG. 1 is a schematic diagram of the structure of the image processing device in an embodiment of the present application. Figure 9 As shown, the image signal processing module of the terminal includes a first type of statistical module and a second type of statistical module, and the image processing device 90 includes:

[0152] The processing unit 901 is configured to input a current frame image collected by a first sensor into the first statistical module through a micro controller unit (MCU) to obtain first statistical data, and input the current frame image into the second statistical module through the MCU to obtain second statistical data, wherein a real-time requirement of the first statistical data is higher than that of the second statistical data, and predict a first image parameter of a next frame image according to the first statistical data of the current frame image through the MCU, and acquire the second statistical data of the current frame image through the application processor (AP), and predict a second image parameter of the next frame image according to the second statistical data of the current frame image.

[0153] The configuration unit 902 is configured to acquire the second image parameter through the MCU, and reconfigure a configuration parameter of the first statistical module according to the first image parameter, and reconfigure a configuration parameter of the second statistical module according to the second image parameter.

[0154] By using the above technical solution, the first statistical data with high real-time requirement is processed through the MCU, and the second statistical data with low real-time requirement is processed through the AP, so that the timeliness of image information processing can be improved.

[0155] In some embodiments, the processing unit 901 is specifically configured to input N frame exposure images corresponding to the current frame image into N second statistical modules through the MCU to obtain N second statistical data, and 1≤N≤3.

[0156] In some embodiments, the processing unit 901 is further configured to predict N first sub-image parameters of the next frame image according to the N second statistical data of the current frame image.

[0157] The N first sub-image parameters are fused to obtain the second image parameter.

[0158] In some embodiments, the processing unit 901 is further configured to buffer the N second statistical data into N first buffers corresponding to N first buffer addresses through the MCU.

[0159] The N first buffer addresses are transmitted to the AP.

[0160] The N second statistical data buffered in the N first buffers are acquired through the AP according to the N first buffer addresses.

[0161] In some embodiments, the processing unit 901 is further configured to send a resource allocation request to the AP through the MCU.

[0162] The AP responds to the resource allocation request, configures the N first buffer addresses corresponding to the second type of statistical module, and transmits to the MCU.

[0163] In some embodiments, the AP creates a set of nodes for the first sensor, including at least a source node and a first statistical node;

[0164] The processing unit 901 is further configured to allocate the N first buffer addresses to the first statistical node through the first output port of the source node.

[0165] The first statistical node acquires N second statistical data cached in the N first buffers according to the N first buffer addresses.

[0166] In some embodiments, the processing unit 901 is further configured to return the N first buffers to a buffer pool.

[0167] In some embodiments, the processing unit 901 is further configured to create a set of nodes for the second sensor at the AP in the case that the second sensor is enabled.

[0168] In some embodiments, the processing unit 901 is further configured to destroy the set of nodes created for the third sensor in the case that the third sensor is detected to be in an off state; wherein the third sensor is the first sensor or the second sensor.

[0169] In some embodiments, the processing unit 901 is further configured to acquire first statistical data of the current frame image at the AP.

[0170] The AP caches the first statistical data and the second statistical data in a preset storage space for debugging analysis.

[0171] The embodiments of the present application also provide another image processing device, Figure 10 The schematic diagram of the composition structure of the image processing device in the embodiments of the present application is shown in FIG. 1. Figure 10 As shown in FIG. 1, the image processing device 100 includes a processor 1001 and a memory 802 configured to store a computer program capable of running on the processor.

[0172] When the processor 1001 is configured to run the computer program, the method steps in the foregoing embodiments are executed.

[0173] Of course, in actual application, the image processing device 100 can also include other components, such as a display, a keyboard, a mouse, a storage device, a communication interface, etc. Figure 10As shown, the various components in the image processing device 100 are coupled together by a bus system 1003. It is understood that the bus system 1003 is used for communicating data between the components. The bus system 1003 includes a data bus, a power bus, a control bus, and a state signal bus. However, for clarity, only the data bus is shown in Figure 10

[0174] In practical applications, the processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understood that the electronic device used to implement the functions of the processor can also be other devices, and the embodiments of the present application do not make specific limitations.

[0175] The memory can be a volatile memory (such as a random access memory (RAM)), a non-volatile memory (such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid state disk (SSD)), or a combination of the above types of memories, and provides instructions and data to the processor.

[0176] In exemplary embodiments, the embodiments of the present application also provide a computer readable storage medium for storing a computer program.

[0177] Optionally, the computer readable storage medium can be applied to any one of the methods in the embodiments of the present application, and the computer program causes the computer to execute the corresponding processes realized by the processor in the various methods of the embodiments of the present application. For brevity, details are not repeated here.

[0178] ​In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The embodiments described above are merely exemplary, and the unit division is merely logical function division, and there can be other division manners in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, and can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0180] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit. Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the above-mentioned program can be stored in a computer readable storage medium, and the program executes the steps including the above-mentioned method embodiments when executed; and the above-mentioned storage medium includes mobile storage equipment, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and various storage program codes.

[0181] The methods disclosed in several method embodiments provided in the present application can be combined arbitrarily without conflict to obtain new method embodiments.

[0182] The features disclosed in several product embodiments provided in the present application can be combined arbitrarily without conflict to obtain new product embodiments.

[0183] The features disclosed in several method or device embodiments provided in the present application can be combined arbitrarily without conflict to obtain new method embodiments or device embodiments.

[0184] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, applied to a terminal, characterized in that: The image signal processing module of the terminal includes a first type of statistical module and a second type of statistical module; the method includes: Inputting a current frame image captured by a first sensor into the first type of statistical module through a microcontroller MCU to obtain first statistical data; and inputting the current frame image into the second type of statistical module through the MCU to obtain second statistical data; wherein the real-time requirement of the first statistical data is higher than the real-time requirement of the second statistical data, the first type of statistical module includes an AE statistical module and an AF statistical module, and the second type of statistical module includes an AWB statistical module, an ATF statistical module, and an ALSC statistical module; Predicting, by the MCU, first image parameters of a next frame image according to first statistical data of the current frame image; acquiring, through the application processor AP, second statistical data of the current frame image, and predicting second image parameters of a next frame image based on the second statistical data of the current frame image; Acquire the second image parameters through the MCU, and reconfigure the configuration parameters of the first type of statistical modules according to the first image parameters, and reconfigure the configuration parameters of the second type of statistical modules according to the second image parameters; The method further includes: when the camera APP is started, transmitting a configuration file required for executing the first prediction algorithm to the MCU through the AP; obtaining, by the MCU, a new first prediction algorithm based on the first prediction algorithm and the configuration file; wherein the first prediction algorithm includes an AE prediction algorithm and an AF prediction algorithm; and predicting, by the MCU, first image parameters of the next frame image based on first statistical data of the current frame image, including: obtaining, by the MCU, first image parameters of the next frame image using the first statistical data of the current frame image as input to the new first prediction algorithm; wherein the first image parameters include exposure time and step length; The obtaining of the second statistical data of the current frame image through the application processor AP and predicting the second image parameters of the next frame image based on the second statistical data of the current frame image includes: using the second statistical data of the current frame image as input of a second prediction algorithm through the AP to obtain the second image parameters of the next frame image; wherein the second prediction algorithm includes an AWB prediction algorithm, an ATF prediction algorithm and an ALSC prediction algorithm, and the second image parameters include a color temperature value, a fluorescent lamp flicker frequency and a shadow parameter.

2. The method according to claim 1, characterized in that The step of inputting the current frame image into the second type statistical module through the MCU to obtain second statistical data includes: The N exposure images corresponding to the current frame image are input into N second-type statistical modules through the MCU to obtain N second statistical data; 1≤N≤3.

3. The method according to claim 2, characterized in that The predicting the second image parameter of the next frame image according to the second statistical data of the current frame image includes: Predicting N first sub-image parameters of the next frame image according to the N second statistical data of the current frame image; The N first sub-image parameters are fused to obtain the second image parameters.

4. The method according to claim 2, characterized in that Also includes: caching the N second statistical data into N first buffers corresponding to the N first buffer addresses respectively through the MCU; transmitting the N first buffer addresses to the AP; The acquiring, by the application processor AP, second statistical data of the current frame image includes: The N second statistical data cached in the N first buffers are obtained through the AP according to the N first buffer addresses.

5. The method according to claim 4, characterized in that Also includes: Sending a resource allocation request to the AP through the MCU; The AP responds to the resource allocation request, configures N first buffer addresses corresponding to the second-type statistical module, and transmits them to the MCU.

6. The method according to claim 4, characterized in that Creating a group of nodes for the first sensor at the AP, including at least a source node and a first statistics node; The acquiring, by the AP according to the N first buffer addresses, N second statistical data cached in the N first buffers includes: Allocating the N first buffer addresses to the first statistics node through the first output port of the source node; The N second statistical data cached in the N first buffers are obtained through the first statistical node according to the N first buffer addresses.

7. The method according to claim 6, characterized in that Also includes: Return the N first buffers to the buffer pool.

8. The method according to claim 6, characterized in that Also includes: When the second sensor is enabled, a group of nodes is created for the second sensor at the AP.

9. The method according to claim 8, characterized in that Also includes: When it is detected that a third sensor is in a closed state, a group of nodes created for the third sensor is destroyed; wherein the third sensor is the first sensor or the second sensor.

10. The method according to claim 1, characterized in that Also includes: acquiring, through the AP, first statistical data of the current frame image; The first statistical data and the second statistical data are cached in a preset storage space through the AP for debugging and analysis.

11. An image processing device, characterized in that: The image processing device includes: a processor and a memory configured to store a computer program that can be run on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 10 when running the computer program.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images