Image processing method and device and storage medium

Abstract

The invention discloses an image processing method and device and a storage medium, and the method comprises the steps: inputting a current frame image into a first-class statistical module through an MCU, and obtaining first statistical data; inputting the current frame image into a second-class statistical module through the MCU to obtain second statistical data; the real-time requirement of the first statistical data is higher than that of the second statistical data; predicting a first image parameter of the next frame of image through the MCU according to the first statistical data; obtaining second statistical data through the AP, and predicting a second image parameter of the next frame of image according to the second statistical data; and obtaining a second image parameter through the MCU, reconfiguring the configuration parameter of the first type of statistical module according to the first image parameter, and reconfiguring the configuration parameter of the second type of statistical module according to the second image parameter. The first statistical data with a high real-time requirement is processed through the MCU, and the second statistical data with a low real-time requirement is processed through the AP, so that the timeliness of image information processing can be improved.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] At present, the Image Signal Processing (ISP) module in the camera system directly interacts with the application processing system (AP-System), and the AP core is responsible for completing the 5A control of the image, which directly affects the image quality. Under this system architecture, since the system running on the AP core is a non-real-time operating system and has a heavy business load, there is a response processing delay, which in turn affects the image quality. Summary of the invention

[0003] The present application hopes to provide an image processing method, device and storage medium.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, an image processing method is provided, which is applied to a terminal, wherein an image signal processing module of the terminal includes a first type of statistical module and a second type of statistical module; the method includes:

[0006] Inputting the current frame image collected by the first sensor into the first type statistical module through the microcontroller MCU to obtain first statistical data; and inputting the current frame image into the second type statistical module through the MCU to obtain second statistical data; wherein the real-time requirement of the first statistical data is higher than the real-time requirement of the second statistical data;

[0007] Predicting, by the MCU, a first image parameter of a next frame image according to first statistical data of the current frame image;

[0008] Acquire second statistical data of the current frame image through the AP, and predict second image parameters of the next frame image according to the second statistical data of the current frame image;

[0009] The second image parameters are obtained through the MCU, and the configuration parameters of the first type of statistical modules are reconfigured according to the first image parameters, and the configuration parameters of the second type of statistical modules are reconfigured according to the second image parameters.

[0010] In a second aspect, an image processing device is provided, which is applied to a terminal, wherein an image signal processing module of the terminal includes a first type of statistical module and a second type of statistical module; the device includes:

[0011] A processing unit is used to input the current frame image collected by the first sensor into the first type of statistical module through a microcontroller MCU to obtain first statistical data; and input the current frame image into the second type of statistical module through the MCU to obtain second statistical data; wherein the real-time requirement of the first statistical data is higher than the real-time requirement of the second statistical data; and predict the first image parameter of the next frame image according to the first statistical data of the current frame image through the MCU; and obtain the second statistical data of the current frame image through the AP, and predict the second image parameter of the next frame image according to the second statistical data of the current frame image;

[0012] A configuration unit is used to obtain the second image parameters through the MCU, and reconfigure the configuration parameters of the first type of statistical modules according to the first image parameters, and reconfigure the configuration parameters of the second type of statistical modules according to the second image parameters.

[0013] According to a third aspect, an image processing device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the method of the first aspect when running the computer program.

[0014] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the method according to the first aspect when executed by a processor.

[0015] By adopting the above technical solution, the first statistical data with high real-time requirements are processed by the MCU, and the second statistical data with low real-time requirements are processed by the AP, which can improve the timeliness of image information processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A simplified schematic diagram of an image processing system in an embodiment of the present application;

[0017] Figure 2 This is a schematic diagram of a first process of the image processing method in an embodiment of the present application;

[0018] Figure 3 The software control architecture of the Camera system 5A based on the cross-core framework in the embodiment of the present application;

[0019] Figure 4 A schematic diagram of the deployment of the Camera system image 5A algorithm based on the cross-core framework in an embodiment of the present application;

[0020] Figure 5 This is a second flow chart of the image processing method in the embodiment of the present application;

[0021] Figure 6 This is a schematic diagram of the Node structure of the Camera system 5A software based on the cross-core framework in an embodiment of the present application;

[0022] Figure 7 The figure is a schematic diagram of the configuration and process control of the Camera system 5A Buffer based on the cross-core framework in the embodiment of the present application;

[0023] Figure 8 This is a schematic diagram of implementing the release of buffers between different cores based on the AP core and MCU core ACK mechanism in an embodiment of the present application;

[0024] Fig. 9 A schematic diagram of the composition structure of an image processing device in an embodiment of the present application;

[0025] Fig.10 Schematic diagram of the composition structure of the image processing device in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0027] The following is a detailed introduction to the image 5A control involved in this application:

[0028] Image 5A controls include: Auto Exposure (AE), Auto Focus (AF), Auto White Balance (AWB), Anti-Flicker (ATF), and Auto Lens Shading Correction (ALSC).

[0029] Among them, AE is a mechanism for the camera to automatically adjust the exposure and gain according to the strength of the external light to prevent overexposure or underexposure. The parameters that AE ultimately needs to adjust include exposure time, aperture size, and ISO sensitivity of the sensor. The requirements for AE are "fast, accurate, and stable", that is, it must be able to converge to the vicinity of the target quickly, and the brightness must be accurate, the convergence process must be smooth, and it must be stable after convergence.

[0030] AF determines the focus situation based on the clarity of the image (generally edge information) or object distance information, thereby calculating the movement direction and size of the lens, and then the driving circuit moves the lens to change the focus position.

[0031] AWB uses an algorithm to restore the white imaged under ambient light of different color temperatures to true white (usually the white observed by the human eye under natural daylight).

[0032] ATF, CMOS-Sensor exposes line by line, and the exposure time of different lines of pixels is different. When power frequency flicker occurs, you will find regular light and dark stripes in the video, and they will scroll when previewing the video, which looks like flickering.

[0033] ALSC, due to the physical properties of the lens itself, causes the brightness of the image periphery to gradually decrease relative to the brightness of the center. When the image light shines on the pixel through the lens, the focal angle at the corner is greater than the central focal angle, causing light loss at the corner. The effect on the image is that the brightness gradually decays from the center of the image to the periphery, and the farther away from the center of the image, the darker the brightness. In order to compensate for the brightness of the periphery, lens shading correction is required.

[0034] Before describing the image processing method of the present application, the image processing system of the present application is introduced. Figure 1 As shown, the image processing system includes a camera system Camera-System and an application processing system AP-System, the Camera-System includes an ISP and an MCU, the ISP includes a first type of statistical module and a second type of statistical module; the MCU controls the first type of statistical module in the ISP to pre-process the current frame image to obtain the first statistical data, and the MCU controls the second type of statistical module in the ISP to pre-process the current frame image to obtain the second statistical data; the MCU processes the first statistical data with high real-time requirements; the cross-core communication technology transmits the second statistical data with low real-time requirements to the AP (ACPU), and the AP (ACPU) processes the second statistical data with low real-time requirements; this can improve the timeliness of image information processing.

[0035] Figure 2 FIG. 1 is a first flow chart of an image processing method in an embodiment of the present application, which is applied to a terminal. The image signal processing ISP module of the terminal includes a first type of statistical module and a second type of statistical module, such as Figure 2 As shown, the image processing method may specifically include:

[0036] Step 201: input the current frame image collected by the first sensor into the first type of statistical module through the microcontroller MCU to obtain first statistical data; and input the current frame image into the second type of statistical module through the MCU to obtain second statistical data; wherein the real-time requirement of the first statistical data is higher than the real-time requirement of the second statistical data.

[0037] It should be noted that since the real-time requirement of the first statistical data is higher than that of the second statistical data, the first type of statistical modules used to obtain the first statistical data may include an AE statistical module and an AF statistical module, and the second type of statistical modules used to obtain the second statistical data may include an AWB statistical module, an ATF statistical module and an ALSC statistical module.

[0038] Step 202: predicting the first image parameter of the next frame image according to the first statistical data of the current frame image through the MCU.

[0039] In the embodiment of the present application, the first image parameter of the next frame image predicted by the MCU according to the first statistical data output by the AE statistical module includes the exposure time. The first image parameter of the next frame image predicted by the MCU according to the first statistical data output by the AF statistical module includes the step length.

[0040] In some embodiments, it also includes: when the camera APP is started, transmitting a configuration file required to execute a preset algorithm to the MCU through the AP;

[0041] Obtain a new preset algorithm through the MCU according to the preset algorithm and configuration file;

[0042] The predicting, by the MCU, the first image parameter of the next frame image according to the first statistical data of the current frame image comprises:

[0043] The first statistical data of the current frame image is used as an input of a new preset algorithm through the MCU to obtain the first image parameters of the next frame image.

[0044] Here, when the preset algorithm is the AE algorithm, the MCU obtains a new AE algorithm based on the AE algorithm and the configuration file required for executing the AE algorithm transmitted by the AP, and then the MCU uses the first statistical data as the input of the new AE algorithm to obtain the first image parameter of the next frame image. When the preset algorithm is the AF algorithm, the MCU obtains a new AF algorithm based on the AF algorithm and the configuration file required for executing the AF algorithm transmitted by the AP, and then the MCU uses the first statistical data as the input of the new AF algorithm to obtain the first image parameter of the next frame image.

[0045] Step 203: obtaining second statistical data of the current frame image through the application processor AP, and predicting second image parameters of the next frame image according to the second statistical data of the current frame image.

[0046] In the embodiment of the present application, the second image parameter of the next frame image predicted by the MCU according to the second statistical data output by the AWB statistical module includes a color temperature value. The second image parameter of the next frame image predicted by the MCU according to the second statistical data output by the ATF statistical module includes a fluorescent lamp flicker frequency. The second image parameter of the next frame image predicted by the MCU according to the second statistical data output by the ALSC statistical module includes a shadow parameter.

[0047] Step 204: Acquire the second image parameter through the MCU, and reconfigure the configuration parameters of the first type of statistical module and the second type of statistical module according to the first image parameter and the second image parameter of the next frame of image.

[0048] In a feasible implementation manner, the second image parameters of the next frame image are transmitted to the MCU through the AP, so that the MCU obtains the second image parameters of the next frame image.

[0049] In the embodiment of the present application, the MCU reconfigures the configuration parameters of the first and second statistical modules according to the predicted first and second image parameters of the next frame image, thereby improving the image effect of the next frame image.

[0050] In some embodiments, it also includes:

[0051] Acquire first statistical data of the current frame image through the AP;

[0052] The first statistical data and the second statistical data are cached in a preset storage space through the AP for debugging and analysis.

[0053] Here, when it is determined that the debugging requirement is met, the first statistical data and the second statistical data are obtained from a preset storage space through the AP, and the first statistical data and the second statistical data are used to perform debugging analysis.

[0054] Here, the execution subject of step 201 to step 204 may be a processor of the terminal.

[0055] By adopting the above technical solution, the first statistical data with high real-time requirements are processed by the MCU, and the second statistical data with low real-time requirements are processed by the AP, which can improve the timeliness of image information processing.

[0056] The following embodiment of the present application takes the control of image 5A as an example and is described as follows:

[0057] Since the ISP module in the camera system directly interacts with the AP system, the AP core is responsible for completing the 5A control of the image, which directly affects the image quality. Under this system architecture, since AE control and AF control have high real-time requirements, the corresponding AE algorithm and AF algorithm are deployed in the AP core. The system running on the AP core is a non-real-time operating system with heavy business load. Therefore, there is a response processing delay, which in turn affects the image quality.

[0058] Based on this, the embodiment of the present application provides a Camera system image 5A software control architecture based on a cross-core framework, such as Figure 3 As shown, the order from bottom to top is as follows:

[0059] Hardware layer: ISP includes AE statistical module, AF statistical module, AWB statistical module, ATF statistical module and ALSC statistical module; each statistical module is used to process the current frame image collected by the first sensor to obtain corresponding statistical data. Among them, statistical data has its special physical function, mainly that the statistical module divides the image into m*n image blocks, calculates statistical data through hardware, and outputs it to the algorithm unit of 5A software as the basis for control decision.

[0060] Firmware layer: manages the control pipeline of the Camera hardware, such as controlling preview, taking pictures, recording, mode switching, etc. AE and AF control and software control are directly controlled by the underlying MCU core, reducing inter-core communication to achieve comprehensive control. The MCU interacts with the AP side for uplink data through the frame buffer and metadata.

[0061] Kernel layer: Android software Linux kernel, loaded with driver devices related to ISP, sensor, etc., used to realize the interaction between the Firmware layer and the User layer.

[0062] User layer: The HW Layer layer implements the interaction between the Kernel and User layers, such as the Message Manager, ISP service and other software modules; the Camera HAL layer controls the specific business logic of the Camera.

[0063] Figure 4 Schematic diagram of the deployment of the Camera system image 5A algorithm based on the cross-core framework in the embodiment of the present application, as shown in FIG. Figure 4As shown in the figure, the balanced classification deployment scheme of algorithms and control strategies is as follows: AE and AF control have higher requirements for real-time performance, and their core processing algorithms are deployed in the underlying MCU core control which is closer to the ISP image signal processing hardware system; AWB, ALSC and ATF control have slightly lower requirements for real-time performance and higher requirements for computing power, and their core algorithms are deployed in the AP core, saving MCU core resources and making full use of the comprehensive processing capabilities of the AP core.

[0064] On the upper AP side, Camera HAL is the specific carrier of Camera business functions, and 5A software is its sub-function module. AWB prediction algorithm, ATF prediction algorithm and ALSC prediction algorithm are deployed, and each prediction algorithm is used to predict the second image parameter of the next frame image. AE channel algorithm and AF channel algorithm are also deployed. Each channel algorithm is used to transmit the configuration files required to execute the AE prediction algorithm and the AF prediction algorithm to the MCU side, and is also used to receive the first statistical data transmitted from the MCU side for debugging and analysis.

[0065] On the bottom MCU side, the AE prediction algorithm and the AF prediction algorithm are deployed, and each prediction algorithm is used to predict the first image parameter of the next frame image. It is also used to receive the algorithm results (second image parameters) of the AWB prediction algorithm, the ATF prediction algorithm, and the ALSC prediction algorithm transmitted from the AP side. Under the unified frame parameter matching of the bottom IQC module, the hardware register parameters related to the Sensor are reconfigured according to the first image parameter and the second image parameter.

[0066] Based on the above embodiments, the embodiments of the present application take into account that the sensor is equipped with Digital Overlap-High-Dynamic Range (DOL-HDR) technology, and outputs multiple frames of images with different exposure times in a "quasi-simultaneous" manner. By performing image fusion processing on multiple frames of images with different exposure times, an HDR image is generated. Based on this, the present application further provides an image processing method, Figure 5 : is a second flow chart of the image processing method in the embodiment of the present application, such as Figure 5 As shown, the image processing method specifically includes:

[0067] Step 501: inputting N exposure images corresponding to the current frame image into N second-type statistical modules through the MCU to obtain N second statistical data; 1≤N≤3.

[0068] Step 502: inputting N exposure images corresponding to the current frame image into N first-type statistical modules through the MCU to obtain N first statistical data; the real-time requirement of the first statistical data is higher than the real-time requirement of the second statistical data.

[0069] Here, Table 1 is a schematic diagram showing the relationship between the Sensor DOL type, the statistical module, and the number of buffers.

[0070]

[0071] As shown in Table 1, when the DOL type of the sensor is QBIN, one frame image corresponds to one exposure image, then the number N of the second type of statistical modules is equal to 1, and the number N of the first type of statistical modules is equal to 1; when the DOL type of the sensor is 2DOL, one frame image corresponds to two exposure images, then the number N of the second type of statistical modules can be 1 or 2, and the number N of the first type of statistical modules can be 1 or 2 (N can be set according to specific circumstances); when the DOL type of the sensor is 3DOL, one frame image corresponds to two exposure images, then the number N of the second type of statistical modules can be 1 or 2 or 3, and the number N of the first type of statistical modules can be 1 or 2 or 3 (N can be set according to specific circumstances); when the DOL type of the sensor is 4DOL, one frame image corresponds to two exposure images, then the number N of the second type of statistical modules can be 1 or 2 or 3, and the number N of the first type of statistical modules can be 1 or 2 or 3 (N can be set according to specific circumstances).

[0072] It should be noted that, according to the Sensor DOL type on the market, no matter which type, due to the limitation of Camera ISP hardware resources, the Camera hardware provides at most 3 statistical modules for each A (Example 5A statistical processing) for processing long exposure, medium exposure (or medium-long exposure, medium-short exposure), short exposure, etc. from the image.

[0073] Step 503: predicting the first image parameters of the next frame image according to the N first statistical data of the current frame image through the MCU.

[0074] In the embodiment of the present application, the first image parameter of the next frame image predicted by the MCU according to the first statistical data output by the AE statistical module includes the exposure time. The first image parameter of the next frame image predicted by the MCU according to the first statistical data output by the AF statistical module includes the step length.

[0075] In some embodiments, predicting the first image parameter of the next frame image according to N first statistical data of the current frame image by the MCU includes:

[0076] Predicting N second sub-image parameters of the next frame image according to the N first statistical data of the current frame image;

[0077] The N second sub-image parameters are fused to obtain the first image parameters.

[0078] That is to say, the MCU predicts N second sub-image parameters of the next frame image according to N first statistical data of the current frame image, and then performs fusion processing on the N second sub-image parameters to predict the first image parameters of the next frame image.

[0079] Step 504: Obtain N second statistical data of the current frame image through the AP, and predict the second image parameters of the next frame image according to the N second statistical data of the current frame image.

[0080] In the embodiment of the present application, the second image parameter of the next frame image predicted by the MCU according to the second statistical data output by the AWB statistical module includes a color temperature value. The second image parameter of the next frame image predicted by the MCU according to the second statistical data output by the ATF statistical module includes a fluorescent lamp flicker frequency. The second image parameter of the next frame image predicted by the MCU according to the second statistical data output by the ALSC statistical module includes a shadow parameter.

[0081] In some embodiments, the method further includes: caching the N second statistical data into N first buffers corresponding to the N first buffer addresses respectively through the MCU;

[0082] transmitting the N first buffer addresses to the AP;

[0083] The acquiring, by the AP, N second statistical data of the current frame image includes:

[0084] The N second statistical data cached in the N first buffers are obtained through the AP according to the N first buffer addresses.

[0085] Here, the MCU caches N second statistical data in N first buffers according to the N first buffer addresses, and transmits the N first buffer addresses to the AP, so that the AP obtains the N second statistical data cached in the N first buffers according to the N first buffer addresses.

[0086] In a feasible implementation, the MCU and the AP pre-agreed to configure N first buffer addresses for the second statistics module, where N is 1, 2, or 3.

[0087] In another feasible implementation manner, the N first buffer addresses configured for the second statistical module are configured on the AP side and transmitted to the MCU.

[0088] Based on this, in some embodiments, it also includes:

[0089] Sending a resource allocation request to the AP through the MCU;

[0090] The AP responds to the resource allocation request, configures N first buffer addresses corresponding to the second type of statistical modules, and transmits them to the MCU.

[0091] It should be noted that the Camera hardware provides up to 3 statistical modules for each A (example 5A statistical processing) to process long exposure, medium exposure (or medium-long exposure, medium-short exposure), short exposure, etc. from the image. Therefore, for example, the 5A statistical module corresponding to each frame of the image can be configured with up to 3 buffers to meet all modes, and there is no need to consider the re-application, loading, distribution and configuration of the buffer when switching the DOL mode. If different buffer management is performed for different DOLs, cross-core interactive APs and MCUs are prone to timing mismatches when dynamically processing different buffer configurations. Considering that the current DDR memory resources are relatively abundant, a balanced consideration is taken, and "space" is exchanged for "security and stability."

[0092] In some embodiments, predicting the second image parameter of the next frame image according to N second statistical data of the current frame image by the MCU includes:

[0093] Predicting N first sub-image parameters of the next frame image according to the N second statistical data of the current frame image;

[0094] The N first sub-image parameters are fused to obtain the second image parameters.

[0095] That is to say, the MCU predicts N first sub-image parameters of the next frame image according to N second statistical data of the current frame image, and then performs fusion processing on the N first sub-image parameters to predict the second image parameters of the next frame image.

[0096] In some embodiments, it also includes:

[0097] Acquire N first statistical data of the current frame image through the AP;

[0098] The N first statistical data and the N second statistical data are cached in a preset storage space through the AP for debugging and analysis.

[0099] In some embodiments, the method further includes: caching the N first statistical data into N second buffers corresponding to the N second buffer addresses respectively through the MCU;

[0100] transmitting the N second buffer addresses to the AP;

[0101] The acquiring, by the AP, N first statistical data of the current frame image includes:

[0102] The N first statistical data cached in the N second buffers are obtained through the AP according to the N second buffer addresses.

[0103] Here, the MCU caches the first statistical data in N second buffers according to the N second buffer addresses, and transmits the N second buffer addresses to the AP, so that the AP obtains the N first statistical data cached in the second buffers according to the N second buffer addresses.

[0104] In a feasible implementation, the MCU and the AP pre-agreed to configure N second buffer addresses for the first statistics module, where N is 1, 2, or 3.

[0105] In another feasible implementation manner, the N second buffer addresses configured for the first statistical module are configured on the AP side and transmitted to the MCU.

[0106] Based on this, in some embodiments, it also includes:

[0107] Sending a resource allocation request to the AP through the MCU;

[0108] The AP responds to the resource allocation request, configures N second buffer addresses corresponding to the first type of statistical modules and transmits them to the MCU.

[0109] It should be noted that when shooting with a camera, different cameras are often working at the same time, switching back and forth, and various scenes are changing dynamically. Therefore, the control of various statistical data on the AP side must also be flexible and dynamically created or destroyed. Only in this way can the flexibility of the control strategy and the timely allocation or release of resources be guaranteed.

[0110] Based on this, in some embodiments, a group of nodes is created for the first sensor at the AP in the Camera HAL layer, including at least a source node and a first statistical node;

[0111] The acquiring, by the AP according to the N first buffer addresses, N second statistical data cached in the N first buffers includes:

[0112] Allocating the N first buffer addresses to the first statistics node through the first output port of the source node;

[0113] The N second statistical data cached in the N first buffers are obtained through the first statistical node according to the N first buffer addresses.

[0114] Here, the source node is used to allocate N first buffer addresses to the corresponding first statistical nodes through the first output port; the first statistical node is used to obtain N second statistical data cached in the N first buffers according to the N first buffer addresses, and predict the second image parameters of the next frame image based on the N second statistical data of the current frame image.

[0115] It should be noted that the number of source nodes is 1, the number of first statistical nodes is consistent with the number of second-type statistical modules, the number of first output ports of source nodes is consistent with the number of second-type statistical modules, and the number of first buffer addresses is consistent with the number of first statistical nodes. Figure 6 When the second type of statistical modules may include an AWB statistical module, an ATF statistical module and an ALSC statistical module, a group of nodes including a first statistical node 1, a first statistical node 2 and a first statistical node 3 are created for the first sensor, wherein the source node is connected to the first statistical node 1 through the first output port 1, the source node is connected to the first statistical node 2 through the first output port 2, and the source node is connected to the first statistical node 3 through the first output port 3.

[0116] Combination Figure 6 As shown, N first buffer addresses 1 are allocated to the first statistical node 1 through the first output port 1 of the source node, and N second statistical data 1 cached in the N first buffers 1 are obtained through the first statistical node 1 according to the N first buffer addresses 1, and then the N second statistical data 1 are input into the AWB prediction algorithm through the first statistical node 1 to predict the second image parameter 1. N first buffer addresses 2 are allocated to the first statistical node 2 through the first output port 2 of the source node, and N second statistical data 2 cached in the N first buffers 2 are obtained through the first statistical node 2 according to the N first buffer addresses 2, and then the N second statistical data 2 are input into the ATF prediction algorithm through the first statistical node 2 to predict the second image parameter 2. N first buffer addresses 3 are allocated to the first statistical node 3 through the first output port 3 of the source node, and N second statistical data 3 cached in the N first buffers 3 are obtained through the first statistical node 3 according to the N first buffer addresses 3, and then the N second statistical data 3 are input into the ALSC prediction algorithm through the first statistical node 3 to predict the second image parameter 3.

[0117] In some embodiments, the method further includes: caching the N second statistical data in a preset storage space through the first statistical node.

[0118] Here, N second statistical data are obtained from a preset storage space through a source node, and the N second statistical data are used for debugging and analysis.

[0119] In some embodiments, the method further includes: returning the N first buffers to the buffer pool.

[0120] That is to say, the N first buffers are returned to the buffer pool to avoid invalid occupation of memory resources.

[0121] Accordingly, in some embodiments, a group of nodes created in the AP for the first sensor also includes a second statistical node;

[0122] The acquiring, by the AP according to the N second buffer addresses, N first statistical data cached in the N second buffers includes:

[0123] Allocating the N second buffer addresses to the second statistics node through the second output port of the source node;

[0124] The N first statistical data cached in the N second buffers are obtained through the second statistical node according to the N second buffer addresses.

[0125] Here, the second statistics node is used to obtain N first statistics data cached in the N second buffers according to the N second buffer addresses, and to predict the first image parameters of the next frame image according to the N first statistics data of the current frame image.

[0126] It should be noted that the number of the second statistical nodes is consistent with the number of the first type of statistical modules, and the number of the second output ports of the source node is consistent with the number of the first type of statistical modules. Figure 6 When the first type of statistical module may include an AE statistical module and an AF statistical module, a group of nodes created for the first sensor also includes a second statistical node 1 and a second statistical node 2, wherein the source node is connected to the second statistical node 1 through the second output port 1, and the source node is connected to the second statistical node 2 through the second output port 2.

[0127] Combination Figure 6 As shown, N second buffer addresses 1 are allocated to second statistical node 1 through the second output port 1 of the source node, N first statistical data 1 cached in N second buffers 1 are obtained through the second statistical node 1 according to the N second buffer addresses 1, and then the N first statistical data 1 are input into the AE prediction algorithm through the second statistical node 1 to predict the first image parameter 1. N second buffer addresses 2 are allocated to second statistical node 2 through the second output port 2 of the source node, N first statistical data 2 cached in N second buffers 2 are obtained through the second statistical node 2 according to the N second buffer addresses 2, and then the N first statistical data 2 are input into the AF prediction algorithm through the second statistical node 2 to predict the first image parameter 2.

[0128] In some embodiments, the method further includes: caching the N first statistical data in a preset storage space through the second statistical node.

[0129] Here, N first statistical data are obtained from a preset storage space through a source node, and the N first statistical data are used for debugging and analysis.

[0130] In some embodiments, the method further includes: returning the N second buffers to the buffer pool.

[0131] That is to say, the N second buffers are returned to the buffer pool to avoid invalid occupation of memory resources.

[0132] In some embodiments, the further step further includes: when the second sensor is enabled, creating a group of nodes for the second sensor on the AP.

[0133] That is to say, every time a sensor is enabled, a group of nodes is dynamically created on the AP side for the currently enabled sensor, thus ensuring the flexibility of the control strategy and the timely allocation of resources.

[0134] In some embodiments, the method further includes: when it is detected that a third sensor is in a closed state, destroying a group of nodes created for the third sensor; wherein the third sensor is the first sensor or the second sensor.

[0135] That is to say, when it is detected that an enabled sensor is in the off state, a group of nodes created for the sensor are dynamically destroyed. This ensures the flexibility of the control strategy and the timely release of resources.

[0136] Step 505: Obtain the second image parameters through the MCU, and reconfigure the configuration parameters of the first type of statistical modules according to the first image parameters, and reconfigure the configuration parameters of the second type of statistical modules according to the second image parameters.

[0137] In a feasible implementation manner, the second image parameters of the next frame image are transmitted to the MCU through the AP, so that the MCU obtains the second image parameters of the next frame image.

[0138] Here combined Figure 6 , reconfigure the configuration parameters of the AE statistical module according to the first image parameter 1 through the MCU, reconfigure the configuration parameters of the AF statistical module according to the first image parameter 2 through the MCU, reconfigure the configuration parameters of the AWB statistical module according to the second image parameter 1 through the MCU, reconfigure the configuration parameters of the ATF statistical module according to the second image parameter 2 through the MCU, and reconfigure the configuration parameters of the ALSC statistical module according to the second image parameter 3 through the MCU, so that the image effect of the next frame image can be improved.

[0139] Figure 75A is a schematic diagram of the configuration and process control of the Camera system Buffer based on the cross-core framework in the embodiment of the present application. Figure 7 As shown, when the mobile phone camera is turned on, the relevant process runs as follows:

[0140] First, the APP&FWK layer creates a preview session (Create Previw Session) and sends it to the Camera HAL for stream configuration (Configure Streams). The Camera HAL then sends a Connect Sensor command, which is passed to the Firmware layer to execute the Connect Sensor command to implement the initial configuration of the image sensor and return ACK to inform that the configuration is successful.

[0141] Secondly, the APP&FWK layer sets the repeating request (Set Repeating Request) and sends it to the Camera HAL layer to process the preview request first (Process Preview Request-First). The Camera HAL then sends the HAL pipeline configuration (Pipeline Congfiguration) to prepare the resources and configuration parameters required for the ISP hardware pipeline to run, including at least Buffer resource allocation.

[0142] When allocating buffer resources, it is necessary to consider that when the subsequent digital overlap DOL is switched, one frame of image corresponds to at least one exposure image. The change in the number of exposure images causes the number of one statistical module (such as AE statistical module) to change, and the number of corresponding buffers also changes. Therefore, it is necessary to apply for and allocate the required number of buffers in advance to avoid errors caused by untimely synchronization between the AP side and the MCU side. Return them in time after use to avoid invalid occupation of memory resources.

[0143] According to the types of Sensor DOLs on the market, no matter which type, due to the limitation of Camera ISP hardware resources, Camera hardware provides up to 3 statistical modules for each A (example 5A statistical processing) to process long exposure, medium exposure (or medium-long exposure, medium-short exposure), short exposure, etc. from the image. Therefore, for example, for each frame of the image, the 5A statistical module is configured with up to 3 buffers, which can meet all modes, and there is no need to consider the re-application, loading, distribution and configuration of the buffer when the DOL mode is switched. If different buffers are managed for different DOLs, cross-core interactive APs and MCUs are prone to timing mismatches when dynamically processing different buffer configurations. Considering that the current DDR memory resources are relatively abundant, a balanced consideration is taken, and "space" is exchanged for "security and stability".

[0144] Based on this, for each frame of image collected by the first sensor, three groups of 5A statistical modules (AE, AF, AWB, ATF and ALSC) are set in the ISP, and five buffers are allocated for each group of 5A statistical modules. Figure 7 Also shown is the sixth Buffer, which is used to cache metadata MetaData. This parameter will be used in the process of predicting the same image parameters of the next frame.

[0145] After receiving the HAL pipeline configuration command, the Firmware executes the HAL pipeline configuration command and uses the five buffers allocated based on each group of 5A statistical modules to cache the statistical data output by a group of 5A statistical modules of the Hardware layer ISP.

[0146] Combination Figure 7 If the DOL type of the sensor is 4DOL, that is, it corresponds to 4 exposure images, here you can select 3 exposure images to input 3 identical statistical modules (for example, any one of the 5A statistical modules), and cache them in the 3 buffers allocated for the statistical module. If the DOL type of the sensor is 2DOL, that is, it corresponds to 2 exposure images, here you can select 2 exposure images to input 2 identical statistical modules (for example, any one of the 5A statistical modules), and cache them in 2 of the 3 buffers allocated for the statistical module, so that the remaining buffer is vacant.

[0147] Figure 8 FIG. 1 is a schematic diagram of implementing the release of buffers between different cores based on the AP core and the MCU core ACK mechanism in an embodiment of the present application. Figure 8 As shown, the 5A statistics module is used as an example for explanation:

[0148] The AP core sends a cluster of related buffers to the underlying MCU core for periodic storage of various statistical data according to the flow of each frame of the image. The return of the buffer is guaranteed by the ACK mechanism of the upper and lower layers to ensure the safe return, release and flow of the 5A statistical data buffer.

[0149] After the HardWare layer generates various statistical data and stores them in the corresponding Buffer, the MCU core Firmware packages the AWB / ATF / ALSC Buffer address and the Preview data metadata, sends an Online Request Response ACK command to the Camera HAL, enters the Process preview process (for details, refer to the 5A software control of the Node dynamic node), further performs algorithm processing, and returns the Buffer to the BufferPool.

[0150] The MCU core firmware obtains the AE / AF Buffer and inputs it into the AE / AF prediction algorithm. After the processing is completed, the AE / AF Buff is further returned to the Camera HAL through the Online Request Response ACK command to execute this command and enter the process preview process (for details, refer to the 5A software control of the Node dynamic node). No further algorithm processing is required. After the debugging requirements are met, the Buffer is returned to the Buffer Pool.

[0151] In order to implement the method of the embodiment of the present application, based on the same inventive concept, an image processing device is also provided in the embodiment of the present application. Fig. 9 Schematic diagram of the composition structure of the image processing device in the embodiment of the present application. Fig. 9 As shown, the image signal processing module of the terminal includes a first type of statistical module and a second type of statistical module, and the image processing device 90 includes:

[0152] The processing unit 901 is configured to input the current frame image collected by the first sensor into the first type statistical module through the microcontroller MCU to obtain first statistical data; and input the current frame image into the second type statistical module through the MCU to obtain second statistical data; wherein the real-time requirement of the first statistical data is higher than the real-time requirement of the second statistical data; and predict the first image parameter of the next frame image according to the first statistical data of the current frame image through the MCU; and obtain the second statistical data of the current frame image through the AP, and predict the second image parameter of the next frame image according to the second statistical data of the current frame image;

[0153] The configuration unit 902 is used to obtain the second image parameters through the MCU, and reconfigure the configuration parameters of the first type of statistical modules according to the first image parameters, and reconfigure the configuration parameters of the second type of statistical modules according to the second image parameters.

[0154] By adopting the above technical solution, the first statistical data with high real-time requirements are processed by the MCU, and the second statistical data with low real-time requirements are processed by the AP, which can improve the timeliness of image information processing.

[0155] In some embodiments, the processing unit 901 is specifically configured to input N exposure images corresponding to the current frame image into N second-type statistical modules through the MCU to obtain N second statistical data; 1≤N≤3.

[0156] In some embodiments, the processing unit 901 is further configured to predict the N first sub-image parameters of the next frame image according to the N second statistical data of the current frame image;

[0157] The N first sub-image parameters are fused to obtain the second image parameters.

[0158] In some embodiments, the processing unit 901 is further configured to cache the N second statistical data into N first buffers corresponding to the N first buffer addresses respectively through the MCU;

[0159] transmitting the N first buffer addresses to the AP;

[0160] The N second statistical data cached in the N first buffers are obtained through the AP according to the N first buffer addresses.

[0161] In some embodiments, the processing unit 901 is further configured to send a resource allocation request to the AP through the MCU;

[0162] The AP responds to the resource allocation request, configures N first buffer addresses corresponding to the second type of statistical modules, and transmits them to the MCU.

[0163] In some embodiments, a group of nodes is created at the AP for the first sensor, including at least a source node and a first statistical node;

[0164] The processing unit 901 is further configured to distribute the N first buffer addresses to the first statistics node through the first output port of the source node;

[0165] The N second statistical data cached in the N first buffers are obtained through the first statistical node according to the N first buffer addresses.

[0166] In some embodiments, the processing unit 901 is further configured to return the N first buffers to the buffer pool.

[0167] In some embodiments, the processing unit 901 is further configured to, when the second sensor is enabled, create a group of nodes for the second sensor on the AP.

[0168] In some embodiments, the processing unit 901 is further configured to, when detecting that the third sensor is in an off state, destroy a group of nodes created for the third sensor; wherein the third sensor is the first sensor or the second sensor.

[0169] In some embodiments, the processing unit 901 is further configured to obtain first statistical data of the current frame image through the AP;

[0170] The first statistical data and the second statistical data are cached in a preset storage space through the AP for debugging and analysis.

[0171] The present application also provides another image processing device, Fig.10 Schematic diagram of the composition structure of the image processing device in the embodiment of the present application. Fig.10 As shown, the image processing device 100 includes: a processor 1001 and a memory 802 configured to store a computer program that can be run on the processor;

[0172] The processor 1001 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0173] Of course, in practical applications, Fig.10As shown, the components in the image processing device 100 are coupled together via a bus system 1003. It is understood that the bus system 1003 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1003 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.10 Various buses are labeled as bus system 1003.

[0174] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0175] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0176] In an exemplary embodiment, the present application also provides a computer-readable storage medium for storing a computer program.

[0177] Optionally, the computer-readable storage medium can be applied to any one of the methods in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the processor in each method in the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0178] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0179] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0180] In addition, all functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art may understand that all or part of the steps of implementing the above method embodiments may be completed by hardware related to program instructions, and the above program may be stored in a computer-readable storage medium, which, when executed, executes the steps of the above method embodiments; and the above storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0181] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0182] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0183] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0184] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image processing method, applied to a terminal, characterized in that: The image signal processing module of the terminal includes a first type of statistical module and a second type of statistical module; the method includes: Inputting the current frame image collected by the first sensor into the first type statistical module through the microcontroller MCU to obtain first statistical data; and inputting the current frame image into the second type statistical module through the MCU to obtain second statistical data; wherein the real-time requirement of the first statistical data is higher than the real-time requirement of the second statistical data; Predicting, by the MCU, a first image parameter of a next frame image according to first statistical data of the current frame image; Acquire second statistical data of the current frame image through the application processor AP, and predict second image parameters of the next frame image according to the second statistical data of the current frame image; The second image parameters are obtained through the MCU, and the configuration parameters of the first type of statistical modules are reconfigured according to the first image parameters, and the configuration parameters of the second type of statistical modules are reconfigured according to the second image parameters.

2. The method according to claim 1, characterized in that: The step of inputting the current frame image into the second type statistical module through the MCU to obtain second statistical data includes: The N exposure images corresponding to the current frame image are input into N second-type statistical modules through the MCU to obtain N second statistical data; 1≤N≤3.

3. The method according to claim 2, characterized in that The predicting the second image parameter of the next frame image according to the second statistical data of the current frame image includes: Predicting N first sub-image parameters of the next frame image according to the N second statistical data of the current frame image; The N first sub-image parameters are fused to obtain the second image parameters.

4. The method according to claim 2, characterized in that: Also includes: The N second statistical data are cached respectively into N first buffers corresponding to the N first buffer addresses by the MCU; transmitting the N first buffer addresses to the AP; The obtaining, by using the application processor AP, second statistical data of the current frame image includes: The N second statistical data cached in the N first buffers are obtained through the AP according to the N first buffer addresses.

5. The method according to claim 4, characterized in that Also includes: Sending a resource allocation request to the AP through the MCU; The AP responds to the resource allocation request, configures N first buffer addresses corresponding to the second type of statistical modules, and transmits them to the MCU.

6. The method according to claim 4, characterized in that Creating a group of nodes for the first sensor at the AP, including at least a source node and a first statistical node; The acquiring, by the AP according to the N first buffer addresses, N second statistical data cached in the N first buffers includes: Allocating the N first buffer addresses to the first statistics node through the first output port of the source node; The N second statistical data cached in the N first buffers are obtained through the first statistical node according to the N first buffer addresses.

7. The method according to claim 6, characterized in that Also includes: Return the N first buffers to the buffer pool.

8. The method according to claim 6, characterized in that Also includes: When the second sensor is enabled, a group of nodes is created at the AP for the second sensor.

9. The method according to claim 8, characterized in that Also includes: When it is detected that the third sensor is in an off state, a group of nodes created for the third sensor is destroyed; wherein the third sensor is the first sensor or the second sensor.

10. The method according to claim 1, characterized in that Also includes: Acquire first statistical data of the current frame image through the AP; The first statistical data and the second statistical data are cached in a preset storage space through the AP for debugging and analysis.

11. An image processing device, characterized in that: The image processing device comprises: a processor and a memory configured to store a computer program that can be run on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 10 when running the computer program.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images