Image output control method, device and electronic equipment

Abstract

The application provides an image output control method and device and electronic equipment. The method comprises the following steps: in response to a calling instruction of a target application to a first image acquisition module, providing a target image for the target application according to an image output by a second image acquisition module; after the first image acquisition module is started and completed, providing a target image for the target application according to an image output by the first image acquisition module; wherein the second image acquisition module works in a normal open power domain.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image output control method, apparatus and electronic device. Background Technology

[0002] In the existing camera startup scheme, the main camera needs to go through multiple stages from receiving the call command to outputting a valid image, such as sensor startup, initialization pipeline, automatic exposure, automatic white balance, automatic focus convergence, etc., which takes a long time. As a result, users have to wait a long time to see the preview screen after opening the camera application. Summary of the Invention

[0003] In view of the above, this application provides an image output control method, apparatus, and electronic device, as follows:

[0004] An image output control method, comprising:

[0005] In response to a call command from a target application to the first image acquisition module, a target image is provided to the target application based on the image output by the second image acquisition module;

[0006] After the first image acquisition module is started, a target image is provided to the target application based on the image output by the first image acquisition module;

[0007] The second image acquisition module operates in the normally open power domain.

[0008] Optionally, the above method provides a target image for the target application based on the image output by the second image acquisition module, including:

[0009] Determine whether the historical image and the image output by the second image acquisition module meet the feature similarity condition; the historical image is the image recorded before the first image acquisition module was last turned off.

[0010] In response to the historical image and the image output by the second image acquisition module satisfying the feature similarity condition, the target image is obtained based on the historical image;

[0011] The target image is provided to the target application.

[0012] Optionally, the target image can be obtained from the historical images, as described above, including one of the following:

[0013] Based on the image output by the second image acquisition module, the historical image is processed to obtain the target image;

[0014] Use the historical image as the target image.

[0015] Optionally, the above method involves processing the historical image based on the image output by the second image acquisition module to obtain the target image, including:

[0016] Obtain at least one difference attribute on the target object between the historical image and the image output by the second image acquisition module;

[0017] Based on the difference attributes, the corresponding object display parameters of the target object in the historical image are adjusted to obtain the target image.

[0018] Optionally, the above method provides a target image for the target application based on the image output by the second image acquisition module, including:

[0019] The image output by the second image acquisition module is provided as the target image to the target application.

[0020] Optionally, the above method provides a target image for the target application based on the image output by the second image acquisition module, including:

[0021] According to the image request parameters of the target application, the image output by the second image acquisition module is processed to obtain the target image;

[0022] Wherein, the image display parameters of the target image match the request display parameters in the image request parameters;

[0023] The target image is provided to the target application.

[0024] Optionally, the above method provides a target image to the target application based on the image output by the first image acquisition module, including:

[0025] Determine whether the first image output by the first image acquisition module and the second image output by the second image acquisition module meet the condition of similar effect; the first image and the second image correspond to the same timestamp;

[0026] In response to the first image and the second image satisfying the effect similarity condition, the image output by the first image acquisition module is provided as the target image to the target application;

[0027] In response to the first image and the second image not satisfying the effect similarity condition, the image output by the second image acquisition module is provided to the target application as the target image, and the image acquisition control parameters of the first image acquisition module or the second image acquisition module are adjusted until the first image and the second image satisfy the effect similarity condition.

[0028] Optionally, the similarity condition in the above method includes: the difference between the effect values ​​of the first image and the second image in at least one effect dimension is less than or equal to a similarity threshold;

[0029] The effect dimensions include at least one of the following: focus dimension, exposure dimension, and white balance dimension.

[0030] An image output control device, comprising:

[0031] The first providing unit is configured to, in response to a call instruction from a target application to a first image acquisition module, provide a target image to the target application based on an image output by a second image acquisition module;

[0032] The second providing unit is configured to provide a target image to the target application based on the image output by the first image acquisition module after the first image acquisition module is started.

[0033] The second image acquisition module operates in the normally open power domain.

[0034] An electronic device, comprising:

[0035] First image acquisition module;

[0036] Second image acquisition module;

[0037] The controller is configured to respond to a call command from a target application to the first image acquisition module, and provide a target image to the target application based on the image output by the second image acquisition module; after the first image acquisition module has been started, it provides a target image to the target application based on the image output by the first image acquisition module.

[0038] The second image acquisition module operates in the normally open power domain. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart illustrating an image output control method provided in this application embodiment;

[0041] Figure 2 This is a partial flowchart of an image output control method provided in an embodiment of this application;

[0042] Figure 3This is another part of the flowchart of an image output control method provided in an embodiment of this application;

[0043] Figure 4 This is another part of a flowchart of an image output control method provided in an embodiment of this application;

[0044] Figure 5 This is a partial flowchart of an image output control method provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of the structure of an image output control device provided in an embodiment of this application;

[0046] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0047] Figure 8 This is an example diagram illustrating how this application can output images in a mobile phone scenario to save the time from sensor startup to outputting a valid frame.

[0048] Figure 9 This is an example image of the image correction method used in this application for mobile phones;

[0049] Figure 10 This application provides a data flow architecture diagram for cold startup of a camera, including the AON secondary camera and the main camera, in the scenario applicable to mobile phones.

[0050] Figure 11 This is an example diagram illustrating frame interpolation processing in a scenario applicable to mobile phones, as described in this application.

[0051] Figure 12 This is another data flow architecture diagram for camera cold start, including AON secondary camera and main camera, in the scenario applicable to mobile phones in this application;

[0052] Figure 13 This is a flowchart illustrating the cold start process for a camera, including both the AON secondary camera and the main camera, in a scenario applicable to mobile phones. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] refer to Figure 1This is a flowchart illustrating the implementation of an image output control method provided in this application embodiment. This method can be applied to electronic devices equipped with a first image acquisition module and a second image acquisition module. The electronic device can be a smartphone, tablet, wearable device, or vehicle-mounted terminal, or other terminal device with image acquisition capabilities. The technical solution in this embodiment utilizes the second image acquisition module, operating in the normally open power domain, to provide images to the target application during the startup of the first image acquisition module. This helps to eliminate the waiting time for the first image acquisition module from startup to outputting a valid frame, allowing the user to see the image data more quickly.

[0055] Specifically, the method in this embodiment may include the following steps:

[0056] Step 101: In response to the target application's call command to the first image acquisition module, provide the target image to the target application based on the image output by the second image acquisition module.

[0057] The target application can be a camera application, video call application, QR code scanning application, or social application, or any other application that needs to capture or display images. The target application's instruction to the first image acquisition module refers to a control instruction used to request the activation of the first image acquisition module to acquire images.

[0058] Specifically, the first image acquisition module may include a main imaging sensor and its associated links in the electronic device for providing high-resolution, high-quality image output. The second image acquisition module may include an auxiliary imaging sensor and its associated links in the electronic device for low-power, standby acquisition. The second image acquisition module operates in an always-on (AON) power domain. Here, the always-on power domain refers to a power management area that maintains at least partial hardware power supply even in normal standby or low-power states of the electronic device. The always-on power domain enables the second image acquisition module to perform low-power standby sampling in the background. The always-on power domain ensures that the second image acquisition module and its associated miniature image signal processor remain in a low-power background standby sampling state when there is no camera call, and switch to a pre-startup state for full image output upon detecting a camera call intent.

[0059] For example, the first image acquisition module and the second image acquisition module are two physically independent camera modules. The second image acquisition module is a low-resolution, low-power, low-latency, medium-frame-rate auxiliary camera, while the first image acquisition module is a high-resolution, high-quality, high-bandwidth, slow-starting main camera. The first and second image acquisition modules differ significantly in attributes such as image output resolution, frame rate, and first-frame latency. For instance, there is a difference of several to tens of times in image output resolution, a difference of 100% in frame rate during stable operation, and an order of magnitude difference in first-frame latency.

[0060] Based on this, the controller in the electronic device responds to the call command and provides a target image to the target application based on the image output by the second image acquisition device. The controller can be the central processing unit (CPU) in the electronic device. For example, after the second image acquisition module outputs an image, the electronic device acquires the image output by the second image acquisition module and provides the target image to the target application based on that image. Correspondingly, the target application outputs the target image. Alternatively, the target application performs subsequent processing on the target image after acquiring it.

[0061] The target image can be output in the preview or display interface of the target application to be presented to the user. For example, after the electronic device obtains the image output by the second image acquisition module, it can directly use the image as the target image, or it can perform processing such as resolution alignment, color format conversion, cropping, or super-resolution on the image before using it as the target image and then providing it to the target application.

[0062] It should be noted that after receiving the call command, the electronic device provides the target image to the target application based on the image output by the first image acquisition module, and at the same time performs power-on initialization and other processes to start the first image acquisition module. Meanwhile, because the second image acquisition module operates in the normally open power domain, the time delay between receiving the call command and providing the target image to the target application based on the image output by the second image acquisition module is low.

[0063] As can be seen, during the startup of the first image acquisition module, the target image can be provided to the target application in advance through the second image acquisition module without waiting for the first image acquisition module to finish starting up, thereby reducing the time for users to wait to see the image in the target application, such as the preview screen.

[0064] Step 102: After the first image acquisition module is started, the target image is provided to the target application based on the image output by the first image acquisition module.

[0065] The completion of the first image acquisition module startup refers to the engineering state in which the first image acquisition module and its supporting main image signal processor complete the processes of sensor startup, pipeline initialization, and 3A convergence. Correspondingly, after these processes are completed, the first image acquisition module is in a state capable of outputting one valid image frame. At this point, the first image acquisition module startup is complete. The 3A here includes: automatic exposure (AE), automatic white balance (AWB), and automatic focus (AF). The 3A convergence state refers to the state where exposure convergence (i.e., stable exposure time and ISO parameters), exposure convergence (e.g., stable gain values ​​in the red, green, and blue channels), and passive focus completion (e.g., a fixed lens position resulting in a clear image) are achieved.

[0066] For example, the first image acquisition module needs to go through several stages from receiving the call command to outputting the image, including sensor activation, register configuration, pixel array warm-up, analog front-end calibration, digital gain establishment, and initial convergence of the 3A loop, taking approximately 160 milliseconds. The second image acquisition module, however, is in a normally-on power domain standby state, and its latency from detecting the call intent (i.e., receiving the call command) to outputting the image is much lower than the aforementioned time. Therefore, after the first image acquisition module has started up, the electronic device switches to providing the target image to the target application based on the image output by the first image acquisition module. This reduces the time the user waits for the target application to output the image and also ensures the image quality of the subsequent images output by the target application.

[0067] As can be seen, in the image output control method provided in the embodiments of this application, by utilizing the second image acquisition module operating in the normally open power domain to provide an image to the target application in advance during the startup of the first image acquisition module, the waiting time from the start-up of the first image acquisition module to the output of a valid frame can be eliminated, thereby significantly shortening the waiting time from the user initiating camera call to seeing the preview screen, and thus improving the user's experience of using electronic devices.

[0068] based on Figure 1 In one implementation of the technical solution shown, in step 101, when providing a target image to the target application based on the image output by the second image acquisition module, the image output by the second image acquisition module can be provided to the target application as the target image.

[0069] Specifically, during the startup of the first image acquisition module, the electronic device does not need to perform additional image processing (such as resolution alignment, super-resolution, etc.) on the image output by the second image acquisition module. The image output by the second image acquisition module can be directly provided to the target application as the target image.

[0070] It should be noted that the implementation method of providing the image output by the second image acquisition module as the target image to the target application in this embodiment can be applied to application scenarios that have high requirements for camera startup speed and relatively lenient requirements for the initial preview image.

[0071] For example, if the target application has low requirements for the preview screen resolution, or if the output resolution of the second image acquisition module can already match the preview window size of the target application, the electronic device can directly provide the image output by the second image acquisition module to the target application. It can be seen that in this embodiment, the image processing process can be omitted to minimize the image processing latency, thereby further shortening the waiting time from when the user starts the camera to when the user sees the preview screen.

[0072] based on Figure 1 In one implementation of the technical solution shown, when providing a target image for the target application based on the image output by the second image acquisition module in step 101, it can be achieved in the following way: Figure 2 As shown:

[0073] Step 201: Process the image output by the second image acquisition module according to the image request parameters of the target application to obtain the target image.

[0074] Step 202: Provide the target image to the target application.

[0075] In this process, the target application generates both a call instruction for the first image acquisition module and corresponding image request parameters. Based on these parameters, the electronic device can obtain both the call instruction and the image request parameters. The image display parameters of the target image match the requested display parameters in the image request parameters.

[0076] It should be noted that image request parameters refer to configuration information used to indicate the target application's requirements for the display format, size, or quality of an image. Specifically, image request parameters may include at least one of the following display parameters: output resolution, aspect ratio, color format, frame rate, and region of interest (ROI). For example, the image request parameters may include a request from the target application for an image with a resolution of 1920×1920.

[0077] Specifically, after the electronic device obtains the image output by the second image acquisition module, it processes the image output by the second image acquisition module according to the image request parameters, including but not limited to: field of view (FOV) distortion correction, center effective area cropping, super-resolution upsampling, color format conversion, and frame rate adaptation, at least one of these.

[0078] For example, the electronic device may first perform distortion correction on the image output by the second image capture module to eliminate geometric deformation caused by lens distortion, then perform central effective region cropping on the corrected image to match the aspect ratio requested by the target application, and finally perform super-resolution upsampling on the cropped image through a super-resolution model to match the output resolution requested by the target application. The super-resolution model may be preferentially deployed on an artificial intelligence hardware accelerator built in a micro image signal processor supporting the second image capture module.

[0079] It should be noted that in this embodiment, the field of view distortion correction and central effective region cropping are completed first, and then super-resolution upsampling is performed on the cropped image, which can reduce the waste of computing power for super-resolution upsampling, and avoid problems such as geometric alignment accuracy degradation and increased bandwidth power consumption.

[0080] It can be seen that in this embodiment, processing the image output by the second image capture module according to the image request parameters of the target application can make the image provided to the target application adapt to the image requirements of the target application, avoiding problems such as picture distortion, insufficient resolution or mismatched aspect ratio caused by directly displaying the unprocessed original image. As a result, the preview picture seen by the user during the activation of the first image capture module is closer to the viewing requirement in visual effect, thereby further improving the user's usage experience of the electronic device.

[0081] Based on Figure 1 the illustrated technical solution, in an implementation manner, in step 101, when providing the target image for the target application according to the image output by the second image capture module, it may be implemented in the following manner, such as Figure 3 shown in:

[0082] Step 301: Determine whether the historical image and the image output by the second image capture module satisfy a feature similarity condition; if the historical image and the image output by the second image capture module satisfy the feature similarity condition, perform step 302 to step 303, otherwise perform step 304.

[0083] The historical image is an image recorded before the first image capture module was last shut down.

[0084] Specifically, after the target application generates a call instruction for the first image capture module, the electronic device can obtain the call instruction, and in response to the call instruction, control the second image capture module to output an image, so that the electronic device can obtain the image output by the second image capture module. Meanwhile, in response to the call instruction, the electronic device also reads the historical image recorded before the first image capture module was last shut down. On this basis, the electronic device can call the feature comparison module to compare the historical image with the image output by the second image capture module, so as to determine whether the historical image and the image output by the second image capture module satisfy the feature similarity condition.

[0085] Among them, the feature similarity condition refers to the judgment criteria used to determine whether the historical image and the image output by the second image acquisition module are consistent in terms of scene content, shooting object or environmental state.

[0086] In one implementation, the feature similarity condition can be: the feature similarity between the historical image and the image output by the second image acquisition module is greater than or equal to a similarity threshold (e.g., 80%), or the vector distance between the image feature vectors generated by the historical image and the image output by the second image acquisition module is less than or equal to a distance threshold.

[0087] Image feature vectors refer to the feature vectors converted from image feature data obtained by feature extraction from historical images or images output by the second image acquisition module. Image feature data can include local salient feature data and global statistical feature data. Local salient feature data is represented by feature point data of at least one feature point in the image, such as the location of the feature point, the direction and intensity of brightness changes of pixels around the feature point, and the descriptor of the feature point. Global statistical feature data can include the image's color histogram features, texture energy features, and depth abstraction features.

[0088] In another implementation, the feature similarity condition can be: among the feature points extracted from the images output by the second image acquisition module of the historical image, the proportion of matching feature points to the total number of feature points exceeds a threshold (e.g., 80%). A feature point refers to a local pixel region description in an image that has significant visual features and can be used for image matching or scene recognition. Matching the first feature point in the historical image with the second feature point in the second image acquisition module means that the first vector distance (e.g., Hamming distance or Euclidean distance) between the descriptors of the first and second feature points is less than a preset distance threshold (specifically, the smallest among the feature points in the second image acquisition module), and the ratio between the first and second vector distances is less than a preset threshold (for removing false positives). The second vector distance is the vector distance between the first feature point and the third feature point with the second smallest vector distance in the image output by the second image acquisition module.

[0089] For example, when the camera is off, an electronic device saves a historical image. This historical image contains multiple feature points, such as scene content, the subject being photographed, or the environmental state. When the camera is reopened, the electronic device acquires the current image through a second image acquisition module and extracts the feature points from the current image. These feature points are then compared with the feature points in the historical image. After filtering out matching feature points, the percentage of matching feature points is determined. If the percentage of matching feature points exceeds 80% of the total number of feature points, it can be determined that the scene being photographed has not changed significantly. Therefore, it can be confirmed that the historical image and the image output by the second image acquisition module meet the feature similarity condition, and thus the historical image can be reused. In this embodiment, by setting the above thresholds, both the image reuse success rate and display reliability are considered, improving camera startup speed while ensuring the stability of the user experience.

[0090] As can be seen, this embodiment can accurately identify whether the current scene is consistent with the historical scene by using the feature similarity condition, avoid misusing historical images when the scene has changed, thus ensuring the accuracy of the scene in the preview.

[0091] Step 302: Obtain the target image based on historical images.

[0092] In one implementation, the electronic device can directly use historical images as the target image.

[0093] In another implementation, the electronic device can process historical images based on the images output by the second image acquisition module to obtain the target image.

[0094] Specifically, the electronic device can obtain at least one difference attribute between the historical image and the image output by the second image acquisition module on the target object; then, based on the difference attribute, it adjusts the corresponding object display parameters of the target object in the historical image to obtain the target image.

[0095] The difference attributes can include: the displacement difference, brightness difference, and hue difference of the target object between the historical image and the image output by the second image acquisition module. Correspondingly, the object display parameters adjusted by the electronic device can include: the displacement, brightness, and hue of the target object in the historical image.

[0096] For example, in response to a planar position offset between the target object in the historical image and the image output by the second image acquisition module, the electronic device can adjust the rendering position coordinates of the target object in the historical image according to the displacement difference corresponding to the planar position offset.

[0097] For example, in response to a brightness difference between the target object in the historical image and the image output by the second image acquisition module, the electronic device can adjust the brightness compensation value of the target object in the historical image according to the brightness difference corresponding to the brightness difference.

[0098] For example, in response to a color temperature shift between the target object in the historical image and the image output by the second image acquisition module, the electronic device can adjust the tone mapping parameters of the target object in the historical image according to the color temperature difference corresponding to the color temperature shift.

[0099] Furthermore, electronic devices can also perform frame interpolation on historical images based on differences in object display parameters and target objects within those images. For example, frame interpolation can be performed on historical images based on displacement differences, brightness differences, and hue differences. The target image after frame interpolation and object display parameter adjustments can include at least one transition image and the historical image itself, with the transition image and the historical image exhibiting a gradual change in displacement, brightness, hue, and other parameters in a temporal sequence.

[0100] Based on this, in this embodiment, frame interpolation is used to output a transition image before outputting the historical image. This can smooth out abrupt changes in displacement, tone, and brightness between the target object in the image output by the second image acquisition module and the target object in the historical image. This can eliminate visual jumps and avoid screen jumps caused by directly providing the historical image as the target image to the target application for output. It can also eliminate the sense of discontinuity in the screen caused by the movement of the target object or changes in the environment.

[0101] Moreover, in this embodiment, the transition image can fill the timing gap before the output image of the second image acquisition module, thereby further filling the timing gap during the startup of the first image acquisition module, so that the preview screen is guaranteed to be continuous and not black screen during the startup of the first image acquisition module.

[0102] Step 303: Provide the target image to the target application.

[0103] Furthermore, after step 303, the image output by the second image acquisition module can be provided as the target image to the target application, such as... Figure 4 As shown in step 305, before providing the target application with the image output by the second image acquisition module as the target image, the target application is first provided with the target image obtained from the historical image, thereby further reducing the delay of the target application outputting the preview screen.

[0104] Step 304: Provide the image output by the second image acquisition module as the target image to the target application.

[0105] As can be seen, in this embodiment, by saving the historical images of the first image acquisition module before it was last closed and comparing them with the images output by the second image acquisition module when it is called again, the historical images can be directly used as the target images, or adjusted for differences, even when the acquisition scene has not changed significantly. This further shortens the processing latency of the second image acquisition module from detecting the calling intent to outputting a usable target image, thus reducing the time for the target application to obtain the target image (such as the time for the user to see the preview or the time to wake up the monitoring camera through the target image) compared to a normal image processing pipeline. Therefore, this embodiment, based on the second image acquisition module using the normally open power domain to output images first, further improves the continuity of image acquisition through historical image reuse and frame interpolation compensation.

[0106] based on Figure 1 In one implementation of the technical solution shown, step 102, when providing a target image to the target application based on the image output by the first image acquisition module, can be achieved in the following way: Figure 5 As shown:

[0107] Step 501: Determine whether the first image output by the first image acquisition module and the second image output by the second image acquisition module meet the effect similarity condition. If the first image and the second image meet the effect similarity condition, proceed to step 502; if the first image and the second image do not meet the effect similarity condition, proceed to step 503.

[0108] In this embodiment, the first image and the second image correspond to the same timestamp. A timestamp is a temporal marker used to identify the moment an image is captured. In this embodiment, the electronic device aligns the two images from the first image acquisition module and the second image acquisition module based on the timestamp to ensure that the first image and the second image correspond to the same shooting moment in time, providing a benchmark for judging subsequent similarity conditions.

[0109] Specifically, an electronic device can use a pre-deployed effect evaluation module to compare a first image and a second image to obtain an evaluation result on whether the first image and the second image meet the effect similarity condition. The effect similarity condition is a criterion used to determine whether the first image output by the first image acquisition module and the second image output by the second image acquisition module are visually similar.

[0110] In one implementation, the effect similarity condition may include: the difference between the effect values ​​of the first image and the second image in at least one effect dimension is less than or equal to a similarity threshold.

[0111] The effect dimensions can include at least one of the following: focus dimension, exposure dimension, and white balance dimension. The effect value for the focus dimension can be an image sharpness value (such as the Laplacian operator variance or gradient magnitude) or a lens position value. The effect value for the exposure dimension can be the average brightness, the exposure value, or the product of ISO and shutter speed. The effect value for the white balance dimension can be the color temperature value, the red-green-blue gain ratio, or the grayscale world deviation value.

[0112] Specifically, the similarity threshold can be a preset fixed value, or it can be dynamically adjusted adaptively based on the current scene conditions such as lighting and gain level. For example, in a low-light, high-gain acquisition scene, the similarity threshold can be appropriately relaxed; in a uniformly lit acquisition scene, the similarity threshold can be tightened to improve the smoothness of switching.

[0113] Step 502: Use the image output by the first image acquisition module as the target image and provide it to the target application.

[0114] Step 503: Use the image output by the second image acquisition module as the target image and provide it to the target application. Adjust the image acquisition control parameters of the first image acquisition module or the second image acquisition module until the first image and the second image meet the condition of similar effect.

[0115] Specifically, the electronic device can send a first adjustment command to the first image acquisition module, so that the first image acquisition module adjusts the corresponding image acquisition control parameters in response to the first adjustment command; or, the electronic device can send a second adjustment command to the second image acquisition module, so that the second image acquisition module adjusts the corresponding image acquisition control parameters in response to the second adjustment command.

[0116] In this embodiment, adjusting the image acquisition control parameters of the first image acquisition module ensures that, before switching to providing the target image based on the image output by the first image acquisition module, the image output by the first image acquisition module is close to the image output by the second image acquisition module. This reduces the visual difference between the two images, achieving a smooth transition between the two image acquisition modules. During this process, the image output effect of the target image output by the target application remains unchanged. Similarly, adjusting the image acquisition control parameters of the second image acquisition module ensures that, before switching to providing the target image based on the image output by the first image acquisition module, the image output by the second image acquisition module is close to the image output by the first image acquisition module. This reduces the visual difference between the two images, achieving a smooth transition between the two image acquisition modules. During this process, the image output effect of the target image output by the target application gradually changes with the adjustment of the image acquisition control parameters of the second image acquisition module.

[0117] Accordingly, after the electronic device adjusts the image acquisition control parameters of the first image acquisition module or the second image acquisition module, it re-compares whether the first image newly output by the first image acquisition module and the second image newly output by the second image acquisition module meet the effect similarity condition until the first image and the second image meet the effect similarity condition.

[0118] Among them, image acquisition control parameters refer to configuration parameters used to control the imaging effect of the image acquisition module. Image acquisition control parameters may include at least one of focus parameters, exposure parameters, white balance parameters, distortion correction parameters, and horizontal deflection correction parameters.

[0119] In one implementation, the focusing parameters refer to the configuration amount that controls the relative distance or refractive state of the lens between the lens optical system and the imaging sensor in the image acquisition module. In this embodiment, the position of the imaging focal point on the optical axis in the image acquisition module can be determined by adjusting the focusing parameters of the image acquisition module. The focusing parameters may include at least one of lens position and focal length. Lens position is the axial displacement of the lens optical center relative to the photosensitive surface of the imaging sensor, which can be measured in steps of a stepper motor or millimeters. Focal length is the equivalent focal length value of the lens optical system.

[0120] For example, if the sharpness value of the first image is lower than that of the second image, the first image acquisition module is controlled to increase the step value of the lens position to shorten the object distance and improve sharpness; if the sharpness value of the first image is higher than that of the second image, the first image acquisition module is controlled to decrease the step value of the lens position.

[0121] In another implementation, exposure parameters refer to the configuration values ​​used to control the amount of light received by the imaging sensor. Exposure parameters may include at least one of ISO sensitivity and shutter speed.

[0122] Specifically, in this embodiment, the exposure parameter adjustment strategy can be determined based on the difference in average brightness between the first image and the second image in the exposure dimension. For example, if the average brightness of the first image is 80 and the average brightness of the second image is 120, and the brightness difference corresponding to the similarity threshold is 20, then the electronic device can control the sensitivity of the first image acquisition module to increase from 100 to 160, or control the shutter speed of the first image acquisition module to increase from 10 milliseconds to 16 milliseconds, so that the average brightness of the first image approaches 120.

[0123] In another implementation, the white balance parameter refers to the configuration amount used to correct the differences in the response of the red, green, and blue channels under different light source color temperatures. The white balance parameter may include at least one of red, green, and blue gain and color temperature. In this embodiment, by adjusting the white balance parameter of the image acquisition module, the overall color cast of the image caused by differences in the spectral distribution of the light source can be eliminated.

[0124] For example, if the color temperature of the first image is 5000K and the color temperature of the second image is 6500K, and the difference between the two is 1500K, which exceeds the similarity threshold of 500K, then the electronic device controls the white balance parameters of the first image acquisition module to increase the red channel gain from 1.0 to 1.2 and decrease the blue channel gain from 1.0 to 0.85, in order to bring the color temperature of the first image closer to 6500K.

[0125] In another implementation, the distortion correction parameter refers to the configuration amount used to correct the geometric deformation introduced by the lens optical system. The distortion correction parameter may include at least one of radial distortion coefficient and tangential distortion coefficient. In this embodiment, by adjusting the distortion correction parameter of the image acquisition module, the bending of image edges or abnormal central magnification caused by lens surface shape and assembly errors can be corrected.

[0126] For example, the second image output by the second image acquisition module has a straight line curvature of 5 pixels at the edge of the image, while the first image output by the first image acquisition module has a straight line curvature of 1 pixel at the edge. The electronic device can adjust the radial distortion coefficient k1 of the second image acquisition module from -0.3 to -0.25 distortion correction parameter to reduce the difference in edge curvature.

[0127] In another implementation, the horizontal deflection correction parameter refers to the amount of configuration used to correct the horizontal coordinate offset of the same world coordinate point in the respective image plane caused by differences in the physical placement of the two image acquisition modules. The horizontal deflection correction parameter may include at least one of horizontal translation and rotation angle. In this embodiment, by adjusting the horizontal deflection correction parameter of the image acquisition modules, horizontal parallax in the image caused by non-parallelism of the module optical axes or differences in assembly pose can be eliminated.

[0128] For example, if the second image is 8 pixels horizontally offset to the right relative to the first image in the center area of ​​the screen, the electronic device controls the horizontal translation of the second image acquisition module to adjust the horizontal deflection correction parameter from +8 pixels to +4 pixels, thereby gradually reducing the horizontal parallax between the two images.

[0129] As can be seen, in this embodiment, after the first image acquisition module is started, the image output is not directly hard switched. Instead, the image acquisition control parameters of the image acquisition module are adjusted according to whether the condition of similar effect is met, so as to gradually reduce the visual difference between the two images, realize the smooth flow switching between the two image acquisition modules, and avoid screen jumps.

[0130] refer to Figure 6This is a schematic diagram of an image output control device provided in an embodiment of this application. This device can be installed in an electronic device equipped with a first image acquisition module and a second image acquisition module. The technical solution in this embodiment utilizes the second image acquisition module, which operates in the normally-on power domain, to provide images to the target application during the startup of the first image acquisition module. This helps to eliminate the waiting time for the first image acquisition module from startup to outputting a valid frame, allowing the user to see the image data more quickly.

[0131] Specifically, the apparatus in this embodiment may include the following units:

[0132] The first providing unit 601 is configured to provide a target image to the target application in response to a call command from the target application to the first image acquisition module, based on the image output by the second image acquisition module;

[0133] The second providing unit 602 is used to provide a target image to the target application based on the image output by the first image acquisition module after the first image acquisition module is started.

[0134] The second image acquisition module operates in the normally open power domain.

[0135] As can be seen, in the image output control device provided in the embodiments of this application, by utilizing the second image acquisition module operating in the normally open power domain to provide an image to the target application in advance during the startup of the first image acquisition module, the waiting time from the start-up of the first image acquisition module to the output of a valid frame can be eliminated, thereby significantly shortening the waiting time from the user initiating camera call to seeing the preview screen, and thus improving the user's experience of using electronic devices.

[0136] In one implementation, the first providing unit 601 is specifically configured to: determine whether a historical image and an image output by a second image acquisition module satisfy a feature similarity condition; the historical image is an image recorded before the first image acquisition module was last closed; in response to the historical image and an image output by the second image acquisition module satisfying the feature similarity condition, obtain a target image based on the historical image; and provide the target image to the target application.

[0137] The first providing unit 601 obtains the target image based on the historical image by one of the following methods:

[0138] Use the historical image as the target image;

[0139] Based on the image output by the second image acquisition module, the historical image is processed to obtain the target image; for example, at least one difference attribute exists on the target object between the historical image and the image output by the second image acquisition module; based on the difference attribute, the corresponding object display parameters of the target object in the historical image are adjusted to obtain the target image.

[0140] In one implementation, the first providing unit 601 is specifically used to: provide the target application with the image output by the second image acquisition module as the target image.

[0141] In one implementation, the first providing unit 601 is specifically used to: process the image output by the second image acquisition module according to the image request parameters of the target application to obtain a target image; wherein the image display parameters of the target image match the request display parameters in the image request parameters; and provide the target image to the target application.

[0142] In one implementation, the second providing unit 602 is specifically configured to: determine whether a first image output by the first image acquisition module and a second image output by the second image acquisition module satisfy an effect similarity condition; the first image and the second image correspond to the same timestamp; in response to the first image and the second image satisfying the effect similarity condition, provide the image output by the first image acquisition module as the target image to the target application; in response to the first image and the second image not satisfying the effect similarity condition, provide the image output by the second image acquisition module as the target image to the target application, and adjust the image acquisition control parameters of the first image acquisition module or the image acquisition control parameters of the second image acquisition module until the first image and the second image satisfy the effect similarity condition.

[0143] The similarity condition includes: the difference between the effect values ​​of the first image and the second image in at least one effect dimension is less than or equal to a similarity threshold; wherein the effect dimension includes at least one of the following: focus dimension, exposure dimension, and white balance dimension.

[0144] It should be noted that the specific implementation of each unit in this embodiment can be referred to the corresponding content above, and will not be described in detail here.

[0145] refer to Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include the following structure:

[0146] First image acquisition module 701 and second image acquisition module 702;

[0147] The controller 703, such as a CPU, provides a target image to the target application based on the image output by the second image acquisition module in response to a call instruction from the target application to the first image acquisition module; and provides the target image to the target application based on the image output by the first image acquisition module after the first image acquisition module has been started.

[0148] It should be noted that the first and second image acquisition modules each have their own physically independent and power-domain isolated image signal processor pipelines, and they do not share the same set of image signal processor hardware pipelines. Only the upper-layer application processor software has unified scheduling logic; the hardware processing paths are completely separated.

[0149] The second image acquisition module operates in the normally-on power domain, enabling it to be in a low-power startup state and directly acquire and output images in response to call commands. The first image acquisition module provides high-resolution, high-quality images after startup.

[0150] As can be seen, in the electronic device provided in this application embodiment, by utilizing the second image acquisition module operating in the normally open power domain to provide an image to the target application in advance during the startup of the first image acquisition module, the waiting time from the start-up of the first image acquisition module to the output of a valid frame can be eliminated, thereby significantly shortening the waiting time from the user initiating camera call to seeing the preview screen, and thus improving the user's experience of using the electronic device.

[0151] Taking a mobile phone equipped with a main camera (first image acquisition module) and an AON secondary camera (second image acquisition module) as an example, the technical solution of this application is illustrated below:

[0152] First, in this application, during the waiting period from the main camera's sensor opening to the output of a valid frame, the image output by the AON secondary camera is returned to the camera application (i.e., the target application) ahead of time. This reduces the time it takes for the user to see the camera's preview compared to a normal image output pipeline. Figure 8 As shown in the image.

[0153] Furthermore, this application compares historical images from when the main camera exits with images from the AON secondary camera. If the features are similar (i.e., the feature similarity condition is met), then the historical image is directly reported to the camera application. Figure 8 As shown, the time it takes for the user to see the camera preview is also reduced compared to a normal pipeline, thus saving the time from sensor startup to outputting a valid frame.

[0154] Based on this, the technical solution of this application may include the following implementation process:

[0155] I. Data reporting process using AON:

[0156] 1. 3A Synchronization: When the data stream from the image output by the AON secondary camera is switched to the main camera, due to hardware differences, the 3A performance of the image will change abruptly, resulting in a poor user experience when viewing the preview screen provided by the camera application. Therefore, the implementation of this application needs to solve the problem of smooth 3A transition. In view of this, for the universality of this application and to reduce the coupling between hardware, a soft synchronization scheme can be adopted, specifically: using timestamps to obtain the 3A status of two frames at the same time, and performing gain mapping, exposure compensation, color temperature alignment, and color correction matrix synchronization.

[0157] (1) Gain Mapping:

[0158] To address the differences in sensor type, lens aperture, and sensitivity curve of the AON secondary camera, a gain mapping table is established to convert the main camera parameters into gain values ​​that can be executed by the AON secondary camera.

[0159] (2) Exposure Compensation (AE Compensation):

[0160] By leveraging the exposure sensitivity characteristics of the AON secondary camera, the shutter speed and ISO output by the main camera are compensated to ensure consistent exposure. For example, if the linearity of the AON secondary camera sensor gain is low, the main camera parameters need to be adjusted to avoid overexposure or underexposure.

[0161] (3) Color temperature alignment:

[0162] The AON secondary camera estimates the ambient color temperature using algorithms such as the grayscale world method and perfect reflection method. The main camera adjusts the gain of the red and blue channels based on the color temperature value output by the AON secondary camera to eliminate color cast.

[0163] (4) Color correction matrix synchronization:

[0164] The Color Correction Matrix (CCM) table index output by the AON secondary camera is used for color correction of the main camera, ensuring color consistency in multi-camera imaging. For example, in scenarios where ultra-wide-angle and main camera images are stitched together, CCM table synchronization can avoid color differences at stitching edges.

[0165] 2. Field of view synchronization:

[0166] Due to factors such as the placement and field of view of the main camera and the AON secondary camera, the two data sources will exhibit inconsistent fields of view and horizontal deflection when in the same coordinate system. Without synchronization, this will result in a poor user experience.

[0167] (1) The relative pose calibration of the main camera and the AON secondary camera needs to be performed.

[0168] (2) Distortion correction can be achieved by using cv2.undistort() to correct the images output by the main camera and the AON sub-camera.

[0169] (3) Epipolar optimization: cv2.stereoRectify() is used to generate a correction mapping table to characterize the mapping relationship between the images output by the main camera and the AON sub-camera.

[0170] After correction is completed, distortion correction parameters and horizontal deflection correction parameters can be generated based on the correction mapping table, such as... Figure 9 As shown, the image is sent to the backend for image processing.

[0171] 3. Super-resolution and resolution alignment:

[0172] Specifically, the resolution is aligned with the image request parameters of the target application. Therefore, super-resolution can narrow the performance difference between the AON secondary camera and the main camera.

[0173] 4. Pipeline data flow:

[0174] like Figure 10 The diagram shown is a data flow architecture diagram for a camera cold start, including the AON secondary camera and the main camera. The specific process includes:

[0175] First, after the target application (APP) sends a camera start request, the AON secondary camera captures an image and sends it to the frame buffer via Direct Memory Access (DMA). The frame buffer image is then processed by field distortion correction, center effective region cropping, and super-resolution upsampling to generate the target image (matching the request display parameters in the image request parameters), which is then provided to the target application (APP) to fill the main camera's startup waiting window.

[0176] Meanwhile, the main camera goes through a startup process including sensor startup, initialization pipeline, and 3A convergence until startup is complete, outputs an image and provides it to the target application to achieve preview screen switching.

[0177] II. Using historical data to assist the reporting process:

[0178] 1. Historical data assistance:

[0179] When the camera exits, a historical image and its feature points from that time are saved. When the camera reopens, the image captured by the AON secondary camera is compared with the saved historical image for feature point analysis. If the feature points are similar (i.e., the feature similarity condition is met), it is assumed that the scene has not changed, and the stored historical image can be directly reported to the target application. Based on the displacement difference between the AON secondary camera images, frame interpolation is performed, such as... Figure 11 As shown in the image.

[0180] 2. Pipeline data flow:

[0181] like Figure 12 The diagram shown is a data flow architecture diagram for a camera cold start, including the AON secondary camera and the main camera. Specifically, it includes the following processes:

[0182] After the target application sends a camera activation request, the AON secondary camera captures an image and sends it to the frame buffer via DMA. At the same time, it compares the image with the historical image recorded when the main camera was last turned off. The historical image is then adjusted using the image output by the AON secondary camera and returned to the application.

[0183] Furthermore, in this embodiment, interpolated frame data, i.e., transition images, are generated based on historical images and reported to the APP to fill the main camera startup delay window.

[0184] Meanwhile, the main camera goes through a startup process including sensor activation, initialization pipeline, and 3A convergence; once the main camera has finished starting up, it outputs an image to the APP to complete the switch of the preview screen.

[0185] Specifically, such as Figure 13 The diagram shown is a flowchart of the camera cold start process, including the AON secondary camera and the main camera. The specific process is as follows:

[0186] After the app initiates the action of opening the camera, it first determines whether the AON secondary camera is currently enabled;

[0187] If the AON secondary camera is not currently enabled or is unavailable, the main camera will be activated in response to the call command, and the image will be provided to the APP after the main camera outputs a valid image.

[0188] If the AON secondary camera is currently enabled, the AON secondary camera is notified to save the data to the specified DMA area via DMA. At the same time, the main camera is notified to execute the startup process without waiting for the result. Furthermore, the open operation is quickly returned, causing the APP to send a request to the camera to quickly start the camera.

[0189] At the same time, after the APP sends the request, it will determine again whether the AON secondary camera is currently enabled;

[0190] If the AON secondary camera is not enabled, wait for the main camera to output an image and send it back to the app;

[0191] If the AON secondary camera is enabled, determine whether the main camera has already output an image. Before the main camera outputs an image, read the image output by the AON secondary camera from the specified DMA area using the timestamp, and determine whether it is close to the historical image in terms of feature point measurement (whether it meets the feature similarity condition).

[0192] If the feature point measurements are not close, the AON image will replace the historical image, and after processing such as super-resolution and cropping, it will be returned to the APP.

[0193] If the feature point measurements are similar, historical image assistance is used. Specifically, the difference attributes between the image output by the AON secondary camera and the historical image are calculated, such as displacement difference. Frame interpolation is performed based on the displacement difference, and after cropping and other processing, the image is returned to the APP.

[0194] If the main camera outputs an image, the image is then obtained from the main camera's path, and processes such as 3A convergence, field distortion correction, super-resolution, and cropping are performed before finally providing the image to the APP to complete the preview screen switching.

[0195] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0196] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0197] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0198] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image output control method, comprising: In response to a call command from a target application to the first image acquisition module, a target image is provided to the target application based on the image output by the second image acquisition module; After the first image acquisition module is started, a target image is provided to the target application based on the image output by the first image acquisition module; The second image acquisition module operates in the normally open power domain.

2. The method according to claim 1, wherein a target image is provided to the target application based on the image output by the second image acquisition module, comprising: Determine whether the historical image and the image output by the second image acquisition module meet the feature similarity condition; The historical image is the image recorded before the first image acquisition module was last turned off; In response to the historical image and the image output by the second image acquisition module satisfying the feature similarity condition, the target image is obtained based on the historical image; The target image is provided to the target application.

3. The method according to claim 2, wherein obtaining the target image based on the historical image includes one of the following: Based on the image output by the second image acquisition module, the historical image is processed to obtain the target image; Use the historical image as the target image.

4. The method according to claim 3, wherein the historical image is processed based on the image output by the second image acquisition module to obtain the target image, comprising: Obtain at least one difference attribute on the target object between the historical image and the image output by the second image acquisition module; Based on the difference attributes, the corresponding object display parameters of the target object in the historical image are adjusted to obtain the target image.

5. The method according to claim 1, wherein a target image is provided for the target application based on the image output by the second image acquisition module, comprising: The image output by the second image acquisition module is provided as the target image to the target application.

6. The method according to claim 1, wherein a target image is provided to the target application based on the image output by the second image acquisition module, comprising: According to the image request parameters of the target application, the image output by the second image acquisition module is processed to obtain the target image; Wherein, the image display parameters of the target image match the request display parameters in the image request parameters; The target image is provided to the target application.

7. The method according to claim 1, wherein a target image is provided to the target application based on the image output by the first image acquisition module, comprising: Determine whether the first image output by the first image acquisition module and the second image output by the second image acquisition module meet the condition of similar effect; The first image and the second image correspond to the same timestamp; In response to the first image and the second image satisfying the effect similarity condition, the image output by the first image acquisition module is provided as the target image to the target application; In response to the first image and the second image not satisfying the effect similarity condition, the image output by the second image acquisition module is provided to the target application as the target image, and the image acquisition control parameters of the first image acquisition module or the second image acquisition module are adjusted until the first image and the second image satisfy the effect similarity condition.

8. The method according to claim 7, wherein the conditions for similar effects include: The difference between the effect values ​​of the first image and the second image in at least one effect dimension is less than or equal to a similarity threshold; The effect dimensions include at least one of the following: focus dimension, exposure dimension, and white balance dimension.

9. An image output control device, comprising: The first providing unit is configured to, in response to a call instruction from a target application to a first image acquisition module, provide a target image to the target application based on an image output by a second image acquisition module; The second providing unit is configured to provide a target image to the target application based on the image output by the first image acquisition module after the first image acquisition module is started. The second image acquisition module operates in the normally open power domain.

10. An electronic device, comprising: First image acquisition module; Second image acquisition module; The controller is configured to respond to a call command from a target application to the first image acquisition module, and provide a target image to the target application based on the image output by the second image acquisition module; after the first image acquisition module has been started, it provides a target image to the target application based on the image output by the first image acquisition module. The second image acquisition module operates in the normally open power domain.

Images