Image processing method and device, terminal equipment and storage medium

Abstract

The invention relates to an image processing method and device, terminal equipment and a storage medium. The method comprises the following steps: determining a zooming scene according to zooming operation; the method comprises the following steps: before acquiring an image through a first image acquisition module, starting the first image acquisition module; wherein the first image acquisition module is an image acquisition module of a to-be-acquired image determined according to the zoom scene; generating a preview image based on the image acquired by the second image acquisition module and the image acquired by the started first image acquisition module; wherein the second image acquisition module is an image acquisition module for acquiring an image before the first image acquisition module acquires the image. According to the method and the device, the situation of preview lagging in the switching process of the image acquisition module is reduced, the smoothness of generating the preview image is improved, and the use experience of a user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, terminal device and storage medium. Background Art

[0002] With the development of imaging technology, more and more electronic devices have imaging functions. Electronic devices with imaging functions usually have image acquisition modules, which are used to acquire images and record videos. The number of image acquisition modules in an electronic device is determined by the configuration of the electronic device, and the number of image acquisition modules in different electronic devices may be different. The image acquisition module may be a module including at least one image acquisition unit, and the acquisition unit may be a camera.

[0003] In an electronic device having multiple image acquisition units, such as an electronic device having three or more image acquisition units, in some specific application modes (such as a multi-camera mode), images may be acquired through two or more image acquisition units to obtain the required images. Summary of the invention

[0004] The present disclosure provides an image processing method, apparatus, terminal device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: determining a zoom scene according to a zoom operation; before acquiring an image through a first image acquisition module, starting the first image acquisition module; wherein the first image acquisition module is: an image acquisition module for acquiring an image to be acquired according to the zoom scene; generating a preview image based on an image acquired by a second image acquisition module and an image acquired by the first image acquisition module; wherein the second image acquisition module is: an image acquisition module for acquiring an image before the first image acquisition module acquires an image.

[0006] In one embodiment, determining the zoom scene according to the zoom operation includes: determining the change in zoom ratio according to the zoom operation; when the absolute value of the change is less than a preset threshold, determining that the zoom scene is a first zoom scene; and when the absolute value of the change is greater than the preset threshold, determining that the zoom scene is a second zoom scene.

[0007] In one embodiment, the zoom scene is a first zoom scene; the second image acquisition module includes: a first image acquisition unit and a second image acquisition unit, and the first image acquisition module includes a third image acquisition unit; generating a preview image based on the image acquired by the second image acquisition module and the image acquired by the first image acquisition module that has been turned on includes: generating the preview image through the image acquired by the first image acquisition unit and the image acquired by the third image acquisition unit; wherein the second image acquisition unit is in a closed state; or generating the preview image through the image acquired by the second image acquisition unit and the image acquired by the third image acquisition unit; wherein the first image acquisition unit is in a closed state.

[0008] In one embodiment, the zoom scene is a second zoom scene; the second image acquisition module includes: a first image acquisition unit and a second image acquisition unit, the first image acquisition module includes at least a third image acquisition unit and a fourth image acquisition unit; the preview image is generated based on the image acquired by the second image acquisition module and the image acquired by the first image acquisition module that has been turned on, including: according to the zoom operation, determining three target image acquisition units among the first image acquisition unit, the second image acquisition unit, the third image acquisition unit and the fourth image acquisition unit; acquiring images by two of the three target image acquisition units to generate the preview image; wherein the remaining one of the target image acquisition units remains in an turned-on state.

[0009] In one embodiment, determining the change in zoom factor according to a zoom operation includes: determining a first zoom factor corresponding to a current preview image; determining a second zoom factor after zooming according to the zoom operation; and determining the change according to the first zoom factor and the second zoom factor.

[0010] In one embodiment, the method further includes: before generating a preview image based on the image captured by the second image acquisition module and the image captured by the first image acquisition module, generating the preview image by using the image captured by the first image acquisition unit and the image captured by the second image acquisition unit.

[0011] In one embodiment, starting the first image acquisition module includes: configuring target resources required by the first image acquisition module; wherein the target resources include at least: an image sensor, an image signal processing unit, a motor, an optical image stabilizer, and transmission resources, storage resources and / or computing resources required in the process of acquiring images; and starting the first image acquisition module based on the target resources.

[0012] In one embodiment, configuring the target resources required by the first image acquisition module includes: creating a target thread; and configuring the target resources required by the first image acquisition module through the target thread.

[0013] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing device, the device comprising: a zoom scene determination module, for determining a zoom scene according to a zoom operation; an activation module, for activating a first image acquisition module before acquiring an image through the first image acquisition module; wherein the first image acquisition module is: an image acquisition module for acquiring an image to be acquired according to the zoom scene determination; a preview module, for generating a preview image based on an image acquired by a second image acquisition module and an image acquired by the activated first image acquisition module; wherein the second image acquisition module is: an image acquisition module for acquiring an image before the first image acquisition module acquires an image.

[0014] According to a third aspect of the embodiments of the present disclosure, a terminal device is provided, including:

[0015] A processor and a memory for storing executable instructions that can be run on the processor, wherein: when the processor is used to run the executable instructions, the executable instructions execute the method described in any one of the above embodiments.

[0016] According to a fourth aspect of the embodiments of the present disclosure, a non-temporary computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0017] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0018] In the embodiment of the present disclosure, in a multi-camera mode, that is, a mode in which multiple image acquisition units acquire images, when zooming is performed through a zoom operation, a zoom scene is determined according to the zoom operation, and different zoom scenes correspond to matching first image acquisition modules. By opening the first image acquisition module in advance before the first image acquisition module acquires images, for example, configuring the target resources required by the first image acquisition module in advance. In this way, when acquiring images through the first image acquisition module in various zoom scenes, there is no need to wait for the first image acquisition module to be opened, thereby reducing the time delay for acquiring images through the first image acquisition module, thereby reducing the time for generating preview images based on the images acquired by the first image acquisition module, reducing the preview freeze that occurs during the switching of the image acquisition modules, and improving the smoothness of generating preview images, thereby improving the user experience.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] Figure 1 is a schematic diagram of an image processing method according to an exemplary embodiment;

[0022] Figure 2 is a schematic diagram showing a method of determining a zoom scene according to an exemplary embodiment;

[0023] Figure 3 is a schematic diagram showing a method of generating a preview image according to an exemplary embodiment;

[0024] Figure 4 is a schematic diagram of an image processing device according to an exemplary embodiment;

[0025] Figure 5 is a schematic diagram of another image processing method according to an exemplary embodiment;

[0026] Figure 6 The present invention is a block diagram of a terminal device according to an exemplary embodiment. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0028] Generally, electronic devices with multiple image acquisition modules use a multi-camera mode when acquiring images, that is, a mode in which images are acquired by multiple image acquisition units. For example, the rear image acquisition module defaults to a multi-camera mode. For example, the back of the electronic device has 3, 4, or 5 cameras, or more cameras.

[0029] When the image acquisition scene changes or the image acquisition unit for acquiring images changes due to zooming and other operations according to the image acquisition operation, the image acquisition module that was not used as the current acquired image before will switch to the image acquisition module for the current acquired image. That is, the image acquisition module that is not the current acquired image will be used as the image acquisition module for acquiring images after zooming. In this case, it is necessary to open the image acquisition module used after zooming, and then acquire images through the opened image acquisition module. Since the opening process takes time, for example, it takes time to configure resources for the image acquisition module that is switched to the current acquired image, and resource configuration takes time, there will be delays, resulting in delays in generating preview images, resulting in preview freezes, and affecting the user experience.

[0030] On the other hand, if there is a switch of the image acquisition unit during the zoom process, the image acquisition unit needs to be turned on and off. Due to hardware characteristics or power consumption, 4 or 5 or more image acquisition units cannot be turned on at the same time. When turning on the image acquisition unit, it is necessary to wait for some of the previously turned on image acquisition units to be turned off and release the corresponding resources before using the released resources to turn on the image acquisition unit to be turned on. In addition, the switching of the image acquisition unit takes time, which will cause delays, resulting in delays in generating preview images, causing preview freezes, and affecting the user experience.

[0031] refer to Figure 1 , is a schematic diagram of an image processing method, the method comprising:

[0032] S100: Determine a zoom scene according to a zoom operation.

[0033] S200, before acquiring an image through the first image acquisition module, start the first image acquisition module; wherein the first image acquisition module is: an image acquisition module for the image to be acquired determined according to the zoom scene.

[0034] S300, generating a preview image based on the image captured by the second image capture module and the image captured by the enabled first image capture module, wherein the second image capture module is an image capture module that captures images before the first image capture module captures images.

[0035] The method can be executed at least in a terminal device having a first image acquisition module and a second image acquisition module, and the first image acquisition module and the second image acquisition module can respectively include a plurality of image acquisition units. The terminal device can include a mobile terminal device and a fixed terminal device, that is, the execution subject of the method can include at least a mobile terminal device and a fixed terminal device. The mobile terminal device can include a mobile phone, a tablet computer, a vehicle-mounted central control device, a wearable device, a smart device, etc., and the smart device can include smart office equipment, smart home equipment, and a robot, etc.

[0036] The terminal device has multiple image acquisition units, which may be cameras, and the models and parameters of different image acquisition units may be different. For example, the image acquisition unit may be a wide-angle camera, a telephoto camera, a periscope telephoto camera, an ultra-wide-angle camera, a black-and-white camera, a macro camera, a depth-of-field camera (Time-of-Flight, ToF) or other cameras with different parameters, and the terminal device may have multiple of these cameras, such as 3, 4 or 5, or more cameras.

[0037] The application scenario of this embodiment is to capture images in a multi-camera mode to generate preview images of an electronic device. The multi-camera mode is a mode in which multiple image capture units capture images and a preview image is generated based on the images captured by the multiple image capture units. For example, the number of image capture units that capture images in the multi-camera mode may be 2 or 3.

[0038] For S100, when zooming in the multi-camera mode, since different image acquisition modules include image acquisition units with different performance parameters, the performance parameters here may include parameters related to zooming such as focal length and / or focal length. In this way, there may be a switching of image acquisition modules, and the corresponding image acquisition module is switched to capture images according to the zoom operation.

[0039] For example, the image acquisition unit is a camera, the focal length of the ultra-wide-angle camera is 12 to 18 mm, the focal length of the wide-angle camera is 22-30 mm, and the focal length of the telephoto camera is 50-80 mm. During the zooming process, it may be possible to switch between different cameras.

[0040] An application for collecting images, such as a camera APP, is installed in the electronic device. When the camera APP is open, a preview interface is provided, that is, an interface for displaying a preview image when the APP is open. The preview interface may also have a zoom control, and the zoom operation on the zoom control may be performed to zoom in or out the object being photographed in the preview image.

[0041] The zoom operation can be determined according to actual zoom needs.

[0042] Exemplarily, the electronic device has a touch display screen and has a function of detecting a zoom operation, and can detect a zoom operation acting on a zoom control in a preview interface.

[0043] For example, the electronic device has a zoom button, and zooming is achieved by adjusting the zoom button. The electronic device can determine the zoom operation according to the change of the zoom button. The zoom button has a dial indicating the zoom multiple, indicating the current zoom multiple.

[0044] After the zoom operation is detected, the zoom scene may be determined according to the zoom operation. For example, the zoom scene may be determined according to the zoom amount adjusted by the zoom operation. Different zoom amounts may correspond to different zoom scenes.

[0045] Exemplarily, the zoom scene may include a first zoom scene and a second zoom scene, and the zoom amount corresponding to the first zoom scene is smaller than the zoom amount corresponding to the second zoom scene. The first zoom scene may also be referred to as a slow zoom scene, and the second zoom scene may also be referred to as a fast zoom scene.

[0046] Since the zoom amounts of the first zoom scene and the second zoom scene are different, the image acquisition units corresponding to the zooming may also be different.

[0047] For S200, the terminal device can determine the second image acquisition module and the first image acquisition module. The first image acquisition module is an image acquisition module for the image to be acquired according to the zoom scene. The second image acquisition module is: an image acquisition module that acquires images before the first image acquisition module acquires images, and the second image acquisition module is also an image acquisition module that acquires images before zooming according to the zoom operation.

[0048] For example, if the image currently displayed in the preview interface is the first image, the image acquisition module that acquires the image at this time is the second image acquisition module. If a zoom operation is detected thereafter, when zooming is performed according to the zoom operation and the image acquisition module changes, it is necessary to open an image acquisition module other than the second image acquisition module for image acquisition, and the opened image acquisition module other than the second image acquisition module is the first image acquisition module.

[0049] Exemplarily, the method further includes:

[0050] The first image acquisition module is determined according to the zoom operation. For example, the first image acquisition module can be determined according to the zoom amount of the zoom operation. The first image acquisition module may be determined differently for different zoom amounts. The first image acquisition module may include an image acquisition unit, such as a camera. Different first image acquisition modules have different specifications and parameters.

[0051] Exemplarily, a preview request can be determined according to a zoom operation, and a first image acquisition module can be determined according to the preview request, wherein the preview request carries an identifier of the first image acquisition module. For example, the preview request is determined according to a change trend of the zoom ratio adjusted by the zoom operation. When the change trend is that the zoom ratio gradually increases, the preview request is determined to be a request to turn on an image acquisition module with a larger focal length. When the change trend is that the zoom ratio gradually decreases, the preview request is determined to be a request to turn on an image acquisition module with a smaller focal length. The image acquisition module requested to be turned on is the first image acquisition module, which is also the image acquisition module of the image to be acquired.

[0052] Exemplarily, the second image acquisition module includes an image acquisition unit for currently acquiring an image, and the image acquisition units for currently acquiring an image all belong to the second image acquisition module. The image acquisition unit included in the first image acquisition module is an image acquisition unit for an image to be acquired, and the image acquisition unit in the first image acquisition module does not acquire an image under the current circumstances.

[0053] The first image acquisition module and the second image acquisition module respectively include at least one image acquisition unit.

[0054] In one embodiment, the first image acquisition module includes an image acquisition unit outside the second image acquisition module.

[0055] During the zooming process, the first image acquisition module may switch its state to serve as an image acquisition module for acquiring images, including one or more image acquisition units in the first image acquisition module switching to serve as image acquisition units for acquiring images.

[0056] Before capturing images through the first image capturing module, the first image capturing module is turned on.

[0057] Exemplarily, starting the first image acquisition module may include: adjusting the first image acquisition module to a state capable of acquiring images.

[0058] Exemplarily, starting the first image acquisition module may include: configuring target resources required by the first image acquisition module, and starting the first image acquisition module according to the target resources. Before the terminal device acquires an image through the image acquisition unit in the image acquisition module, it is necessary to configure various resources required by the image acquisition unit for acquiring the image, and then acquire the image through the image acquisition unit based on these resources.

[0059] The target resources may include various resources required by the first image acquisition module when acquiring images, including software resources and hardware resources. The hardware resources may include image sensors, image signal processing units, motors and / or optical image stabilizers, etc. The software resources may include transmission resources, storage resources and computing resources required for various interactive information, etc., which are not listed here one by one. The target resources may include a pipeline corresponding to the first image acquisition module. After the target resources are configured, the first image acquisition module can be turned on according to the target resources. The first image acquisition module can be in an image acquisition state and can directly acquire images.

[0060] Exemplarily, once the target resources are configured, the first image acquisition module can be enabled.

[0061] For S300, since the target resources required by the first image acquisition module have been configured, when acquiring images through the first image acquisition module, images can be acquired directly through the first image acquisition module based on the target resources, and there is no need to wait for the process of starting the first image acquisition module, that is, there is no need to wait for the process of configuring the required target resources for the first image acquisition module, which reduces the time between switching to the first image acquisition module and the first image acquisition module starting to acquire images, and images can be acquired through the already turned on first image acquisition module after switching the image acquisition module.

[0062] A preview image is generated according to the image captured by the second image capture module and the image captured by the enabled first image capture module.

[0063] It takes time to start the first image acquisition module, for example, to configure resources. Usually, the resource configuration of the corresponding image acquisition module is performed after the image acquisition module is switched, which is a serial operation. In this way, the image can only be acquired through the first image acquisition module after the first image acquisition module is started, which will cause delays and freezes in the process of generating preview images, reducing the user experience.

[0064] In the embodiment of the present disclosure, in a multi-camera mode, that is, a mode in which multiple image acquisition units acquire images, when zooming is performed through a zoom operation, a zoom scene is determined according to the zoom operation, and different zoom scenes correspond to matching first image acquisition modules. By opening the first image acquisition module in advance before the first image acquisition module acquires images, for example, configuring the target resources required by the first image acquisition module in advance. In this way, when acquiring images through the first image acquisition module in various zoom scenes, there is no need to wait for the first image acquisition module to be opened, thereby reducing the time delay for acquiring images through the first image acquisition module, thereby reducing the time for generating preview images based on the images acquired by the first image acquisition module, reducing the preview freeze that occurs during the switching of the image acquisition modules, and improving the smoothness of generating preview images, thereby improving the user experience.

[0065] In one embodiment, reference Figure 2 , is a schematic diagram of determining a zoom scene, and the method may include:

[0066] S101, determining a change in zoom factor according to a zoom operation;

[0067] S102: when the absolute value of the change amount is less than a preset threshold, the zoom scene is determined to be the first zoom scene;

[0068] S103: If the absolute value of the change amount is greater than a preset threshold, it is determined that the zoom scene is the second zoom scene.

[0069] The switching from one zoom factor to another zoom factor can be determined according to the zoom operation, and the amount of change can be determined according to the difference between the two zoom factors.

[0070] Exemplarily, a first zoom factor corresponding to the current preview image may be determined, and a second zoom factor after zooming according to the zoom operation may be determined, and then the change amount may be determined according to the first zoom factor and the second zoom factor.

[0071] The zoom factor can be displayed on the preview interface, that is, both the first zoom factor and the second zoom factor can be displayed through the preview interface. The current preview image may include a preview image of the image before zooming, and the first zoom factor may be the zoom factor of the preview image before zooming. The electronic device can identify the first zoom factor and the second zoom factor.

[0072] The second zoom factor may be greater than the first zoom factor, for example, the first zoom factor is 1X and the second zoom factor is 3X. The second zoom factor may also be less than the first zoom factor, for example, the first zoom factor is 2X and the second zoom factor is 0.5X. Here, X represents the magnification factor, and the zoom factor includes the magnification factor.

[0073] The change amount is determined according to the absolute value of the difference between the first zoom factor and the second zoom factor. Since the difference may be a positive number or a negative number, the absolute value of the difference may be taken as the change amount.

[0074] The preset threshold can be determined according to actual usage requirements, and can be a positive integer or a decimal greater than 0, such as 0.2 or 1.

[0075] If the absolute value of the change is less than the preset threshold, the zoom scene is determined to be the first zoom scene, indicating that the zoom change is relatively small. If the absolute value of the change is greater than the preset threshold, the zoom scene is determined to be the second zoom scene, indicating that the zoom change is relatively large.

[0076] Since the amount of change of zoom is different, the first image acquisition modules corresponding to different zoom scenes may be different, so it is necessary to distinguish the zoom scenes and determine the first image acquisition module that better matches the zoom.

[0077] In one embodiment, the second image acquisition module includes: a first image acquisition unit and a second image acquisition unit, and the first image acquisition module includes a third image acquisition unit. The first image acquisition unit, the second image acquisition unit, and the third image acquisition unit are image acquisition units with different parameters. For example, the first image acquisition unit is a wide-angle camera, the second image acquisition unit is a telephoto camera, and the third image acquisition unit is an ultra-wide-angle camera.

[0078] Step S200, before acquiring an image through the first image acquisition module, starting the first image acquisition module includes:

[0079] Before the first image acquisition unit or the second image acquisition unit is changed to the third image acquisition unit, the first image acquisition module is started and target resources required by the first image acquisition module are configured.

[0080] The first image acquisition unit and the second image acquisition unit are used as units for acquiring images before zooming the zoom operation. In the multi-camera mode, the first image acquisition unit and the second image acquisition unit are used to acquire images, and a preview image is generated based on the images acquired by the first image acquisition unit and the second image acquisition unit. The preview image is a preview image corresponding to the first zoom factor.

[0081] After zooming, the third image acquisition unit replaces the first image acquisition unit or the second image acquisition unit, through the first image acquisition unit and the third image acquisition unit, or, the second image acquisition unit and the third image acquisition unit.

[0082] Before the first image acquisition unit or the second image acquisition unit is changed to the third image acquisition unit, the target resources required by the third image acquisition unit are configured, and the target resources required by the third image acquisition unit are configured in advance. In this way, when acquiring images through the third image acquisition unit, there is no need to wait for the configuration of the target resources of the third image acquisition unit, which reduces the time delay of acquiring images through the third image acquisition unit, and can directly acquire images through the third image acquisition unit based on the configured target resources, thereby reducing the time for generating preview images, reducing the jamming in the middle when the image acquisition module is switched, and improving the fluency of generating preview images, thereby improving the user experience.

[0083] In one embodiment, before generating the preview image based on the image captured by the second image capture module and the image captured by the first image capture module, the preview image is generated using the image captured by the first image capture unit and the image captured by the second image capture unit.

[0084] Before the first image acquisition module acquires an image, a preview image is generated by the image acquired by the second image acquisition module. This can reduce preview delays or freezes caused by not generating preview images based on the images acquired by the first image acquisition module.

[0085] In another embodiment, configuring the target resources required by the first image acquisition module includes:

[0086] A target thread is created, and target resources required by the first image acquisition module are configured through the target thread.

[0087] The target resources required by the first image acquisition module can be configured by creating a separate thread, recording the thread as a target thread, and then configuring the target resources required by the first image acquisition module through the target thread. The target thread has the function of executing the target resources required by the first image acquisition module.

[0088] The target thread may be a system-level thread in the terminal device.

[0089] In one embodiment, the zoom scene is a first zoom scene; the second image acquisition module includes: a first image acquisition unit and a second image acquisition unit, and the first image acquisition module includes a third image acquisition unit.

[0090] S300, generating a preview image based on the image captured by the second image capture module and the image captured by the enabled first image capture module, including:

[0091] A preview image is generated by using the image acquired by the first image acquisition unit and the image acquired by the third image acquisition unit; wherein the second image acquisition unit is in a closed state.

[0092] After zooming is performed according to the zoom operation and the first image acquisition module is turned on, the first image acquisition unit in the second image acquisition module and the third image acquisition unit in the first image acquisition module are both in the turned-on state, and images can be directly acquired. The third image acquisition unit in the first image acquisition module and the first image acquisition unit in the second image acquisition module acquire images, thereby generating a preview image after zooming.

[0093] Since the change amount of the first zoom scene during zooming is small, the possibility of zooming between the two image acquisition units is the greatest during zooming, and due to hardware characteristics, images are acquired through two image acquisition units at most at the same time. Therefore, by turning off the second image acquisition unit, the various resources occupied and consumed by the second image acquisition unit when it is in the turned-on state can be reduced.

[0094] Exemplarily, the zoom operation is an operation of zooming in, and the focal length of the first image acquisition unit, the focal length of the second image acquisition unit, and the focal length of the third image acquisition unit increase in sequence.

[0095] or,

[0096] S300, generating a preview image based on the image captured by the second image capture module and the image captured by the enabled first image capture module, including:

[0097] A preview image is generated by using the image acquired by the second image acquisition unit and the image acquired by the third image acquisition unit; wherein the first image acquisition unit is in a closed state.

[0098] After zooming is performed according to the zoom operation and the first image acquisition module is turned on, the second image acquisition unit in the second image acquisition module and the third image acquisition unit in the first image acquisition module are both in the turned-on state, and images can be directly acquired. The third image acquisition unit in the first image acquisition module and the second image acquisition unit in the second image acquisition module acquire images, thereby generating a preview image after zooming.

[0099] Since the change amount of the first zoom scene during zooming is small, the possibility of zooming between the two image acquisition units is the greatest during zooming, and due to hardware characteristics, images are acquired through at most two image acquisition units at the same time. Therefore, by turning off the first image acquisition unit, the various resources occupied and consumed by the first image acquisition unit when it is in the turned-on state can be reduced.

[0100] Exemplarily, the zoom operation is an operation of reducing the zoom factor, and the focal length of the first image acquisition unit, the focal length of the second image acquisition unit, and the focal length of the third image acquisition unit are reduced in sequence.

[0101] In one embodiment, the zoom scene is a second zoom scene; the second image acquisition module includes: a first image acquisition unit and a second image acquisition unit, and the first image acquisition module includes at least a third image acquisition unit and / or a fourth image acquisition unit.

[0102] refer to Figure 3 , is another schematic diagram of generating a preview image, generating a preview image based on an image captured by a second image acquisition module and an image captured by the first image acquisition module that has been turned on, including:

[0103] S301, determining three target image acquisition units among a first image acquisition unit, a second image acquisition unit, a third image acquisition unit and a fourth image acquisition unit according to a zoom operation;

[0104] S302, capturing images by two of the three target image capturing units to generate a preview image; wherein the remaining target image capturing unit remains in an open state.

[0105] Since the second zoom scene has a large amount of changes, there may be multiple image acquisition units with different focal lengths zooming during the zooming process, and due to hardware characteristics, only three image acquisition units can be turned on at the same time, and images are acquired through two of them. Therefore, three target image acquisition units are determined from the four image acquisition units, and the three target image acquisition units are kept in an open state, and images are acquired through two of the target image acquisition units to generate preview images.

[0106] In this way, when zooming in the second zoom scene, since another target image acquisition unit is in an open state, there is no need to open the image acquisition unit again when the zoom is switched to the target image acquisition unit, thereby reducing the opening operation of the image acquisition module during the switching between the three image acquisition units, reducing delays and freezes, and improving the user experience.

[0107] Exemplarily, the first image acquisition unit is a wide-angle camera, the second image acquisition unit is a non-telephoto camera (such as an ultra-wide-angle camera), the third image acquisition unit is a telephoto camera, and the fourth image acquisition unit is a periscope telephoto camera.

[0108] The zoom operation is a magnification zoom, then according to the zoom operation, the first image acquisition unit, the second image acquisition unit and the fourth image acquisition unit are determined as target image acquisition units, images are acquired by two of the image acquisition units, and the remaining image acquisition unit remains in an open state.

[0109] Exemplarily, the first image acquisition unit is a telephoto camera, the second image acquisition unit is a non-wide-angle camera (such as a periscope telephoto camera), the third image acquisition unit is an ultra-wide-angle camera, and the fourth image acquisition unit is a wide-angle camera.

[0110] The zoom operation is a magnification zoom, then according to the zoom operation, the first image acquisition unit, the second image acquisition unit and the third image acquisition unit are determined as target image acquisition units, images are acquired through two of the image acquisition units, and the remaining image acquisition unit remains in an open state.

[0111] Exemplarily, the first image acquisition unit is a telephoto camera, the second image acquisition unit is a wide-angle camera, the third image acquisition unit is an ultra-wide-angle camera, and the fourth image acquisition unit is a periscope telephoto camera.

[0112] If the zoom operation is a zoom-in operation, the first image acquisition unit, the third image acquisition unit, and the fourth image acquisition unit are determined as target image acquisition units according to the zoom operation. If the zoom operation is a zoom-out operation, the first image acquisition unit, the second image acquisition unit, and the fourth image acquisition unit are determined as target image acquisition units according to the zoom operation, and images are acquired through two of the image acquisition units, and the remaining image acquisition unit remains in an open state.

[0113] In one embodiment, reference Figure 4 , is a schematic diagram of an image processing device, the device comprising:

[0114] A zoom scene determination module 1, used to determine a zoom scene according to a zoom operation;

[0115] The start module 2 is used to start the first image acquisition module before acquiring an image through the first image acquisition module; wherein the first image acquisition module is: an image acquisition module for the image to be acquired determined according to the zoom scene;

[0116] A preview module 3, used to generate a preview image based on the image captured by the second image acquisition module and the image captured by the first image acquisition module that has been turned on;

[0117] The second image acquisition module is an image acquisition module that acquires images before the first image acquisition module acquires images.

[0118] In one embodiment, the zoom scene determination module 1 includes:

[0119] a change amount determining unit, used to determine a change amount of the zoom factor according to the zoom operation;

[0120] The zoom scene determining unit is used to determine that the zoom scene is a first zoom scene when the absolute value of the change amount is less than a preset threshold; and to determine that the zoom scene is a second zoom scene when the absolute value of the change amount is greater than the preset threshold.

[0121] In one embodiment, the zoom scene is a first zoom scene; the second image acquisition module includes: a first image acquisition unit and a second image acquisition unit, and the first image acquisition module includes a third image acquisition unit;

[0122] Preview module 3 is also used to:

[0123] The preview image is generated by using the image acquired by the first image acquisition unit and the image acquired by the third image acquisition unit; wherein the second image acquisition unit is in a closed state;

[0124] or,

[0125] The preview image is generated by using the image acquired by the second image acquisition unit and the image acquired by the third image acquisition unit; wherein the first image acquisition unit is in a closed state.

[0126] In one embodiment, the zoom scene is a second zoom scene; the second image acquisition module includes: a first image acquisition unit and a second image acquisition unit, and the first image acquisition module includes at least a third image acquisition unit and / or a fourth image acquisition unit;

[0127] Preview module 3, including:

[0128] A first determining unit, configured to determine three target image acquisition units among the first image acquisition unit, the second image acquisition unit, the third image acquisition unit, and the fourth image acquisition unit according to the zoom operation;

[0129] The preview unit is used to acquire images through two of the three target image acquisition units to generate the preview image; wherein the remaining one of the target image acquisition units remains in an open state.

[0130] In one embodiment, the apparatus further comprises:

[0131] The image generation module is used to generate the preview image by using the image captured by the first image acquisition unit and the image captured by the second image acquisition unit before generating the preview image based on the image captured by the second image acquisition module and the image captured by the first image acquisition module.

[0132] In one embodiment, the variation determination unit includes:

[0133] A first determining subunit, used to determine a first zoom factor corresponding to the current preview image;

[0134] A second determining subunit, used to determine a second zoom factor after zooming according to the zoom operation;

[0135] The variation determination subunit is used to determine the variation according to the first zoom factor and the second zoom factor.

[0136] In one embodiment, starting the first image acquisition unit includes:

[0137] A configuration subunit, configured to configure target resources required by the first image acquisition module; wherein the target resources include at least: an image sensor, an image signal processing unit, a motor, an optical image stabilizer, and transmission resources, storage resources and / or computing resources required in the process of acquiring images;

[0138] An enabling subunit is used to enable the first image acquisition module based on the target resource.

[0139] In one embodiment, the configuration subunit is used to: create a target thread, and configure target resources required by the first image acquisition module through the target thread.

[0140] In one embodiment, the device comprises:

[0141] The first image acquisition module determination module is used to determine the first image acquisition module according to the zoom operation.

[0142] In another embodiment, a terminal device is provided, including:

[0143] A processor and a memory for storing executable instructions capable of running on the processor, wherein:

[0144] When the processor is used to run the executable instructions, the executable instructions execute the method described in any one of the above embodiments.

[0145] In another embodiment, a non-transitory computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0146] In one embodiment, reference Figure 5 , which is a schematic diagram of another image processing method.

[0147] Usually, the rear image acquisition module of electronic devices such as mobile phones and tablets has multiple image acquisition units, such as more than 3, and the default mode is to use the multi-camera mode, such as supporting 4 cameras or 5 cameras for switching. Due to hardware characteristics, the image processing hardware platform of electronic devices only has 3 front-end image processing and analysis units (CSID), which supports opening 3 cameras at the same time, but cannot open 4 or 5 or more cameras at the same time.

[0148] In this multi-camera scenario, the number of cameras is greater than the number of cameras that can be simultaneously turned on supported by the image processing hardware platform of the electronic device, such as the number of cameras is greater than the number of front-end image data acquisition and processing units (Image Front Eng, ife) in the image signal processing unit (Image Signal Processing, ISP).

[0149] In the zooming scene, because the camera (sensor) needs to be switched, the camera needs to be frequently turned on and off, which may cause the newly opened camera to wait for the closed camera to complete processing and release the resources required for opening, such as CSID ifeHW resources, before it can be normally opened (active) and work. This waiting scene will cause preview jamming during the zooming process, reducing the user experience.

[0150] The camera application in the electronic device can detect the user's zoom operation, identify and determine the current zoom scene. For example, it can be determined whether a zoom operation is detected, thereby determining whether it is a zoom scene. In the zoom scene, the zoom operation is determined to be which zoom scene, which can be a first zoom scene or a second zoom scene. The first zoom scene is a slow zoom scene, and the second zoom scene is a fast zoom scene.

[0151] Whether it is a fast zoom scene or a slow zoom scene can be determined based on the change in zoom multiple and a preset threshold, and based on whether the change meets the preset threshold. Then, the first image acquisition module to be used after zooming is determined based on the zoom scene.

[0152] Before starting the first image acquisition module, a new path EarlyActivePatch is added to open the first image acquisition module in an offline background, such as opening a real-time rendering pipeline (realtimePipeline).

[0153] After the first image acquisition module is turned on, the camera that captures images in different zoom scenes can be controlled by the multi-camera control center.

[0154] Zoom scene 1: Determine the zoom scene as the first zoom scene, that is, the slow zoom scene.

[0155] The camera application sends a preview request to the multi-camera control center. After receiving the preview request, the multi-camera control center compares the image acquisition unit opened in the current preview request with that opened in the previous preview request. If it is found that the current preview request requires a new image acquisition unit to be opened, the EarlyActivePatch channel is used to open the image acquisition unit required by the current preview request in the background. At this time, the preview image in the current preview interface remains consistent with the preview image of the previous frame.

[0156] The multi-camera control center belongs to the bottom-level control module in the electronic device, and the image acquisition module belongs to the bottom-level processing module. The multi-camera control center can perform multi-camera front-end control and multi-camera back-end control. The image acquisition unit can capture images and output images to the front-end image processing module (including real-time rendering channel) for front-end cropping and image quality processing. The front-end image processing module sends the processed image data to the multi-camera control center, and the multi-camera control center sends the received data to the back-end processing module (including offline rendering channel, offlinePipeline). The back-end processing module performs alignment calculation, cropping and image quality processing on the image data to obtain a preview image, and then sends it to the camera APP to display the preview image.

[0157] For the image acquisition unit opened by calling the EarlyActivePatch channel for the last preview request, if the current preview request still needs to continue to control the opening, the current preview request sends a preview request to the multi-camera control center for collecting images to generate preview images.

[0158] Zoom scene 2: determines the zoom scene to be the second zoom scene, that is, the fast zoom scene.

[0159] Determining the second image acquisition module, that is, detecting the image acquisition unit opened by the previous request, can be divided into the following three situations for processing:

[0160] If the second image acquisition module includes a wide-angle camera and a non-telephoto camera, it is determined that the periscope telephoto camera needs to be turned on for the current preview request, and the periscope telephoto camera, wide-angle camera and ultra-wide-angle camera are kept turned on at the same time. Two of them are controlled to capture images according to the actual zoom operation, and the other one is ensured not to be turned off by calling the EarlyActivePatch channel.

[0161] If the second image acquisition module includes a non-wide-angle camera and a telephoto camera, it is determined that the ultra-wide-angle camera needs to be turned on for the current preview request, and the ultra-wide-angle camera, telephoto camera, and periscope telephoto camera are kept turned on at the same time. Two of them are controlled to capture images according to the actual zoom operation, and the other one is ensured not to be turned off by calling the EarlyActivePatch channel.

[0162] If the second image acquisition module includes a wide-angle camera and a telephoto camera, when the zoom operation is a zoom-in operation, it is determined that the ultra-wide-angle camera and the periscope telephoto camera need to be turned on for the current preview request, and the ultra-wide-angle camera, the telephoto camera, and the periscope telephoto camera remain turned on at the same time. According to the actual zoom operation, two of them are controlled to capture images, and the other is ensured not to be turned off by calling the EarlyActivePatch channel. When the zoom operation is a zoom-out operation, it is determined that the ultra-wide-angle camera and the periscope telephoto camera need to be turned on for the current preview request, and the ultra-wide-angle camera, the wide-angle camera, and the periscope telephoto camera remain turned on at the same time.

[0163] Slow zoom scenario: By adding an offline pre-allocation solution, the time consumption of switching on and off the image acquisition unit is transferred to the offline thread to complete, and the lag problem caused by the time overhead required to reopen the image acquisition unit will not be caused.

[0164] Fast zoom scene: According to the actual scene, three image acquisition units are selected to be turned on at the same time, and two of the three fixed image acquisition units are selected for each frame of data processing, and the remaining image acquisition unit is controlled not to be turned off, ensuring that there is no frequent switching of the image acquisition units in the fast zoom scene, and the preview field of view can meet user needs.

[0165] By checking the above two scenarios, the smoothness of multi-camera zoom is ensured as much as possible under the limitation of existing software resources, which improves the user experience during the zoom process.

[0166] It should be noted that the “first” and “second” in the embodiments of the present disclosure are only for the convenience of description and distinction and have no other specific meanings.

[0167] Figure 6 1 is a block diagram of a terminal device according to an exemplary embodiment. For example, the terminal device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0168] Reference Figure 6 The terminal device may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0169] The processing component 802 generally controls the overall operation of the terminal device, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0170] The memory 804 is configured to store various types of data to support operations on the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0171] The power component 806 provides power to various components of the terminal device. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0172] The multimedia component 808 includes a screen that provides an output interface between the terminal device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the terminal device is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0173] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the terminal device is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0174] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0175] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the terminal device. For example, the sensor assembly 814 can detect the open / closed state of the terminal device, the relative positioning of components, such as the display and keypad of the terminal device, and the sensor assembly 814 can also detect the position change of the terminal device or a component of the terminal device, the presence or absence of user contact with the terminal device, the orientation or acceleration / deceleration of the terminal device, and the temperature change of the terminal device. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0176] The communication component 816 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0177] In an exemplary embodiment, the terminal device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0178] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0179] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Determine a zoom scene according to a zoom operation; Before acquiring an image through the first image acquisition module, the first image acquisition module is turned on; wherein the first image acquisition module is: an image acquisition module for the image to be acquired determined according to the zoom scene; Generate a preview image based on the image captured by the second image capture module and the image captured by the first image capture module that has been turned on; The second image acquisition module is an image acquisition module that acquires images before the first image acquisition module acquires images.

2. The method according to claim 1, characterized in that The step of determining the zoom scene according to the zoom operation includes: Determining a change in zoom factor according to the zoom operation; The absolute value of the change amount is less than a preset threshold, and the zoom scene is determined to be the first zoom scene; The absolute value of the change amount is greater than the preset threshold, and the zoom scene is determined to be the second zoom scene.

3. The method according to claim 2, characterized in that The zoom scene is a first zoom scene; the second image acquisition module includes: a first image acquisition unit and a second image acquisition unit, and the first image acquisition module includes a third image acquisition unit; The generating of a preview image based on the image captured by the second image capture module and the image captured by the enabled first image capture module includes: The preview image is generated by using the image acquired by the first image acquisition unit and the image acquired by the third image acquisition unit; wherein the second image acquisition unit is in a closed state; or, The preview image is generated by using the image acquired by the second image acquisition unit and the image acquired by the third image acquisition unit; wherein the first image acquisition unit is in a closed state.

4. The method according to claim 2, characterized in that: The zoom scene is a second zoom scene; the second image acquisition module includes: a first image acquisition unit and a second image acquisition unit, and the first image acquisition module includes at least a third image acquisition unit and / or a fourth image acquisition unit; The generating of a preview image based on the image captured by the second image capture module and the image captured by the enabled first image capture module includes: According to the zoom operation, determining three target image acquisition units among the first image acquisition unit, the second image acquisition unit, the third image acquisition unit and the fourth image acquisition unit; The preview image is generated by collecting images through two of the three target image collection units; wherein the remaining one of the target image collection units is kept in an open state.

5. The method according to claim 3 or 4, characterized in that: The method further comprises: Before generating a preview image based on the image captured by the second image acquisition module and the image captured by the first image acquisition module, the preview image is generated by using the image captured by the first image acquisition unit and the image captured by the second image acquisition unit.

6. The method according to claim 2, characterized in that Determining the change amount of the zoom factor according to the zoom operation includes: Determine a first zoom factor corresponding to the current preview image; Determining a second zoom factor after zooming according to the zoom operation; The change amount is determined according to the first zoom factor and the second zoom factor.

7. The method according to claim 1, characterized in that The starting of the first image acquisition module includes: Configuring target resources required by the first image acquisition module; wherein the target resources include at least: an image sensor, an image signal processing unit, a motor, an optical image stabilizer, and transmission resources, storage resources and / or computing resources required in the process of acquiring images; The first image acquisition module is started based on the target resource.

8. The method according to claim 7, characterized in that The configuring of the target resources required by the first image acquisition module includes: Create a target thread; The target resources required by the first image acquisition module are configured through the target thread.

9. An image processing device, characterized in that: The device comprises: A zoom scene determination module, used to determine a zoom scene according to a zoom operation; An opening module, used for opening the first image acquisition module before acquiring an image through the first image acquisition module; wherein the first image acquisition module is: an image acquisition module for an image to be acquired determined according to the zoom scene; A preview module, used to generate a preview image based on the image captured by the second image acquisition module and the image captured by the first image acquisition module that has been turned on; The second image acquisition module is an image acquisition module that acquires images before the first image acquisition module acquires images.

10. A terminal device, characterized in that: include: A processor and a memory for storing executable instructions capable of running on the processor, wherein: When the processor is used to run the executable instructions, the executable instructions execute the method described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 8.

Images