Image processing method, device and equipment and readable storage medium

Abstract

The invention relates to an image processing method, device and equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: in response to a zooming instruction, obtaining a current zooming magnification and a target zooming magnification; determining a corresponding target zoom curve according to the current zoom ratio and the target zoom ratio; a to-be-processed image sequence is obtained, and the to-be-processed image sequence comprises images of which the acquisition time is earlier than the receiving time of the zooming instruction; and processing the images in the to-be-processed image sequence according to the transition zoom magnification in the target zoom curve to obtain a target image sequence. According to the embodiment of the invention, the image data acquired at the moment corresponding to the zooming instruction does not need to wait for response, but the target image sequence is obtained by processing the image between the receiving moments of the zooming instruction in time to respond to the zooming instruction, so that the timeliness of zooming response is improved.

Description

Technical Field

[0001] The present application relates to the field of imaging technologies, and in particular, to an image processing method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] With the continuous intelligence of electronic devices, the shooting function has become more and more popular. Many electronic devices are equipped with multiple cameras. Different cameras generally correspond to different focal lengths, and the focal length ranges corresponding to each focal length are different, and the focuses presented by the captured images are also different. Therefore, when using an electronic device to take pictures, zooming can be achieved by switching cameras.

[0003] In the traditional technology, in the case of switching cameras to achieve zooming, the zoomed image does not change in time, and the timeliness of the zoom response is poor. Summary of the Invention

[0004] The present application provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the timeliness of the zoom response.

[0005] In a first aspect, the present application provides an image processing method, the method including:

[0006] Responding to a zoom instruction, obtaining a current zoom ratio and a target zoom ratio;

[0007] Determining a corresponding target zoom curve according to the current zoom ratio and the target zoom ratio;

[0008] Obtaining a sequence of images to be processed, where the sequence of images to be processed includes images whose acquisition time is earlier than the reception time of the zoom instruction;

[0009] Processing the images in the sequence of images to be processed according to the transition zoom ratio in the target zoom curve to obtain a target sequence of images.

[0010] In a second aspect, the present application further provides an image processing apparatus, including:

[0011] A magnification acquisition module, configured to respond to a zoom instruction and obtain a current zoom ratio and a target zoom ratio;

[0012] A zoom curve determination module, configured to determine a corresponding target zoom curve according to the current zoom ratio and the target zoom ratio;

[0013] An image acquisition module, configured to obtain a sequence of images to be processed, where the sequence of images to be processed includes images whose acquisition time is earlier than the reception time of the zoom instruction;

[0014] An image processing module, configured to process images in the to-be-processed image sequence according to the transition zoom ratio in the target zoom curve, so as to obtain a target image sequence.

[0015] In a third aspect, the present application further provides an electronic device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps of the image processing method provided in the first aspect are implemented.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image processing method provided in the first aspect are implemented.

[0017] In a fifth aspect, the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the image processing method provided in the first aspect are implemented.

[0018] For the above-mentioned image processing method, device, electronic device, computer-readable storage medium, and computer program product, in response to a zoom instruction, a corresponding target zoom curve is determined according to the current zoom ratio and the target zoom ratio, a to-be-processed image sequence is obtained, and images in the to-be-processed image sequence are processed according to the transition zoom ratio in the target zoom curve to obtain a target image sequence. Since the acquisition time of the to-be-processed images obtained is earlier than the reception time of the zoom instruction, it is not necessary to wait for the image data acquired at the corresponding time of the zoom instruction to come out before responding, but the images between the reception time of the zoom instruction are promptly used for processing to obtain a target image sequence to respond to the zoom instruction, thereby improving the timeliness of the zoom response. Description of the Drawings

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for describing the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0020] Figure 1 It is a schematic flowchart of an image processing method in an embodiment;

[0021] Figure 2 It is a schematic flowchart of dual-open frame fusion during zooming in an embodiment;

[0022] Figure 3 It is a schematic flowchart of an image processing method in another embodiment;

[0023] Figure 4Schematic diagram of the front capture frame in an embodiment;

[0024] Figure 5 Schematic diagram of the follow - hand duration in an embodiment;

[0025] Figure 6 Schematic diagram of the target point cut - scene capture frame in another embodiment;

[0026] Figure 7 Schematic diagram of the process for determining the fusion indication information in an embodiment;

[0027] Figure 8 Schematic diagram of the algorithm processing decision for the fusion indication in an embodiment;

[0028] Figure 9 Structural block diagram of the image processing device in an embodiment;

[0029] Figure 10 Structural block diagram of the image processing device in another embodiment;

[0030] Figure 11 Structural block diagram of the image processing device in yet another embodiment;

[0031] Figure 12 Internal structure diagram of the electronic device in an embodiment. Detailed implementation manners

[0032] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] In related electronic devices, there may be 3 - 5 different types of lenses. For example, the operation page of a multi - camera mobile phone camera will display focal length selections such as 0.6x, 1x, 2x, or 5x, 10x. The smaller the number, the wider the corresponding viewing angle range, and the farther the object being photographed. This roughly corresponds to the focal lengths of the ultra - wide - angle UW (Ultra Wide), wide - angle W (Wide), and telephoto T (Telephoto) lenses.

[0034] Generally, the main camera of most mobile phones is a wide - angle lens with a focal length of about 28 millimeters because this focal length is close to the viewing range seen by the human eye, and the main camera is the most frequently used camera. The operation page of the mobile phone camera will display a value of 1x, as Figure 1 shown, 1x is the focal length corresponding to the main camera, and the image quality is relatively clear, suitable for shooting portraits, buildings, landscapes, and documentaries, etc.

[0035] Ultra-wide-angle lenses can provide a wider viewing angle than wide-angle lenses, that is, they can capture a wider picture than wide-angle lenses, allowing more landscape elements to be captured, making them suitable for shooting scenery, architecture, etc. The images captured can have an impactful sense of picture. Ultra-wide-angle lenses have the characteristics of lens distortion, and objects at the edge of the captured image are elongated and magnified. This distortion can be fully utilized to shoot from above, creating a visual impact of "near objects are larger and far objects are smaller". When shooting buildings, the characteristics of lens distortion can be used to make the buildings more magnificent.

[0036] A telephoto lens with a focal length of 1x or more is usually called a telephoto lens. The larger the number in front of the x, the narrower the shooting range, but the farther you can shoot, and the clearer the distant objects you shoot. When shooting farther objects or zooming in on objects in the picture, the image quality will not drop like with digital zoom. A telephoto lens can "pull in" the distance between the background and the foreground, thus creating a sense of compression in the spatial distance, making the overall picture more substantial. A telephoto lens has little distortion and a weak perspective effect, and can shorten the distance between the foreground and the background, enhance the relationship between the foreground and the background, and create unique visual effects. This feature can be used to use straight-extending scenery such as roads and railings as guide lines to draw the audience's attention back to the subject in the depth of the picture.

[0037] Lenses corresponding to different focal lengths have different characteristics. The camera shooting solution is designed with the SAT (SpatialAlignment Transform) solution to achieve real-time lens switching according to user needs in different shooting scenes. The hand tracking, zoom duration and zoom smoothness during the zoom process are explicit evaluation indicators for evaluating the zoom experience, among which hand tracking is a relatively important quantifiable measurement indicator.

[0038] In the related technology, the zoom event and the target zoom ratio are generally sent to the hardware abstraction layer along with the capture request. The hardware abstraction layer starts to perform corresponding cropping processing on the result image data corresponding to the request only after receiving the zoom event and the target zoom ratio. The request is sent down layer by layer in sequence. The result image data corresponding to the corresponding request will go through a series of processing at the bottom layer before reaching the SAT-related processing module. This results in an overall delay of more than 300 milliseconds. In addition, the time taken to send the image to the camera application for display after cropping takes more time, resulting in the camera application displaying a picture that fails to respond in time after the user's zoom operation, and the tracking performance is poor.

[0039] In view of the problem of poor timeliness of zoom response during the above zoom process, an embodiment of the present application provides an image processing method. By responding to a zoom instruction, determining a corresponding target zoom curve according to the current zoom ratio and the target zoom ratio, obtaining a sequence of images to be processed, and processing the images in the sequence of images to be processed according to the target zoom curve to obtain a sequence of target images. Since the acquisition time of the images to be processed is earlier than the reception time of the zoom instruction, there is no need to wait for the images acquired at the time corresponding to the zoom instruction to respond, but instead, the images between the reception times of the zoom instructions are timely used for processing to obtain a sequence of target images to respond to the zoom instruction, thereby improving the timeliness of the zoom response.

[0040] The image processing method provided by the embodiment of the present application can be applied to an electronic device. Among them, the electronic device can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart TV, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. It should be noted that the electronic device can be a terminal or a server.

[0041] In an exemplary embodiment, as Figure 1 shown, an image processing method is provided that can be applied to an electronic device, including the following steps 102 to step 108. Among them:

[0042] Step 102, in response to a zoom instruction, obtain the current zoom ratio and the target zoom ratio.

[0043] Among them, the zoom instruction can refer to a focal length change instruction generated according to a zoom operation. Exemplarily, the zoom operation can refer to an indeterminate continuous zoom operation, that is, the user does not have a clear target zoom ratio during the zoom process, and adjusts the zoom ratio and views the result of the zoomed screen to achieve a better preview or recording screen. The zoom operation can also be sliding to a certain target zoom ratio. The zoom operation can be implemented by sliding a zoom control on the camera application interface. The zoom control can be a disk, a strip, or other forms of zoom controls. In some possible implementation manners, the zoom operation can also be implemented by sliding or rotating a hardware device. For example, a rotatable device is configured on the electronic device, and the zoom ratio is adjusted by rotating the rotatable device.

[0044] Exemplarily, the camera application page of the electronic device may be provided with zoom controls, and there may be one or more zoom controls. Different zoom controls correspond to different zoom ratios. By triggering the zoom control, zooming at the corresponding zoom ratio can be achieved. For example, by clicking to trigger the zoom control, it is possible to determine the start and stop points of the zoom operation by point-cutting the zoom control. The current zoom ratio is the starting point, and the target zoom ratio is the stop point. Alternatively, when the zoom ratios corresponding to the point-cuttable zoom controls are limited, the start and stop points of the zoom operation can also be determined by swiping.

[0045] Understandably, the current zoom ratio is usually for the current image frame, that is, the current zoom ratio corresponding to the current image frame is obtained. The current image frame refers to the image frame that needs to be previewed, photographed, or recorded currently. Correspondingly, the zoom instruction may be a zoom instruction in the camera preview scenario, the photo-taking scenario, or the recording scenario. The current zoom ratio refers to the zoom ratio of the current image frame. The target zoom ratio may be the zoom ratio currently issued by the camera application. The camera application will obtain the zoom ratio obtained by the zoom operation and issue the obtained zoom ratio as the target zoom ratio to the hardware abstraction layer. The target zoom ratio currently issued by the camera application refers to the target zoom ratio of the user's sliding operation or the user's point-cutting operation currently obtained by the camera application. Exemplarily, the camera application of the electronic device generates a zoom instruction in response to the zoom operation generated by the user by sliding the disk, or by rotating the hardware device, or by point-cutting the zoom control, and issues the zoom instruction to the hardware abstraction layer. In response to the zoom instruction, the hardware abstraction layer can obtain the current zoom ratio of the current image frame and the target zoom ratio currently issued by the camera application.

[0046] Step 104, determine the corresponding target zoom curve according to the current zoom ratio and the target zoom ratio.

[0047] The zoom curve is a pre-configured zoom processing method to make the zooming process smoother and can be expressed by different functions, such as linear functions, power functions, exponential functions, etc. In one possible implementation, a zoom curve between different zoom ratios is established in advance, and according to the current zoom ratio and the target zoom ratio, the corresponding target zoom ratio is obtained from the zoom curves between different zoom ratios. In another possible implementation, the corresponding zoom scene type is determined according to the current zoom ratio and the target zoom ratio, and the corresponding target zoom curve is obtained according to the zoom scene type. Different zoom scene types can be configured with different zoom curves. Among them, the zoom scene type refers to the scene type corresponding to the zoom operation. The zoom scene type includes, for example, a magnification scene (zoom in scene), a reduction scene (zoom out scene), or a magnification value scene. Among them, the magnification scene refers to the zoom operation scene where the zoom ratio changes from a small value to a larger value, the reduction scene refers to the zoom operation scene where the zoom ratio changes from a large value to a smaller value, and the magnification value scene includes, for example, specific value scenes such as the zoom operation scene from 0.6X to 2X, the zoom operation scene from 1X to 3X, or the zoom operation scene from 6X to 0.6X. Exemplarily, the electronic device can determine the zoom scene type according to the difference between the current zoom ratio and the target zoom ratio, or can also determine the zoom ratio according to the current zoom ratio and the target zoom ratio, determine the zoom scene type according to the zoom ratio, and then determine the corresponding target zoom curve according to the zoom scene type.

[0048] Step 106, obtain the image sequence to be processed, where the image sequence to be processed includes images whose acquisition time is earlier than the reception time of the zoom instruction.

[0049] Exemplarily, the image sequence to be processed may include one or more images to be processed. Multiple frames include two or more frames. In the cases of camera application for shooting preview, recording video, or taking pictures, the image sensor outputs image data at a preset frame rate, and the output image data is sequentially processed by multiple processing modules in the Image Signal Processor (ISP for short) and then stored in the buffer. For example, in the case of camera application for shooting preview, the image data will be sequentially obtained from the buffer for cropping and then displayed on the camera application interface. It can be understood that the multiple processing modules in the image signal processor can also be divided into multiple segments, and the result of each segment after processing can be stored in the segmented buffer, and the image sequence to be processed is obtained from the segmented buffer.

[0050] Images in the image sequence to be processed that are captured earlier than the reception time of the zoom instruction can also be referred to as pre-captured frame images. The number of pre-captured frame images can be determined according to the activation duration of the target image sensor or according to the buffer quantity of the image processing platform. The shorter the activation duration of the target image sensor, the fewer the number of pre-captured frame images; the longer the activation duration of the target image sensor, the more the number of pre-captured frame images. The target image sensor is the image sensor determined according to the target zoom ratio.

[0051] Step 108: Process the images in the image sequence to be processed according to the transition zoom ratios in the target zoom curve to obtain the target image sequence.

[0052] There are multiple transition zoom ratios on the target zoom curve. The transition zoom ratios can include intermediate zoom ratios between the current zoom ratio and the target zoom ratio. The transition zoom ratios can also include the current zoom ratio and the target zoom ratio. Any two transition zoom ratios can be the same or different, and the change step between the transition zoom ratios can be the same or different. The images in the image sequence to be processed can include a single frame of image output by a single image sensor or an image pair output by two image sensors. The image pair includes a single frame of image output by each of the two image sensors. A single frame of image or an image pair in the image sequence to be processed is processed to obtain a corresponding single frame of target image.

[0053] The target image sequence includes multiple target images. The timing of each target image in the target image sequence corresponds to the timing of the images in the image to be processed.

[0054] Exemplarily, the hardware abstraction layer of the electronic device processes each single frame of image or an image pair in the image sequence to be processed one by one according to the transition zoom ratios in the target zoom curve to obtain the target image sequence. Processing a single frame of image or an image pair in the image sequence to be processed with one transition zoom ratio to obtain a corresponding single frame of target image.

[0055] In this embodiment, in response to the zoom instruction, the corresponding target zoom curve is determined according to the current zoom ratio and the target zoom ratio, the image sequence to be processed is obtained, and the images in the image sequence to be processed are processed according to the target zoom curve to obtain the target image sequence. Since the capture time of the image to be processed obtained is earlier than the reception time of the zoom instruction, there is no need to wait for the image data captured at the corresponding time of the zoom instruction to come out before responding. Instead, the images between the reception time of the zoom instruction are promptly used for processing to obtain the target image sequence to respond to the zoom instruction, improving the timeliness of the zoom response.

[0056] In some exemplary embodiments, before responding to the zoom instruction, the method further includes: receiving the zoom instruction and the target zoom ratio sent by the camera application through the cross-process communication interface.

[0057] The cross-process communication interface (Android Interface Definition Language, abbreviated as AIDL) is an interface definition language for inter-process communication (Inter-Process Communication, abbreviated as IPC). By defining a standardized AIDL interface, cross-process data transfer and control between upper-layer applications or frameworks (such as camera services) and the underlying hardware abstraction layer (Hardware Abstraction Layer, abbreviated as HAL) can be achieved.

[0058] The hardware abstraction layer receives the zoom instruction and the target zoom ratio sent by the camera application through the cross-process communication interface. The camera application sending the zoom instruction and the target zoom ratio through the cross-process communication interface, compared with sending the zoom instruction and the target zoom ratio through the request method, enables the hardware abstraction layer to obtain the zoom instruction and the target zoom ratio in advance, so as to perform field-of-view cropping on the pre-capture frame image in a timely manner, respond to the zoom instruction in a timely manner, and achieve a significant improvement in followability.

[0059] In some exemplary embodiments, in response to the zoom instruction, a pre-capture frame image is obtained, and the pre-capture frame image is processed by the transition zoom ratio in the target zoom curve. Wait for the target image sensor to output a data stream. The algorithm side will perform fusion processing on the data stream output by the current image sensor and the data stream output by the target image sensor to obtain a first fused image, and then process the first fused image by the transition zoom ratio corresponding to the first fused image object in the target zoom curve to obtain a new target image. As Figure 2 shown, the above image processing method further includes:

[0060] Step 202, determining the target image sensor to be switched according to the target zoom ratio.

[0061] Different image sensors have different corresponding zoom ratio ranges. Comparing the current zoom ratio with the zoom ratio ranges corresponding to each image sensor can determine the current image sensor corresponding to the current zoom ratio. Comparing the target zoom ratio with the zoom ratio ranges corresponding to each image sensor can determine the target image sensor corresponding to the target zoom ratio. When the target image sensor is the same as the current image sensor, there is no need to turn on a new image sensor. When the target image sensor is different from the current image sensor, a new image sensor needs to be turned on.

[0062] In an exemplary embodiment, the zoom direction and the optical change point are determined according to the current zoom ratio and the target zoom ratio; the target image sensor is determined according to the zoom direction and the optical change point. The zoom direction may be the zoom-in direction or the zoom-out direction. For example, if the current zoom ratio is 1X and the target zoom ratio is 3X, the zoom direction is the zoom-in direction; if the current zoom ratio is 3X and the target zoom ratio is 1X, the zoom direction is the zoom-out direction. The optical change point is the zoom ratio threshold for switching from one camera to another. For example, in the stable state, the optical change point corresponding to the switch between the ultra-wide-angle lens and the wide-angle lens is 1.0, the optical change point corresponding to the switch between the wide-angle lens and the telephoto lens is 3.0, and the optical change point corresponding to the switch between the telephoto lens and the ultra-telephoto lens is 6.0. According to the current zoom ratio and the target zoom ratio, it can be determined which optical change points are passed through, and then the target image sensor can be determined.

[0063] Step 204, when it is detected that the target image sensor outputs a data stream, obtain the first image data output by the current image sensor and the second image data output by the target image sensor; the acquisition times of the first image data and the second image data are later than the reception time of the zoom command.

[0064] The target image sensor outputs a data stream, indicating that the target image sensor has been powered on and started to collect image data. At this time, both the current image sensor and the target sensor are performing exposure to output data streams, and the hardware abstraction layer will receive the first image data output by the current image sensor and the second image data output by the target image sensor. The first image data and the second image data are image data collected after the zoom command, so the acquisition times of the first image data and the second image data are both later than the reception time of the zoom command.

[0065] Step 206, perform fusion processing on the first image data and the second image data to obtain a first fused image.

[0066] Exemplarily, with the second image data as the reference, align the first image data to the second image data, and then perform fusion processing to obtain a first fused image.

[0067] Step 208, process the first fused image according to the transition zoom ratio corresponding to the first fused image in the target zoom curve to obtain a new target image.

[0068] Exemplarily, obtain the transition zoom ratio corresponding to the first fused image from the target zoom curve, and process the first fused image according to the transition zoom ratio corresponding to the first fused image to obtain a new target image.

[0069] In this embodiment, after the data stream output by the target image sensor for the currently switched scenario is detected, the hardware abstraction layer calls an algorithm to fuse the first image data output by the current image sensor and the second image data output by the target image sensor received, to obtain a first fused image, and then processes the first fused image using the transition zoom ratio corresponding to the target zoom curve to obtain a new target image. When the zoom ratio corresponding to the image to be processed reaches the target zoom ratio, this zooming is completed.

[0070] In some exemplary embodiments, as Figure 3 shown, taking the image preview / recording scenario as an example, the implementation process of this image processing method is described. This image processing method includes steps 302 to 318.

[0071] Step 302, start the camera application to start image preview / recording.

[0072] The user triggers the camera application, and the electronic device runs the camera application to perform image preview on the camera application interface. It can be understood that video recording can also be performed on the camera application interface.

[0073] Step 304, in response to a zoom operation, the camera application recognizes the zoom instruction and the corresponding target zoom ratio.

[0074] The camera application responds to the zoom operation implemented by the user's click or pinch or swipe operation method, recognizes the zoom event to generate a zoom instruction, and obtains the target zoom ratio.

[0075] Step 306, the hardware abstraction layer receives the zoom instruction and the target zoom ratio sent by the camera application through the cross-process communication interface.

[0076] Step 308, after receiving the zoom instruction, the hardware abstraction layer determines the target zoom curve based on the current zoom ratio and the target zoom ratio.

[0077] It can be understood that the specific implementation of step 308 can be the same as that of step 104. The hardware abstraction layer can determine the corresponding zoom scenario type according to the current zoom ratio and the target zoom ratio, obtain the corresponding target zoom curve according to the zoom scenario type, or obtain the corresponding target zoom ratio from the zoom curves between different zoom ratios according to the current zoom ratio and the target zoom ratio.

[0078] Step 310, determine the target image sensor according to the target zoom ratio, and control the target image sensor to turn on.

[0079] Exemplarily, by comparing the target zoom ratio with the zoom ratio ranges corresponding to each image sensor respectively, the target image sensor corresponding to the target zoom ratio can be determined.

[0080] Step 312: Obtain the pre-taken frame image in the image sequence to be processed, obtain the corresponding transitional zoom ratio of the pre-taken frame image according to the target zoom curve, process the pre-taken frame image according to the transitional zoom ratio to obtain the target image, and respond to the zoom command according to the target image.

[0081] Exemplarily, the pre-taken frame image can be sequentially obtained from the image sequence to be processed, the corresponding transitional zoom ratio of each pre-taken frame image can be obtained from the target zoom curve, and the field of view angle of the pre-taken frame image can be cropped using the transitional zoom ratio to obtain the target image corresponding to each pre-taken frame image.

[0082] Step 314: Detect whether it is a dual-frame scene. If so, execute Step 316; if not, execute Step 312.

[0083] The dual-frame scene refers to a scene where both the current image sensor and the target image sensor output data streams. Detecting whether it is a dual-frame scene includes detecting whether the target image sensor outputs a data stream. When it is detected that the target image sensor outputs a data stream, the dual-frame scene is detected. If it is not detected that the target image sensor outputs a data stream, the dual-frame scene is not detected.

[0084] When it is not detected that the target image sensor outputs a data stream, the image to be processed can continue to be obtained from the image sequence to be processed, the corresponding transitional zoom ratio of the image to be processed can be obtained from the target zoom curve, and the image to be processed can be cropped using the corresponding transitional zoom ratio of the image to be processed to obtain a new target image. The image sequence to be processed may include image frames whose acquisition time is later than the time when the zoom command is received. The image frame whose acquisition time is later than the time when the zoom command is received can be referred to as a post-taken image frame. The corresponding transitional zoom ratio of the post-taken frame image is obtained according to the target zoom curve, and the post-taken frame image is processed according to the corresponding transitional zoom ratio of the post-taken frame image to obtain a new target image.

[0085] Step 316: When it is detected that the current scene is a dual-frame scene, obtain the first image data output by the current image sensor and the second image data output by the target image sensor, fuse the first image data and the second image data to obtain the first fused image, and then crop the first fused image according to the target zoom curve to obtain a new target image.

[0086] Step 318: When the zoom ratio reaches the target zoom ratio, complete the current zoom process.

[0087] Step 320: Resume normal preview or recording.

[0088] In this embodiment, when a zoom operation generates a zoom command, the camera application sends the zoom command and the target zoom ratio to the hardware abstraction layer through the cross-process communication interface, enabling the hardware abstraction layer to determine the target zoom curve in advance based on the current zoom ratio and the target zoom ratio, and to obtain the pre-capture frame image, obtain the zoom ratio corresponding to the pre-capture frame image from the target zoom curve, crop the pre-capture frame image using the transitional zoom ratio, and then display it in the camera application, promptly responding to the zoom command, and greatly improving the responsiveness of the zoom operation.

[0089] After receiving the zoom event, the hardware abstraction layer can directly crop and validate the cached image frames according to the target zoom curve, and at the same time trigger the activation of the secondary camera (i.e., the camera corresponding to the target image sensor). As Figure 4As shown, in the image preview or video recording scenario, the image data collected by the current image sensor and the image data collected by the target image sensor are sequentially stored in the cache according to the time sequence. The cache may include a first cache and a second cache. The first cache is used to store the image data collected by the current image sensor, and the second cache is used to store the image data collected by the target image sensor. The first cache stores the 1st, 2nd, and 3rd frames collected by the current image sensor. When it is detected that the user triggers a zoom operation, the target image sensor is determined according to the target zoom ratio, and the target image sensor is pulled up. Since it generally takes 3 to 4 frames for the target image sensor to output a data stream from the trigger, at this time, the 1st to 3rd frame images are obtained from the first cache for processing, and the 1st to 3rd frames in the first cache are cropped frame by frame using the transitional zoom ratio in the target zoom curve, and the output images are displayed. At the same time, the 4th to 5th frame image data collected by the current image sensor continue to be cached in the first cache, and then the 4th and 5th frame image data are continuously read from the first cache and cropped using the corresponding transitional zoom ratio in the target zoom curve to output images for display; the 6th to 9th frame image data collected by the current image sensor are stored in the first cache, and the target image sensor starts to output a data stream and collects the 1st to 4th frame image data and stores them in the second cache. The hardware abstraction layer obtains the 6th frame image data from the first cache and the 1st frame image data from the second cache at the same time, starts the alignment and fusion processing to obtain the first fused image, crops the first fused image using the transitional zoom curve corresponding to the first fused image in the target zoom curve to obtain a new target image, and displays the new target image. Then, the 2nd frame collected by the target image sensor and the 7th frame collected by the current image sensor are sequentially aligned and fused, and then displayed after FOV cropping; the 3rd frame collected by the target image sensor and the 8th frame collected by the current image sensor are aligned and fused, and then displayed after FOV cropping; the 4th frame collected by the target image sensor and the 9th frame collected by the current image sensor are aligned and fused, and then displayed after FOV cropping.

[0090] As Figure 5As shown, the hand - following duration includes the zoom event monitoring duration, the duration for issuing the zoom event and the target zoom ratio, the power - on duration of the image sensor, the start - of - flow duration of the image sensor (or the upstream duration of the data stream), and the FOV (Field Of View) cropping and display - sending duration. Among them, the zoom event monitoring duration is generally within 50 ms (milliseconds); the duration for issuing the zoom event and the target zoom ratio is generally about 100 ms; the power - on duration of the image sensor varies from 60 ms to 100 ms; the start - of - flow duration of the image sensor (or the upstream duration of the data stream) is generally from 100 ms to 130 ms, and the FOV cropping and display - sending duration refers to the latter - half pipeline delay duration, generally several tens of milliseconds. Through the image - processing method provided by this application, the duration for issuing the zoom command and the target zoom ratio, the power - on duration of the image sensor, and the start - of - flow duration of the image sensor can be saved, and the zoom command can be responded to in a timely manner.

[0091] If the backend of the pipeline also supports cropping, the subsequent modules of the pipeline can be directly controlled to perform cropping processing, saving some of the pipeline delay time, further optimizing the hand - following property of zooming, and improving the timeliness of zoom response.

[0092] In some exemplary embodiments, the above - mentioned image - processing method further includes: the image - processing pipeline includes multiple cropping modules, and the cropping module closest to the outlet of the image - processing pipeline is selected to crop the pre - fetched frame image. For example, the image - processing pipeline includes 20 processing modules, and each frame of image data collected needs to be processed by 20 processing modules. Among them, the 7th processing module and the 15th processing module are both cropping modules. When a zoom command is generated, there may be image data being processed in all 20 processing modules. Then, the image data in the 15th cropping module can be selected for processing, and then continue to be processed by the remaining 5 processing modules and then displayed, which can save the delay time of the image - processing pipeline and improve the timeliness of zoom response.

[0093] Since this application performs pre - fetch frame processing based on the received zoom events, for the target point - cut scenario (e.g., from 1X to 0.6X, after a preset number of single frames of stable interval in the middle, and then point - cut to 1X, 3X, etc. again), it may capture the tail - remaining frames of the previous switch (although the previous zoom is completed, the dual cameras will still be on for a while). This may cause the algorithm side to mis - perform alignment or fusion (blending) animations (the algorithm side generally performs alignment or fusion processing using the dual - channel data after receiving the dual - channel data and cannot distinguish whether it is the dual - open frame of the current scenario or the dual - open frame left over from the previous switch), thus causing FOV alignment jitter or fusion jitter. Therefore, for this scenario, a special strategy and algorithm side need to be designed to synchronize this event to distinguish whether it is the dual - open frame of the current scenario (only perform alignment or fusion processing on the dual - open frames of the current scenario). As Figure 6 shown, when zooming from the 1X zoom ratio to the 0.6X zoom ratio, the first frame is the image data output by the wide - angle image sensor W corresponding to the wide - angle camera. The second to fifth frames are the dual - frame image data output by the wide - angle image sensor W corresponding to the wide - angle camera and the ultra - wide - angle image sensor UW corresponding to the ultra - wide - angle camera, that is, a pair of image pairs. At the 6th frame, the ultra - wide - angle camera becomes the main camera, the wide - angle camera becomes the secondary camera, and the corresponding wide - angle image sensor is turned off, completing the first zoom switch. At the 7th and 8th frames, it continues to be output by the ultra - wide - angle image sensor. When a second zoom switch operation from 0.6X to 1X is received at the 9th frame output, the pre - fetched 5 frames of images include the 4th, 5th, 6th, 7th, and 8th frames, where the 4th and 5th frames belong to the dual - open frames generated during the first zoom switch. Assuming that a new dual - open frame starts at the 10th frame, multiple image pairs continue to be output later. When the algorithm side obtains these dual - open frames, it will perform alignment or fusion, while the single - open frames in between cannot perform alignment or fusion, so it will cause a split in the effect (do it in the front, not in the middle, and do it again in the back). If the main and secondary cameras for the two consecutive switches are not the same pair of image sensors, it will cause the alignment directions to be completely in two directions, resulting in multi - direction offset.

[0094] Therefore, for this scenario, after a new dual - open frame starts, the main and secondary camera identification information synchronized to the algorithm needs to be set to two different image sensors (sensors), and even if the pre - fetched frames capture dual - open frame images before the new dual - open frame starts, since the main and secondary camera identification information is the same image sensor or the secondary camera identification information is empty, no alignment or fusion processing is required, thus avoiding the algorithm from mis - performing alignment or fusion processing.

[0095] In some exemplary embodiments, as Figure 7 shown, the above - mentioned processing of the images in the to - be - processed image sequence according to the transition zoom ratio in the target zoom curve to obtain the target image sequence includes:

[0096] Step 702: When obtaining a pair of images from the image sequence to be processed, obtain main camera identification information and secondary camera identification information; wherein, the pair of images includes one frame of image output by different image sensors respectively.

[0097] The image sequence to be processed may include images with a collection time earlier than the reception time of the zoom command, and images with a collection time later than the reception time of the zoom command. Images with a collection time earlier than the reception time of the zoom command may be referred to as pre-captured frame images, and images with a collection time later than the reception time of the zoom command may be referred to as post-captured frame images. The pre-captured frame images may include a pair of images output by different image sensors during the previous zoom process; the post-captured frame images may include a pair of images output by different image sensors during the current zoom process. For example, as shown in Figure 6 When zooming from the 1X point to 0.6X and then from the 0.6X point to 1X, the pre-captured frame images may include an image pair output by the fourth frame, and the image pair includes one frame of image data output by the ultra-wide-angle image sensor and the wide-angle image sensor respectively.

[0098] Step 704: Determine fusion indication information according to the main camera identification information and the secondary camera identification information.

[0099] The main camera identification information is used to record the identification of the main camera. The secondary camera identification information is used to record the identification of the secondary camera. For example, if the wide-angle camera is used as the main camera, the main camera identification information records the ID (Identity) information of the wide-angle image sensor corresponding to the wide-angle camera. If the ultra-wide-angle camera is used as the secondary camera, the secondary camera identification information records the ID information of the ultra-wide-angle image sensor corresponding to the ultra-wide-angle camera.

[0100] The fusion indication information may include at least one of the fusion indication information indicating to perform fusion and the fusion indication information indicating not to perform fusion. The first fusion identifier may be used to indicate to perform fusion, and the second fusion identifier may be used to indicate not to perform fusion. There are also various ways such as using the first fusion identifier to indicate to perform fusion, and the non-occurrence of the first fusion identifier indicates not to perform fusion.

[0101] Exemplarily, when the pair of images includes two frames of pre-captured frame images, the fusion indication information is not to perform fusion. When the pair of images includes two frames of post-captured frame images, the fusion indication information is to perform fusion.

[0102] Step 706: Process the pair of images according to the fusion indication information and the transition zoom ratio in the target zoom curve to obtain the target image corresponding to the pair of images.

[0103] The fusion indication information is used to indicate whether to fuse an image pair. Through the fusion indication information, it is possible to distinguish which image pairs need to be fused and which do not, avoiding incorrect fusion processing and thus avoiding discontinuity in the output image. In some exemplary embodiments, determining the fusion indication information based on the main camera identification information and the secondary camera identification information includes: generating fusion indication information indicating execution of fusion when both the main camera identification information and the secondary camera identification information are not empty and are different; generating fusion indication information indicating non-execution of fusion when the main camera identification information and the secondary camera identification information are the same or the secondary camera identification information is empty.

[0104] When both the main camera identification information and the secondary camera identification information are not empty and are different, the generated fusion indication information indicates execution of fusion. After the first zoom operation is completed and after waiting for the target number of frames, the second zoom operation is received, and fusion processing is required only after waiting for the secondary camera to output a stream during the second zoom operation.

[0105] In some exemplary embodiments, the above method further includes: determining a target image sensor to be switched according to the target zoom ratio; when no data stream output by the target image sensor is detected, configuring the secondary camera identification information to be empty, or configuring the main camera identification information and the secondary camera identification information to be the same current image sensor identification; when a data stream output by the target image sensor is detected, configuring the main camera identification information to be the identification of the current image sensor and the secondary camera identification information to be the identification of the target image sensor.

[0106] By detecting the result of the data stream output by the target image sensor, the main camera identification information and the secondary camera identification information are configured. Only when the data stream output by the target image sensor is detected, the main camera identification information is configured to be the identification of the current image sensor and the secondary camera identification information is configured to be the identification of the target image sensor. Thus, according to the main camera identification information, the secondary camera identification information, the sensor enable flag, and the situation of the two data streams, fusion indication information indicating execution of fusion can be generated, and the fusion indication information can be determined more accurately.

[0107] In some exemplary embodiments, the above image processing method further includes: obtaining a sensor activation flag and the number of data flow paths; step 704 includes: determining fusion indication information according to the main camera identification information, the secondary camera identification information, the sensor activation flag, and the number of data flow paths. The sensor activation flag is represented by a mask, and it can be known which sensors are activated. The number of data flow paths information may include one or more paths. When the sensor activation flag indicates that multiple sensors are activated, the number of data flow paths information is multiple, and both the main camera identification information and the secondary camera identification information are not empty and different, a fusion indication information indicating to perform fusion is generated. When the main camera identification information and the secondary camera identification information are the same or the secondary camera identification information is empty, a fusion indication information indicating not to perform fusion is generated. The "multiple" in "multiple or multiple paths" includes two or more.

[0108] The fusion indication information can be generated more accurately through the sensor activation flag, the number of data flow paths, the main camera identification information, and the secondary camera identification information.

[0109] As Figure 8 shown, the algorithm side determines the fusion indication information according to the main camera identification information, the secondary camera identification information, the sensor activation flag, and the number of data flow paths information. If the fusion indication information indicates to perform fusion, the image data output by the main and secondary cameras is aligned and / or fused to obtain a second fused image, and then the second fused image is cropped using the transition zoom ratio corresponding to the target zoom curve to obtain a new target image. If the fusion indication information indicates not to perform fusion, the image data output by the main image sensor corresponding to the main camera is obtained and cropped to obtain a new target image for display.

[0110] In some exemplary embodiments, the target image sequence includes multiple frames of target images. Processing the image sequence to be processed according to the fusion indication information and the transition zoom ratio in the target zoom curve to obtain a target image sequence includes: when the fusion indication information is to perform fusion, fusing the image pair to obtain a second fused image; processing the second fused image according to the transition zoom ratio corresponding to the second fused image in the target zoom curve to obtain the target image corresponding to the image pair.

[0111] The image pair includes the third image data output by the main image sensor corresponding to the main camera identification information and the fourth image data output by the secondary image sensor corresponding to the secondary camera identification information; the acquisition time of the third image data and the fourth image data is later than the reception time of the zoom command; the third image data and the fourth image data are fused to obtain a second fused image.

[0112] Obtain the transition zoom ratio corresponding to the image pair from the target zoom curve, and use the transition zoom ratio corresponding to the image pair to crop the image pair to obtain the target image corresponding to the image pair.

[0113] The main camera identification information is used to identify the ID identification of the main image sensor corresponding to the main camera, that is, the master id. The secondary camera identification information is used to identify the ID identification of the secondary image sensor corresponding to the secondary camera, that is, the slave id. Determine the current image sensor according to the current zoom ratio, and determine the target image sensor according to the target zoom ratio. The current image sensor is the main image sensor, and the target image sensor is the secondary image sensor. When the zoom ratio reaches the optical zoom point, the main image sensor switches to the target image sensor, and the current image sensor switches to the secondary image sensor.

[0114] Exemplarily, during the zooming process, the zoom ratio changes gradually. The electronic device obtains the current zoom ratio and compares the current zoom ratio with the optical zoom point of the target image sensor. When the current zoom ratio reaches the optical zoom point, the target image sensor is switched to the main image sensor, the current image sensor is switched to the secondary image sensor, and the secondary image sensor is turned off. Obtain the transition zoom ratio corresponding to the second fused image from the target zoom curve, and crop the second fused image according to the transition zoom ratio corresponding to the second fused image to obtain a new target image.

[0115] In this embodiment, when the fusion indication information is to perform fusion, the image pair is fused to obtain the second fused image. The transition zoom ratio corresponding to the second fused image is obtained from the target zoom curve, and the second fused image is processed through the corresponding transition zoom ratio to obtain the target image corresponding to the image pair. The dual-open frame image fusion is started only when the fusion indication information is to perform fusion, avoiding misalignment or misfusion, and avoiding the splitting of the image screen during the zooming process.

[0116] In some exemplary embodiments, processing the image pair according to the fusion indication information and the transition zoom ratio in the target zoom curve to obtain the target image corresponding to the image pair includes: when the fusion indication information is not to perform fusion, obtaining the third image data output by the main image sensor corresponding to the main camera identification information from the image pair, and processing the third image data according to the transition zoom ratio corresponding to the third image data in the target zoom curve to obtain the target image corresponding to the third image data.

[0117] The third image data can be pre-taken frame image data or post-taken frame image data. When the fusion indication information is not to perform fusion, the third image data output by the main image sensor is included in the image sequence to be processed, and the third image data is processed to obtain the corresponding target image. Even if there is double-open frame image data for the pre-taken frame, since the fusion indication information is not to perform fusion, the third image data output by the main image sensor can be obtained for processing, and there is no need to obtain the image data output by the secondary image sensor. This avoids the situation of incorrect fusion display due to obtaining the remaining tail double-open frame of the previous zoom scene by the pre-taken frame strategy.

[0118] In an exemplary embodiment, the above image processing method further includes: in response to a point-cut zoom instruction, the current zoom ratio and the target zoom ratio, obtaining a target zoom curve according to the current zoom ratio and the target zoom ratio; in response to the point-cut zoom instruction, obtaining a pre-taken frame image collected by the current image sensor, and processing the pre-taken frame image according to the transition zoom ratio in the target zoom curve to obtain the corresponding target image; after detecting the data stream output by the target image sensor, obtaining the first image data output by the current image sensor and the second image data output by the target image sensor, fusing the first image data and the second image data to obtain a first fused image, and processing the first fused image according to the transition zoom ratio corresponding to the first fused image in the target zoom curve to obtain the corresponding new target image.

[0119] When a point-cut zoom event occurs, only the image data output by the main camera in the current zoom scene is obtained, that is, the pre-taken frame image output by the current image sensor. After the data stream output by the target image sensor is detected, the two data streams output by the current image sensor and the target sensor are obtained for fusion processing, and then cropping and display are performed. There is no need for the algorithm side to make a distinction, and the secondary camera frames left over from the previous switch will not be sent to the algorithm side, which can further save power.

[0120] In some exemplary embodiments, obtaining the image sequence to be processed includes: in response to a zoom instruction, obtaining the image sequence to be processed from the first buffer corresponding to the current image sensor.

[0121] The image data collected by the current image sensor is stored in the first buffer. After receiving the zoom instruction, the image sequence to be processed is obtained from the first buffer. The image sequence to be processed includes pre-taken frame images whose acquisition time is earlier than the reception time of the zoom instruction.

[0122] Correspondingly, the above method further includes: in the case of detecting the data stream output by the target image sensor, obtaining images from the first buffer corresponding to the current image sensor and the second buffer corresponding to the target image sensor to obtain the image sequence to be processed.

[0123] Different image sensors correspond to different caches. The first cache is used to store the image data collected by the current image sensor. The second cache is used to store the image data collected by the target image sensor. In the case of detecting a double-opening frame, the first image data output by the current image sensor is obtained from the first cache, and the second image data output by the target image sensor is obtained from the second cache. The first image data and the second image data are used as a pair of images to be processed.

[0124] In some exemplary embodiments, the above method further includes: in response to a startup instruction of the camera application, controlling the power-on of the image sensors corresponding to the wide-angle camera and the ultra-wide-angle camera.

[0125] In the case of responding to the startup instruction of the camera application, controlling the power-on of the image sensors corresponding to the wide-angle camera and the ultra-wide-angle camera, and temporarily not powering on other cameras can save power. Moreover, in a scenario where a zoom event occurs, since the image sensors corresponding to the ultra-wide-angle camera and the wide-angle camera are both powered on, the zoom response can be better improved.

[0126] In some exemplary embodiments, the above method further includes: determining a zoom direction according to the current zoom ratio and the target zoom ratio, and in the case where the zoom direction is the zoom-in direction, controlling the power-on of the image sensor corresponding to the target zoom ratio.

[0127] In some exemplary embodiments, the above method further includes: the hardware abstraction layer receives the shooting mode sent by the camera application in response to a shooting operation through the cross-process communication interface, and controls the corresponding image sensor to perform a shooting action according to the shooting mode.

[0128] The shooting operation may be an image preview or a video recording operation or a photo-taking operation. The shooting mode may include a video recording mode, a shooting mode, etc. Sending the shooting mode through the cross-process communication interface can transmit the shooting mode information faster and improve the response efficiency of the shooting operation.

[0129] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps in other steps.

[0130] Based on the same inventive concept, an embodiment of the present application further provides an image processing apparatus for implementing the above-mentioned image processing method. The solution provided by this apparatus for solving problems is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the following image processing apparatuses can refer to the limitations on the image processing method in the foregoing, and will not be repeated here.

[0131] In an exemplary embodiment, as Figure 9 shown, an image processing apparatus 900 is provided, including a magnification acquisition module 902, a zoom curve determination module 904, an image acquisition module 906, and an image processing module 908.

[0132] The magnification acquisition module 902 is configured to acquire a current zoom magnification and a target zoom magnification in response to a zoom instruction.

[0133] The zoom curve determination module 904 is configured to determine a corresponding target zoom curve according to the current zoom magnification and the target zoom magnification.

[0134] The image acquisition module 906 is configured to acquire an image sequence to be processed, where the image sequence to be processed includes an image whose acquisition time is earlier than the reception time of the zoom instruction.

[0135] The image processing module 908 is configured to process the images in the image sequence to be processed according to the transition zoom magnification in the target zoom curve to obtain a target image sequence.

[0136] In this embodiment, in response to the zoom instruction, a corresponding target zoom curve is determined according to the current zoom magnification and the target zoom magnification, the image sequence to be processed is acquired, and the images in the image sequence to be processed are processed according to the target zoom curve to obtain a target image sequence. Since the acquisition time of the image to be processed is earlier than the reception time of the zoom instruction, there is no need to wait for the image data acquired at the corresponding time of the zoom instruction to come out before responding, but instead, the images between the reception time of the zoom instruction are promptly used for processing to obtain a target image sequence to respond to the zoom instruction, improving the timeliness of the zoom response.

[0137] In some exemplary embodiments, the above apparatus further includes a reception module. The reception module is configured to receive a zoom instruction and a target zoom magnification sent by a camera application through a cross-process communication interface.

[0138] In some exemplary embodiments, as Figure 10 shown, the above apparatus further includes a sensor determination module 910 and a fusion module 912.

[0139] The sensor determination module 910 is configured to determine a target image sensor to be switched according to the target zoom magnification.

[0140] The image acquisition module 906 is further configured to, when detecting that the target image sensor outputs a data stream, acquire first image data output by the current image sensor corresponding to the current zoom ratio and second image data output by the target image sensor; the acquisition times of the first image data and the second image data are later than the reception time of the zoom command.

[0141] The fusion module 912 is configured to perform a fusion process on the first image data and the second image data to obtain a first fused image.

[0142] The image processing module 908 is further configured to process the first fused image according to the transition zoom ratio corresponding to the first fused image in the target zoom curve to obtain a new target image.

[0143] In some exemplary embodiments, as Figure 11 shown, the above device further includes an information acquisition module 914 and an indication information determination module 916.

[0144] The information acquisition module 914 is configured to acquire main camera identification information and sub-camera identification information when acquiring a pair of image pairs from the to-be-processed image sequence.

[0145] The indication information determination module 916 is configured to determine fusion indication information according to the main camera identification information and the sub-camera identification information.

[0146] The image processing module 908 is further configured to process the image pair according to the fusion indication information and the transition zoom ratio of the target zoom curve to obtain the target image corresponding to the image pair.

[0147] In some exemplary embodiments, the indication information determination module 916 is further configured to generate fusion indication information indicating to perform fusion when both the main camera identification information and the sub-camera identification information are not empty and different; and generate fusion indication information indicating not to perform fusion when the main camera identification information and the sub-camera identification information are the same or the sub-camera identification information is empty.

[0148] In some exemplary embodiments, the above device further includes a sensor determination module and a configuration module.

[0149] The sensor determination module is configured to determine the target image sensor to be switched according to the target zoom ratio;

[0150] The configuration module is used to configure the secondary camera identification information as empty, or the main camera identification information and the secondary camera identification information as the same current image sensor identification when the data stream output by the target image sensor is not detected; and to configure the main camera identification information as the identification of the current image sensor and the secondary camera identification information as the identification of the target image sensor when the data stream output by the target image sensor is detected.

[0151] In some exemplary embodiments, the image processing module 908 is further configured to, when the fusion indication information is to perform fusion, perform fusion processing on the image pair to obtain a second fused image; and process the second fused image according to the transition zoom ratio corresponding to the second fused image in the target zoom curve to obtain the target image corresponding to the image pair. The image pair includes third image data output by the main image sensor corresponding to the main camera identification information and fourth image data output by the secondary image sensor corresponding to the secondary camera identification information; the acquisition time of the third image data and the fourth image data is later than the reception time of the zoom instruction; when the fusion indication information is to perform fusion, perform fusion processing on the third image data and the fourth image data to obtain a second fused image; and process the second fused image according to the target zoom curve to obtain the corresponding target image.

[0152] In some exemplary embodiments, the target image sequence includes target images; the image processing module 908 is further configured to, when the fusion indication information is not to perform fusion, obtain the third image data output by the main image sensor corresponding to the main camera identification information from the image pair, and process the third image data according to the transition zoom ratio corresponding to the third image data in the target zoom curve to obtain the target image corresponding to the third image data.

[0153] In some exemplary embodiments, the image acquisition module 906 is further configured to, in response to a zoom instruction, obtain a sequence of images to be processed from the first buffer corresponding to the current image sensor; and to obtain images from the first buffer corresponding to the current image sensor and the second buffer corresponding to the target image sensor when the data stream output by the target image sensor is detected, to obtain a sequence of images to be processed.

[0154] In some exemplary embodiments, the above device further includes a control module. The control module is configured to, in response to a start instruction of the camera application, control the image sensors corresponding to the wide-angle camera and the ultra-wide-angle camera to power on.

[0155] In some exemplary embodiments, the above device further includes a receiving module and a control module. The receiving module is configured to receive the shooting mode sent by the camera application in response to a shooting operation through the cross-process communication interface. The control module is configured to control the corresponding image sensor to perform a shooting action according to the shooting mode.

[0156] It can be understood that the functions implemented by the respective modules in the above device are the same as the corresponding steps in the above image processing method, and the implementation processes are the same. Therefore, they will not be elaborated here. For the specific implementation, refer to the implementation of the corresponding steps in the above image processing method.

[0157] Each module in the above image processing device can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in the processor in the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0158] In an exemplary embodiment, an electronic device is provided. The electronic device can be a terminal, and its internal structure diagram can be as Figure 12 shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used for exchanging information between the processor and external devices. The communication interface of the electronic device is used for communicating with external terminals in a wired or wireless manner. The wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the outer shell of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0159] Those skilled in the art can understand that Figure 12 the structure shown in

[0160] In an exemplary embodiment, an electronic device is provided, which includes a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps of the image processing method in the above embodiment are implemented.

[0161] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method in the above embodiment are implemented.

[0162] In an embodiment, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the steps of the image processing method in the above embodiment are implemented.

[0163] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0164] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0165] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0166] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: In response to a zoom instruction, obtaining a current zoom ratio and a target zoom ratio; Determining a corresponding target zoom curve according to the current zoom magnification and the target zoom magnification; Acquire a sequence of images to be processed, wherein the sequence of images to be processed includes images whose acquisition time is earlier than the reception time of the zoom instruction; The images in the to-be-processed image sequence are processed according to the transition zoom ratio in the target zoom curve to obtain a target image sequence.

2. The method according to claim 1, characterized in that Before responding to the zoom instruction, the method further includes: Receive the zoom command and target zoom ratio sent by the camera application through the cross-process communication interface.

3. The method according to claim 1, characterized in that The method further comprises: Determining a target image sensor to be switched according to the target zoom ratio; In the case of detecting the output data stream of the target image sensor, obtaining first image data output by the current image sensor corresponding to the current zoom ratio and second image data output by the target image sensor; the acquisition time of the first image data and the second image data is later than the reception time of the zoom instruction; fusing the first image data and the second image data to obtain a first fused image; The first fused image is processed according to the transition zoom ratio corresponding to the first fused image in the target zoom curve to obtain a new target image.

4. The method according to claim 1, characterized in that: The step of processing the images in the to-be-processed image sequence according to the transition zoom ratio in the target zoom curve to obtain a target image sequence includes: In the case of obtaining a pair of images from the sequence of images to be processed, obtaining primary camera identification information and secondary camera identification information; wherein the image pair includes a frame of image output by different image sensors; Determining fusion indication information according to the main camera identification information and the secondary camera identification information; The image pair is processed according to the fusion indication information and the transition zoom ratio in the target zoom curve to obtain a target image corresponding to the image pair.

5. The method according to claim 4, characterized in that The determining, according to the main camera identification information and the secondary camera identification information, fusion indication information includes: When both the main camera identification information and the secondary camera identification information are not empty and are different, generating fusion indication information indicating execution of fusion; When the main-photographing identification information and the secondary-photographing identification information are the same or the secondary-photographing identification information is empty, fusion indication information indicating that fusion is not to be performed is generated.

6. The method according to claim 4, characterized in that The method further comprises: Determining a target image sensor to be switched according to the target zoom ratio; In the case that the target image sensor output data stream is not detected, the secondary camera identification information is configured to be empty, or the primary camera identification information and the secondary camera identification information are the same current image sensor identification; In the case where the target image sensor is detected to output a data stream, the main camera identification information is configured as an identification of the current image sensor, and the secondary camera identification information is configured as an identification of the target image sensor.

7. The method according to claim 4, characterized in that The step of processing the image pair according to the fusion indication information and the transition zoom ratio in the target zoom curve to obtain a target image corresponding to the image pair includes: When the fusion instruction information indicates to perform fusion, Performing fusion processing on the image pair to obtain a second fused image; The second fused image is processed according to the transition zoom ratio corresponding to the second fused image in the target zoom curve to obtain a target image corresponding to the image pair.

8. The method according to claim 4, characterized in that The step of processing the image pair according to the fusion indication information and the transition zoom ratio in the target zoom curve to obtain a target image corresponding to the image pair includes: When the fusion indication information indicates not to perform fusion, third image data output by the main image sensor corresponding to the main camera identification information is obtained from the image pair, and the third image data is processed according to the transition zoom ratio corresponding to the third image data in the target zoom curve to obtain a target image corresponding to the third image data.

9. The method according to claim 1, characterized in that: The step of obtaining a sequence of images to be processed comprises: In response to the zoom instruction, acquiring a sequence of images to be processed from a first buffer corresponding to the current image sensor; The method further comprises: When it is detected that the target image sensor outputs a data stream, images are acquired from a first buffer corresponding to the current image sensor and a second buffer corresponding to the target image sensor to obtain an image sequence to be processed.

10. The method according to claim 1, characterized in that The method further comprises: In response to a start-up instruction of the camera application, an image sensor corresponding to the wide-angle camera and an image sensor corresponding to the ultra-wide-angle camera are controlled to be powered on.

11. The method according to claim 1, characterized in that: The method further comprises: The hardware abstraction layer receives a shooting mode sent by the camera application through the cross-process communication interface in response to a shooting operation, and controls the corresponding image sensor to perform a shooting action according to the shooting mode.

12. An image processing device, characterized in that: The device comprises: A zoom ratio acquisition module, used to acquire a current zoom ratio and a target zoom ratio in response to a zoom instruction; A zoom curve determination module, used to determine a corresponding target zoom curve according to the current zoom ratio and the target zoom ratio; An image acquisition module, used for acquiring a sequence of images to be processed, wherein the sequence of images to be processed includes images whose acquisition time is earlier than the reception time of the zoom instruction; The image processing module is used to process the images in the to-be-processed image sequence according to the transition zoom ratio in the target zoom curve to obtain a target image sequence.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images