Zoom response method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a zoom response method, electronic equipment and a storage medium. In the method, the electronic equipment generates a preview image frame request carrying a zoom magnification value in response to a zoom operation of a user, and the zoom magnification value takes effect preferentially on preview image frame data generated based on previous exposure, so that an effective link of the zoom magnification value is shortened, the zoom response delay is reduced, and the user experience is improved. And the zooming operation experience of the user is improved.

Description

Zoom response method, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on February 13, 2023, with application number 202310136978.X and application name “Zoom Response Method, Electronic Device and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of smart terminal technology, and in particular to a zoom response method, electronic device, and storage medium. Background Art

[0003] With the development of smart devices such as mobile phones and tablets, the camera function is becoming increasingly important to users. Taking mobile phones as an example, in order to provide users with a better photo taking experience, the terminal is usually equipped with multiple cameras with different focal lengths to capture scenes at different distances.

[0004] When shooting scenes at different distances, users can zoom (or adjust focus) through related operations to obtain clear images or videos. However, the system response delay of smart devices such as mobile phones to user zoom operations may be long, resulting in a long time difference between the user's zoom operation and the corresponding change in the FOV (Field of View) on the shooting preview interface, resulting in a poor user experience.

[0005] Summary of the Invention

[0006] To address the above technical issues, embodiments of the present application provide a zoom response method, electronic device, and storage medium. In this method, the electronic device generates a preview image frame request carrying a zoom magnification value in response to a user's zoom operation. The zoom magnification value is preferentially applied to preview image frame data generated based on previously exposed data, thereby shortening the zoom magnification value validation process, reducing zoom response latency, and thereby improving the user's zoom operation experience.

[0007] In a first aspect, embodiments of the present application provide a zoom response method. The method is applied to an electronic device and includes:

[0008] In the process of the electronic device processing the preview image frame requests in sequence according to the generation order to generate and display the preview image frames, in response to the received zoom operation, the electronic device generates a first preview image frame request; the first preview image frame request includes a first zoom ratio value, which is used to generate the first preview image frame; when the electronic device performs spatial alignment processing on the second preview image frame data to calculate the cropping data, the first cropping data corresponding to the second preview image frame data is calculated according to the first zoom ratio value; wherein, the second preview image frame data is generated by controlling the exposure of the first camera in response to the second preview image frame request, and the generation time of the second preview image frame request is earlier than the generation time of the first preview image frame request; the electronic device crops the second preview image frame data according to the first cropping data to obtain a second preview image frame corresponding to the second preview image frame request, and displays the second preview image frame; wherein, the field of view angle of the second preview image frame matches the first zoom ratio value.

[0009] The first zoom magnification value is the zoom magnification value that needs to take effect in advance.

[0010] In this way, the zoom ratio value corresponding to the first preview image frame request takes effect in advance on the preview image frame data corresponding to the second preview image frame request, thereby shortening the zoom ratio value's effectiveness link and reducing the zoom response delay, so that the user's zoom operation adjusts the camera application preview interface to a certain zoom ratio, and the time difference between the preview image corresponding to the zoom ratio and the display of the electronic device screen is small, thereby improving the user's zoom operation experience.

[0011] According to the first aspect, the zoom magnification corresponding to the second preview image frame request is a second zoom magnification value; the second preview image frame data is obtained by preliminary cropping the original exposure image frame data based on the second zoom magnification value; the electronic device calculates the first cropping data corresponding to the second preview image frame data based on the first zoom magnification value, which may include: if it is determined that the target cropping of the second preview image frame data can be completed based on the first zoom magnification value based on the first zoom magnification value, the second zoom magnification value and the second preview image frame data, then the first cropping data corresponding to the second preview image frame data is calculated based on the first zoom magnification value.

[0012] Considering that whether in the Zoom In scenario or the Zoom Out scenario, when the user's zoom operation involves lens switching, the preview image frame obtained by exposure with the original lens (i.e., the lens used for image exposure before the lens switching) may not meet the new zoom magnification value, the electronic device first determines whether the current cropping can be completed based on the new zoom magnification value to ensure the smooth execution of the preview image data processing flow.

[0013] According to the first aspect, or any implementation of the first aspect above, the method also includes: if it is determined that the target cropping of the second preview image frame data cannot be completed based on the first zoom magnification value according to the first zoom magnification value, and the second preview image frame data, the electronic device obtains the third preview image frame data corresponding to the second preview image request and collected by the second camera; the electronic device calculates the second cropping data corresponding to the third preview image frame data according to the first zoom magnification value, and crops the third preview image frame data according to the second cropping data to obtain the third preview image frame corresponding to the second preview image frame request, and displays the third preview image frame.

[0014] In this way, the electronic device performs image exposure operations through multiple cameras at the same time, and obtains multiple preview image frame data corresponding to the same preview image frame request, so that when the zoom scene involves lens switching, the new zoom ratio value can be effective on the preview image frame data collected by the target camera, solving the problem that the new zoom ratio value cannot take effect in advance.

[0015] According to the first aspect, or any implementation of the first aspect above, the method further includes: in response to a zoom operation by the user, the electronic device turns on the second camera.

[0016] In this way, when the electronic device anticipates that the first zoom ratio value needs to be effective in advance on the image data collected by the second camera, the second camera is turned on in advance to meet the condition that the first zoom ratio can be effective in advance.

[0017] According to the first aspect, or any implementation method of the first aspect above, the electronic device obtains third preview image frame data collected by the second camera corresponding to the second preview image request, including: if the first zoom magnification value belongs to the zoom range of the second camera, the electronic device obtains the third preview image frame data collected by the second camera corresponding to the second preview image frame request.

[0018] In this way, if the zoom ratio value that needs to take effect in advance belongs to the zoom range of other cameras, the electronic device will turn on the target camera in advance and perform image exposure operations through multiple cameras at the same time to obtain multiple preview image frame data corresponding to the same preview image frame request. The new zoom ratio value can be taken effect on the preview image frame data collected by the target camera, solving the problem that the new zoom ratio value cannot take effect in advance.

[0019] According to the first aspect, or any implementation of the first aspect above, the electronic device obtains third preview image frame data collected by the second camera corresponding to the second preview image request, including: if the first zoom magnification value and the second zoom magnification value both belong to the zoom range of the first camera, and the first zoom magnification value is less than the second zoom magnification value, then the electronic device obtains the third preview image frame data collected by the second camera corresponding to the second preview image frame request; wherein the zoom magnification value corresponding to the second camera is less than the zoom magnification value corresponding to the first camera.

[0020] In this way, in the Zoom out scenario, if the first zoom magnification value and the second zoom magnification value belong to the zoom range of the same camera, the first zoom magnification value can be taken effect in advance on the corresponding image data collected by the target camera with a smaller zoom magnification value, shortening the effective path of the first zoom magnification value and reducing the response delay of the zoom operation.

[0021] According to the first aspect, or any implementation of the first aspect above, the method also includes: when the third preview image frame data collected by the second camera corresponding to the second preview image frame request is not obtained, the electronic device calculates the third cropping data corresponding to the second preview image frame data according to the current zoom magnification value, and crops the second preview image frame data according to the third cropping data to obtain a fourth preview image frame corresponding to the second preview image frame request, and displays the fourth preview image frame.

[0022] The current zoom factor value indicates the most recent zoom factor value in effect as of the current time. As preview image frame requests are processed sequentially in the order in which they are generated to generate and display preview image frames, the current zoom factor value is updated in real time. For example, after the first preview image frame request is processed and before the second preview image frame request is processed, the current zoom factor value is the zoom factor value in effect for the image frame data corresponding to the first preview image frame request.

[0023] According to the first aspect, or any implementation of the first aspect above, the method further includes: if the first zoom ratio value is greater than the second zoom ratio value, and the difference between the two is greater than a preset threshold, determining that the target cropping of the second preview image frame data cannot be completed based on the first zoom ratio value.

[0024] According to the first aspect, or any implementation of the first aspect above, the method also includes: if the first zoom magnification value is less than the second zoom magnification value, and the reserved edge data of the second preview image frame data cannot meet the image field angle corresponding to the first zoom magnification value, then the electronic device determines that the target cropping of the second preview image frame data cannot be completed based on the first zoom magnification value.

[0025] According to the first aspect, or any implementation of the first aspect above, the electronic device includes: a camera service and a camera HAL; the camera HAL includes an interface module, a real-time pipeline processing module, and an offline pipeline processing module; the real-time pipeline processing module includes a sensor node and an IFE module, and the offline pipeline processing module includes a SAT module and an IPE module;

[0026] Among them, the camera service continuously generates preview image frame requests and sends each preview image frame request to the camera HAL; after receiving each preview image frame request, the interface module in the camera HAL sends each preview image frame request to the sensor node; the sensor node controls the camera to perform exposure and image output operations according to each preview image frame request in turn; after receiving the original preview image frame data corresponding to each preview image frame request, the IFE module preliminarily crops the original preview image frame data according to the zoom magnification value corresponding to the preview image frame request and sends it to the SAT module; the SAT module performs spatial alignment processing on the received preview image frame data to calculate the cropped data, and sends the cropped data calculation result to the IPE module; the IPE module performs target cropping processing on the corresponding preview image frame data according to the cropped data calculation result to obtain the preview image frame data to be displayed.

[0027] According to the first aspect, or any implementation of the first aspect above, the method also includes: after the interface module receives each preview image frame request, if the preview image frame request includes a zoom magnification value, the zoom magnification value is written into the target queue; the target queue is a first-in-first-out queue.

[0028] The target queue is used to store zoom ratio values ​​that need to take effect in advance.

[0029] In this way, by globally maintaining a target queue for storing zoom magnification values ​​that need to take effect in advance, the problem that the speed at which the camera HAL receives preview image frame requests and the speed at which the Offline Pipeline processes preview image frame requests cannot be synchronized frame by frame can be solved.

[0030] According to the first aspect, or any implementation of the first aspect above, the method also includes: when the SAT module performs spatial alignment processing on the received fifth preview image frame data to calculate the cropping data, if the third zoom magnification value is read in the target queue, the fourth cropping data corresponding to the fifth preview image frame data is calculated according to the third zoom magnification value; the IPE module performs target cropping processing on the fifth preview image frame data according to the fourth cropping data to obtain the fifth preview image frame to be displayed.

[0031] According to the first aspect, or any implementation of the first aspect above, the method further includes: when the SAT module performs spatial alignment processing on the received fifth preview image frame data to calculate the cropping data, if the zoom magnification value is not read in the target queue, the fifth cropping data corresponding to the fifth preview image frame data is calculated according to the current zoom magnification value; the IPE module performs target cropping processing on the fifth preview image frame data according to the fifth cropping data to obtain the sixth preview image frame to be displayed; wherein the fifth preview image frame data is obtained by the sensor node controlling the exposure of the third camera based on the third preview image frame request, and the current zoom magnification value is used to indicate the zoom magnification value that is most recently effective from the current time.

[0032] According to the first aspect, or any implementation of the first aspect above, the fifth preview image frame data is obtained by the IFE module performing preliminary cropping on the original exposure image frame data corresponding to the third preview image frame request based on the fourth zoom magnification value corresponding to the third preview image frame request; if the SAT module determines, based on the third zoom magnification value, the fourth zoom magnification value and the fifth preview image frame data, that the IPE module cannot complete the target cropping of the fifth preview image frame data based on the third zoom magnification value, then the SAT module obtains the seventh preview image frame data corresponding to the third preview image frame request and collected by the fourth camera; the SAT module calculates the sixth cropped data corresponding to the seventh preview image frame data based on the third zoom magnification value; the IPE module crops the seventh preview image frame data based on the sixth cropped data to obtain the seventh preview image frame to be displayed corresponding to the third preview image frame request.

[0033] According to the first aspect, or any implementation of the first aspect above, if the third zoom ratio value belongs to the zoom range of the fourth camera, the SAT module obtains the seventh preview image frame data collected by the fourth camera corresponding to the third preview image frame request.

[0034] According to the first aspect, or any implementation of the first aspect above, if the third zoom magnification value and the fourth zoom magnification value both belong to the zoom range of the third camera, and the third zoom magnification value is less than the fourth zoom magnification value, the SAT module obtains the seventh preview image frame data collected by the fourth camera corresponding to the third preview image frame request; wherein the zoom magnification value corresponding to the fourth camera is less than the zoom magnification value corresponding to the third camera.

[0035] According to the first aspect, or any implementation of the first aspect above, the electronic device further includes a multi-camera decision module; the method further includes: in response to the user's zoom operation, the multi-camera decision module controls the electronic device to turn on the fourth camera in advance.

[0036] According to the first aspect, or any implementation of the first aspect above, the zoom operation includes: clicking the zoom control, sliding the zoom control, and sliding the preview interface with two fingers.

[0037] When the user's zoom operation is a sliding operation (such as sliding the zoom control with one finger or sliding the preview interface with two fingers, etc.), reducing the system's zoom response delay can effectively improve the tracking performance of the user's zoom operation; when the user's zoom operation is a click operation (such as clicking the zoom control, etc.), shortening the zoom ratio value's effective link can effectively improve the system's response speed to the user's zoom operation.

[0038] In a second aspect, embodiments of the present application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the zoom response method of the first aspect and any one of the first aspects.

[0039] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0040] In a third aspect, embodiments of the present application provide a computer-readable storage medium comprising a computer program, which, when executed on an electronic device, causes the electronic device to execute the zoom response method according to the first aspect and any one of the first aspects.

[0041] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0042] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed, enables a computer to execute the zoom response method according to the first aspect or any one of the first aspects.

[0043] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0044] In a fifth aspect, the present application provides a chip comprising a processing circuit and a transceiver pin. The transceiver pin and the processing circuit communicate with each other via an internal connection path, and the processing circuit executes the zoom response method according to the first aspect or any one of the first aspects to control the receiving pin to receive a signal and to control the transmitting pin to send a signal.

[0045] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figures 1a-1d are schematic diagrams of exemplary application scenarios;

[0047] FIG2 is a schematic diagram illustrating a hardware structure of an electronic device;

[0048] FIG3 is a schematic diagram illustrating a software architecture of an electronic device;

[0049] FIG4 is a schematic diagram illustrating module interactions involved in a zoom response process;

[0050] FIG5 is an exemplary diagram showing a zoom response delay phenomenon;

[0051] FIG6 is a schematic diagram illustrating module interactions involved in a zoom response method;

[0052] 7a-7b are schematic diagrams showing exemplary zoom ratio values ​​taking effect first;

[0053] FIG8 is a schematic diagram illustrating an exemplary zoom ratio queue;

[0054] FIG9a-FIG9b are exemplary comparative examples of image field angle ranges;

[0055] Figures 10a-10b are schematic diagrams of module interaction when the multi-camera decision module turns on the target camera in advance;

[0056] FIG10c is an exemplary diagram illustrating exposure and transmission of image frame data when multiple cameras are simultaneously turned on;

[0057] FIG11 is a schematic diagram showing an exemplary zoom ratio value in effect;

[0058] FIG12 is a schematic diagram showing an exemplary flow of the SAT module in the Offline Pipeline processing preview image frame data. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0061] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0062] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0063] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0064] With the advancement of device and semiconductor technologies, the integration of cameras with different focal lengths into a single device has become a trend to meet user photography needs. By integrating these cameras into the same device, the preview interface's FOV can be adjusted simply by adjusting the zoom factor (also known as the zoom ratio), achieving the effect of zooming in or out.

[0065] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) or a dedicated camera (such as a SLR camera, a card camera), etc. This application does not impose any restrictions on the specific types of the above-mentioned terminal devices.

[0066] The following explanations are given using a mobile phone as an example of a terminal device.

[0067] FIG1a exemplifies an application scenario. The mobile phone detects a user operation (such as a touch / click operation) on the camera application icon, and in response to the operation, the shooting interface 10 shown in (1) of FIG1a can be displayed. The shooting interface 10 can be the user interface of the default shooting mode of the camera application, and the user can complete the shooting on this interface. The camera application is an image shooting application on electronic devices such as smartphones and tablets. This application does not limit the name of the application. That is, the user can open the shooting interface 10 of the camera application by clicking the camera application icon. It can be understood that the default camera in the default shooting mode is not limited to the rear camera, and the mobile phone can also set the front camera as the default camera. That is, after opening the camera application, the mobile phone can display the image captured by the front camera in the preview area 102, which can be used for the user to take pictures through the default front camera.

[0068] As shown in (1) of FIG1a, the shooting interface 10 may include a parameter adjustment area 101, a preview area 102, a camera mode option area 103, a gallery shortcut control 105, a camera flip control 104, a shutter control 106, and a zoom control 107. Among them, the various controls in the parameter adjustment area 101 are used to adjust corresponding shooting parameters, including but not limited to: a flash setting control, an AI recognition switch setting control, a color standard setting control, and a more detailed camera setting control. The preview area 102 can be used to display a preview image, which is an image captured in real time by the mobile phone through the camera. The mobile phone can refresh the display content in the preview area 102 in real time to facilitate the user to preview the image currently captured by the camera. One or more shooting mode options can be displayed in the camera mode option 103. The one or more shooting mode options may include but are not limited to: an aperture mode option, a night scene mode option, a portrait mode option, a photo mode option, a video mode option, a professional option mode, and more options. It is understandable that one or more shooting mode options can be displayed on the interface as text information, such as "aperture", "night scene", "portrait", "photo", "video", "professional", "more", and can also be displayed as icons or other forms of interactive elements (IE), which are not limited in this application.

[0069] Continuing with reference to (1) in FIG1a , the zoom control 107 is used to trigger the mobile phone to adjust the zoom ratio, thereby adjusting the FOV of the shooting preview interface 102. The "1x" displayed on the zoom control 107 indicates that the current optical zoom ratio of the mobile phone is 1x, and the subsequent x represents the zoom ratio.

[0070] In one example, as shown in (1) of FIG1a , a user clicks on the zoom control 107, and the mobile phone adjusts the current zoom ratio of the mobile phone in response to the user's click operation on the zoom control 107. For example, the user clicks on the zoom control 107 to trigger the zoom ratio of the mobile phone to be adjusted from "1x" to "3x", as shown in (2) of FIG1a . At this time, the FOV of the mobile phone's shooting preview interface becomes smaller, and the shooting scene in the shooting preview interface becomes larger. Similarly, the user can click on the zoom control 107 multiple times to adjust (increase or decrease) the current zoom ratio of the mobile phone multiple times according to the preset ratio difference.

[0071] In another example, as shown in (1) of FIG1b, the user slides the zoom control 107, and the mobile phone adjusts the current zoom magnification of the mobile phone in response to the user's sliding operation on the zoom control 107, and the FOV of the shooting preview interface and the size of the captured scene change accordingly. When the user slides the zoom control 107 to adjust the current zoom magnification of the mobile phone, the display style of the zoom control 107 changes to indicate that the zoom magnification of the mobile phone is being adjusted, as shown in (2) of FIG1b. Continuing to refer to (2) of FIG1b, in response to the user's sliding operation, the current zoom magnification 108 of the mobile phone changes and is displayed on the preview interface, so that the user can clearly know the current zoom magnification of the mobile phone. When the current zoom magnification of the mobile phone is adjusted to 3.0x, if the user stops sliding the zoom control and lets go, the current zoom magnification of the mobile phone is set to 3.0x, the display style of the zoom control 107 is restored and indicates that the current zoom magnification is "3x", and the zoom magnification adjustment of the mobile phone is completed, as shown in (2) of FIG1a. In the example shown in FIG1 b , taking the case where the user slides the zoom control 107 upward to increase the zoom ratio as an example, the case where the user slides the zoom control 107 downward to decrease the zoom ratio is similar, which will not be described in detail here.

[0072] In another example, as shown in (1) in FIG1c, the user slides the preview interface with two fingers facing away from each other, and the mobile phone responds to the user's two-finger sliding operation by increasing the current zoom ratio of the mobile phone, the FOV of the shooting preview interface becomes smaller, and the captured scene becomes larger. Similarly, if the user slides the preview interface with two fingers facing toward each other, the mobile phone responds to the user's two-finger sliding operation by reducing the current zoom ratio of the mobile phone, the FOV of the shooting preview interface becomes larger, and the captured scene becomes smaller. When the user slides the preview interface with two fingers (slides with two fingers facing away from each other or slides with two fingers facing toward each other) to adjust the current zoom ratio of the mobile phone, the display style of the zoom control 107 changes to indicate that the zoom ratio of the mobile phone is being adjusted, as shown in (2) in FIG1c. Continuing to refer to (2) in FIG1c, in response to the user's two-finger sliding operation, the current zoom ratio 109 of the mobile phone changes accordingly and is displayed on the preview interface, so that the user can clearly know the current zoom ratio of the mobile phone. When the current zoom ratio of the mobile phone is adjusted to 3.0x, if the user stops the two-finger sliding operation and releases the hand, the current zoom ratio of the mobile phone is set to 3.0x, the display style of the zoom control 107 is restored and indicates that the current zoom ratio is "3x", and the zoom ratio adjustment of the mobile phone is completed. At this time, you can refer to (2) in Figure 1a.

[0073] However, in the zoom scenarios involved in, for example, FIG1a, FIG1b, and FIG1c, the mobile phone system has a high response delay to the user's zoom operation (such as clicking the zoom control, sliding the zoom control, sliding the preview interface with two fingers, etc.). Specifically, it can be manifested as a large difference between the time when the user's zoom operation (or magnification adjustment operation) adjusts the mobile phone to a certain zoom magnification and the time when the preview image corresponding to the zoom magnification is displayed on the mobile phone screen. Taking the user sliding the zoom control 107 to adjust the current zoom magnification of the mobile phone as an example, combined with FIG1d (1) and FIG1d (2), when the user slides the zoom control to adjust the zoom magnification of the mobile phone to 3.0x, the current zoom magnification 108 of the mobile phone on the preview interface is displayed as "3.0x", but the FOV of the shooting preview interface is not adjusted synchronously. Instead, the preview image corresponding to the zoom magnification "3.0x" is displayed on the mobile phone screen after a response delay. In the example shown in (2) of Figure 1d, the current zoom magnification 108 of the mobile phone displayed on the mobile phone interface has been adjusted, while the FOV of the shooting preview interface has not been adjusted accordingly. At this time, the zoom magnification corresponding to the mobile phone preview image is inconsistent with the current zoom magnification 108 of the mobile phone "3.0x" displayed on the mobile phone interface. Continuing to refer to (3) of Figure 1d, when the preview image corresponding to the zoom magnification "3.0x" is displayed on the mobile phone screen, due to the user's continuous sliding of the zoom control, the current zoom magnification 108 of the mobile phone set by the user may have become "3.2x". At this time, the zoom magnification "3.0x" corresponding to the mobile phone preview image is also inconsistent with the current zoom magnification 108 of the mobile phone "3.2x" displayed on the mobile phone interface.

[0074] The ideal response latency for a mobile phone system to a user's zoom operation is 0ms. This means that the FOV of the camera preview interface and the current zoom factor displayed on the phone interface adjust synchronously with the user's zoom operation. However, due to the limitations of the terminal's hardware processing latency and software architecture design, the mobile phone system cannot achieve a 0ms response latency for user zoom operations. Understandably, the lower the mobile phone system's response latency to user zoom operations, the better the user experience. When adjusting the FOV of the camera preview interface based on a zoom operation, the response latency of the first and last preview frames is most significant in impacting the user experience. A higher response latency for the first preview frame will indicate that the user's zoom operation has begun, but the preview interface's FOV has not adjusted in real time. A higher response latency for the last preview frame will indicate that the user's zoom operation has stopped, but the preview interface's FOV is still changing.

[0075] With the prevalence of touchscreen devices, clicks and swipes have become the primary means of human-computer interaction. When users click or slide zoom controls, or swipe through the preview interface with two fingers, the tracking of the FOV of the device's camera preview interface becomes a key factor affecting the user experience. Therefore, reducing the device's response latency to zoom operations to improve the user experience is an urgent issue.

[0076] As shown in Figure 2, a schematic diagram of the structure of the electronic device 100 is shown. Optionally, the electronic device 100 can be a terminal, which can also be called a terminal device. The terminal can be a device with a camera such as a cellular phone or a tablet computer (pad), which is not limited in this application. It should be noted that the schematic diagram of the structure of the electronic device 100 can be applicable to the mobile phones in Figures 1a to 1d. It should be understood that the electronic device 100 shown in Figure 2 is only an example of an electronic device, and the electronic device 100 can have more or fewer components than shown in the figure, can combine two or more components, or can have different component configurations. The various components shown in Figure 2 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0077] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0078] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0079] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0080] The processor 110 may further include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0081] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0082] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0083] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0084] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0085] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0086] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0087] The wireless communication module 160 can provide wireless communication solutions for application on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0088] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0089] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0090] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0091] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0092] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0093] Camera 193 is used to capture still images or videos. The scene generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV.

[0094] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1. A camera comprises a photosensor, a lens, and a package that secures the photosensor and lens. The camera is used to capture images within a specific magnification range. The smaller the focal length of the camera 193, the larger its field of view, the wider the viewing area, and the more objects that can be captured. Conversely, the larger the focal length of the camera 193, the smaller its field of view, the smaller the viewing area, and the fewer but more distant objects that can be captured. For example, if N is 3, the three cameras are a wide-angle camera, a main camera, and a telephoto camera, respectively, with magnification ranges of 0.4-1.0, 1.0-3.5, and 3.5-30. These magnification ranges are for illustrative purposes only. The performance of these cameras in terms of field of view is as follows: the wide-angle camera outperforms the main camera, and the main camera outperforms the telephoto camera.

[0095] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0096] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, such as enabling the electronic device 100 to implement the zoom response method in the embodiment of the present application. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0097] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0098] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0099] The pressure sensor is used to sense pressure signals and convert the pressure signals into electrical signals. In some embodiments, the pressure sensor can be provided on the display screen 194. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor.

[0100] The gyroscope sensor can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) can be determined by the gyroscope sensor.

[0101] The accelerometer can detect the magnitude of acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the accelerometer can detect the magnitude and direction of gravity. The accelerometer can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0102] A touch sensor, also known as a "touch panel," can be mounted on display screen 194. The touch sensor and display screen 194 together form a touch screen, also known as a "touch screen." The touch sensor detects touch operations applied to or near the touch sensor. The touch sensor can communicate the detected touch operations to the application processor to determine the type of touch event.

[0103] Keys 190 include a power button (also known as a power button), a volume button, and the like. Keys 190 may be mechanical keys or touch-sensitive keys. Electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100.

[0104] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or touch vibration feedback. For example, touch operations on different applications (such as taking photos, playing audio, etc.) can correspond to different vibration feedback effects.

[0105] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0106] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0107] FIG3 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0108] The layered architecture of electronic device 100 divides the software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), and the kernel layer (also known as the driver layer).

[0109] It is understood that Figure 3 is merely an example. For example, between the application framework layer and the HAL layer, the Android runtime and system library layers may also be included. The Android runtime includes core libraries and a virtual machine, responsible for scheduling and management of the Android system. The system library may include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGL ES), and a 2D graphics engine (e.g., SGL).

[0110] The application layer can include a series of application packages. As shown in Figure 3, an application package can include applications such as a camera application, a gallery application, and applications with camera functions. Application packages can also include applications such as calls, calendars, maps, navigation, music, videos, and short messaging applications.

[0111] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0112] As shown in Figure 3, the application framework layer may include a camera service that can be called by the camera application to implement functions related to shooting. In addition, the application framework layer may also include a window manager, content provider, view system, telephony manager, resource manager, notification manager, etc.

[0113] The window manager is used to manage window programs. It can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0114] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0115] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0116] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0117] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0118] The Notification Manager allows applications to display notifications in the status bar. These messages can be used to convey informational messages and can disappear automatically after a short pause, without requiring user interaction. For example, notifications are used to notify the completion of downloads, message reminders, and so on. Notifications can also appear in the system's top status bar as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Notifications can also take the form of text messages displayed in the status bar, beeping sounds, vibrations on electronic devices, or flashing indicator lights.

[0119] It should be noted that the camera application can also call the content provider, resource manager, notification manager, window manager, view system, etc. according to actual business needs, and the embodiments of the present application do not impose any restrictions on this.

[0120] The kernel layer is the layer between hardware and software. The kernel layer includes at least a camera driver. This camera driver can be used to drive hardware modules with shooting functions, such as a camera sensor. In other words, the camera driver is responsible for exchanging data with the camera sensor. Of course, the kernel layer may also include display drivers, audio drivers, sensor drivers, etc., and this embodiment of the application does not impose any restrictions on this.

[0121] In addition, the HAL layer can encapsulate the drivers in the kernel layer and provide a calling interface to the application framework layer, shielding the implementation details of the low-level hardware. As shown in Figure 3, the above HAL layer can include the Camera HAL, the multi-camera decision module, etc.

[0122] Camera HAL is the core software framework of Camera, which includes an interface module, a sensor node, and an image processing module. In one implementation, the image processing module may include an image signal processing front end (IFE) module, a spatial alignment transform (SAT) module, and an image signal processing back end (IPE) module. Among them, the related processing involved in the sensor node and IFE module belongs to the real-time pipeline processing, and the related processing involved in the SAT module and IPE module belongs to the offline pipeline processing.

[0123] The sensor node, image processing module, and interface module are components of the image data and control command transmission pipeline within the Camera HAL. Different components have different functions. For example, a sensor node can be a control node for a camera sensor, which can control the camera sensor through a camera driver. Another example is an interface module, which can be a software interface for the application framework layer, used to exchange data with the application framework layer. The interface module can also exchange data with other modules within the HAL (such as the multi-camera decision module, image processing module, and sensor node). Another example is the image processing module, which processes raw image data transmitted by the camera sensor. The IFE module performs preliminary cropping of preview image frames captured by the camera sensor, preserving image margins. The SAT module performs spatial alignment of image data based on the zoom factor and the SAT algorithm, determining cropping and warping data to smooth the preview image and reduce artifacts. The IPE module performs cropping and warping of the preview image frame data based on the SAT module's calculation results.

[0124] The multi-camera decision module can determine the camera sensor to be used for image generation based on the application scenario, such as the front camera sensor or the rear camera sensor, or the main camera sensor, wide-angle camera sensor, or telephoto camera sensor in the rear camera. The camera application can pass user-selected camera mode, zoom parameters, and other information to the camera service at the application framework layer, which then passes it to the multi-camera decision module via the HAL layer interface module.

[0125] In an embodiment of the present application, the multi-camera decision module can also estimate the target camera to be switched based on the user's zoom operation and activate the camera sensor of the target camera in advance through the camera driver. In addition, the multi-camera decision module can also set the camera that currently outputs the image from the multiple cameras whose camera sensors are activated.

[0126] In this way, the multi-camera decision module can determine the camera that actually outputs the image and turn on the camera based on the camera mode, zoom parameters, etc., or it can estimate and turn on the camera to be switched in advance in the zoom scene.

[0127] It is understood that the layers in the software structure shown in FIG3 and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, and this application does not limit this.

[0128] In addition, it is understandable that in order to implement the zoom response method in the embodiment of the present application, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0129] In one implementation, the display of the preview image frame can be based on the Android architecture provided by Google and the software implementation of the Camera HAL layer of the Qualcomm platform. Figure 4 exemplifies the module interaction process involved in a zoom operation response method. In the preview scenario of the camera application, referring to Figure 4, when the user performs a zoom operation, the touch module of the mobile phone generates a touch point reporting operation in response to the user's zoom operation and sends it to the camera application. After the camera application receives the touch point reporting event, it converts the point reporting coordinate data corresponding to the touch point reporting event into a zoom ratio (ZoomRatio) value and sends it to the camera service in the application framework layer. The camera service writes the zoom ratio value into the preview image frame request and sends it to the camera HAL in the HAL layer. Among them, in the preview scenario of the camera application, the camera service will continue to send preview image frame requests to the camera HAL. The following example uses the camera service to write the zoom ratio value into the 10th preview image frame request (referred to as preview image frame request 10) and send it to the camera HAL. In the camera HAL, each preview image frame request is processed sequentially through the Realtime Pipeline and the Offline Pipeline. The Realtime Pipeline includes processing by the Sensor node and the IFE module, while the Offline Pipeline includes processing by the SAT module and the IPE module. As shown in Figure 4, a preview image frame request 10 is sent to the Sensor node. Based on the preview image frame request 10, the Sensor node controls the camera sensor through the camera driver to perform exposure and image output. After the camera sensor completes the exposure and image output, the raw image data corresponding to the preview image frame request 10 is sent to the IFE module through the camera driver. The IFE module performs preliminary cropping on the raw image data corresponding to the preview image frame request 10 and sends the processed image data to the SAT module. The IFE module can perform preliminary cropping on the raw image data corresponding to the preview image frame request 10 based on the zoom factor value corresponding to the preview image frame request 10, preserving the image edge area to facilitate spatial alignment of the image by the SAT module. The SAT module performs image spatial alignment based on the zoom factor value corresponding to preview image frame request 10 and the relevant SAT algorithm, calculates cropping and distortion data corresponding to the image data, and sends the calculation results and the image data corresponding to preview image frame request 10 to the IPE module. Based on the SAT calculation results, the IPE module performs further cropping and image distortion operations on the image data corresponding to preview image frame request 10, obtaining a preview image frame corresponding to preview image frame request 10 (referred to as preview image frame 10). Preview image frame 10 can then be processed by the application framework layer, such as the camera service, and fed back to the camera application for UI display.

[0130] It can be seen that when the user performs a zoom operation, the preview image frame request carrying the relevant zoom magnification value sent by the camera service has a long processing link in the camera HAL, resulting in a high response of the zoom operation.

[0131] In addition, due to the rotating buffer mechanism between the application framework layer and the HAL layer of the camera system in the Android architecture. When the user opens the camera application, the camera service will first continuously send multiple (for example, 8) preview image frame requests to the camera HAL, and send another preview image frame request to the camera HAL after receiving a preview image frame feedback from the camera HAL. In the camera HAL, the Sensor node will sequentially control the camera sensor through the camera driver to perform exposure and image output operations based on these multiple preview image frame requests. In other words, for any preview image frame request, the Sensor node will control the camera sensor to perform exposure and image output operations for this preview image frame request only after all previous preview image frame requests have been processed and exposed by the Sensor node.

[0132] The following still takes the example of the camera service writing the zoom magnification value set based on the user's zoom operation into the preview image frame request 10. The preview image frame request 10 is sent by the camera service to the camera HAL. As shown in (1) in Figure 5, after the interface module in the camera HAL receives the preview image frame request 10, it sends the preview image frame request 10 to the sensor node for processing. Assuming that at the same time, the sensor node is processing the preview image frame request 6 to expose the preview image frame 6 (in the embodiment of the present application, the image frame obtained by processing the preview image frame request n is referred to as the preview image frame n), the preview image frame request 10 needs to wait for the sensor node to process the preview image frame requests 6, 7, 8, and 9 in sequence before it is processed by the sensor node to expose the preview image frame 10. Taking the frame rate as 30fps as an example, the exposure time of the camera sensor for one image frame is 30ms, and the time for the Sensor node to process the preview image frame request 6-9 to expose the preview image frame 6-9 is 4*30=120ms, that is, the preview image frame request 10 needs to wait for 120ms before it is processed by the Sensor node. In the scenario shown in Figure 5 (1), when the interface module in the camera HAL receives the preview image frame request 10, the Sensor node is controlling the exposure of the preview image frames 6-9 in sequence, the IFE module is processing the preview image frame 5, the SAT module is performing calculations on the cropping of the preview image frame 4, the IPE module is completing the cropping and distortion of the preview image frame 3, and the preview image frame 2 is transmitted to the application framework layer and synthesized by SurfaceFlinger for UI display. Continuing to refer to Figure 5 (2), in the camera HAL, since the preview image frame requests 2-10 are processed in sequence, the preview image frames 2-10 are also exposed, produced, and displayed in sequence. When the user performs a zoom operation to trigger a preview image frame request 10 for a zoom ratio change, the phone displays preview image frame 2. Preview image frame 10 corresponding to preview image frame request 10 can only be displayed after preview image frames 2-9. The processing chain for each preview image frame is the same, as shown in FIG4 . Due to the aforementioned queuing exposure mechanism, the chain processing time for each preview image frame request in the camera HAL is relatively long. Furthermore, when the user adjusts the zoom ratio through a zoom operation, the processing delay of the preview image frame request corresponding to the target zoom ratio will also be longer, resulting in problems such as those shown in FIG1d , leading to a poor zoom tracking chirality experience for the user.

[0133] To address the above issues, embodiments of the present application provide a zoom response method. In this method, when a user adjusts the zoom ratio through a zoom operation, the terminal system shortens the zoom ratio's effective link, thereby significantly reducing the zoom ratio response delay, thereby increasing the terminal system's response speed to the user's zoom operation and improving the user experience.

[0134] The following still takes the example of the camera service writing the zoom magnification value set based on the user's zoom operation into the preview image frame request 10, and the preview image frame request 10 is sent by the camera service to the camera HAL. In an embodiment of the present application, as shown in Figure 6, in the camera HAL, after the interface module receives the preview image frame request 10, it directly forwards the zoom magnification value carried in the preview image frame request 10 to the Offline Pipeline, so that the relevant modules in the Offline Pipeline can process based on the latest zoom magnification carried by the preview image frame request 10. In this way, since the zoom magnification value is directly forwarded to the Offline Pipeline, the effective link of the zoom magnification value skips the processing of the Sensor node and the IFE module, so that the delay of the zoom response can be greatly reduced.

[0135] Continuing with FIG6 , when the interface module in the camera HAL receives preview image frame request 10 , it is assumed that the sensor node is sequentially controlling the exposure of preview image frames 6-9, the IFE module is processing preview image frame 5, the SAT module is performing calculations for cropping preview image frame 4, the IPE module is completing the cropping and warping of preview image frame 3, and preview image frame 2 is transmitted to the application framework layer for UI display through SurfaceFlinger synthesis. After receiving preview image frame request 10 , the interface module in the camera HAL forwards the zoom factor value carried in preview image frame request 10 to the SAT module, so that the SAT module performs image space alignment processing on the preview image frame data exposed earlier (as shown in FIG6 , the image data of preview image frame 4) based on the zoom factor value corresponding to preview image frame request 10 and the relevant SAT algorithm, calculates the cropping data and warping data corresponding to the image data of preview image frame 4, and sends the calculation results and the image data of preview image frame 4 to the IPE module, which then completes the actual cropping and warping operations. As a result, the zoom magnification value corresponding to preview image frame request 10, that is, the zoom magnification value corresponding to preview image frame 10, takes effect in advance on preview image frame 4. Assuming the exposure time of each image frame is 30 milliseconds, the zoom response delay in this scenario can be reduced by (10-4)*30=180 milliseconds.

[0136] It should be noted that, when the user performs a zoom operation to adjust the zoom magnification value (for example, the scene shown in (2) in Figure 1b and (2) in Figure 1c), the preview image frame request sent by the camera service to the camera HAL will carry the zoom magnification value, otherwise the preview image frame request sent by the camera service to the camera HAL will not carry the zoom magnification value. After the camera HAL receives the preview image frame request sent by the camera service, if the preview image frame request does not carry the zoom magnification value, the camera HAL can determine the zoom magnification value corresponding to the preview image frame request based on the default or the zoom magnification value corresponding to the previous preview image frame request. For example, the default zoom magnification value of the camera application is 1.0, then after the camera application is started, the camera HAL will use the default zoom magnification of 1.0 as the zoom magnification value corresponding to the first preview image frame request received. As another example, if the preview image frame request 9 received by the camera HAL does not carry a zoom magnification value, the zoom magnification value corresponding to the preview image frame request 8 is used as the zoom magnification value corresponding to the preview image frame request 9 .

[0137] In the application scenario shown in Figure 6, preview image frame requests 4-9 do not carry a zoom factor value. The zoom factors corresponding to preview image frame requests 4-9 are all determined by the camera HAL based on the zoom factor value of the corresponding previous preview image frame request (i.e., preview image frame requests 3-8). In this embodiment of the present application, to reduce the zoom factor response delay, the SAT module performs image space alignment processing on the image data of preview image frame 4 based on the zoom factor value carried in preview image frame request 10, rather than the zoom factor value corresponding to preview image frame request 4 set by the camera HAL.

[0138] Similarly, if the preview image frame request 11 received by the camera HAL also carries a zoom ratio value (such as a zoom ratio of 1), in order to reduce the response delay of the zoom ratio, assuming that the SAT module is performing image space alignment processing on the image data of the preview image frame 5, the SAT module performs image space alignment processing on the image data of the preview image frame 5 according to the zoom ratio 1 carried by the preview image frame request 11, and so on.

[0139] In one scenario, as shown in Figure 7a, the SAT module performs spatial image alignment on the image data of preview image frame 5 according to the zoom magnification 1 carried in preview image frame request 10, performs spatial image alignment on the image data of preview image frame 6 according to the zoom magnification 2 carried in preview image frame request 11, performs spatial image alignment on the image data of preview image frame 7 according to the zoom magnification 3 carried in preview image frame request 13, and so on. In other words, the zoom magnification 1 carried in preview image frame request 10 will take effect on preview image frame 4 in advance, the zoom magnification 2 carried in preview image frame request 11 will take effect on preview image frame 5 in advance, the zoom magnification 3 carried in preview image frame request 12 will take effect on preview image frame 6 in advance, and so on. If the user continues to perform zoom operations, preview frame request 16 issued by the camera service will still carry a zoom magnification value (e.g., zoom magnification 7). The camera HAL can then preemptively apply the zoom magnification 7 carried in preview image frame request 16 to preview image frame 10. Therefore, when the user does not stop the zoom operation, the zoom magnification value set due to the user's zoom operation will take effect in advance on the preview image frame exposed earlier.

[0140] In another scenario, as shown in Figure 7b, the zoom factor 1 carried in preview image frame request 10 will take effect in advance on preview image frame 4, the zoom factor 2 carried in preview image frame request 11 will take effect in advance on preview image frame 5, the zoom factor 3 carried in preview image frame request 12 will take effect in advance on preview image frame 6, and the zoom factor 4 carried in preview image frame request 13 will take effect in advance on preview image frame 7. If the user has stopped the zoom operation at this time, the subsequent preview image frame requests sent by the camera service to the camera HAL will no longer carry a zoom factor value, such as preview image frame requests 14, 15, and 16. After the user stops the zoom operation, the camera HAL can determine the zoom factor value corresponding to each subsequent preview image frame request (which does not carry a zoom factor value) based on the zoom factor value set by the user (i.e., the currently effective zoom factor value). In the example shown in Figure 7b, the zoom magnification value set by the user is zoom magnification 4, so the SAT module can apply the zoom magnification 4 set by the user to the subsequently processed preview image frames, such as preview image frames 8, 9, 10, 11, 12, 13, 14, 15, etc.

[0141] It should be noted that, referring to FIG7b , taking preview image frame request 8 without a zoom factor value as an example, the zoom factor value effective on the corresponding preview image frame 8 is the currently most recently effective zoom factor of 4 (i.e., the zoom factor value effective on preview image frame 7); taking preview image frame request 9 without a zoom factor value as an example, the zoom factor value effective on the corresponding preview image 9 is also the currently most recently effective zoom factor of 4 (i.e., the zoom factor value effective on preview image frame 8). The same is true for preview image frame requests 14-16 without a zoom factor value, and will not be further described. Continuing with preview image frame request 10 with a zoom factor of 1 as an example, zoom factor 1 has already been effective on preview image frame 4, and the zoom factor value effective on the corresponding preview image 10 is the currently most recently effective zoom factor of 4 (i.e., the zoom factor value effective on preview image frame 10). The same is true for preview image frame requests 11-13 with zoom factors, and will not be further described. In this way, when there is no zoom ratio value that needs to take effect first, the camera HAL always applies the current latest zoom ratio value to the next frame image, which can prevent the field of view angle of the preview interface from jumping and avoid giving the user a poor visual experience.

[0142] In some scenarios, the speed at which the camera HAL receives preview image frame requests and the speed at which the Offline Pipeline processes them cannot guarantee frame-by-frame synchronization. For example, if the camera HAL receives preview image frame requests faster than the Offline Pipeline processes them, the zoom magnification values ​​carried in some preview image frame requests may be skipped and not take effect. Conversely, the zoom magnification value carried in the same preview image frame request may take effect on more than two preview image frames.

[0143] To address the issue of the camera HAL receiving preview image frame requests and the Offline Pipeline processing them at different speeds, the camera HAL maintains a global zoom factor queue. This first-in, first-out queue is used to store the zoom factor values ​​carried by received preview image frame requests. As shown in Figure 8, after the camera HAL (e.g., the interface module) receives a preview image frame request and parses it to find that it carries a zoom factor value, it writes the zoom factor value to the zoom factor queue. When a zoom factor value exists in the zoom factor queue, the Offline Pipeline sequentially removes the zoom factor value from the queue and makes it effective.

[0144] In the example shown in Figure 7a, the zoom magnifications 1 to 7 carried by preview image frame requests 10 to 16, respectively, are sequentially written to the zoom magnification queue. When the Offline Pipeline processes the image data of preview image frame 4 and reads zoom magnification 1 from the zoom magnification queue, it removes zoom magnification 1 from the zoom queue and processes the image data of preview image frame 4 based on zoom magnification 1. Similarly, when the Offline Pipeline processes the image data of preview image frame 5 and reads zoom magnification 2 from the zoom queue, it removes zoom magnification 2 from the zoom queue and processes the image data of preview image frame 5 based on zoom magnification 2, and so on.

[0145] In the example shown in Figure 7b, the zoom magnifications 1 to 4 carried by preview image frame requests 10 to 13, respectively, are written into the zoom magnification queue in sequence. When the Offline Pipeline processes the image data of preview image frame 4, it reads that zoom magnification 1 exists in the zoom magnification queue. It then removes zoom magnification 1 from the zoom queue and processes the image data of preview image frame 4 based on zoom magnification 1. Similarly, when the Offline Pipeline processes the image data of preview image frame 7, it reads that zoom magnification 4 exists in the zoom magnification queue. It then removes zoom magnification 4 from the zoom queue and processes the image data of preview image frame 7 based on zoom magnification 4. Subsequently, when the Offline Pipeline processes the image data of preview image frame 8, it reads that no zoom magnification value exists in the zoom magnification queue. At the same time, the zoom magnification value currently set by the user (i.e., the most recently effective) is zoom magnification 4. The most recently effective zoom factor is the zoom factor that was in effect for the previous frame of preview image data. For example, when the Offline Pipeline is processing the image data for preview image frame 8, the most recently effective zoom factor is the zoom factor that was in effect for preview image frame 7. Therefore, the Offline Pipeline processes the image data for preview image frame 8 at a zoom factor of 4. When processing the image data for preview image frames 8-15, the Offline Pipeline reads that there are no zoom factors in the zoom factor queue. Since the most recently effective zoom factor is 4, the Offline Pipeline continues processing the image data at zoom factor 4.

[0146] In one possible implementation, the camera HAL further includes a zooming state detection module for detecting whether the electronic device is currently in a zooming state. If the camera HAL receives a preview image frame request carrying a zoom magnification value, it can be determined that the electronic device is currently in a zooming state.

[0147] Exemplarily, if the multiple preview image frame requests received consecutively by the camera HAL do not carry a zoom ratio value, the Zooming status detection module can determine that the electronic device is not currently in a zoom scene; otherwise, the Zooming status detection module determines that the electronic device has not exited the zoom scene.

[0148] In this embodiment, when the zooming state detection module determines that the electronic device is currently in a zooming state, if the preview image frame request received by the camera HAL does not include a zoom factor value, the camera HAL uses the zoom factor value corresponding to the previous preview image frame request as the zoom factor value corresponding to the current preview image frame request and writes this zoom factor value to the zoom factor queue. In this way, the continuous use of multiple zoom factor values ​​can prevent the preview interface's field of view from jumping, avoiding a poor visual experience for the user.

[0149] When the zooming state detection module determines that the electronic device has exited the zooming state, the zooming state detection module can instruct the zoom factor queue to be cleared. This way, when the user performs a zoom operation again to put the electronic device into the zoom state, the camera HAL can again write the zoom factor value carried in the preview image frame request into the zoom factor queue for priority, reducing the zoom factor value validation process and the zoom operation response delay.

[0150] The zoom scenarios involved in the user's zoom operation can be roughly divided into Zoom In scenarios and Zoom Out scenarios. Among them, the Zoom In scenario refers to adjusting the focal length of the lens to magnify the scene, that is, the shooting scenario of pushing the scene from far to near; the Zoom Out scenario refers to adjusting the focal length of the lens to reduce the scene, that is, the shooting scenario of pulling the scene from near to far. However, whether in the Zoom In scenario or the Zoom Out scenario, the mobile phone may need to switch the camera used to expose the image in response to the user's zoom operation, or it may not need to switch the camera used to expose the image. For example, in the Zoom In scenario, assuming that the user adjusts the zoom magnification value from 1.0x to 2.0x, there is no need to switch the camera, and the main camera can continue to be used for image exposure. For another example, in the Zoom In scenario, assuming that the user adjusts the zoom magnification value from 1.0x to 5.0x, the camera used for image exposure needs to be switched from the main camera to the telephoto camera. For example, in a Zoom Out scenario, if the user adjusts the zoom ratio from 2.0x to 1.0x, there is no need to switch cameras; the main camera can continue to be used for image exposure. For another example, in a Zoom Out scenario, if the user adjusts the zoom ratio from 1.0x to 0.8x, the camera used for image exposure needs to be switched from the main camera to the wide-angle camera.

[0151] However, whether in the Zoom In scenario or the Zoom Out scenario, when the user's zoom operation involves lens switching, it may happen that the preview image frame obtained by exposure of the original lens (i.e., the lens used for image exposure before the lens switching) cannot meet the new zoom ratio value.

[0152] For example, in the example shown in Figure 6, the SAT module applies the zoom magnification value (i.e., the new zoom magnification value) carried by the preview image frame request 10 to the image data of the preview image frame 4, but the image data of the preview image frame 4 obtained by exposure may not meet the requirements for cropping based on the zoom magnification value.

[0153] One possible scenario is that cropping cannot be completed because the cropping operation exceeds the limits of the interpolation algorithm involved in image processing. In the Zoom In scenario, when the user adjusts the zoom magnification value quickly, if a lens switch event is required (such as quickly adjusting the zoom magnification value from 1.0x to 40.0x, switching the lens from the main camera to the telephoto camera), the IPE module needs to crop the target image with a smaller field of view from the original image with a larger field of view. Assume, as shown in Figure 9a, preview image frame 4 is an original image frame with a larger field of view (assuming a zoom magnification value of 1.0x), and preview image frame 10 corresponding to preview image frame request 10 (assuming a zoom magnification value of 40.0x) is a target image frame with a smaller field of view. In this case, the IPE hardware will crop the target image frame based on the zoom magnification value of 40.0x on preview image frame 4, which will exceed the limits of the image interpolation algorithm, resulting in the IPE hardware possibly being unable to complete the corresponding cropping operation.

[0154] It should be pointed out that in the Zoom In scenario, when the user adjusts the zoom ratio value slowly (such as adjusting the zoom ratio value from 1.0x to 1.1x, or even adjusting the zoom ratio value from 1.0x to 10.0x), or when no lens switching event occurs, the IPE module can crop a target image with a relatively small field of view from the original image with a relatively large field of view, and there is no problem of being unable to complete the cropping operation.

[0155] Another possible scenario is that the cropping cannot be completed due to the small field of view of the image to be cropped. In the Zoom Out scenario, it is necessary to crop a target image with a larger field of view from the original image with a smaller field of view. Especially when lens switching is required, the cropping operation may not be completed. Assume that, as shown in Figure 9b, preview image frame 4 is an original image frame with a smaller field of view (such as a zoom magnification value of 10.0x), and preview image frame 10 corresponding to preview image frame request 10 (such as a zoom magnification value of 1.0x) is a target image frame with a larger field of view, then the operation of cropping the target image frame according to the zoom magnification value of 1.0x on preview image frame 4 cannot be completed at all.

[0156] It should be pointed out that in the Zoom Out scenario, if the user adjusts the zoom ratio slowly and does not involve lens switching, given that the IFE module reserves the image edges when performing the cropping operation, the IPE module may also crop a target image with a relatively large field of view from the original image with a relatively small field of view.

[0157] In order to solve the above problem, in an embodiment of the present application, when the user performs a zoom operation, the multi-camera decision module in the HAL layer starts the target camera in advance through the camera driver, so as to perform image exposure operations through the original camera and the target camera at the same time, and obtains the second preview image frame data collected by the original camera and the target camera corresponding to the same preview image frame request. In this way, after the SAT module takes out a zoom magnification value from the zoom magnification queue, if the cropping data calculation cannot be completed based on the zoom magnification value and the first preview image frame data, or if it is determined that the IPE module cannot complete the cropping operation based on its calculation results, the relevant cropping data is recalculated based on the zoom magnification value and the second preview image frame data, so that the IPE module completes the cropping operation on the second preview image frame based on the calculation results of the SAT module.

[0158] In one possible implementation, when the user performs a zoom operation, the multi-camera decision module predicts that the camera to be switched is the target camera based on the zoom magnification value, and the camera sensor of the target camera is turned on in advance through the camera driver. Exemplarily, as shown in FIG10a, when the user performs a zoom operation, the camera application sends the zoom magnification to the multi-camera decision module in real time through the camera service. The multi-camera decision module estimates whether the current zoom scene requires switching the camera that actually outputs the image based on a preset decision algorithm, and when it is estimated that the current zoom scene requires switching the camera that actually outputs the image to the target camera, the camera sensor of the target camera is turned on in advance through the camera driver. For example, if the multi-camera decision module estimates that the current zoom scene requires switching the camera that actually outputs the image from the main camera to the telephoto camera based on a preset decision algorithm, the camera sensor of the wide-angle camera is turned on in advance through the camera driver.

[0159] In another possible embodiment, when the user performs a zoom operation, the multi-camera decision module predicts based on the zoom magnification value that the current zoom scene requires switching the camera that actually outputs the image, and then the camera driver pre-activates the camera sensors of all cameras other than the current camera (with the same shooting direction as the current camera). For example, as shown in FIG10b , assuming that the current camera that outputs the image is the primary camera, when the user performs a zoom operation, the camera application sends the zoom magnification to the multi-camera decision module in real time through the camera service. The multi-camera decision module estimates whether the current zoom scene requires switching the camera that actually outputs the image based on a preset decision algorithm, and then pre-activates the camera sensors of all cameras other than the primary camera through the camera driver, for example, pre-activating the camera sensors of the telephoto camera and the wide-angle camera.

[0160] When the camera sensors of multiple cameras are simultaneously enabled, the Sensor node in the camera HAL processes a preview image frame request. The Sensor node controls each camera's camera sensor through the camera driver to expose and output the image. Referring to Figure 10c, when both the main camera's camera sensor and the telephoto camera's camera sensor are enabled, the Sensor node in the camera HAL processes a preview image frame request. The camera driver controls each of the main camera's camera sensor and the telephoto camera's camera sensor to expose and output the image, respectively, to obtain first image data captured by the main camera's camera sensor and second image data captured by the telephoto camera's camera sensor.

[0161] It should be noted that among the multiple cameras of the mobile phone, only one will be used as the current image output camera. In an embodiment of the present application, the multi-camera decision module can set and switch the current image output camera. For example, when the current image output camera is the main camera, if the multi-camera decision module predicts that the current zoom scene needs to switch the actual image output camera based on the zoom magnification value, and the camera to be switched is a telephoto camera, the multi-camera decision module can set the current image output camera to the telephoto camera.

[0162] The first image data and the second image data correspond to the same preview image frame request, that is, to the same shooting scene. After the camera driver obtains the first image data and the second image data, the first image data and the second image data are simultaneously sent to the IFE module. Of the first image data and the second image data, one set of image data is captured by the current image output camera, and the other set of image data is not captured by the current image output camera.

[0163] After receiving the first and second image data, the IFE module performs preliminary cropping on the image data captured by the current output camera and sends the cropped image data and the uncropped image data to the SAT module. Assuming that the first image data is image data captured by the current output camera, the IFE module performs preliminary cropping on the first image data and then sends the cropped first image data and the uncropped second image data to the SAT module.

[0164] In a specific example, as shown in (1) in FIG11 , when both the main camera and the wide-angle camera are turned on, the Sensor node controls the main camera and the wide-angle camera to expose and output the image through the camera driver at the same time when processing the preview image frame request 6, so as to generate the image data of the first preview image frame 6 and the image data of the second preview image frame 6 respectively. Among them, the image data of the first preview image frame 6 is generated by exposing the main camera, and the image data of the second preview image frame 6 is generated by exposing the wide-angle camera. In the case where the current output camera is the main camera, when the SAT module in the Offline Pipeline processes the preview image frame data corresponding to the preview image frame request 6, the zoom magnification value corresponding to the preview image frame request 12 is read in the zoom magnification queue, and the zoom magnification value corresponding to the preview image frame request 12 is used to replace the zoom magnification value corresponding to the preview image frame request 6 and take effect on the image data of the preview image frame 6. The SAT module determines whether the IPE module can complete the cropping operation based on the zoom magnification value corresponding to the preview image frame request 12 and the image data of the first preview image frame 6 (that is, the image data collected by the current output camera). If so, the SAT module calculates cropping data and distortion data based on the zoom magnification value and the image data of the first preview image frame 6, and sends the calculation results to the IPE module. If not, it indicates that the zoom magnification value corresponding to the preview image frame request 12 cannot be applied to the image data of the first preview image frame 6. The SAT module then determines whether the target camera corresponding to the zoom magnification value (e.g., a wide-angle camera) is turned on, and if the target camera is turned on, whether the image data of the second preview image frame 6 corresponding to the preview image frame request 6 has been acquired by the target camera. After the SAT module acquires the image data of the second preview image frame 6, the SAT module applies the zoom magnification value corresponding to the preview image frame request 12 to the image data of the second preview image frame 6. In other words, the SAT module calculates cropping data and distortion data based on the zoom magnification value corresponding to the preview image frame request 12 and the image data of the second preview image frame 6, and sends the calculation results to the IPE module, so that the IPE module performs cropping processing on the image data of the second preview image frame 6 based on the calculation results.

[0165] Continuing with (2) in FIG11 , when the zoom magnification value corresponding to the preview image frame request 12 cannot be effective on the image data of the first preview image frame 6, assuming that the target camera (such as the wide-angle camera) is not turned on when the sensor node processes the preview image frame request 6, the SAT module will not obtain the image data of the second preview image frame 6 corresponding to the preview image frame request 6, and thus the zoom magnification value corresponding to the preview image frame request 12 cannot be effective in advance. At this time, the SAT module will take the zoom magnification value that is currently effective into effect on the image data of the first preview image frame 6, that is, the SAT module will take the zoom magnification value that is effective on the image data of the preview image frame 5 into effect on the image data of the first preview image frame 6. Thus, the SAT module calculates the cropping data and the distortion data based on the zoom magnification value that is effective on the image data of the preview image frame 5 and the image data of the first preview image frame 6, and sends the calculation result to the IPE module so that the IPE module performs cropping processing on the image data of the first preview image frame 6 according to the calculation result.

[0166] It should be noted that when the camera HAL receives a preview image frame request, if the preview image frame request does not carry a zoom ratio value, then when the camera HAL receives the preview image frame request, it will use the zoom ratio value corresponding to the previous preview image frame request as the zoom ratio value corresponding to the preview image frame request.

[0167] As shown in Figure 12, the process of the SAT module in the Offline Pipeline processing preview image frame data specifically includes:

[0168] S101: The SAT module obtains first target image data corresponding to a target preview image frame request, where the zoom magnification of the target preview image frame request is a first zoom magnification value.

[0169] The target preview image frame request can be any preview image frame request sent by the camera service to the camera HAL. If the target preview image frame request carries a zoom magnification value when it is sent, the first zoom magnification value is the zoom magnification value carried by the target preview image frame request; if the target preview image frame request does not carry a zoom magnification value, the first zoom magnification value is the system default zoom magnification value or the most recently effective zoom magnification value. If the target preview image frame request does not carry a zoom magnification value when it is sent, if the target preview image frame request is the first preview image frame request sent by the camera service to the camera HAL after the camera application is started, the first zoom magnification value is the system default zoom magnification value; otherwise, the first zoom magnification value is the zoom magnification value corresponding to the previous preview image frame request.

[0170] The first target image data refers to the image data obtained by controlling the exposure of the camera sensor of the current output camera through the camera driver when the Sensor node in the camera HAL processes the target preview image frame request.

[0171] At the same time, if the camera sensor of the mobile phone is turned on in addition to the camera sensor of the current camera, the camera sensors of other cameras are also turned on, such as the camera sensor of the target camera that is turned on in advance by the multi-camera decision module according to the zoom scene is also turned on, then when the Sensor node processes the target preview image frame request, it will also control the exposure of the camera sensors of other cameras through the camera driver to obtain the second target image data corresponding to the target preview image frame request.

[0172] S102, the SAT module reads whether there is a zoom magnification value in the zoom magnification queue, if so, execute S103, if not, execute S110.

[0173] Before the SAT module spatially aligns the image data using the preset SAT algorithm to calculate the cropped data, it first checks the zoom factor queue for any existing zoom factors. The zoom factor values ​​stored in the zoom factor queue are prioritized. The SAT module prioritizes the zoom factor values ​​stored in the zoom factor queue for the currently processed image data, thereby shortening zoom response time and improving the user's zoom experience.

[0174] S103: The SAT module takes out a zoom magnification value from the zoom magnification queue as the second zoom magnification value.

[0175] When there are zoom magnification values ​​in the zoom magnification queue, the SAT module takes out a zoom magnification value from the zoom magnification queue and prioritizes the zoom magnification value validation process. In this embodiment, the zoom magnification value taken out from the zoom magnification queue is temporarily referred to as the second zoom magnification value.

[0176] S104, the SAT module determines whether the IPE module can complete the cropping operation on the first target image data based on the second zoom magnification value according to the first target image data, the first zoom magnification value and the second zoom magnification value. If so, execute S105, otherwise execute S106.

[0177] After the SAT module takes out a zoom magnification value from the zoom magnification queue, it needs to determine whether the IPE module can complete the cropping operation based on the second zoom magnification value and the first target image data.

[0178] For example, if the second zoom ratio value is greater than the first zoom ratio value, and the difference between the two is large (such as greater than a preset threshold), the corresponding cropping operation may exceed the limitations of the image interpolation algorithm involved in image processing and cannot be implemented. At this time, the SAT module can determine that the IPE module cannot complete the cropping operation based on the second zoom ratio value and the first target image data.

[0179] As another example, if the second zoom magnification value is smaller than the first zoom magnification value, and the reserved edge image data of the first target image data cannot satisfy the image field of view corresponding to the second zoom magnification value, then since an image with a larger field of view angle cannot be cropped from an image with a smaller field of view angle, the SAT module can determine that the IPE module cannot complete the cropping operation based on the second zoom magnification value and the first target image data.

[0180] S105 , the SAT module calculates cropping data according to the second zoom magnification value and the first target image data, and instructs the IPE module to perform a cropping operation on the first target image data according to the calculation result.

[0181] When the SAT module determines that the IPE module can complete the cropping operation based on the second zoom magnification value and the first target image data, the SAT module calculates the cropping data and the distortion data based on the second zoom magnification value and the first target image data, and instructs the IPE module to perform the cropping operation and the distortion operation on the first target image data according to the calculation results.

[0182] S106, the SAT module determines whether the target camera is turned on according to the first zoom magnification value and the second zoom magnification value, and if so, executes S107, and if not, executes S110.

[0183] When the SAT module determines that the IPE module cannot complete the cropping operation based on the second zoom magnification value and the first target image data, if the first zoom magnification value belongs to the zoom range of the first camera and the second zoom magnification value belongs to the zoom range of the second camera, the SAT module determines that the second camera corresponding to the second zoom magnification is the target camera, and determines whether the target camera is turned on.

[0184] When the SAT module determines that the IPE module cannot complete the cropping operation based on the second zoom magnification value and the first target image data, if the first zoom magnification value is within the zoom range of the first camera, the second zoom magnification value is also within the zoom range of the first camera, and the second zoom magnification value is less than the first zoom magnification value, then the current zoom scene is a Zoom Out scene, and the SAT module determines that the third camera is the target camera and determines whether the target camera is turned on. The zoom magnification value corresponding to the third camera (e.g., the wide-angle camera) is less than the zoom magnification value corresponding to the first camera (e.g., the main camera).

[0185] S107 , whether the SAT module obtains the second target image data corresponding to the target preview image frame request and captured by the target camera, if so, execute S108 , if not, execute S110 .

[0186] When the second zoom ratio value cannot be effective on the first target image data, if the SAT module has obtained the second target image data corresponding to the target preview image frame request captured by the target camera, the SAT module can apply the second zoom ratio value to the second target image data to calculate the corresponding cropping data.

[0187] When the second zoom ratio value cannot be effective on the first target image data, if the SAT module does not obtain the second target image data corresponding to the target preview image frame request captured by the target camera, the second zoom ratio value cannot replace the current latest effective zoom ratio value and take effect in advance. At this time, the SAT module will apply the current latest effective zoom ratio value (which may be the first zoom ratio value or other zoom ratio values) to the first target image data to calculate the corresponding cropping data.

[0188] S108 , the SAT module determines whether the IPE module can complete the cropping operation on the second target image data based on the second zoom magnification value, and if so, executes S109 , and if not, executes S110 .

[0189] Exemplarily, the SAT module determines whether the cropping operation performed by the IPE module on the second target image data based on the second zoom magnification value will exceed the limitations of the image interpolation algorithm involved in image processing. If so, the SAT module determines that the IPE module cannot complete the cropping operation on the second target image data based on the second zoom magnification value. If not, the SAT module determines that the IPE module can complete the cropping operation on the second target image data based on the second zoom magnification value.

[0190] S109: The SAT module calculates cropping data according to the second zoom magnification value and the second target image data, and instructs the IPE module to perform a cropping operation on the second target image data according to the calculation result.

[0191] In this case, since the camera sensor of the target camera corresponding to the zoom scene is turned on in advance, when the second zoom ratio value that needs to take effect in advance cannot take effect on the first target image data, the SAT module will take effect on the second target image data collected by the target camera to ensure that the second zoom ratio can be taken effect in advance and reduce the response delay of the zoom operation.

[0192] S110 , the SAT module calculates cropping data according to the latest effective zoom magnification value and the first target image data, and instructs the IPE module to perform a cropping operation on the first target image data according to the calculation result.

[0193] The most recently effective zoom magnification value is: the zoom magnification value effective on the image frame data corresponding to the preview image frame request before the target preview image frame request.

[0194] In one case, when the SAT module processes the first target image data requested with the target preview image frame, if it reads that there is no zoom magnification value in the zoom magnification queue, the SAT module calculates the cropping data based on the current latest effective zoom magnification value and the first target image data, and instructs the IPE module to perform a cropping operation on the first target image data based on the calculation result.

[0195] In one situation, when the SAT module processes the first target image data requested by the target preview image frame, if the second zoom magnification value is read from the zoom magnification queue, the second zoom magnification value cannot be effective on the first target image data, and the target camera is not turned on or the SAT module does not obtain the second target image data corresponding to the target preview image frame request captured by the target camera or the second zoom magnification value cannot be effective on the second target image data, then the SAT module calculates the cropping data based on the current latest effective zoom magnification value and the first target image data, and instructs the IPE module to perform a cropping operation on the first target image data based on the calculation result.

[0196] In an embodiment of the present application, when the camera HAL receives a preview image frame request carrying a zoom magnification value, the zoom magnification value it carries is stored in the zoom magnification queue and takes effect in the Offline Pipeline first. This breaks the current situation in which the preview image frame request is bound to the zoom magnification value in the Android and camera HAL architectures, allowing the zoom magnification value's effectiveness link to skip the exposure processing of the Sensor node and the processing of the IFE module, thereby significantly reducing the system's zoom response delay. When the user's zoom operation is a sliding operation (such as sliding the zoom control with one finger or sliding the preview interface with two fingers, etc.), reducing the system's zoom response delay can effectively improve the user's zoom operation's tracking performance; when the user's zoom operation is a click operation (such as clicking the zoom control, etc.), shortening the zoom magnification value's effectiveness link can effectively improve the system's response speed to the user's zoom operation.

[0197] In this way, in the zoom scenarios involved in, for example, Figures 1a, 1b, and 1c, the mobile phone system has a low response delay to user zoom operations (such as clicking the zoom control, sliding the zoom control, sliding the preview interface with two fingers, etc.). Specifically, it can be manifested as the user's zoom operation (or magnification adjustment operation) adjusting the mobile phone camera application preview interface to a certain zoom magnification, and the time difference between the preview image corresponding to the zoom magnification and being displayed on the mobile phone screen is small.

[0198] In addition to the aforementioned application scenario where a user adjusts the zoom in the camera app's photo preview interface, the technical solutions provided by the embodiments of this application are also applicable to application scenarios where a user adjusts the zoom in the camera app's video preview interface, application scenarios where a user adjusts the zoom after the camera app starts recording, and other application scenarios involving camera zoom. In other application scenarios involving camera zoom, the implementation of the zoom response method provided by the embodiments of this application is similar to that described above and will not be repeated here.

[0199] This embodiment further provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the zoom response method in the above-mentioned embodiment.

[0200] This embodiment further provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the zoom response method in the above-mentioned embodiment.

[0201] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the zoom response method in the above-mentioned method embodiments.

[0202] Among them, the electronic devices (such as mobile phones, etc.), computer storage media, computer program products or chips provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0203] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0204] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0205] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquire first image data, where the first image data corresponds to a first preview image frame request, and a zoom ratio corresponding to the first preview image frame request is a first zoom ratio value; Reading a second zoom ratio value, where the second zoom ratio value corresponds to a second preview image frame request, and a generation time of the second preview image frame request is later than a generation time of the first preview image frame request; The first image data is processed based on the second zoom magnification value.

2. The method according to claim 1, characterized in that Before reading the second zoom ratio value, the method further includes: Upon receiving the zoom operation, a second preview image frame request is generated.

3. The method according to claim 1, characterized in that The electronic device includes a display screen, and the method further includes: A first preview image frame is displayed on the preview interface of the display screen, where the first preview image frame is obtained by the electronic device processing the first image data by replacing the first zoom magnification value with the second zoom magnification value.

4. The method according to claim 1, characterized in that: The method further comprises: Determining that a zoom ratio value exists in a zoom ratio queue, and determining the second zoom ratio value in the zoom ratio queue; The zoom ratio queue is a first-in-first-out queue.

5. The method according to any one of claims 1 to 4, characterized in that The electronic device includes a first camera, the first image data is acquired by the first camera, and the processing of the first image data based on the second zoom magnification value includes: If the first image data meets the condition of being processed based on the second zoom magnification value, the first image data is processed based on the second zoom magnification value.

6. The method according to claim 5, characterized in that The electronic device further includes a second camera, and the method further includes: If the first image data does not meet the condition of being processed based on the second zoom ratio value, determining whether the second camera has captured the second image data; If the second camera captures second image data, the second image data is processed based on the second zoom ratio value.

7. The method according to claim 5, characterized in that The electronic device further includes a second camera, and the method further includes: If the second camera does not capture the second image data, processing the first image data based on the current zoom ratio value; The current zoom magnification value is used to indicate the most recently effective zoom magnification value.

8. The method according to any one of claims 5 to 7, characterized in that The first image data meets a condition to be processed based on the second zoom magnification value, including: According to the first zoom magnification value, the second zoom magnification value and the first image data, it is determined that target cropping can be completed for the first image data based on the second zoom magnification value.

9. The method according to any one of claims 5 to 8, characterized in that The first camera is different from the second camera, and the second camera is a telephoto camera or a wide-angle camera.

10. The method according to any one of claims 6 to 9, characterized in that Upon receiving the zoom operation, the second camera is started.

11. The method according to any one of 2 to 10, characterized in that: The zoom operation includes: clicking a zoom control, sliding a zoom control, and sliding a preview interface with two fingers.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the method as claimed in any one of claims 1 to 11.

13. A chip system, applied to an electronic device including a display screen, characterized in that: The chip system includes a processing circuit and a transceiver pin. When the transceiver pin and the processing circuit communicate with each other through an internal connection path, the processing circuit executes the method as described in any one of claims 1-11 to control the receiving pin to receive a signal and the transmitting pin to send a signal.

14. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 11.

Images