Zooming method and device for equipment with multiple cameras

Abstract

The embodiment of the invention discloses a multi-camera equipment zooming method and device, and relates to the technical field of terminals. According to the method, the touch screen operation duration from the initial first touch screen position to the second touch screen position is determined based on the user behavior, and then the optimal time point for powering on the to-be-started second camera is determined. The method comprises the following steps: the electronic equipment comprising a plurality of cameras receives a zoom magnification adjusting operation which is used for indicating that a first zoom magnification is adjusted to a third zoom magnification through a second zoom magnification, a second camera is started according to the zoom magnification, a starting time point of the second camera is determined based on the zoom magnification adjusting operation, and the second camera is started according to the starting time point of the second camera. And starting to collect image data, and displaying the collected image in the display area. According to the method, the problem of shot picture jumping caused by the switching process of the multiple cameras in the zooming process of the electronic equipment can be solved, so that smooth zooming of the multiple cameras is realized.

Description

Technical Field

[0001] This application relates to the technical field of mobile terminals, and in particular, to a method and device for multi-camera zooming. Background Art

[0002] With the continuous development of mobile terminal devices (such as mobile phones, tablets, and computers), the camera shooting function has become increasingly important, and the shooting effect has also become better and better. During the user's shooting process, the user can perform zoom shooting by switching cameras with different focal lengths, and can also process the collected image data in combination with digital zooming to meet various magnification shooting scenarios. However, during the process of zooming the camera, switching between different cameras may cause problems such as frame loss, jitter, and unnatural transitions. Summary of the Invention

[0003] The embodiments of this application provide a method and device for device zooming. In the method provided by the embodiments of this application, the device has multiple cameras, and each camera has a different working focal length. During the process of the user adjusting the zoom magnification operation, by determining the start time point of the newly added working camera for the next focal length, the newly added camera is pre-loaded in advance. When the zoom magnification is adjusted to the second zoom magnification, the newly added working camera directly displays the collected image on the screen, ensuring that problems such as frame loss, jitter, or unnatural transitions may occur when switching between different cameras.

[0004] In a first aspect, a method for device zooming provided by an embodiment of the present application is applied to an electronic device with multiple cameras, including: The device receives an instruction to open a camera application, and the first camera of the electronic device starts to work, and the zoom ratio of the device is the first zoom ratio; Receiving a zoom ratio adjustment operation instruction, adjusting the zoom ratio of the device from the first zoom ratio through the second zoom ratio to the third zoom ratio, where the second zoom ratio is the zoom ratio critical value at which the second camera needs to be switched next according to the displacement direction of the zoom ratio adjustment operation at this time. The second zoom ratio is different from the first zoom ratio. If the zoom adjustment operation stops when reaching the position of the second zoom ratio, the second zoom ratio is the same as the third zoom ratio; otherwise, the second zoom ratio is different from the third zoom ratio; determining the startup time point of the second camera based on the zoom ratio adjustment operation, powering on the second camera at the startup time point, and completing the preloading work of the second camera in advance; When the zoom ratio is adjusted to the second zoom ratio, the second camera has completed the preloading work and starts to work, and the collected image data is sent to the display screen of the device; At this time, some or all of the image data displayed on the device display screen comes from the second camera. Among them, the application receiving the open instruction is any application that needs to start the camera, including common systems such as the application camera and video call, and also including non-system applications such as WeChat video call. By the user's zoom ratio operation, the time point for the device to power on the second lens is determined. On the one hand, it ensures that when the zoom ratio is adjusted to the second zoom ratio, the displayed picture of the camera will not cause frame loss, stuttering or unnatural transition due to the switching of the working lens. On the other hand, by preloading the second camera in advance at a reasonable time in combination with the user's operation behavior, the overall power consumption of the device is also reduced, and the device power consumption is controlled while improving the display performance.

[0005] In the above embodiment, it is necessary to determine the startup time point of the second camera based on the zoom ratio adjustment operation. In some embodiments, it is necessary to first determine the total time required for the zoom ratio to be adjusted from the first zoom ratio to the second zoom ratio; It is also necessary to determine the time required for the second camera to complete the preloading. Based on the total time of the magnification adjustment operation and the preloading time, the startup time point of the second camera can be determined. The total time determined based on the zoom ratio operation can ensure a more appropriate startup time point for the second camera, and reduce the power consumption as much as possible while ensuring the quality of the displayed image.

[0006] In the above embodiments, it is necessary to determine the total time required to adjust the zoom ratio from the first zoom ratio to the second zoom ratio. In some embodiments, the total time is determined according to the operation speed and displacement of the zoom ratio adjustment. Among them, the speed of the zoom ratio adjustment operation is the speed of the user's touch screen operation, and the displacement is the displacement generated by the user's movement starting from the initial touch screen position. The displacement of the user from the initial touch screen position to the switching of the second zoom ratio is called the total displacement. Based on the user's touch screen speed and touch screen displacement, the total time of the zoom ratio adjustment can be better determined to ensure that the second camera starts at the most appropriate time point.

[0007] In the above embodiments, there are many ways to determine the total time of the zoom ratio based on the user's touch screen speed and touch screen displacement. In some embodiments, the user's zoom ratio adjustment operation time is usually very short, and the user's adjustment operation can be approximately regarded as a uniform motion, that is, the touch screen speed at the start of the user is the touch screen speed of the entire zoom ratio adjustment operation. After determining the total displacement, the total time of the zoom ratio operation can be directly determined. The calculation of the uniform motion model is relatively simple, and the load on the device is small, which can ensure the display screen effect of the device while minimizing the computing power consumption of the device.

[0008] In the above embodiments, there are many ways to determine the total time of the zoom ratio based on the user's touch screen speed and touch screen displacement. In some embodiments, the user's zoom ratio adjustment operation is usually a process of gradually accelerating, so the user's adjustment operation can be regarded as a uniformly accelerated motion model. Among them, the touch screen speed at the start of the user is the initial velocity of the user. According to the multi-frame data sampling when the user just starts, the current acceleration of the user can be determined, so as to determine the uniformly accelerated mathematical curve of the user's zoom ratio adjustment. After determining the total displacement that needs to be operated currently, the total time of the zoom ratio operation can be directly determined. The uniformly accelerated model is relatively more in line with the user's actual zoom ratio adjustment operation. Through the uniformly accelerated model, the total time of the user's zoom ratio operation can be determined more accurately, so as to determine the start time point of the second camera more accurately.

[0009] In the above embodiments, there are many ways to determine the total duration of the zoom ratio based on the user's touch screen speed and touch screen displacement. In some embodiments, the user's zoom ratio adjustment operation is usually a process of gradually accelerating. Therefore, the user's adjustment operation can be regarded as a uniformly accelerated motion model. Among them, the touch screen speed and acceleration at the start of the user may not be data determined by real-time sampling of the current user's operation. They can be values determined by the user according to big data collection. By obtaining the common touch screen speed and acceleration through a large number of data of the regular user's zoom ratio adjustment operations, the uniformly accelerated mathematical curve of the user's zoom ratio adjustment can be determined. After determining the total displacement that needs to be operated currently, the total duration of the zoom ratio operation can be directly determined. Among them, one of the touch screen speed and acceleration can also be determined in advance based on big data, and the other can be determined based on the user's current zoom ratio operation. By determining the relevant values in advance through big data, the computing pressure of the electronic device can be reduced to a certain extent, and the total duration of the user's zoom ratio operation can be estimated more quickly.

[0010] In the above embodiments, there are many ways to determine the total duration of the zoom ratio based on the user's touch screen speed and touch screen displacement. In some embodiments, some discrete data are directly used to determine the mathematical relationship between the touch screen speed and the operation displacement. These discrete data can be empirical values obtained by sampling the user's multiple zoom ratio operations, or data empirical values obtained through a large number of sample collections. By determining the total duration of the zoom ratio operation through the mathematical relationship already stored in the electronic device, the computing pressure of the electronic device can be reduced, and the total duration of the user's zoom ratio operation can be estimated more quickly.

[0011] In the above embodiments, it is necessary to determine the preloading duration of the second camera. In some embodiments, a large number of preparatory works are required before the electronic device camera officially works, including power-on, pre-focusing, exposure preparation, image data acquisition, image data transmission, resource loading, and buffer rotation. Among them, the preloading duration includes the duration of any one stage or the total duration of multiple stages in the above preparatory works of the electronic device lens. By clarifying the preparatory works required before the second camera officially works, the preloading duration of the second camera can be determined more accurately.

[0012] In the above embodiments, it is necessary to determine the preloading duration of the second camera. Since the preloading time-consuming of the electronic device camera is related to the current working state of the electronic device, in some embodiments, when determining the preloading duration of the second camera, the hardware capabilities and load status of the electronic device need to be combined. By combining the current load status and hardware capabilities of the device, the preloading duration of the second camera can be determined more accurately.

[0013] In some embodiments in combination with some embodiments of the first aspect, the electronic device may collect the speed of the current user's zoom magnification adjustment operation according to a certain rule. When the touch screen speed at a certain moment exceeds a pre-set threshold speed, it can be approximately considered that the user is performing a continuous zoom adjustment operation, and the second camera is directly powered on. When the user's touch screen speed is very fast, there is usually no intention of stopping immediately. By directly powering on, unnecessary device operations are reduced, and the loss of device performance is reduced.

[0014] In some embodiments in combination with some embodiments of the first aspect, when the device receives a user's rapid zoom magnification adjustment operation, the rapid zoom magnification refers to the change value of the touch screen speed of the user's zoom magnification operation exceeding a pre-set threshold speed change value, and the speed change value is the speed difference at different times. By detecting the calculated change in the touch screen speed, it can be approximately considered that the user is performing an irregular interaction operation, and the camera on the same side as the first camera is directly powered on, thereby reducing unnecessary device operations and reducing the loss of device performance.

[0015] In some embodiments in combination with some embodiments of the first aspect, the device receives a user's zoom magnification adjustment operation with different round trips. The zoom magnification operations in different directions for round trips include multi-directional sliding back and forth or multi-directional pinch zooming, resulting in the zoom magnification repeatedly shrinking and magnifying. By detecting that the user is performing a multi-directional zoom magnification adjustment operation, it can be approximately considered that the user is performing an irregular interaction operation, and the camera on the same side as the first camera is directly powered on, thereby reducing unnecessary device operations and reducing the loss of device performance.

[0016] In the above embodiments, the camera on the same side as the first camera refers to the front or rear camera of the electronic device. In this way, the number of cameras powered on unnecessarily is minimized, ensuring a reduction in the power consumption of the device.

[0017] In some embodiments in combination with some embodiments of the first aspect, when an instruction to exit the application is received for the first time, relevant data at the time of exit is stored, including but not limited to the user's zoom magnification adjustment operation, the first zoom magnification, the second zoom magnification, the second camera, and the start time point; when an instruction to return to the application is received for the second time, the interval between the first time and the second time is determined; if the time interval is less than a pre-set interval threshold, it indicates that the user probably hopes to continue the previous zoom magnification operation. Therefore, the start time point of the current second camera can be determined based on the data saved at the time of exit. By saving the data before exiting the application, when the user probably continues the previous task, the start time point of the second camera can be determined with low or zero computational effort, reducing the computing power consumption of the device.

[0018] In a second aspect, embodiments of the present application provide an electronic device, which includes: one or more processors, a memory, a display, and multiple cameras. The memory is used to store computer program code, and the computer program code includes computer instructions. The one or more processors call the computer instructions to cause the electronic device to perform the following operations. Among them, the device receives an instruction to open the camera application, and the first camera of the electronic device starts to work, and the zoom ratio of the device is the first zoom ratio; receives a zoom ratio adjustment operation instruction, and adjusts the zoom ratio of the device from the first zoom ratio through the second zoom ratio to the third zoom ratio. The second zoom ratio is the next zoom ratio critical value closest to the direction of the zoom ratio adjustment operation displacement at this time that needs to switch to the second camera. The second zoom ratio is different from the first zoom ratio; if the zoom adjustment operation stops when reaching the position of the second zoom ratio, the second zoom ratio is the same as the third zoom ratio, otherwise the second zoom ratio is different from the third zoom ratio; based on the zoom ratio adjustment operation, determine the startup time point of the second camera, and power on the second camera at the startup time point to complete the preloading work of the second camera in advance; when the zoom ratio is adjusted to the second zoom ratio, the second camera has completed the preloading work and starts to work, and sends the collected image data to the display screen of the device; at this time, part or all of the image data displayed on the device display screen comes from the second camera. Among them, the application that receives the opening instruction is any application that needs to start the camera, including common systems such as the application camera and video phone, and also including non-system applications such as WeChat video call, etc. By the user's zoom ratio operation, the time point for the device to power on the second lens is determined. On the one hand, it ensures that when the zoom ratio is adjusted to the second zoom ratio, the displayed picture of the camera will not cause frame loss, stuttering, or unnatural transition due to the switching of the working lens. On the other hand, preloading the second camera in advance at a reasonable time in combination with the user's operation behavior also reduces the overall power consumption of the device, while improving the display performance and controlling the power consumption of the device.

[0019] Combined with some embodiments of the second aspect, it is necessary to determine the startup time point of the second camera based on the zoom ratio adjustment operation. In some embodiments, it is necessary to first determine the total duration required for the zoom ratio to be adjusted from the first zoom ratio to the second zoom ratio; it is also necessary to determine the duration required for the second camera to complete the preloading. Based on the total duration of the magnification adjustment operation and the preloading duration, the startup time point of the second camera can be determined. The total duration determined based on the zoom ratio operation can ensure a more appropriate startup time point for the second camera, while minimizing the power consumption while ensuring the quality of the displayed image.

[0020] In connection with some embodiments of the second aspect, it is necessary to determine the total duration required to adjust the zoom ratio from the first zoom ratio to the second zoom ratio. In some embodiments, the total duration is determined according to the operation speed and displacement of the zoom ratio adjustment operation. Among them, the speed of the zoom ratio adjustment operation is the speed of the user's touch screen operation, and the displacement is the displacement generated by the user's movement starting from the initial touch screen position. The displacement of the user from the initial touch screen position to the switching of the second zoom ratio is called the total displacement. Based on the user's touch screen speed and touch screen displacement, the total duration of the zoom ratio adjustment can be better determined to ensure that the second camera is activated at the most appropriate time point.

[0021] In connection with some embodiments of the second aspect, there are many ways to determine the total duration of the zoom ratio based on the user's touch screen speed and touch screen displacement. In some embodiments, the user's zoom ratio adjustment operation time is usually very short, and the user's adjustment operation can be approximately regarded as a uniform motion, that is, the touch screen speed at the start of the user is the touch screen speed of the entire zoom ratio adjustment operation. After determining the total displacement, the total duration of the zoom ratio operation can be directly determined. The calculation of the uniform motion model is relatively simple, and the load on the device is small. While ensuring the display screen effect of the device, the computing power consumption of the device can be minimized.

[0022] In connection with some embodiments of the second aspect, there are many ways to determine the total duration of the zoom ratio based on the user's touch screen speed and touch screen displacement. In some embodiments, the user's zoom ratio adjustment operation is usually a process of gradually accelerating, so the user's adjustment operation can be regarded as a uniformly accelerated motion model. Among them, the touch screen speed at the start of the user is the initial velocity of the user. According to the multi-frame data sampling when the user just starts, the current acceleration of the user can be determined, so as to determine the uniformly accelerated mathematical curve of the user's zoom ratio adjustment. After determining the total displacement that needs to be operated currently, the total duration of the zoom ratio operation can be directly determined. The uniformly accelerated model is relatively more in line with the user's actual zoom ratio adjustment operation. Through the uniformly accelerated model, the total duration of the user's zoom ratio operation can be determined more accurately, so as to determine the activation time point of the second camera more accurately.

[0023] In combination with some embodiments of the second aspect, there are many ways to determine the total duration of the zoom ratio based on the user's touch screen speed and touch screen displacement. In some embodiments, the user's zoom ratio adjustment operation is usually a gradually accelerating process. Therefore, the user's adjustment operation can be regarded as a uniformly accelerated motion model. Among them, the touch screen speed and acceleration at the start of the user may not be data determined by real-time sampling of the current user's operation. They can be values determined by the user based on big data collection. By obtaining the common touch screen speed and acceleration through a large number of data of the regular user's zoom ratio adjustment operations, the uniformly accelerated mathematical curve of the user's zoom ratio adjustment can be determined. After determining the total displacement that needs to be operated currently, the total duration of the zoom ratio operation can be directly determined. Among them, one of the touch screen speed and acceleration can also be determined in advance based on big data, and the other can be determined based on the user's current zoom ratio operation. By determining the relevant values in advance through big data, the computing pressure of the electronic device can be reduced to a certain extent, and the total duration of the user's zoom ratio operation can be estimated more quickly.

[0024] In combination with some embodiments of the second aspect, there are many ways to determine the total duration of the zoom ratio based on the user's touch screen speed and touch screen displacement. In some embodiments, some discrete data are directly used to determine the mathematical relationship between the touch screen speed and the operation displacement. These discrete data can be empirical values obtained by sampling the user's multiple zoom ratio operations, or data empirical values obtained by collecting a large number of samples. By determining the total duration of the zoom ratio operation through the mathematical relationship already stored in the electronic device, the computing pressure of the electronic device can be reduced, and the total duration of the user's zoom ratio operation can be estimated more quickly.

[0025] In combination with some embodiments of the second aspect, it is necessary to determine the preloading duration of the second camera. In some embodiments, a large number of preparatory works are required before the electronic device camera officially works, including power-on, pre-focusing, exposure preparation, image data acquisition, image data transmission, resource loading, and buffer rotation. Among them, the preloading duration includes the duration of any one stage or the total duration of multiple stages in the above preparatory works of the electronic device lens. By clarifying the preparatory works required before the second camera officially works, the preloading duration of the second camera can be determined more accurately.

[0026] In combination with some embodiments of the second aspect, it is necessary to determine the preloading duration of the second camera. Since the preloading time-consuming of the electronic device camera is related to the current working state of the electronic device, in some embodiments, the hardware capabilities and load status of the electronic device need to be combined when determining the preloading duration of the second camera. By combining the current load status and hardware capabilities of the device, the preloading duration of the second camera can be determined more accurately.

[0027] In combination with some embodiments of the second aspect, in some embodiments, the electronic device can collect the speed of the current user's zoom magnification adjustment operation according to a certain rule. When the touch screen speed at a certain moment exceeds a pre-set threshold speed, it can be approximately considered that the user is performing a continuous zoom adjustment operation, and the second camera is directly powered on. When the user's touch screen speed is very fast, there is usually no intention to stop immediately. By directly powering on, unnecessary device operations are reduced, and the loss of device performance is reduced.

[0028] In combination with some embodiments of the second aspect, in some embodiments, the device receives a user's rapid zoom magnification adjustment operation. The rapid zoom magnification refers to the change value of the touch screen speed of the user's zoom magnification operation exceeding a pre-set threshold speed change value, and the speed change value is the speed difference at different times. By detecting the calculated change in the touch screen speed, it can be approximately considered that the user is performing an irregular interaction operation, and the camera on the same side as the first camera is directly powered on, thereby reducing unnecessary device operations and reducing the loss of device performance.

[0029] In combination with some embodiments of the second aspect, in some embodiments, the device receives a user's zoom magnification adjustment operation with different round trips. The zoom magnification operation with different direction round trips includes multi-directional back-and-forth sliding or multi-directional pinch-zoom, resulting in the zoom magnification repeatedly shrinking and enlarging. By detecting that the user is performing a multi-directional zoom magnification adjustment operation, it can be approximately considered that the user is performing an irregular interaction operation, and the camera on the same side as the first camera is directly powered on, thereby reducing unnecessary device operations and reducing the loss of device performance.

[0030] In combination with some embodiments of the second aspect, the camera on the same side as the first camera refers to the front or rear camera of the electronic device. In this way, the number of cameras powered on unnecessarily is minimized, ensuring a reduction in the power consumption of the device.

[0031] In combination with some embodiments of the second aspect, in some embodiments, when an instruction to exit the application is received for the first time, the relevant data at the time of exit is stored, including but not limited to the user's zoom magnification adjustment operation, the first zoom magnification, the second zoom magnification, the second camera, and the start time point; when an instruction to return to the application is received for the second time, the interval between the first time and the second time is determined; if the time interval is less than a pre-set interval threshold, it indicates that the user probably hopes to continue the previous zoom magnification operation. Therefore, the start time point of the current second camera can be determined based on the data saved at the time of exit. By saving the data before exiting the application, when the user probably continues the previous task, the start time point of the second camera can be determined with low or zero computational effort, reducing the computing power consumption of the device.

[0032] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, which, when running on an electronic device, cause the electronic device to execute the method described in any possible implementation manner of the first aspect and the second aspect.

[0033] It can be understood that both the electronic device provided in the second aspect and the computer storage medium provided in the third aspect are used to execute the method provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the hardware structure of a terminal in an embodiment of the present invention;

[0035] Figure 2 It is a schematic diagram of the software structure of a terminal in an embodiment of the present invention;

[0036] Figure 3 It is a schematic diagram of the shooting fields of different cameras in an embodiment of the present invention;

[0037] Figure 4 It is a schematic diagram of a user adjusting the zoom ratio for taking pictures in an embodiment of the present invention;

[0038] Figure 5 It is another schematic diagram of a user adjusting the zoom ratio for taking pictures in an embodiment of the present invention;

[0039] Figure 6 It is a schematic diagram of rotating a zoom ratio control in an embodiment of the present invention;

[0040] Figure 7 It is a schematic diagram of the steps of a method for implementing the zoom ratio of multiple cameras in an embodiment of the present invention;

[0041] Figure 8 It is another schematic diagram of the steps of a method for implementing the zoom ratio of multiple cameras in an embodiment of the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] At present, most mobile terminals are equipped with multiple cameras. The hardware parameters and positions of each camera are different, and they have different field of view ranges, so that different image pictures can be captured, meeting the shooting demands of users in different scenarios and improving the shooting quality. During the shooting process, due to the adjustment of the field of view range, it is necessary to switch different cameras, and the processes of powering on / off and data acquisition and display of the cameras all take time. If the camera to be switched is not started in advance, it is very easy to cause problems such as frame dropping, freezing, sudden changes in color or field of view angle during the camera switching process. If the second camera to be added is started earlier, it may have a greater impact on the power consumption of the mobile terminal.

[0043] The following embodiments of the present application provide a multi-camera zoom method applied to a mobile terminal. Based on the zoom operation of the user during the shooting process, the best timing to activate the next camera is estimated, which solves the problems of preview frame loss, lag, and unnatural color transitions during camera switching, balances the relationship between user experience and the power consumption of the mobile terminal, and minimizes the overall power consumption of the mobile terminal as much as possible while improving the user experience.

[0044] The "multiple" referred to in the following embodiments of the present application means greater than or equal to two. It should be noted that the term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after, unless otherwise specified. In the description of the embodiments of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0045] The terms "including" and "having" and any variations thereof mentioned in the description of the embodiments of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include other unlisted steps or units, or may optionally further include other steps or units inherent to these processes, methods, products, or devices.

[0046] In the embodiments of the present application, "terminal device", "electronic device", "mobile terminal", and "terminal" all have the same meaning, and these several expressions can be interchanged.

[0047] The operation interface involved in the embodiments of the present application can also be referred to as a user interface (User Interface, UI) or a graphical interface, or other names. The operation interface is an interface for human-computer interaction between an electronic device and a user. The electronic device can display and output relevant information through the interface, such as displaying images / text / data, etc., and can also receive user operations through the operation interface, such as touch / click / long press / double click / drag / typing, etc. The common manifestation form of the user interface is a graphical user interface (graphic user interface, GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be visual interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. displayed on the display screen of the electronic device.

[0048] The camera application involved in the embodiments of this application is a software application capable of taking photos or videos. The camera application mentioned in the embodiments of this application can be a pre-installed application on the terminal device, such as a camera, or a program with a shooting function downloaded from other networks or obtained and installed in other ways by the user during the use of the electronic device.

[0049] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0050] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application.

[0051] Figure 1 The structural schematic diagram of the electronic device 100 is shown.

[0052] The electronic device 100 can be various devices including multiple cameras, such as mobile phones, tablet computers, large-screen devices, etc. Exemplary embodiments of the portable mobile terminal include, but are not limited to, those equipped with or other operating systems of the portable mobile terminal. The above portable mobile terminal can also be other mobile terminals, such as digital cameras, smart speakers, in-vehicle devices (also known as car machines), laptop computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), wearable mobile terminals, virtual reality devices and other IOT (Internet of Things) devices. It should also be understood that in some other embodiments of this application, the above mobile terminal may not be a portable mobile terminal, but a desktop computer with multiple cameras, a smart refrigerator with multiple cameras, a smart air conditioner or other smart home appliances with multiple cameras. These terminal devices are all configured with touch screens, and users can interact with the terminal devices through touch operations on the touch screens to achieve control of the terminal devices.

[0053] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) connector 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0054] It can be understood that the structure schematically shown in the embodiments of this application does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0055] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0056] The processor may generate operation control signals according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0057] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory may store instructions or data that have been used by the processor 110 or are used frequently. If the processor 110 needs to use such instructions or data, it can directly call them from this memory. This avoids repeated accesses and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0058] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 110 may be connected to modules such as a touch sensor, an audio module, a wireless communication module, a display, a camera, etc. through at least one of the above interfaces.

[0059] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present application are only illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0060] The charging management module 140 is configured to receive a charging input from a charger. The charger may be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 may receive the charging input from the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 may receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 may also supply power to the electronic device through the power management module 141.

[0061] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery charge cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0062] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc. The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100.

[0063] The modulation and demodulation processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor can be an independent device. In some other embodiments, the modulation and demodulation processor can be independent of the processor 110 and be disposed in the same device as the mobile communication module 150 or other functional modules.

[0064] The wireless communication module 160 may provide solutions for wireless communication applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0065] The electronic device 100 may implement a display function through a GPU, a display screen 194, an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0066] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel may adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a Microled, a Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or more display screens 194.

[0067] The electronic device 100 may implement a camera function through a camera module 193, an ISP, a video codec, a GPU, the display screen 194, an application processor AP, a neural network processor NPU, etc.

[0068] The camera module 193 can be used to collect color image data and depth data of a photographed object. The ISP can be used to process the color image data collected by the camera module 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera sensor element, where the optical signal is converted into an electrical signal. The camera sensor element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be disposed in the camera module 193.

[0069] In some embodiments, the camera module 193 may be composed of a color camera module and a 3D sensing module.

[0070] In some embodiments, the sensor element of the camera of the color camera module may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The sensor element converts the optical signal into an electrical signal and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats.

[0071] In some embodiments, the 3D sensing module may be a time of flight (TOF) 3D sensing module or a structured light 3D sensing module. Among them, structured light 3D sensing is an active depth sensing technology. The basic components of the structured light 3D sensing module may include an infrared (IR) emitter, an IR camera module, etc. The working principle of the structured light 3D sensing module is to first emit a specific pattern of light spots to the photographed object, then receive the light spot pattern coding on the surface of the object, and then compare the similarities and differences with the original projected light spots, and use the triangulation principle to calculate the three-dimensional coordinates of the object. The three-dimensional coordinates include the distance between the electronic device 100 and the photographed object. Among them, TOF 3D sensing can be an active depth sensing technology. The basic components of the TOF 3D sensing module may include an infrared (IR) emitter, an IR camera module, etc. The working principle of the TOF 3D sensing module is to calculate the distance (i.e., depth) between the TOF 3D sensing module and the photographed object through the time of infrared return to obtain a 3D depth of field map.

[0072] The structured light 3D sensing module can also be applied to fields such as face recognition, somatosensory game consoles, and industrial machine vision detection. The TOF 3D sensing module can also be applied to fields such as game consoles, augmented reality (AR) / virtual reality (VR), etc.

[0073] In some other embodiments, the camera module 193 can also be composed of two or more cameras. These two or more cameras can include a color camera, which can be used to collect color image data of the object being photographed. These two or more cameras can use stereo vision technology to collect depth data of the object being photographed. Stereo vision technology is based on the principle of human eye parallax. Under natural light, images of the same object are taken from different angles through two or more cameras, and then operations such as triangulation are performed to obtain the distance information, that is, depth information, between the electronic device 100 and the object being photographed.

[0074] In some embodiments, the electronic device 100 can include one or more camera modules 193. Specifically, the electronic device 100 can include one front camera module 193 and one rear camera module 193. Among them, the front camera module 193 is usually used to collect color image data and depth data of the photographer himself facing the display screen 194, and the rear camera module can be used to collect color image data and depth data of the photographed object (such as a person, a landscape, etc.) that the photographer is facing.

[0075] In some embodiments, the CPU or GPU or NPU in the processor 110 can process the color image data and depth data collected by the camera module 193. In some embodiments, the NPU can identify the color image data collected by the camera module 193 (specifically the color camera module) through a neural network algorithm based on the skeleton point recognition technology, such as the convolutional neural network algorithm (CNN), to determine the skeleton points of the person being photographed. The CPU or GPU can also run the neural network algorithm to determine the skeleton points of the person being photographed according to the color image data. In some embodiments, the CPU or GPU or NPU can also be used to confirm the figure of the person being photographed (such as body proportions, the fatness or thinness of the body parts between the skeleton points) based on the depth data collected by the camera module 193 (which can be a 3D sensing module) and the identified skeleton points, and can further determine the body beautification parameters for the person being photographed. Finally, the photographed image of the person being photographed is processed according to the body beautification parameters so that the figure of the person being photographed in the photographed image is beautified. How to perform body beautification processing on the image of the person being photographed based on the color image data and depth data collected by the camera module 193 will be introduced in detail in subsequent embodiments, and will not be elaborated here.

[0076] The digital signal processor is used to process digital signals and can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0077] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0078] The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission pattern between human brain neurons, it can quickly process input information and can also continuously learn on its own. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0079] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card. Or files such as music and videos are transferred from the electronic device to the external memory card.

[0080] The internal memory 121 can be used to store computer-executable program code, and this executable program code includes instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, the image playback function, etc.). The data storage area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional methods or data processing of the electronic device 100 by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor.

[0081] The electronic device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, etc. For example, music playback, recording, etc.

[0082] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0083] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music through the speaker 170A, or output the audio signal of a hands-free call.

[0084] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by bringing the receiver 170B close to the human ear.

[0085] The microphone 170C, also known as the "microphone", "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak by bringing the mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to implement functions such as collecting sound signals, noise reduction, identifying the sound source, and implementing a directional recording function.

[0086] The headphone jack 170D is used to connect a wired headphone.

[0087] The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can include at least two parallel plates having conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0088] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the X, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake during shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the angle of jitter of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and controls the lens to move in the opposite direction to cancel the jitter of the electronic device 100, thereby achieving anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenarios.

[0089] The barometric pressure sensor 180C is used to measure the barometric pressure. In some embodiments, the electronic device 100 calculates the altitude according to the barometric pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0090] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. When the electronic device is a foldable electronic device, the magnetic sensor 180D can be used to detect the folding or unfolding of the electronic device, or the folding angle. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip, features such as automatic unlocking of the flip are set.

[0091] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0092] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance through infrared or laser. In some embodiments, for the shooting scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0093] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode may be an infrared light-emitting diode. The electronic device 100 emits infrared light outward through the light-emitting diode. The electronic device 100 uses the photodiode to detect the infrared reflected light from nearby objects. When the intensity of the detected reflected light is greater than the threshold, it can be determined that there is an object near the electronic device 100. When the intensity of the detected reflected light is less than the threshold, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear for a call, so as to automatically turn off the screen to achieve the purpose of power saving. The proximity light sensor 180G can also be used for automatic unlocking and locking of the leather case mode and pocket mode.

[0094] The ambient light sensor 180L can be used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is blocked, for example, when the electronic device is in the pocket. When it is detected that the electronic device is blocked or in the pocket, some functions (such as the touch function) can be disabled to prevent accidental operation.

[0095] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access the application lock, fingerprint taking pictures, fingerprint answering calls, etc.

[0096] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 utilizes the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature detected by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor in order to reduce the power consumption of the electronic device to implement thermal protection. In other embodiments, when the temperature detected by the temperature sensor 180J is lower than another threshold, the electronic device 100 heats the battery 142. In still other embodiments, when the temperature is lower than yet another threshold, the electronic device 100 can boost the output voltage of the battery 142.

[0097] The touch sensor 180K, also known as a "touch control device". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch control screen". The touch sensor 180K is used to detect a touch operation acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from that of the display screen 194.

[0098] The bone conduction sensor 180M can acquire vibration signals

[0099] The keys 190 can include a power-on key, volume keys, etc. The keys 190 can be mechanical keys. They can also be touch keys. The electronic device 100 can receive key inputs and generate key signal inputs related to the user settings and function control of the electronic device 100.

[0100] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playing, etc.) can correspond to different vibration feedback effects. Touch operations acting on different regions of the display screen 194 can also correspond to different vibration feedback effects for the motor 191. Different application scenarios (such as time reminder, receiving messages, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0101] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0102] The software system of the electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservices architecture, or cloud architecture. In the embodiments of this application, taking the Android system with a layered architecture as an example, the software structure of the electronic device 100 is exemplarily described.

[0103] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0104] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom are the application layer, the application framework layer, Android runtime (ART) and native C / C++ libraries, the Hardware Abstract Layer (HAL), and the kernel layer.

[0105] The application layer may include a series of application packages.

[0106] Such as Figure 2 shown, the application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0107] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0108] Such as Figure 2 shown, the application framework layer may include window manager, content provider, view system, resource manager, notification manager, activity manager, input manager, etc.

[0109] The window manager provides the Window Manager Service (WMS). The WMS can be used for window management, window animation management, surface management, and as a transfer station for the input system.

[0110] The content provider is used to store and obtain data, and make this data accessible to applications. The data may include video, image, audio, dialed and answered calls, browsing history and bookmarks, phone book, etc.

[0111] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a short message notification icon may include a view for displaying text and a view for displaying pictures.

[0112] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0113] The notification manager enables an application to display notification information in the status bar. It can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that a download is complete, a message reminder, etc. The notification can also be in the form of a chart or scrolling text in the system top status bar, such as the notification of a background-running application, or in the form of a dialog window on the screen. For example, it can prompt text information in the status bar, emit a prompt tone, vibrate the electronic device, blink the indicator light, etc.

[0114] The activity manager can provide the Activity Manager Service (AMS). AMS can be used for the startup, switching, scheduling of system components (such as activities, services, content providers, broadcast receivers), as well as the management and scheduling of application processes.

[0115] The input manager can provide the Input Manager Service (IMS). IMS can be used to manage system inputs, such as touch screen input, key input, sensor input, etc. IMS retrieves events from input device nodes and, through interaction with the WMS, distributes the events to the appropriate windows.

[0116] The Android Runtime includes the core libraries and the Android Runtime. The Android Runtime is responsible for converting source code into machine code. The Android Runtime mainly includes the Ahead of Time (AOT) compilation technology and the Just in Time (JIT) compilation technology.

[0117] The core libraries are mainly used to provide the functions of basic Java class libraries, such as libraries for basic data structures, mathematics, IO, tools, databases, networks, etc. The core libraries provide APIs for users to develop Android applications.

[0118] The native C / C++ libraries can include multiple functional modules. For example: surface manager, Media Framework, libc, OpenGL ES, SQLite, Webkit, etc.

[0119] Among them, the surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications. The media framework supports the playback and recording of a variety of common audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides the drawing and operation of 2D and 3D graphics in the application. SQLite provides a lightweight relational database for the applications of the electronic device 100.

[0120] The hardware abstraction layer runs in the user space, encapsulates the kernel layer drivers, and provides call interfaces to the upper layer.

[0121] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0122] Next, in combination with the capture and photographing scenario, the working processes of the software and hardware of the electronic device 100 will be exemplarily described.

[0123] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including information such as touch coordinates and the timestamp of the touch operation). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation and the control corresponding to the click operation as the control of the camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer, and captures a static image or video through the camera 193.

[0124] In the embodiments of the present application, it is exemplified by including N cameras, where N is a positive integer not less than 3. The camera is a package body including a photosensitive sensor, a lens, and a fixing photosensitive sensor, and the camera is used to obtain a photographed image within a specific magnification range. Taking N as 3 as an example, the three cameras are a wide-angle camera, a main camera, and a telephoto camera, and their respective magnification ranges are 0.4X - 1X, 1X - 2.5X, and 2.5X - 10X respectively. The above magnification range values are only for exemplary illustration. It should be noted that in each embodiment of the present application, the magnification is also simply referred to as the zoom ratio.

[0125] In all embodiments of the present invention, all zoom ratio values are examples and not uniquely specified values.

[0126] The camera application (app) involved in the embodiments of this application is a software application capable of implementing shooting-related functions. The camera application can be a system camera program pre-installed on the terminal device, or a camera application program downloaded from other networks or obtained and installed in other ways by the user during the use of the electronic device. Corresponding to the user's click operation on the application icon, a corresponding viewfinder preview interface can be displayed. When the user clicks the corresponding video recording button, the mobile phone can be triggered to display a preview interface for video viewfinding. If the electronic device has a front camera and a rear camera, when the user clicks the camera switching button, the viewfinding display can be switched to the other camera.

[0127] The camera working process of common mobile terminals can be generally summarized into three parts: the camera input device management module, the session management module, and the camera output management module.

[0128] Among them, the camera input device management module is responsible for calling the camera to collect data. After the camera is started, frame capture begins, and the captured frame data is used as the camera input stream. Among them, the action of capturing frames after the camera is started is collectively referred to as frame capture, which mainly includes single-frame capture, multi-frame capture, and loop frame capture.

[0129] Among them, single-frame capture refers to capturing one frame of data in the frame data stream after the camera is started, which is commonly used for ordinary photography.

[0130] Multi-frame capture refers to continuously capturing multiple frames of data in the frame data stream after the camera is started, which is commonly used for continuous shooting.

[0131] Loop frame capture refers to continuously capturing multiple frames of data in the frame data stream after the camera is started, which is commonly used for continuous shooting.

[0132] The session management module can configure the input stream, that is, select which cameras to use for shooting, the usage time of the cameras, etc. In addition, parameters such as the flash, exposure time, focus, and zoom can also be configured to achieve different shooting effects, so as to adapt to different business scenarios. The application can meet the shooting requirements of different scenarios by switching sessions.

[0133] The camera output stream is responsible for outputting the collected content as a preview stream, a photo stream, or a video stream.

[0134] The camera application in the mobile terminal controls the camera to implement basic operations such as image display (preview), photo saving (taking pictures), and video recording (recording videos). During the implementation of the basic operations, the camera service will control the camera device to collect and output data. The collected image data is directly transferred to the specific function module for processing through the graphics buffer at the device hardware interface (HDI, Hardware Device Interfaces) at the bottom layer of the camera.

[0135] Among them, the graphics buffer is used to send the data processed by the bottom layer to the upper layer for image display in a timely manner. When the camera implements different functions, the graphics buffer saves different data. When the camera is previewing, the camera application is in the process of taking pictures. The graphics buffer only needs to cache the data collected by the preview stream and then pass it to the image display system for display. When the camera takes pictures, it is necessary to save the picture after taking the picture, so the graphics buffer needs to cache the data collected by the photo stream and pass the data to the media image receiver for storage.

[0136] When the camera is recording a video, the media recording service first creates a video view (video surface) for transmitting data and provides it to the camera service. The camera service can control the camera device to collect data, generate a video stream, and cache the generated video stream in the graphics buffer. The collected data is processed by the underlying HDI and the video stream data is passed to the media recording service. The media recording service also records the audio data corresponding to the video. The media recording service processes the video data, synthesizes the audio data and the video stream data to generate the corresponding video file, saves it as a video file, and completes the video recording.

[0137] In the embodiments of the present application, "camera" and "lens" have the same meaning and the two expressions can be interchanged.

[0138] The mobile terminal includes multiple cameras, which may include a main camera, a telephoto camera, a wide-angle camera, an infrared camera, a depth camera, a black and white camera and other cameras.

[0139] The field of view of the camera is determined by the field of view (FOV) of the camera. The larger the FOV of the camera, the larger the field of view of the camera. Each camera has its own advantages and disadvantages in different scenarios. The following is an exemplary introduction to the characteristics (advantages and disadvantages) of the cameras that may be involved in the embodiments of the present application and the scenarios to which they are applicable.

[0140] The main camera has the characteristics of large light intake, high resolution and central field of view. The main camera is generally used as the default camera of mobile terminals (such as mobile phones). In other words, when the mobile terminal responds to the user's operation of launching the "Camera" application, it can generally launch the main camera by default, and the image captured by the main camera is also displayed on the preview interface.

[0141] A telephoto camera has a relatively large focal length and is suitable for photographing objects that are far from the mobile phone, that is, distant objects. However, the light intake of a telephoto camera is small. When using a telephoto camera to take pictures in a low-light scene, the picture quality may be affected due to insufficient light intake. Moreover, the field of view of a telephoto camera is small and not suitable for photographing large scenes, that is, not suitable for photographing large objects such as buildings or landscapes.

[0142] A wide-angle camera has a large field of view and is suitable for photographing large objects such as buildings or landscapes. However, the resolution of a wide-angle camera is low, and the images presented by the photographing are prone to distortion, failing to achieve the best photographing effect.

[0143] An infrared camera features a large spectral range. For example, an infrared camera can not only sense visible light but also sense infrared light. In a low-light scene (where visible light is weak), taking advantage of the feature that an infrared camera can sense infrared light, an infrared camera can be used to take pictures to improve the picture quality in a low-light scene. However, an infrared camera also has the drawback of low resolution.

[0144] The resolution of a depth camera is also low. Common depth cameras include time-of-flight (TOF) cameras or structured light cameras. Taking a time-of-flight camera as an example, it has the feature of accurately obtaining the depth information of the photographed object and is suitable for scenarios such as face recognition.

[0145] A black-and-white camera has no color filter. Therefore, compared with a color camera, it has a larger light intake. However, the images captured by a black-and-white camera can only show different levels of gray scale and cannot present the true colors of the photographed object. Among them, the main camera, telephoto camera, and wide-angle camera mentioned above are all color cameras.

[0146] The hardware parameters of a camera (such as the field of view angle) affect the size of the camera's field of view.

[0147] Exemplarily, the field of view of the main camera (such as the rear main camera) can be 1.0X - 8.0X; the field of view of the wide-angle camera (such as the rear wide-angle camera) is 0.6X - 1.0X; the field of view of the telephoto camera (such as the rear telephoto camera) is 8.0X. Among them, 1.0X, 8.0X, and 0.6X are all zoom ratios. "1X" means the zoom ratio is 1 time. "8.0X" means the zoom ratio is 8.0 times. "0.6X" means the zoom ratio is 0.6 times. Among them, the zoom ratio described in the embodiments of the present application can be an optical zoom ratio or a digital zoom ratio.

[0148] In the process of using the mobile terminal, the user will habitually identify the best position camera for image capture or video capture. For example, when used for several people to take selfies together, priority is given to using the front wide-angle camera with a relatively large field of view. For another example, when the user needs to shoot a distant subject, based on the long focal length of the telephoto camera, the mobile terminal can easily use the telephoto camera to shoot the distant subject. For another example, when the user shoots a subject in a dark scene, based on the characteristics of the large amount of light entering the main camera and the high resolution, the mobile terminal can easily use the main camera to shoot. For another example, when the user needs to shoot larger objects such as buildings or landscapes, based on the characteristics of the short focal length and large viewing angle of the wide-angle camera, the mobile terminal can easily use the wide-angle camera to shoot. Therefore, when the shooting scene shot by the user changes, the mobile terminal will also change the camera that matches it accordingly.

[0149] At the same time, users can dynamically switch cameras according to their needs. For example, the camera system application is also used to provide the following functions: in video recording mode, if the user chooses to switch cameras (i.e., turn on the function of switching cameras during video recording), the camera switch icon can be displayed (i.e., appear) in the viewfinder interface of the recorded video. In response to the user's click operation on the switch camera icon, the camera system application can switch the preview camera to refresh the video stream. For example, when the preview camera is the front camera, the preview video stream is video stream 1. In response to the user's click operation on the switch camera icon, the camera system application can switch the preview camera to the rear camera, and the preview video stream is refreshed to video stream 2.

[0150] Cameras can be divided into fixed-focus lenses and zoom lenses according to whether they have a fixed field of view. Fixed-focus lenses refer specifically to lenses with only a fixed focal length, that is, only a fixed field of view. Fixed-focus lenses are simple in design, fast in focusing, and have stable image quality, but they do not have a zoom function and are suitable for large group photos. Zoom lenses can change their focal length within a certain range to obtain different widths and narrownesses of field of view.

[0151] Common zoom methods include optical zoom and digital zoom. Optical zoom is to magnify the image from a physical level through the optical refraction of the camera itself. Since it is done by optical refraction, the image quality is lossless. Also, since optical zoom is a physical technology, the imaging of this part is what you see and what you get.

[0152] Digital zoom uses software technology to enlarge and fill in pixels on an already formed image. Since the pixels are filled in by algorithms at the pixel level, the image quality is compromised.

[0153] Optical zoom is achieved by changing the focal length of the camera, that is, by changing the distance between the center of the lens (i.e., the optical center) and the imaging plane (image sensor); digital zoom is achieved by cropping the captured image. In the following embodiments of the present application, the zoom can be in the form of optical zoom, that is, by switching the camera to achieve zoom; or it can be in the form of a combination of optical zoom and digital zoom (also known as hybrid zoom) to change the preview image perspective and achieve the zoom purpose.

[0154] As Figure 3 shown, it is an example diagram of shooting a target object with a mobile terminal. The mobile terminal includes a wide-angle camera (abbreviated as wide-angle), a main camera (abbreviated as main camera), and a telephoto camera (abbreviated as telephoto). When a magnification adjustment operation is received, the camera is switched.

[0155] The displayed images of the mobile terminal under each camera after switching are as Figure 3 shown. As the user slides the operation on the mobile terminal, for example, when the user performs a zoom-in operation, the camera used by the mobile terminal switches from the wide-angle camera to the main camera, and the image content captured by the mobile terminal also shrinks from the panoramic view. As the user continues the zoom-in operation, the magnification of the mobile terminal continues to increase, and the camera also switches from the main camera to the telephoto camera with the highest magnification. The range of the image content captured by the mobile terminal shrinks to the smallest, but the display size of the captured object shown in the displayed image of the mobile terminal is enlarged. The user can precisely control the most suitable hardware camera through magnification adjustment, flexibly output image and video content, so as to achieve different shooting effects and meet the requirements of multi-service scenario adaptation.

[0156] The multi-camera zoom method provided by each embodiment of the present application can be applied to any shooting scenario such as image browsing, taking pictures, or video recording.

[0157] Generally, the lens of the camera (also known as the lens) is fixed relative to the mobile terminal, so the switching of the camera is also called the switching of the lens.

[0158] Assume that the mobile terminal has three cameras: a wide-angle camera, a main camera, and a telephoto camera, and by default, the main camera is used for work. The mobile terminal stops the main camera and only uses the telephoto camera for work. This process is regarded as the switching of the camera in the present invention, and the telephoto camera at this time is the newly added camera.

[0159] Another possible situation is that the mobile terminal continuously uses the main camera to collect images, and at the same time starts the second camera, the telephoto camera, to collect image data in parallel. The two cameras respectively collect image data and send it to the mobile terminal for processing to generate corresponding pictures. This process is also regarded as the switching of the lens in the present invention, and the telephoto camera at this time is also the newly added camera.

[0160] Therefore, in the embodiments of the present invention, the switching of the lens refers to any scenario that changes the working camera or adds a working camera. In the embodiments of the present invention, the control of lens switching is performed through user touch screen operations.

[0161] When the user touches the touch screen of the terminal device, the finger contacts the screen to generate a corresponding touch screen signal. The detection module of the terminal device can detect the touch screen signal, and can sample the touch screen signal at a fixed period, generate a touch event according to the obtained touch screen signal, and then report the touch event to the application program at the application layer. The application program makes a response according to the touch event.

[0162] Optionally, the touch event may include parameters such as the coordinates of the touch point and the touch state. According to whether the touch screen signal is detected in two adjacent frames, the touch state can be divided into three types. (1) DOWN event, that is, the touch screen signal is not detected in the previous frame of two adjacent frames, but is detected in the latter frame; (2) MOVE event, that is, the touch screen signal is detected in both the previous and the latter frames of two adjacent frames; (3) UP event, that is, the touch screen signal is detected in the previous frame of two adjacent frames, but not detected in the latter frame.

[0163] Each touch operation starts from a DOWN event, goes through a series of MOVE events, and ends until an UP event is detected. In other words, each touch event can be composed of a DOWN event, one or more MOVE events, and an UP event. The electronic device can identify and distinguish the touch state of the touch event based on a specific software algorithm, etc., and further can judge the type of the user's touch operation according to the touch state, such as a sliding operation, a click operation, a long press operation, a double click operation, a zoom operation, etc.

[0164] In the embodiments of the present application, the user adjusts the zoom ratio through continuous sliding operations or zoom operations. These operations for adjusting the zoom ratio can be collectively referred to as touch screen operations.

[0165] Taking the sliding operation as an example, when a DOWN event is detected, taking the frame in which the DOWN event is detected as the starting frame, record the starting coordinates of the user's finger touch point in the starting frame, continuously monitor the MOVE event of each subsequent frame. If the displacement between the coordinates of the user's finger touch point and the starting coordinates in the MOVE event of a certain frame exceeds a preset value, the current operation can be determined as a sliding operation. In response to the user's sliding operation, the control that the user's finger touches can slide following the user's finger.

[0166] When the user touches and slides on the touch screen, the terminal device can detect the control that the user touches according to the touch signal, as well as the position and sliding displacement of the finger touch point in each frame of the sliding operation process. The corresponding sliding speed adopted by the user periodically can be calculated according to the sliding distance in two adjacent frames.

[0167] Among them, the meaning of displacement is the distance in a certain direction, and the smallest measurement unit of displacement on the display screen of the terminal device is 1 pixel. In this application, the "distance between the touch point of each frame and the touch point of the previous adjacent frame in the sliding direction" is the sliding displacement of the user's finger touch point from the previous frame to the current frame. Similarly, the distance between the starting touch point and the ending touch point of the touch operation in the sliding direction is the total sliding displacement of the finger during the user's sliding operation.

[0168] In an embodiment of the present invention, the camera used at the initial work is the first camera. There can be more than one first camera. At this time, the zoom ratio is the first zoom ratio, and the position of the user's finger and the screen touch point is the starting first touch screen position of the finger. According to the touch screen displacement direction of the user (zooming in or out), the zoom ratio of the next camera to be switched is the second zoom ratio. The newly added working camera at the second zoom ratio is the second camera. At this time, the touch screen position where the user's finger needs to touch the screen is the second touch screen position. The position where the finger touches the screen when the user's actual touch screen operation stops is the actual target touch screen position. At this time, the actual zoom ratio is the actual target zoom ratio. If there is a newly added working camera at this time, it is called the actual newly added target camera; if the working cameras required by the user's actual target zoom ratio and the first zoom ratio are the same, that is, there is no need to switch the working camera.

[0169] Next, taking the photographing scenario as an example, a way to trigger the magnification adjustment operation is described in combination with Figure 4 the user interface shown to implement a magnification adjustment operation.

[0170] After the user opens the camera application and selects "Take a Photo" in the function selection area 104, the camera displays the image obtained at the current first zoom ratio. When the mobile terminal receives the touch operation of the user on this interface, the zoom ratio is adjusted. When the user performs a sliding operation as shown in Figure 4 the mobile terminal can zoom according to this sliding operation, and then the displayed image 101 will also change due to factors such as the change of the magnification and the shaking of the hand. Among them, when a zoom-in operation is detected (such as: swiping up), the magnification is increased, and the displayed image 101 shows a zoom-in effect; when a zoom-out operation is detected (such as: swiping down), the magnification is decreased, and the displayed image 101 shows a zoom-out effect; the faster the sliding operation is detected, the faster the change of the magnification, and the faster the change of the displayed image 101. As shown in Figure 4The zoom ratio control 102 in the captured photo interface 10 shown indicates the adjustable range of the zoom ratio of the mobile terminal, and the control 103 on the ratio control 102 indicates the current zoom ratio. As the control 103 moves along the ratio control 102 towards the maximum zoom ratio, the zoom ratio gradually increases; conversely, as the control 103 moves along the ratio control 102 towards the minimum zoom ratio, the zoom ratio gradually decreases. Optionally, the current zoom ratio can also be displayed in text on the displayed image 101 to remind the user of the current zoom ratio.

[0171] Taking the photo-taking scenario as an example below, another implementation manner of triggering the ratio adjustment operation is described in combination with Figure 5 the user interface shown.

[0172] When the mobile terminal receives the ratio adjustment operation input by the user on this interface, such as Figure 5 the two-finger zoom operation shown in, the mobile terminal can perform zooming according to this zoom operation, and then the displayed image 101 will also change due to factors such as the change in the ratio and the shake of the hand. Among them, when it is detected that the two fingers slide outward, that is, during the zoom-in operation, the zoom ratio increases, and the displayed image 101 shows a zoom-in effect; when it is detected that the two fingers slide inward, that is, during the zoom-out operation, the zoom ratio decreases, and the displayed image 101 shows a zoom-out effect; the faster the zoom operation is detected, the faster the change in the zoom ratio, and the faster the change in the displayed image 101.

[0173] When the user triggers the shooting action using the camera application, usually, the camera will default to activate the main camera, and preview the image captured by the main camera in the display area. When the user performs the zoom ratio adjustment operation, when the mobile terminal detects that the current zoom ratio is within a certain range, it will power on other cameras.

[0174] Usually when starting to take a photo, the mobile terminal starts the default mode for data acquisition. Common default modes include only running the main camera, running the main camera and the wide-angle camera, and running all cameras. During the process of the user sliding to change the ratio, the mobile terminal then changes the camera for data acquisition.

[0175] Among them, Figure 4 the ratio control 102 in is an exemplary one. The ratio control can also be called a zoom bar, which is used to display the optical zoom range.

[0176] Common ratio controls can also be in a control style similar to Figure 6 shown in. The zoom bar is in a ring shape. When the user swipes left and right, the zoom bar rotates to present continuous ratio values, and the control 103 displays the currently rotated and selected ratio value. When the user swipes to the ratio they want, they stop swiping.

[0177] Among themFigure 6 The specified magnification control 105 displays several fixed zoom magnifications of the current device. By directly clicking on any magnification that the user wants to select in the specified magnification control 105, the magnification of the device can be quickly switched to the target magnification value, without the need to continuously transition to the target magnification by sliding, allowing the user to switch to the target magnification more quickly. Among them, w in the control 105 represents the wide-angle camera state.

[0178] In addition to the Figure 4 long strip shape in Figure 6 and the ring shape in Figure 4 , the magnification control can also be any other possible shape. At the same time, the zoom bar can be displayed at any position on the display screen of the mobile terminal at any rotation angle, such as

[0179] the zoom bar in

[0180] can be placed horizontally at the top or bottom of the displayed image 101, can be placed vertically on the left or right side of the screen, or can be placed at an arbitrary angle on any blank display area of the screen. Figure 4 As shown by the long strip-shaped zoom control in

[0181] When the user slides from the 1X position of the zoom control 102 to the 2X position, the distance that the finger slides is s1, and the change value of the zoom magnification is 1. According to Formula 1, the change rate of the zoom magnification per unit sliding distance at this time can be calculated as ω1. When the user slides from the 2X position of the zoom control 102 to the 10X position, the distance that the finger slides is s2, and the change value of the zoom magnification is 8. According to Formula 1, the change rate of the zoom magnification per unit sliding distance at this time can be calculated as ω2. Obviously, the value of ω2 is greater than ω1, and the change value of the zoom magnification also increases regularly as the finger slides upward. For the same sliding distance of the finger at different positions of the zoom control 102, the difference in the magnification change is different, and the magnification change value at the bottom is smaller than that at the top.

[0182] Among them, z1 represents the second zoom ratio of the next camera to be switched according to the direction of the user's touch screen displacement, z0 represents the first zoom ratio at the initial stage of the user's sliding, s represents the finger sliding displacement when the user adjusts the magnification from the first touch screen position of the first zoom ratio to the second touch screen position corresponding to the second zoom ratio, and can be calculated from the change in the finger sliding position on the screen. ω represents the change rate of the zoom ratio per unit sliding distance, that is, the change value of the camera magnification within the unit distance of the user's finger sliding.

[0183] This application provides a method for simulating the relationship between the camera zoom ratio and the user's finger sliding distance, and a power function shown in Formula 2 can be used to simulate the corresponding relationship between the two.

[0184] z = 2 ks Formula 2

[0185] Among them, s represents the distance of finger sliding, k represents the camera coefficient, and z represents the zoom ratio of the camera system (i.e., the lens zoom multiple).

[0186] Among them, k is related to the design of the camera system, and the specific value can be calculated from the actual measurement data of the camera device, or can be obtained by simulating the camera device, or can be directly calculated from the camera system design. In this embodiment, through actual data sampling of the used mobile terminal, when the finger movement distance of the user is 5.92 mm, the zoom ratio multiple of the camera application of this mobile terminal becomes 60X. Substituting it into Formula 2, k = 0.9978 can be calculated.

[0187] This application provides another method for simulating the relationship between the camera zoom ratio and the user's finger sliding distance, simplifies the two into a relatively simple linear relationship, and a multiple function shown in Formula 3 can be used to simulate the corresponding relationship between the two.

[0188] z = ks Formula 3

[0189] Among them, s represents the distance of finger sliding, k represents the camera coefficient, and z represents the zoom ratio of the camera system (i.e., the lens zoom multiple).

[0190] Among them, k is related to the design of the camera system, and the specific value can be calculated from the actual measurement data of the camera device, or can be obtained by simulating the camera device, or can be directly calculated from the camera system design.

[0191] This application provides another method for simulating the relationship between the camera zoom ratio and the user's finger sliding distance. Within a certain finger sliding distance range, the zoom ratio remains unchanged, forming a discrete point correspondence relationship. Exemplarily, Table 1 is used to show this relationship between the two.

[0192] Zoom ratio range Sliding distance (mm) 1X 0-200 2X 200-400 10X 400-600

[0193] Table 1 Relationship between zoom ratio and finger sliding distance

[0194] The present application provides another method for simulating the relationship between the zoom ratio of a camera and the finger sliding distance of a user. A large amount of actual data during the user's zooming process can be adopted, specifically measuring the user's sliding distance and the corresponding zoom ratio, and further obtaining the mathematical curve model matching the device through ideal mathematical fitting.

[0195] The present application provides another method for simulating the relationship between the zoom ratio of a camera and the finger sliding distance of a user. Due to the inconsistent performance of cameras, in order to more accurately construct the mathematical relationship between the two, different mathematical relationships can be adopted within different value ranges, and the method of piecewise functions can be used to construct the mathematical relationship between the two. For example, the default zoom ratio of the camera is 0.8X, and within the finger sliding distance of 100 mm, the zoom ratio is fixed at 1X. When the sliding distance exceeds 100 mm, the exponential function is used to construct the mathematical relationship between the zoom ratio and the sliding distance. When the sliding distance exceeds 300 mm, the zoom ratio reaches the maximum of 10X, so the zoom ratio becomes fixed at 10X again.

[0196] The mathematical model adopted in the above embodiments is only for simulating the mathematical relationship between the zoom ratio of the camera of the camera and the finger sliding distance. Any other common mathematical curve models that also conform to the rules can be adopted, and no limitation is made thereto.

[0197] Among them, there are many actual sampling calculation methods for the mobile terminal. A common method is that during the finger sliding process of the user, the touch firmware on the screen of the mobile terminal obtains the sampling point coordinates of the position of the user's hand on the screen and continuously reports the position coordinates to the application layer of the mobile terminal. At the same time, it is also necessary to obtain the reporting point period of the data reported to the application layer currently. Combining the reporting point position information and the reporting point period, the mathematical curve model relationship between the two can be calculated. The finger sliding distance in the embodiments of the present application refers to the displacement distance that occurs on the electronic device when the finger performs the zoom operation.

[0198] In common solutions, usually each camera of the mobile terminal is responsible for the shooting task of the target focal length within a certain range. A possible zoom ratio matching the working camera in the embodiments of the present application is shown in Table 2.

[0199]

[0200] Table 2 A working zoom ratio range of the camera

[0201] When the user starts the camera application, if the user does not trigger a zoom operation, the camera works at a fixed zoom ratio. The session management module of the mobile terminal camera will select a matching camera according to the configuration shown in Figure 2 to work based on the current zoom ratio of the device.

[0202] When the user takes a picture using the camera application, the mobile terminal creates a capture layer (capture surface) in the media image receiver to receive image data for display. Exemplarily, if the fixed zoom ratio currently selected by the user is 2X and within the focal length range of 1.5X - 3X, the mobile terminal powers on only the main camera, and the main camera captures the corresponding image data.

[0203] Exemplarily, if the fixed zoom ratio currently selected by the user is 1X and within the focal length range of 0.9X - 1.5X, the mobile terminal uses both the main camera and the wide-angle camera to work at this time. The mobile terminal powers on the main camera and the wide-angle camera, and each camera captures a stream of image data. The current image data is captured in a dual-stream mode. The mobile terminal transfers the data captured in the dual-stream mode to the media image receiver of the camera application through HDI, and the media image receiver processes the dual-stream data to generate the captured picture for display.

[0204] When the user starts the camera application, if the user triggers a zoom operation, after the mobile terminal detects the user's zoom operation, it will re-match a suitable camera to work according to the configuration shown in Table 2.

[0205] Exemplarily, the user initially takes pictures with a fixed zoom ratio of 2X, and currently only the main camera is working and sending the captured image data for display. If the user then drags the zoom bar to adjust the zoom ratio and wants to increase the zoom ratio from 2X to 5X. During the process of the user dragging the zoom bar and sliding, the zoom ratio gradually increases from 2X, and at this time only the main camera is still working. When the zoom ratio increases to 3X, according to Table 2, the telephoto camera will be used to capture image data. Therefore, at this time, the device powers on the telephoto camera, and the telephoto camera starts to load resources. After the telephoto camera finishes loading, it enters the running mode, and the main camera stops working. The user continues to slide and increase the zoom ratio to 5X, and at this time only the telephoto camera captures the image data.

[0206] Regarding the switching of the camera triggered by a fixed zoom ratio, due to differences in the user's operation methods, the timing of camera switching may not be reasonable enough. For example, in the previous example, as the device's zoom ratio gradually increases and reaches the threshold of 3X, at this time the telephoto camera has not completed initialization and resource loading, resulting in the telephoto camera not being able to immediately capture data for display, causing lag.

[0207] During the process of triggering the camera with a fixed zoom ratio, the mobile terminal can set a fixed preloading time. Before the user slides to 3X, the telephoto camera can be preloaded in advance. This method may cause the premature activation of the telephoto camera, and at the same time, the main camera is also continuously working, thus causing relatively large power consumption to the device.

[0208] The modes shown in Table 2 can be any modes of working with the camera of the mobile terminal, which can be video shooting, time-lapse photography, video preview, picture preview, macro photography mode, portrait mode, landscape mode, slow motion mode, continuous shooting mode, panoramic shooting mode, etc.

[0209] In the present invention, during the zooming process, the best timing for switching the camera is estimated by combining the user's sliding behavior and the operating ability of the mobile terminal, while achieving the best user display experience and ensuring power saving of the mobile terminal.

[0210] The process of the user sliding the zoom bar is the process of adjusting the zoom ratio of the mobile terminal from the initial first zoom ratio to the actual target zoom ratio. During this process, the switching of different cameras of the mobile terminal may be involved. Before the camera officially enters the working mode, it needs to be powered on in advance to load resources, which is called the power-on time of the camera. When the camera completes the power-on and enters the running mode, it is said that the camera has completed the switching.

[0211] After the mobile terminal completes the switching of the camera, the switched camera enters the running mode. The process of the camera collecting image data is the starting current process of the camera. The process of the camera sending the collected image data to the display interface of the mobile terminal is the output current process of the camera. All the preparation processes involved before the output current can be collectively referred to as the preloading process of the camera.

[0212] The preloading time may further include the hardware preparation work of the mobile terminal and the preparation work of the software related to photographing. For example, the preparation work may be the power-on of the hardware responsible for photographing in the mobile terminal. For example, power-on operations may be performed on the executable processor, sensors, and Image Signal Processor (ISP) of the mobile terminal. The hardware preparation may further include the power-on of the camera, etc., which is not limited here. The preparation work of the software related to photographing may include the preloading of software resources involved in the camera application. The software preparation may further include the resource loading of the system buffer, the acquisition and transmission of the image data stream, etc., which is not limited here. In this way, preparations can be made in advance for subsequent shooting, advancing the time including the hardware power-on time, the device focusing time, the exposure preparation time, the image acquisition time, the data stream transmission time, and the software resource preloading time, ensuring that when the user adjusts the zoom ratio to the second zoom ratio, the newly added second camera can directly enter the working mode. The second zoom ratio at this time is the magnification threshold point at which the mobile terminal needs to switch the working camera.

[0213] Taking the photographing scenario as an example, the following briefly describes the power-on working process of the camera in the mobile terminal after receiving the user's trigger magnification operation.

[0214] Exemplarily, assume that the optical zoom ratio of the telephoto camera is 5X - 6X, and the optical zoom ratio of the main camera is 1X - 2X. The mobile terminal turns on the camera in the default mode (e.g., the turned-on camera is the main camera). When the mobile terminal detects the operation of the user sliding the magnification control upward to zoom in, the mobile terminal then turns on the telephoto camera.

[0215] Exemplarily, the mobile terminal can detect the operation of the user increasing the zoom ratio (in response to this operation, the mobile terminal can reduce the FOV presented by the preview image. At the same time, the zoom ratio displayed in the displayed image will gradually increase. When the zoom ratio increases to 5X, the mobile terminal can start the telephoto camera, and the telephoto camera starts to enter the preparation process of resource loading and has not yet entered the running state. Still, only the main camera captures images. After the telephoto camera starts and completes and enters the running state, the main camera is turned off, and the telephoto camera works alone to capture images.

[0216] Exemplarily, the mobile terminal can detect the user's operation of increasing the zoom ratio. In response to this operation, the mobile terminal can reduce the FOV presented by the preview image. At the same time, the zoom ratio displayed in the display image will gradually increase. When the zoom ratio increases to 5X, the mobile terminal can activate the telephoto camera, and the telephoto camera starts to enter the preparation process of resource loading. It has not entered the running state yet, and still only the main camera captures images. When the telephoto camera starts up and enters the running state, both the main camera and the telephoto camera are in the running state and capture images simultaneously. Until the zoom ratio increases to 7X, the main camera is turned off, and the telephoto camera works alone to capture images.

[0217] Exemplarily, during the process of the zoom ratio increasing from 1X to 5X, the main camera is always in the running state. When the zoom ratio reaches the intermediate ratio value (e.g., 3X), the mobile terminal powers on the telephoto camera to start it. The telephoto camera starts to enter the preparation process of resource loading and has not entered the running state yet. Therefore, during this process, only the ordinary camera captures images, and the mobile terminal gradually reduces the FOV presented by the preview image (e.g., can perform centering cropping on the figure shown at 1X). When the zoom ratio increases to 5X, the telephoto camera enters the running state, the main camera is turned off, and the telephoto camera works alone to capture images.

[0218] Exemplarily, during the process of the zoom ratio increasing from 1X to 5X, the main camera is always in the running state. When the zoom ratio reaches the intermediate ratio value (e.g., 3X), the mobile terminal powers on the telephoto camera. The telephoto camera is in the startup state and starts to enter the preparation process of resource loading and has not entered the running state yet. Therefore, during this process, only the main camera captures images, and the mobile terminal gradually reduces the FOV presented by the preview image (e.g., can perform centering cropping on the figure shown at 1X). When the zoom ratio increases to 5X, the telephoto camera enters the running state. At this time, both the main camera and the telephoto camera are in the running state and capture images simultaneously. Until the zoom ratio increases to 7X, the main camera is turned off, and the telephoto camera works alone to capture images.

[0219] Exemplarily, when the zoom ratio increases to 3.1X, the main camera is in the running state, and the telephoto camera starts to enter the running state, that is, the main camera and the telephoto camera start to capture images simultaneously. During the process of the zoom ratio increasing from 3.1X to 4.9X, both the main camera and the telephoto camera are in the running state and capture images simultaneously. Until the zoom ratio increases to 5X, the main camera is turned off, and the telephoto camera is still in the running state. At this time, only the telephoto camera captures images.

[0220] In an embodiment of the present application, a possible estimated preloading duration of the camera includes, taking a photo as an example, before the camera of the mobile terminal sends the collected image data for display (outflow), it is necessary to first complete the power-on of the camera and collect the image data stream (start the stream). The preloading process includes the time for the camera to power on and the time to collect the image data stream (start the stream).

[0221] In an embodiment of the present application, a possible estimated preloading duration of the camera includes, taking a photo as an example, in addition to the power-on time and the start-stream time, the preloading process also includes the time for the camera to focus in advance and the exposure preparation time.

[0222] In another embodiment of the present application, a possible estimated preloading duration of the camera includes that if the mobile terminal needs to continuously send images in a continuous shooting scenario, causing resource tension in the graphics buffer, the preloading time prepared by the camera also needs to consider the rotation time of the buffer.

[0223] In another embodiment of the present application, a possible estimated preloading duration of the camera includes that the CPU load of the mobile terminal is relatively high and cannot fully support the shooting work of the mobile terminal. Therefore, the CPU load rate needs to be considered when calculating the preloading time of the camera. In the case of the mobile terminal being overloaded, the preloading time will increase in a certain mathematical relationship in the form of the operation ability coefficient of the mobile terminal.

[0224] The mobile terminal estimates the duration of the preloading process of the camera in advance based on the user's touch screen operation behavior and the current operation ability of the mobile terminal, and preloads the camera in advance. When the user touches the zoom bar to reach the value of the second zoom ratio, the camera matching the second zoom ratio has completed the preloading and starts to send data for display, which will bring the best shooting display experience to the user.

[0225] The camera startup time is the time when the power is turned on for the camera. Assuming that the time node when the user's finger touches the touch screen of the electronic device to start the touch screen operation is t0, the total duration required for the user's finger to move from the initial first touch screen position to the second touch screen position where the second zoom ratio is located is △t. Assuming that the preloading duration required to complete the second camera switch and make it fully enter the working state is t pre ,then when the finger continuously touches the screen until the time node (t0 + △t - t pre ), the electronic device can start to power on the second camera to be switched to make it enter the pre-start state.

[0226] Among them, due to various actual factors, the duration △t of the finger continuously touching the screen may not be a unique fixed value, but any value within the interval [△t - t, △t + t], and t is a preset time threshold.

[0227] Previously, a mathematical curve model between the camera zoom ratio and the displacement of the user's touch screen operation was constructed, and one of them was selected as an example. Exemplarily, the exponential mathematical curve model shown in Formula 2 is selected here. At the same time, for the sake of simplifying the process, it can be defaulted here that the camera coefficient k in Formula 2 is 1. At this time, the simplified mathematical curve model between the camera zoom ratio and the displacement of the user's touch screen operation is shown in Formula 4.

[0228] z = 2 s Formula 4

[0229] In an embodiment of the present application, a possible way to estimate the startup time of the camera of the device is to simplify the user's touch screen operation into a uniform speed curve model. After the user's finger starts touching the screen for several frames, it approximately reaches a constant uniform touch screen operation speed, that is, the zoom touch screen speed v. After that, the user always performs touch screen operations at this speed until the user adjusts to the actual target zoom ratio. The mobile terminal can collect the MOVE events detected in the first few frames of the user's startup touch screen operation and obtain the current uniform touch screen operation speed v through calculation.

[0230] When the model of the user's touch screen operation is approximately regarded as uniform, the relationship between the touch screen displacement of the user's finger and time is shown in Formula 5.

[0231] s = vt Formula 5

[0232] Wherein, v represents a stable speed quickly reached after the user's touch screen operation, t represents the touch screen duration, and s represents the touch screen displacement of the finger.

[0233] Substituting the mathematical relationship between the touch screen displacement and time in Formula 5 into Formula 4, the mathematical relationship between the zoom ratio and time under the uniform speed curve model can be obtained, and the relationship is shown in Formula 6:

[0234] z = 2 vt Formula 6

[0235] By adopting common finger touch screen moving speeds or actual testing of the current finger touch screen speed and other methods, the theoretical value or experimental value of the finger touch screen operation speed can be substituted into Formula 6, so as to obtain the specific mathematical relationship between the zoom ratio and time.

[0236] Assume that the current device is operating with the first camera. The speed of the user's uniform touchscreen operation is v. The finger touch displacement required from the starting pixel of the finger touch operation (the first touch position) to the position of switching to the nearest camera (the second touch position) is s, and the starting time node of the finger touch operation is t0. Based on the uniform touchscreen operation model, the estimated duration for the touchscreen operation to reach the second zoom ratio is △t (s / v). Assume that the preloading duration required for the second camera to fully enter the working state is t_pre. Then, when the device reaches the time node of (t0 + △t - t_pre) during the continuous finger touch operation, it can start powering on the second camera to be switched to enter pre-startup, so as to ensure that when the user adjusts the zoom ratio to the second zoom ratio, the newly added second camera can officially enter the working mode.

[0237] In this embodiment of the present application, another possible implementation method is that when the device reaches the time node of (t0 + △t - t_pre) during the continuous finger touch operation, it does not immediately power on the second camera to be switched directly. During the user's touchscreen operation, as the user gets closer and closer to the actual target position they want to reach, the touchscreen operation speed will continuously decrease until it stops. Therefore, a minimum speed threshold v can be set. low , when the device reaches the time node of (t0 + △t - t_pre) during the continuous finger touch operation, detect the current user touchscreen operation speed. If the current real-time touchscreen operation speed is still greater than the minimum speed threshold v. low , it is considered that the user still has the intention to continue the touchscreen operation, and then the second camera to be switched can be powered on to enter pre-startup. Otherwise, it can be considered that the user is about to stop the touchscreen operation, and the second camera is not powered on.

[0238] In the above-described embodiment, the duration △t of the finger touch operation may not be a unique fixed value and may be any value within the interval [△t - t, △t + t], where t is a pre-set time threshold.

[0239] Taking the uniform touchscreen operation model as an example, the power-on startup situation of the camera in this case is illustrated.

[0240] Suppose there is an existing camera usage scenario for a mobile device. The starting first zoom ratio of the mobile device is 2X, and the first camera is currently in use. When the user needs to perform a magnification operation, the second zoom ratio at the next critical point is 10X, which means that starting from 10X, the second camera needs to be switched to work. Based on a certain distance of finger touch screen operation, the estimated touch screen operation speed is 100 pixel / s. Suppose the total displacement required for touch screen operation from the first touch screen position on the current screen of the finger to the second touch screen position for switching to 10X is 500 pixels. Through calculation, it can be obtained that the user should switch to the second camera to collect image data after continuously and uniformly touching the screen for 5 s. Suppose the preloading duration estimated by the mobile device for the second camera is about 300 ms - 400 ms. Therefore, the mobile device starts to power on and load resources for the second camera when it detects that the user's finger has continuously touched the screen for about 4.6 s - 4.7 s.

[0241] In another possible way to estimate the camera startup time in the embodiments of the present application, it is assumed that the user's touch screen operation is a uniformly accelerated curve model. To achieve a better user experience, other non-uniform models can be selected for the user's touch screen operation model. When the user performs a touch screen operation, the touch screen operation speed is relatively slow at the beginning and will gradually accelerate and increase later. Therefore, a uniformly variable motion model can be used to simulate the user's touch screen operation. Among them, the relationship between the uniformly variable motion speed and time is shown in Formula 7:

[0242] v = v0 + at Formula 7

[0243] Among them, v0 represents the starting touch screen operation speed, a represents the acceleration of the touch screen operation speed, t represents the duration of the touch screen operation, and the touch screen operation speed v increases uniformly with time. The mobile device can collect the MOVE events detected in the first few frames when the user starts the touch screen operation and obtain the starting touch screen operation speed v0 through calculation.

[0244] Among them, the relationship between the touch screen operation displacement and time in the uniformly variable motion model is shown in Formula 8:

[0245] s = v0 * t + 1 / 2 * a * t^2 Formula 8

[0246] Among them, v0 represents the initial speed, a represents the acceleration, t represents the duration, and s represents the touch screen operation displacement. The displacement also increases uniformly with time.

[0247] Among them, the acceleration a is related to the user's touch screen operation behavior. One way to calculate the acceleration a is actual data sampling. During the user's touch screen operation, the corresponding values of time and touch screen operation displacement are collected, and the value of the acceleration is obtained through actual calculation. Another possible way to calculate the acceleration a is to refer to the empirical data values in the past touch screen operation models.

[0248] In this embodiment, Formula 4 has established the relationship between the displacement and time of the user's touch screen operation. Substituting Formula 8 into Formula 4 gives the relationship between the zoom ratio and time of the current user's touch screen operation. Assuming that the initial speed of the user's touch screen operation is 0, the mathematical relationship between the zoom ratio z and time t is shown in Formula 9.

[0249]

[0250] Through Formula 8, the estimated power-on time of the camera under the current user's touch screen operation can be calculated. Assume that the current device uses the first camera to work, the initial speed of the user's touch screen operation is v, the uniform acceleration is a, the displacement required for the finger to touch the screen from the starting pixel position of the first touch position to the second touch position corresponding to the second camera is s, and the starting time node of the finger touch screen operation is t0. Based on the uniform acceleration touch screen operation model, substituting into the formula to estimate the duration of the touch screen operation to the second zoom ratio is △t. Assume that the preloading duration required for the second camera to fully enter the working state is t pre , then when the device continuously touches the screen with the finger until the time node of (t0 + △t - t pre ), it can start to power on the second camera to be switched to enter pre-startup, so as to ensure that when the user adjusts the zoom ratio to the second zoom ratio, the newly added second camera can officially enter the working mode.

[0251] In this embodiment of the present application, another possible implementation method is that when the device continuously touches the screen with the finger until the time node of (t0 + △t - t pre ), it does not immediately power on the second camera to be switched directly. During the user's touch screen operation, as the user gets closer and closer to the actual target position, the touch screen operation speed will continuously decrease until it stops. Therefore, a minimum speed threshold v low can be set. When the device continuously touches the screen with the finger until the time node of (t0 + △t - t pre ), detect the current user's touch screen operation speed. If the current real-time touch screen operation speed is still greater than the minimum speed threshold v low , it is considered that the user still has the intention to continue the touch screen operation, and then the second camera to be switched can be powered on to enter pre-startup. Otherwise, it can be considered that the user is about to stop the touch screen operation, and the second camera is not powered on.

[0252] In the above-mentioned embodiment, the duration △t of continuous finger touch screen operation may not be a unique fixed value, and may be any value within the interval of [△t - t, △t + t], where t is a preset time threshold.

[0253] Taking the fixed acceleration value a as an example, the power-on startup situation of the camera in this case is illustrated.

[0254] Assume the existing camera usage scenario for a mobile terminal. The starting first zoom ratio is 0.6X, and the wide-angle camera works alone. The critical second zoom ratio is 1X. At 1X, the main camera starts to work, and the wide-angle camera and the main camera jointly collect image data for display. Substitute the relevant data into Equation 8. Assuming a fixed uniform acceleration value a is calculated, it takes approximately 609 ms in total from the initial first zoom ratio touch operation to the second zoom ratio. At the same time, the estimated preloading time for the main camera of the mobile terminal is about 300 ms - 400 ms. Therefore, it can be calculated that the mobile terminal should start powering on and loading resources for the main camera when the user's finger continues the touch operation for about 209 ms - 309 ms.

[0255] In another possible way to estimate the camera startup time of the device in the embodiments of the present application, it is to estimate the power-on time of the second camera by combining the real-time dynamically detected finger touch operation speed. The terminal device can process the detected MOVE events with a fixed sampling period to obtain the real-time touch operation speed. Among them, the fixed period can be any regular fixed time period such as any two frames, three frames, four frames, etc.

[0256] The terminal device regularly calculates the finger touch operation displacement s of the finger's current touch position at this moment reaching the second touch position during the sampling period t , and the real-time touch operation speed calculated at this moment is v t . Therefore, the time △t required for the user's touch operation to reach the second touch position at the sampling moment can be estimated in real time t . Assume that the preloading time required for the second camera to fully enter the working state is t pre . When △t t reaches any value within the interval [t pre - t, t pre + t], the power can be turned on for the second camera to be switched to enter pre-startup, so as to ensure that when the user adjusts the zoom ratio to the second zoom ratio, the newly added second camera can officially enter the working mode. Among them, t is a preset time threshold.

[0257] The relationship between the finger touch operation displacement and time of the user can be matched through a mathematical curve model. In addition to the above-mentioned examples of the uniform speed curve model and the uniformly variable mathematical curve model, the types of curves can also be other touch operation curves, elastic curves, Bezier curves, etc. It is also possible to adopt a large amount of actual data during the user's zooming process, specifically measure the user's touch operation displacement and the corresponding zoom ratio, and further obtain the mathematical curve model matched by the device through ideal mathematical fitting, which is not limited here.

[0258] The following introduces the overall process of the multi-camera zoom method provided by the embodiments of the present application, asFigure 7 The schematic diagram of the method flow shown demonstrates a method for preloading and adding a new second camera in advance based on user operations provided in an embodiment of the present application. This method can be executed by a mobile terminal and includes, but is not limited to, steps S701 - S704, which are described in detail as follows:

[0259] S701: Receive a zoom ratio adjustment operation instruction.

[0260] Among them, the zoom ratio adjustment operation can refer to the Figure 4 user interface shown above, which is used to indicate adjusting the first zoom ratio to the second zoom ratio. Among them, the first zoom ratio is the initial zoom ratio of the image currently displayed on the mobile terminal, and the second zoom ratio is the zoom ratio at which the working camera needs to be switched next according to the current moving direction of the user.

[0261] The mobile terminal may include N cameras. Each time the magnification range indicated by the magnification adjustment operation input by the user (i.e., from the first zoom ratio to the second zoom ratio) may be different, and the multiple cameras sequentially switched by the mobile terminal in response to the magnification adjustment operation may also be different. Here, "multiple cameras" refers to the cameras among the N cameras whose magnification ranges intersect with the range from the first zoom ratio to the second zoom ratio in response to this magnification adjustment operation. "Multiple" means two or more. It should be understood that when there is only one camera among the N cameras whose magnification range intersects with the range from the first zoom ratio to the second zoom ratio, there is no jump caused by camera switching, and digital zooming can be used to achieve it.

[0262] It should also be understood that the adjustment of the zoom ratio is continuous. Here, "continuous" does not mean continuous in the strict sense, but means that the interval between two adjacent magnification ratios is short, such as an interval of 0.1X, 0.2X, etc. Since the magnification ratios indicated by the magnification adjustment operation change continuously from small to large or from large to small, correspondingly, the multiple cameras are also switched sequentially in ascending or descending order of their zoom ratios.

[0263] S702: Determine the duration when the touch screen operation reaches the second touch screen position.

[0264] Based on user behavior, estimate the duration of the touch screen operation from the first touch screen position to the second touch screen position. One estimation method is to, based on the touch screen operation of the user, obtain the mathematical curve 1 between the displacement of the user's finger touch screen operation and time, and at the same time analyze and obtain the mathematical curve 2 between the zoom ratio and the displacement of the user's finger touch screen operation. Substitute curve 1 into curve 2 to obtain the mathematical curve 3 between the zoom ratio of the mobile terminal and the duration under the current operation of the user.

[0265] After obtaining the mathematical relationship between the zoom ratio and the duration in the manner described in the present invention above, during the user's touch screen operation, relevant data required is continuously collected by sensors on the mobile terminal screen, and after reporting the data, relevant calculations are performed. Based on the direction of the user's touch screen operation (zooming in or out), the second zoom ratio is determined, and then through the mathematical curve 3, the approximate duration from the user's current touch screen operation behavior to the second touch screen position corresponding to the second zoom ratio is calculated.

[0266] Another way to estimate the start time of the user's touch screen operation to the second zoom ratio is to detect the user's touch screen operation in real time, calculate the real-time touch screen operation speed based on the real-time touch screen operation, so as to obtain the duration required for the user's finger to reach the second touch screen position at each sampling moment.

[0267] There is no limitation on the calculation method for estimating the duration of the user's touch screen operation to the second zoom ratio. In addition to the above examples, any other method can be used for relevant calculations.

[0268] S703: Activate the second camera at an appropriate time point.

[0269] According to the method described in the above embodiments of the present invention, based on the current power-on of the camera on the mobile terminal, focusing preparation time, exposure preparation time, resource loading time, image acquisition time, data stream transmission time, CPU load rate, buffer rotation time, etc., the preloading time required for the newly added second camera to display and acquire images in the current state is estimated.

[0270] Based on the total duration estimated in step S702 from the current position of the user's touch screen operation to the second zoom ratio position, subtracting the preloading time of the newly added second camera from the total duration, the time interval difference between the two is the time node to activate the newly added second camera.

[0271] S704: Display the acquired image.

[0272] When the user's finger operates to the second touch screen position, the newly added second camera has completed preloading and entered the working mode. The newly added second camera has completed the current image acquisition, and the target data stream is transmitted to the application upper layer through the data interface for image display in the display area of the mobile terminal.

[0273] In a possible implementation manner, all the first cameras stop working, and the second camera works alone. The image acquired by the second camera alone is displayed in the display area.

[0274] In another possible implementation manner, there are still working cameras in the first camera to acquire image data, and the second camera also starts to acquire image data. The image data acquired by all working cameras is processed and then sent to the display area for display.

[0275] In this way, it is ensured that when the user touches the screen and operates to the second touch screen position, the newly added second camera can complete the preloading preparation work in advance, and officially enter the working mode at the switching node, and send the captured image to the display area, so that the color and FOV are smoothly transitioned to the image data captured by the newly added second camera. This prevents the captured picture from being stuck, losing frames or having an unnatural transition due to the switching of the camera.

[0276] The embodiment of the present application also provides a method for determining the second camera to be started and preloading in advance based on the scene. This method can be applied to a mobile terminal including multiple cameras. In a first application scenario, the mobile terminal includes at least three cameras, a front camera a, a rear camera b, a rear camera c, and a rear camera d. Among them, the rear camera b is a wide-angle camera, the rear camera c is the main camera, and the rear camera d is a telephoto camera. In this application scenario, by using the method of this embodiment, when the mobile terminal uses the camera application to take pictures, it can determine the camera to be started based on the current scene and preload the camera in advance. Exemplarily, as Figure 8 shown, the method for taking pictures may include S801-S804:

[0277] S801. Receive a zoom ratio adjustment instruction.

[0278] Exemplarily, the mobile terminal receives an instruction from the user to operate the zoom ratio, the same as step S701 above.

[0279] In addition to the user continuously touching the screen operation in step S701, the user can also directly send a magnification adjustment instruction by directly clicking on the direct magnification control. As Figure 6 shown, the user can directly click on the fixed magnification value in the control 105.

[0280] S802. Anticipate the second camera to be started in the user's current scenario.

[0281] Based on the current scene, anticipate the second camera that the user is about to start, and preload the camera in advance, so that when the user touches the screen and operates to the second zoom ratio, the image stream captured by the second camera can be directly output.

[0282] A possible way to determine the second camera to be started is to identify the best shooting zoom ratio based on the current environment. This method is easy to assist users with general shooting skills to take better pictures. The shooting skills and aesthetic appreciation levels of mobile terminal users are inconsistent. Therefore, the mobile terminal can provide relatively professional optimized shooting guidance to the user during the shooting process.

[0283] For example, when the user opens the camera application, the mobile terminal can clearly identify the current shooting scene (such as indoors, outdoors, rainy days, etc.), shooting time (such as morning, evening, night, etc.), shooting light and other shooting environment information. At the same time, by the user clicking on a certain object area of ​​the screen or through the center position area of ​​the screen, the shooting object that the user may be concerned about can be identified. After the mobile terminal reports and processes a series of information, it identifies the zoom ratio that is most suitable for the shooting object in the current scene. If the camera matched with the ratio is not the currently working camera, the matched camera is used as the camera to be started. After the user executes step S801, the mobile terminal can exemplarily prompt the user with the recommended zoom ratio on the display interface, and guide the user to operate in the direction of a better shooting ratio. When the user operates to the best zoom ratio identified by the mobile terminal, the mobile terminal can prompt the user on the display interface that the ratio is the best ratio.

[0284] Another possible way to determine the camera to be activated is to identify the zoom ratio based on the user's usage habits. This method summarizes the user's habitual shooting method based on the user's regular use and operation over a period of time. When using the camera application, the user usually has a fixed personal shooting habit. The mobile terminal can learn from the user's shooting habits over a period of time. In the next similar scenario, it can determine the camera that the user is most accustomed to using, thereby completing the preparation work for preloading the camera in advance.

[0285] For example, a user is used to taking a landscape picture around 6 o'clock every day after get off work, and the user is used to composing and capturing images at a magnification of 0.6X. The mobile terminal learns the user's shooting habits through a period of information data collection. Assume that the default zoom magnification is 1X when the current camera application is started, and the main camera is used to capture images. When the user opens the camera application at around 6 o'clock in the evening, the mobile terminal determines that the zoom magnification that the user most wants to achieve is 0.6X according to the user's shooting habits, and thus matches the wide-angle camera b to be used when the zoom magnification is 0.6X.

[0286] For another example, the user has been accustomed to shooting at a zoom ratio within a certain range in the recent period, and the mobile terminal identifies the user's shooting habit by collecting information data for a period of time. Assuming that the user has been accustomed to shooting at a zoom ratio of 1.2X-2X in the recent period, the default zoom ratio of the main camera c is 1X when the camera application is started. After the user starts the camera application, the mobile terminal detects that the user has performed a zoom operation in step S801, and based on the user's shooting habits, it determines that the zoom ratio that the user is most likely to adjust is 1.2X-2X, thereby matching the camera that needs to work within this zoom ratio range.

[0287] For another example, during the process of a user adjusting the zoom ratio for shooting in a camera application and then exiting the camera application, all the working cameras stop working simultaneously when exiting. At this time, the mobile terminal records the relevant configurations when the camera application exits, such as the zoom ratio of the camera, the direction of zoom adjustment and the touch screen operation speed before the user exits, the data of the last frame before exiting, the mathematical relationship between the zoom ratio and time, etc. Set a time interval threshold ΔT. When the user returns to the camera application to continue shooting, if the time interval is not greater than ΔT, it is determined that the user needs to continue the previous shooting. The relevant configurations saved when exiting last time will be extracted, the target zoom ratio to be achieved according to the user's previous touch screen operation plan will be identified, and thus the second camera to be started by the mobile terminal will be determined.

[0288] S803. Start the second camera at an appropriate time point.

[0289] In step S802, the mobile terminal has determined the second camera to be started by the user. According to the zoom ratio working range of the mobile terminal, the critical zoom ratio value, i.e., the second zoom ratio, for starting the second camera is determined. Through the position of the user's current finger touching the screen, the displacement to the second touch screen position corresponding to the second zoom ratio is calculated, and at the same time, the change value of the zoom ratio can also be obtained.

[0290] Through the above solution of the present invention, the mathematical relationship between the zoom ratio and time can be obtained. By using a method similar to that shown in step S702, the time length △t for the user to reach the second zoom ratio can be calculated. At the same time, the mobile terminal can use the same method as in step S703 to obtain the preloading time t required to start the second camera. pre 。

[0291] Assume that the starting time node of the finger touch screen operation is t0. Then when the device continuously touches the screen until the time node of (t0 + △t - t pre ), the power can be supplied to the second camera to be switched to enter the pre - startup state.

[0292] In the above - described embodiments, the duration △t of the continuous finger touch screen operation may not be a unique fixed value and may be any value within the interval [△t - t, △t + t], where t is a preset time threshold.

[0293] S804. Display the captured image.

[0294] When the zoom ratio of the mobile terminal reaches the second zoom ratio for switching the lens, at this time, the lens to be working has completed the preloading work and can directly enter the working mode, and the captured image data is transmitted to the display of the mobile terminal for display. Specifically, refer to step S704.

[0295] The embodiment of the present application also provides a method for preloading a second camera in advance based on user operations. This method is applicable to scenarios where users perform rapid zooming, such as performing a high-speed two-finger pinch or a rapid swipe within an extremely short period of time. The speed of such rapid touch-screen operations by users is extremely fast and the time is very short, so the power consumption brought about can be ignored. When the mobile terminal recognizes the operation mode of the user's rapid zooming, in addition to the camera that is currently collecting images, the mobile terminal can directly power on all the remaining same-side cameras (front or rear) and start preloading. When the user finishes the zoom operation and the zoom value stabilizes, then determine the camera that needs to work at the current zoom ratio, and power off the unnecessary cameras to save power consumption.

[0296] One method for determining that a user performs a rapid zoom operation is to set an upper speed threshold V for the touch-screen operation speed of the user's finger, preset a duration threshold T, and detect the user's MOVE event in real time to calculate the real-time touch-screen operation speed. When the touch-screen operation speed is greater than the preset upper speed threshold V and the continuous duration exceeds the duration threshold T, it can be considered that the user is performing a rapid zoom operation.

[0297] Another method for determining that a user performs a rapid zoom operation is to obtain a corresponding gesture operation model through the training of a large number of user rapid zoom operation data sets. When the mobile terminal determines that the user's zoom operation conforms to this gesture operation model, it can be considered that the user has adopted a rapid zoom operation.

[0298] Among them, the operation mode of the user can be any scenario that can cause rapid zooming, such as a rapid two-finger pinch and release scenario, and there is no limitation on this.

[0299] In addition to the above examples, any other method can be used to determine whether the user has performed a rapid zoom operation, and there is no limitation on this.

[0300] The embodiment of the present application also provides a method for preloading a newly added second camera in advance based on user operations. This method is applicable to the scenario where the user rapidly touches and slides back and forth on the screen, such as the user continuously swiping up and down, resulting in the zoom ratio repeatedly and rapidly scaling. In order to ensure the user experience in such extreme scenarios, all the same-side cameras (front or rear) are powered on. After the user stops the zoom operation, based on the zoom ratio in the stable state, then determine the camera that needs to work at the current zoom ratio, and power off the remaining unnecessary cameras to save power consumption.

[0301] One method for determining that a user performs a rapid back-and-forth touch-screen operation is to set a threshold for the change rate of the zoom ratio. When this upper threshold is reached, it means that the zoom ratio of the user has changed suddenly within a time period, and it can be considered that the user has adopted the operation of rapidly and repeatedly scaling the zoom ratio.

[0302] Another method for determining that a user performs a back-and-forth touch screen operation is that if it is detected that the direction of the user's touch screen operation by the finger changes suddenly multiple times within a short period of time, it can be considered that the user has performed an operation of quickly and repeatedly zooming in and out the zoom ratio.

[0303] Another method for determining that a user performs a back-and-forth touch screen operation is to train a corresponding gesture operation model through a dataset of a large number of users' operations of quickly and repeatedly zooming in and out the zoom ratio. When the mobile terminal determines that the user's zoom operation conforms to this gesture operation model, it can be considered that the user has performed an operation of quickly and repeatedly zooming in and out the zoom ratio.

[0304] Among them, the user's operation method is any method that can cause the zoom ratio to be quickly and repeatedly zoomed in and out. For example, it can also be a quick two-finger pinch and release operation, etc., and there is no limitation on this.

[0305] In addition to the above examples, any other method can be used to determine whether the user has performed a back-and-forth touch screen operation, and there is no limitation on this.

[0306] The mobile terminal device continuously detects the working state of the powered-on camera. When a certain camera has not performed image acquisition work for a period of time, it is determined that the user no longer needs to use this camera temporarily, and the application will notify the device to power off this camera, thereby reducing unnecessary power consumption waste.

[0307] An embodiment of the present invention discloses an electronic device, including a processor, as well as a memory, an input device, and an output device connected to the processor. Among them, the input device and the output device can be integrated into one device. For example, a touch sensor can be used as the input device, a display screen can be used as the output device, and the touch sensor and the display screen can be integrated into a touch screen.

[0308] The above-mentioned electronic device may include: a touch screen, the touch screen including a touch sensor and a display screen; one or more processors; multiple cameras; a memory; one or more application programs; and one or more computer programs. The above-mentioned devices can be connected through one or more communication buses. Among them, the one or more computer programs are stored in the above-mentioned memory and are configured to be executed by the one or more processors. The one or more computer programs include instructions, and the above-mentioned instructions can be used to execute each step in the above-mentioned embodiments. Among them, all relevant contents of each step involved in the above-mentioned method embodiments can be cited in the function descriptions of the corresponding entity devices, and will not be elaborated here.

[0309] An embodiment of the present invention also provides a computer storage medium, in which computer instructions are stored. When the computer instructions run on an electronic device, the electronic device executes the above-mentioned relevant method steps to implement the multi-camera zoom method in the above-mentioned embodiments.

[0310] An embodiment of the present invention further provides a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above-mentioned related method steps to implement the multi-camera zoom method in the above embodiment.

[0311] In addition, an embodiment of the present invention further provides a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. The memory is used to store computer execution instructions. When the device runs, the processor may execute the computer execution instructions stored in the memory to enable the chip to execute the multi-camera zoom method in each of the above method embodiments.

[0312] Among them, the electronic device, computer storage medium, computer program product or chip provided by the embodiments of the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0313] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and conciseness of description, only the above-mentioned division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0314] In several embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0315] Those of ordinary skill in the art can understand all or part of the processes in the above embodiment methods. These processes can be completed by relevant hardware instructed by a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of each of the above method embodiments. The foregoing storage media include: various media such as ROM or random access memory RAM, magnetic disks or optical discs that can store program codes.

[0316] The above content is only a specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for device zooming, characterized in that, The device has a first camera and a second camera, and the method includes: Receiving an instruction to open the camera application. At this time, the first camera is working and the second camera is not working, and the zoom ratio of the device is the first zoom ratio. Receiving a zoom ratio adjustment operation, which is used to indicate that the zoom ratio of the device is adjusted from the second zoom ratio to the third zoom ratio through the second zoom ratio. The second zoom ratio is the critical value of the zoom ratio at which the second camera starts to work according to the displacement direction of the zoom ratio adjustment operation. Determining the startup time point of the second camera based on the zoom ratio adjustment operation. At the startup time point, the second camera starts to power on. Among them, the second camera completes preloading before the zoom ratio of the device is adjusted to the second zoom ratio. Displaying image data, where the image data partially or entirely comes from the second camera.

2. The method according to any one of claims 1, characterized in that, The determining the startup time point of the second camera based on the zoom ratio adjustment operation specifically includes: Based on the zoom ratio adjustment operation, determining the total time required for the zoom ratio to be adjusted to the second zoom ratio. Determining the preloading time required for the second camera to complete preloading. Determining the startup time point of the second camera based on the total time and the preloading time.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The preparation process includes any one or more of the stages of camera power-on, pre-focusing, exposure preparation, image data acquisition, image data transmission, resource loading, and buffer rotation. The preparation process duration includes the duration of any one of the above-mentioned preparation processes or the total duration of the multiple stages. Determining the preloading time based on the preparation process duration and the operating ability of the electronic device. The operating ability is related to the hardware configuration and load of the electronic device.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Determining the touch screen operation displacement, which is the displacement generated by the zoom ratio adjustment operation starting from the initial touch screen position.

5. The method according to claim 4, characterized in that, The method further includes: Determining the touch screen speed at startup, which is the speed of the zoom ratio adjustment operation. Determining the total time based on the touch screen operation displacement and the touch screen speed.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: When the speed of the zoom ratio adjustment operation is greater than the first speed, the second camera powers on. The first speed is a preset threshold speed.

7. According to the method described in any one of claims 1-6, characterized in that, The method further includes: Receiving an instruction to exit the camera application. Receiving an instruction to open the camera application again. At this time, the first camera starts to work and the second camera does not work. Receiving a second zoom ratio adjustment operation, where the second zoom ratio adjustment operation includes a rapid zoom ratio adjustment operation, and the rapid zoom ratio adjustment operation refers to that the speed change of the zoom ratio adjustment operation exceeds a preset speed change threshold. Starting all the cameras on the same side of the first camera. The same side means both the front or the back.

8. The method according to any one of claims 1-6, characterized in that The method further includes: Receiving an instruction to exit the application at the first time point. Store the exit data, where the exit data is the information when the application instruction is received, and the information includes at least one of the following: the zoom magnification adjustment operation, the first zoom magnification, the second zoom magnification, the second camera, and the start time point; Receive an instruction to return to the application at a second time point; Determine that the time duration between the second time and the first time is not greater than an interval threshold; Determine the start time point of the second camera based on the exit data.

9. An electronic device, characterized in that, The electronic device includes: one or more processors, a memory, a display, and a plurality of cameras. The memory is used to store computer program code, and the computer program code includes computer instructions. The one or more processors call the computer instructions to cause the electronic device to execute the method according to any one of claims 1-8.

10. A computer-readable storage medium, comprising instructions, characterized in that, When the instruction runs on the electronic device, it causes the electronic device to execute the method according to any one of claims 1-8.

Images