Focus tracking method, electronic device and readable medium

Abstract

The present application provides a focus tracking method, an electronic device and a readable medium. The focus tracking method is applied to an electronic device, and the electronic device includes a camera module with a rotatable reflective prism. The focus tracking method includes: displaying a first interface, the first interface is a camera preview interface or a camera shooting interface in a video recording mode, and the first interface displays a first object; receiving a first operation of the user on the first object on the first interface; determining the target position of the reflective prism in response to the first operation; driving the reflective prism to rotate to the target position; after the reflective prism rotates to the target position, displaying a tracking screen window on the first interface, and the tracking screen window displays the first object. Since the screen displayed in the tracking screen window is obtained by shooting the camera module instead of cropping other images, the clarity is high. In addition, the user only inputs the first operation on the first object, and the camera module can automatically shoot a video, and the problem of blurry video shot due to tracking of the user's mobile device can also be avoided.

Description

Technical Field

[0001] The present application relates to the field of photographing technology, and in particular to a focus tracking method, electronic equipment, computer program product, and computer-readable storage medium. Background Art

[0002] The electronic device shoots a moving object at a medium to long distance, uses a telephoto camera module to capture images, and the user holds the electronic device and follows the moving object to shoot.

[0003] Since the telephoto camera module has a small field of view, users can easily lose sight of the target when composing the image manually. When users move their electronic devices, large movements can affect the anti-shake effect, which in turn affects the image quality. Summary of the invention

[0004] The present application provides a focus tracking method, an electronic device, a computer program product and a computer-readable storage medium, which can enable a user to specify an object to be tracked, and the electronic device can automatically achieve continuous tracking of the user-specified object to shoot high-quality video.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides a focus tracking method, which is applied to an electronic device, wherein the electronic device includes a camera module, and the camera module includes a rotatable reflective prism. The focus tracking method includes: displaying a first interface, wherein the first interface is a camera preview interface or a camera shooting interface in a video recording mode, and the first interface displays a first object; receiving a first operation of a user on the first object on the first interface; determining a target position of the reflective prism in response to the first operation; driving the reflective prism to rotate to the target position; after the reflective prism rotates to the target position, displaying a tracking screen window on the first interface, and the tracking screen window displays the first object.

[0007] It can be seen from the above content that: in the video recording mode, the user inputs the first operation on the first object on the first interface, and the electronic device can respond to the first operation, determine the target position of the reflective prism, and drive the reflective prism to rotate to the target position; after the reflective prism rotates to the target position, the tracking screen window is displayed on the first interface, and the image frame of the video captured by the camera module includes the tracking screen window of the first object. Since the image displayed in the tracking screen window is obtained by the camera module instead of cropping other images, the clarity is high, so the user specifies the object to be tracked and photographed, and the electronic device automatically shoots a high-definition video. In addition, the user only inputs the first operation on the first object, and the camera module can automatically shoot the video without the user holding the device to move the tracking and shooting, and can also avoid the problem of blurred video captured due to the user moving the mobile device to track.

[0008] In one possible embodiment, in response to a first operation, determining the target position of the reflecting prism includes: in response to the first operation, based on the position information of the first operation, the position information of the reflecting prism, and the image data collected by the camera module at the first moment, determining the target position of the reflecting prism at a second moment, the second moment being the next moment of the first moment.

[0009] In a possible implementation, driving the reflective prism to rotate to a target position includes: monitoring a first interrupt signal, driving the reflective prism to rotate to the target position, the first interrupt signal being used to indicate that exposure of the first image ends, and the first image is captured by the camera module at a first operation time.

[0010] In this possible implementation, when the first interrupt signal is monitored, it can be determined that the exposure of the first image output by the camera module is completed, and then the reflective prism is driven to rotate to the target position, thereby ensuring that the image exposure process is not affected by the rotation of the reflective prism.

[0011] In one possible implementation, the camera module also includes an anti-shake component, and further includes: when a first interrupt signal is detected, the anti-shake component is driven not to perform anti-shake, the first interrupt signal is used to indicate that the exposure of the first image is finished, and the first image is collected by the camera module at a first operation time; when a second interrupt signal is detected, the anti-shake component is driven to perform anti-shake, and the second interrupt signal is used to indicate that the camera module starts to output the first image.

[0012] In this possible implementation manner, the anti-shake component performs anti-shake during the output of the first image, thereby ensuring that the clarity of the first image is not affected by device shaking.

[0013] In a possible implementation, the camera module further includes a brake component, wherein: before driving the reflective prism to rotate to the target position, it also includes: driving the brake component to unlock; after driving the reflective prism to rotate to the target position, it also includes: driving the brake component to lock.

[0014] In this possible implementation, the camera module includes a brake component. Before the reflective prism rotates, the brake component can be unlocked. After the reflective prism rotates, the brake component is locked, which can facilitate the reflective prism to quickly stabilize after reaching the position.

[0015] In a possible implementation, the camera module further includes a focusing component, and the method further includes: driving the focusing component to focus on the first object during the process of the reflective prism rotating to the target position.

[0016] In this possible implementation, the camera module includes a focusing component. When the reflective prism rotates to the target position, the focusing component is driven to focus on the first object, thereby further ensuring the clarity of the first object in the video captured by the camera module.

[0017] In a possible implementation, before driving the focusing component to perform focusing, it also includes: using the target position of the reflection prism to compensate for the focusing parameters; during the process of the reflection prism rotating to the target position, driving the focusing component to perform focusing, including: during the process of the reflection prism rotating to the target position, driving the focusing component to focus on the first object with the compensated focusing parameters.

[0018] In a possible implementation, the method further includes: periodically acquiring and storing position information of the reflective prism.

[0019] In a possible implementation manner, the anti-shake component updates the anti-shake parameters using the position information of the reflective prism, and operates with the updated anti-shake parameters.

[0020] In one possible implementation, the hardware abstraction layer of the operating system of the electronic device includes: a control module and an algorithm module for rotating the reflective prism, and the kernel layer of the operating system includes a driver for rotating the reflective prism; determining the target position of the reflective prism includes: the algorithm module determines the target position of the reflective prism; driving the reflective prism to rotate to the target position includes: the control module for rotating the reflective prism controls the driving operation of the reflective prism to drive the reflective prism to rotate to the target position. The control module for rotating the reflective prism refers to the Scan control module, and the driver for rotating the reflective prism refers to the Scan driver.

[0021] In one possible implementation, the hardware abstraction layer of the operating system of the electronic device includes: an anti-shake control module, and the kernel layer of the operating system includes an anti-shake driver; the algorithm module is also used to generate anti-shake parameters, and the electronic device controls the anti-shake component to perform optical anti-shake through the anti-shake control module and the anti-shake driver. The anti-shake control module refers to the OIS control module, and the anti-shake driver refers to the OIS driver.

[0022] In a second aspect, the present application provides an electronic device comprising: one or more processors, a memory, a camera module and a display screen; the camera module comprises a rotatable reflective prism; the memory, the camera module and the display screen are coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and when one or more processors execute the computer instructions, the electronic device performs a focus tracking method as described in any one of the first aspects.

[0023] In a third aspect, the present application provides a computer-readable storage medium for storing a computer program, which, when executed, is specifically used to implement the focus tracking method as described in any one of the first aspects.

[0024] In a fourth aspect, the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the focus tracking method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 An interface diagram of a user performing a continuous tracking object operation in a protagonist mode provided by an embodiment of the present application;

[0026] Figure 2 A hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0028] Figure 4 A running timing diagram of a continuous tracking object process provided by an embodiment of the present application;

[0029] Figure 5 A flowchart of a process for implementing continuous tracking of an object provided in an embodiment of the present application;

[0030] Figure 6 A timing diagram of the Sensor Firmware outputting an exposure interrupt signal to control the operation of the Scan driver, Scan Firmware, OIS Firmware, and brake Firmware provided in an embodiment of the present application;

[0031] Figure 7 A flowchart of the focusing process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0033] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0034] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0035] The commonly used shooting methods for electronic devices to shoot moving objects at medium and long distances are as follows:

[0036] Method 1: Use a wide-angle camera module to shoot images. The size of the objects in the images shot by the wide-angle camera module is small, so the images shot by the wide-angle camera module are cropped and the moving objects are captured through secondary composition. Due to the small size of the camera module sensor, the image quality and clarity of the secondary cropped composition are low, which is difficult to meet the shooting requirements.

[0037] Method 2: Use a telephoto camera module to capture images, and the user holds the electronic device to follow the moving object to capture. Since the telephoto camera module has a small field of view (FOV), it is difficult for the user to manually compose the image, and it is easy to lose the target. When the user holds the electronic device and moves it, the anti-shake capability of the electronic device is required to be high. If the user's movements are too large, the anti-shake effect will be affected, and thus the image quality will be affected.

[0038] Therefore, for moving objects at medium and long distances, electronic devices cannot achieve convenient user operation and capture high-quality images. Based on this, the embodiment of the present application provides a focus tracking shooting method, or a focus tracking method, which allows the user to only specify the object to be tracked, and the electronic device automatically realizes continuous tracking of the user-specified object, and captures high-quality videos through composition, framing, imaging and other actions.

[0039] The focus tracking shooting method provided in the embodiment of the present application can be applied to tracking moving objects in the camera's recording mode. The user can input a continuous tracking operation on the shooting object, and the camera module of the electronic device can automatically track and shoot the moving object selected by the user.

[0040] The following combination Figure 1 , the application scenarios of the focus tracking shooting method provided in the embodiments of the present application are introduced.

[0041] Figure 1 Taking a mobile phone as an example, the interface diagram of the camera shooting video scene is shown. Figure 1 As shown in (a), the mobile phone starts the main character mode in the video recording mode of the camera application, and the mobile phone displays the camera preview interface 101. The camera preview interface 101 displays a picture captured by the mobile phone camera module, and the picture includes three people. If the user wants to track and shoot the person 101, he can click on the person 101 (for example, single-click or double-click) to input the continuous tracking object operation. The mobile phone responds to the operation input by the user, such as Figure 1 As shown in (b), the camera preview interface 103 displays the tracking image 104 of the person 101 in a small window display. Figure 1 As shown in (c), when the person 101 is moving, the mobile phone can continuously track the person 101 by rotating the reflective prism of the camera module. In the camera preview interface 105 displayed by the mobile phone, the tracking screen 104 of the person 101 follows the person 101.

[0042] In some embodiments, the tracking screen window in the camera preview interface 105 can be operated by the user. For example, the user can drag the tracking screen to another location in the camera preview interface.

[0043] In some embodiments, the tracking image window in the camera preview interface 105 is displayed on an upper layer of the camera preview interface.

[0044] In some embodiments, the user selects a tracking object in the protagonist mode of the camera recording mode, and the user activates the camera to start shooting. During the shooting process, the mobile phone can Figure 1 As shown in (b) and (c), the tracking object selected by the user is tracked and photographed, and the tracking image is displayed in a small window. The tracking object may include a moving person, animal or other object.

[0045] In some embodiments, when the camera is shooting a video in the protagonist mode, the user can also Figure 1 The tracking object is selected in the manner shown in (a), and the mobile phone continuously tracks the object selected by the user.

[0046] In some embodiments, the camera is in other video recording modes, not the main mode. The display screen of the mobile phone displays the camera preview interface, which can also be Figure 1 As shown in (a), the person 102 is clicked to input the continuous tracking object operation. The mobile phone responds to the operation input by the user, such as Figure 1 As shown in (b), the camera preview interface 103 displays the tracking image 104 of the person 102 in a small window. Figure 1 As shown in (c), when the person 102 is moving, the mobile phone can continuously track the person 102 by rotating the reflective prism of the camera module.

[0047] When the camera is shooting video in other video modes, users can also Figure 1 The tracking object is selected in the manner shown in (a), and the mobile phone then continuously tracks and shoots the object selected by the user.

[0048] In some embodiments, the camera can be configured with a switch for the object tracking function in the video recording mode. Figure 1 As shown in (a), the person 102 is clicked to input the continuous tracking object operation. The mobile phone responds to the operation input by the user, such as Figure 1 As shown in (b), the camera preview interface 103 displays the tracking image 104 of the person 102 in a small window. Figure 1 As shown in (a), the person 102 is clicked to input a continuous tracking object operation, and the mobile phone does not respond to the operation.

[0049] The focus tracking shooting method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, personal digital assistants (PDA), desktops, laptops, notebook computers, ultra-mobile personal computers (UMPC), handheld computers, netbooks and wearable devices.

[0050] Take mobile phones as an example. Figure 2 This is an example of the composition of an electronic device provided in an embodiment of the present application. Figure 2 As shown, the electronic device 100 may include a processor 110, an internal memory 120, a camera 130, a display screen 140, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, and a camera module 190, etc.

[0051] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a smart sensor hub and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0053] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0054] The internal memory 120 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 120. The internal memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 120, and / or the instructions stored in the memory provided in the processor.

[0055] In some embodiments, the internal memory 120 stores instructions for executing the focus tracking shooting method. The processor 110 can implement moving object tracking in the camera's video recording mode by executing the instructions stored in the internal memory 120, and can control the camera module 190 to automatically track and shoot the moving object selected by the user in response to the user's input operation on the shooting object.

[0056] The electronic device can implement some shooting functions through the ISP, the camera 130, the video codec, the GPU, the display screen 140 and the application processor, etc. The shooting function can be understood as a conventional shooting function.

[0057] The ISP is used to process the data fed back by the camera 130. For example, when taking a photo, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.

[0058] The camera 130 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0059] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0060] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0061] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of electronic devices, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0062] The electronic device implements the display function through a GPU, a display screen 140, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 140 and the application processor. The GPU is used for image rendering by performing mathematical and geometric calculations. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0063] The display screen 140 is used to display images, video interfaces, etc. The display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 140, where N is a positive integer greater than 1.

[0064] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0065] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0066] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 110 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0067] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 150 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0068] The electronic device can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D and the application processor.

[0069] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0070] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device can listen to music or listen to a hands-free call through the speaker 170A.

[0071] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0072] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device can be provided with at least one microphone 170C. In other embodiments, the electronic device can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, and directional recording function, etc.

[0073] The earphone interface 170D is used to connect a wired earphone and can be a USB interface, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0074] In the sensor module 180, the pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 140. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 140, the electronic device detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device can also calculate the position of the touch based on the detection signal of the pressure sensor 180A.

[0075] The touch sensor 180B is also called a "touch control device". The touch sensor 180B can be arranged on the display screen 140, and the touch sensor 180B and the display screen 140 form a touch screen, also called a "touch control screen". The touch sensor 180B is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 140. In other embodiments, the touch sensor 180B can also be arranged on the surface of the electronic device, which is different from the position of the display screen 140.

[0076] The acceleration sensor 180C can detect the magnitude of the acceleration of the electronic device in various directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity, and can also be used to identify the posture of the electronic device.

[0077] The gyro sensor 180D may be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180D.

[0078] In some embodiments, the electronic device implements other shooting functions through the ISP, the camera module 190, the video codec, the GPU, the display screen 140 and the application processor. Other shooting functions may refer to: continuous tracking of moving objects in the camera's video recording mode, implementing operations input by the user on the shooting object, and automatically tracking and shooting the moving object selected by the user.

[0079] In some embodiments, the camera module 190 includes: a lens photosensitive element Sensor, an OIS IC and a Scan IC. The camera module 190 may also include an AF IC, which is not shown in the figure.

[0080] In some embodiments, the camera module 190 is a telephoto module.

[0081] In the processor 110, the ISP is also used to process the data fed back by the camera module 190. For example, when taking a photo, the shutter is opened, and the light is transmitted to the sensor through the lens. The light signal is converted into an electrical signal, and the sensor transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.

[0082] The lens of the camera module 190 includes a reflective prism, which is located in the incident light path of the sensor and is used to reflect light to the sensor. In some embodiments, the reflective prism supports rotational movement in the xz plane, and the range of rotational movement is +-24 degrees. It also supports rotational movement in the yz plane, and the range of rotational movement is +-7. The rotation of the reflective prism can also be understood as the rotation of the camera module lens. The rotation of the camera module lens can achieve that the subject is in the center of the picture during the zooming in or out process.

[0083] Scan IC can be understood as a controller that controls the rotation of the reflective prism. For example, Scan IC can be a chip with logic control capabilities. The Scan motor receives the control instructions of Scan IC and can drive the reflective prism to rotate. In some embodiments, the Scan motor includes two, one drives the reflective prism to rotate in the xz plane, and the other drives the reflective prism to rotate in the yz plane.

[0084] In some embodiments, the brake component can be used to lock and unlock the position of the camera module lens. Before the reflective prism rotates, the brake component can be unlocked, and after the reflective prism rotates, the brake component is locked, which can facilitate the camera module lens to quickly stabilize after reaching the position. In some embodiments, the brake component includes a brake IC and an actuator. The brake IC can receive control instructions from the Scan IC to control the actuator to lock or unlock. The brake IC can be understood as a controller, such as a chip with logic control capabilities.

[0085] In some embodiments, an OIS IC (Optical Image Stabilizer IC) is used for anti-shake. The OIS IC can control the OIS motor for anti-shake. In some embodiments, the OIS motor does not perform anti-shake when the Scan motor drives the reflection prism to rotate. Based on this, the Scan IC can control the OIS motor not to run during the operation of the Scan motor, and to run after the Scan motor finishes running. The OIS IC and the OIS motor can be collectively referred to as an anti-shake component.

[0086] In some embodiments, the AF motor is used for focusing, and the OIS IC can control the AF motor to focus after performing algorithm calculation compensation. The AF motor can also be called a focusing component.

[0087] Electronic equipment in Figure 2 The hardware components shown also have operating systems running on them. operating system, operating system, Operating system, etc. Applications such as camera applications can be installed and run on the operating system.

[0088] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application.

[0089] The layered architecture divides the operating system of the electronic device into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the operating system of the electronic device is the Android system. The Android system can be divided into five layers, from top to bottom, namely the application (APP) layer, the application framework layer (abbreviated as FWK), the system library, the hardware abstraction layer (HAL) and the kernel layer.

[0090] The application layer can include a series of application packages. Figure 3 As shown, the application package may include applications such as camera and call.

[0091] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0092] like Figure 3 As shown, the application framework layer may include window management services, content providers, phone managers, view systems, resource managers, and camera services, etc.

[0093] The window management service is used to manage window programs. The window management service can implement window addition, deletion, display and hiding control, etc. The content provider is used to store and obtain data and make the data accessible to applications. The data can include video, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The phone manager is used to provide communication functions for electronic devices. For example, the management of call status (including connected, hung up, etc.). The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0094] In some embodiments, the camera service may also be referred to as a camera framework, which is used to receive recording requests, image capture requests, and other requests from camera applications, while maintaining the business logic of the internal circulation of recording requests, image capture requests, and other requests, and sending the final result of the request to the camera application.

[0095] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. In some embodiments of the present application, the application cold start will run in the Android runtime, and the Android runtime will obtain the optimized file status parameters of the application. Then, the Android runtime can determine whether the optimized file is outdated due to system upgrades based on the optimized file status parameters, and return the judgment result to the application management module.

[0096] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0097] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0098] The system library may include multiple functional modules, such as a surface manager, a media library, a 3D graphics processing library (such as OpenGL ES), and a 2D graphics engine (such as SGL).

[0099] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0100] The hardware abstraction layer is a software located between the operating system kernel and the hardware circuit. It is usually used to abstract the hardware to realize the interaction between the operating system and the hardware circuit at the logical layer.

[0101] In some embodiments, the hardware abstraction layer includes a camera Hal, which includes an image link Pipeline and CamxHal3. The camera Hal is mainly responsible for the construction, linking, and interactive control of all modules in the image link. CamxHal3 is used to interact with the upper-level module to open or close the camera Hal, and to transmit data to the camera Hal.

[0102] In some embodiments, the image link Pipeline includes: a module control module, an ISP control module and an algorithm module; the module module is responsible for processing the camera module management, and the module module includes a sensor control module, a scan control module, an OIS control module, and a motor actuator control module.

[0103] In some embodiments, the sensor control module is used to connect to the image sensor Sensor, and the sensor control module can control the operation of the sensor through the sensor driver and sensor firmware. The sensor control module can also be called a sensor node.

[0104] In some embodiments, the Scan control module is used to connect to the Scan motor to realize the rotation of the reflective prism. The Scan control module can control the operation of the Scan motor through the Scan driver and Scan Firmware.

[0105] In some embodiments, the OIS control module is used to implement anti-shake processing during the rotation of the camera module, and the OIS control module can control the operation of the OIS motor through the OIS driver and OIS Firmware. In other embodiments, the Scan control module can control the operation of the OIS motor through the Scan driver, Scan Firmware and control OIS Firmware.

[0106] In some embodiments, the Actuator control module is used for AF focusing during the rotation of the camera module. The Actuator control module can control the operation of the AF motor through AF drive, OIS Firmware and AF Firmware.

[0107] The ISP control module is used to receive image data sent by the sensor, process the image data, and then send it to the algorithm module. In some embodiments, the sensor refers to the sensor of the camera module 190.

[0108] The algorithm module is used to receive the image data sent by the ISP control module, analyze the image data, obtain the operation strategy of the lower module, and feed it back to the module in the module control module. In some embodiments, the algorithm module obtains the rotation strategy of the reflective prism and feeds it back to the Scan control module.

[0109] The kernel layer is the layer between hardware and software. The kernel layer receives commands from the control layer and supports the conversion of control commands into actual device operation instructions. Operation instructions include power-on and power-off control, device mode register instruction generation and issuance. The register control commands of the driver layer are passed to the firmware layer through the device bus. The kernel layer includes at least Sensor driver, Scan driver, OIS driver, and AF driver.

[0110] The firmware layer refers to the software system integrated into each component of the camera module, including Sensor Firmware, Scan Firmware, OIS Firmware, AF Firmware and Braking Firmware. In some embodiments, the firmware layer may not include Braking Firmware.

[0111] In some embodiments, Sensor Firmware is a software system running inside the Sensor, responsible for sensor exposure and image data output. Scan Firmware is a software system running inside the Scan IC, used to control the rotation of the camera module lens, and is responsible for converting the code sent down from the upper layer into a control signal, so as to drive the camera module lens to rotate and achieve the focus tracking and framing goals. Brake Firmware is a software system running inside the brake IC, used to lock the position of the camera module, so that the camera module can quickly stabilize after reaching the target position. OIS Firmware is a software system running inside the OIS IC, used for anti-shake. AF Firmware is a software system running inside the AF IC, used for focus tracking, responsible for converting the target position sent down from the upper layer into a motor current, and driving the AF motor to focus.

[0112] Under the above five-layer architecture, the electronic device is also provided with a hardware layer, which may include the above-mentioned electronic device hardware components. Figure 3 The display includes Sensor, Scan IC, OIS IC, AF motor and brake components.

[0113] It should be noted that although the embodiments of the present application are based on The system is used as an example, but the basic principles are also applicable to Electronic devices with other operating systems.

[0114] The camera application starts to display the camera preview interface. The user enters the continuous tracking object operation in the camera preview interface. The camera application sends a frame request to the image link Pipeline through the camera framework to trigger the execution of the continuous tracking object process. The request carries the coordinates of the continuous tracking object operation position.

[0115] Figure 4 Demonstrates the runtime timing of a persistent tracking object process.

[0116] like Figure 4As shown, the image link Pipeline receives the request and executes 1. Calling the algorithm module to process the request. In some embodiments, the algorithm module parses the coordinates of the operating position of the continuously tracked object and the position information of the reflective prism to obtain the target position of the reflective prism. Since the Scan motor drives the reflective prism to operate, the target position of the reflective prism obtained by the algorithm module is usually the operating parameter of the Scan motor, which is expressed in the form of code. In some embodiments, the algorithm module can also obtain the operating parameters of the OIS motor based on the coordinates OIS of the continuously tracked object operating position and the position information of the motor, which can also be expressed in the form of code. In some embodiments, the algorithm module can also calculate the focus parameters of the AF motor, which are also expressed in the form of code.

[0117] The algorithm module calculates the code of the Scan motor and executes 2. to write the code to the Metadata Pool.

[0118] The Scan control module executes 3-1. Monitoring the data written to the Metadata Pool, the OIS control module executes 3-2. Monitoring the data written to the Metadata Pool, and the Actuator control module executes 3-3. Monitoring the data written to the Metadata Pool.

[0119] When the Scan control module detects that the Metadata Pool is written into the Scan motor code, it executes 4-1. The Scan control module verifies the code and sends it to the Scan driver via the CSL bus and the V4L2 bus.

[0120] When the OIS control module detects that the Metadata Pool is written with the code of the OIS motor, it executes 4-2. It sends an OIS command to the OIS driver via the CSL bus and the V4L2 bus. The command can carry a code to drive the OIS driver to perform anti-shake.

[0121] The image link Pipeline also controls the SensorNode to execute 4-3. It sends the exposure sequence and exposure interrupt control sequence to the Sensor driver through the CSL bus and V4L2 bus.

[0122] The Scan driver receives the code and executes 5-1. It writes the received code into the cache sequence.

[0123] The Sensor driver receives the exposure sequence and exposure interrupt control sequence, and executes 5-2. Configure the SensorFirmware to output exposure interrupt. The exposure interrupt can be different according to the exposure time and non-exposure time. Different level identifiers can be used to distinguish the exposure time and non-exposure time. For example, the exposure time is high level and the non-exposure time is low level, or the exposure time is low level and the non-exposure time is high level. The change of high and low levels is used as the interrupt input of the Scan driver, ScanFirmware, and OIS Firmware.

[0124] The OIS driver can convert the code into an OIS command, usually a register instruction, and execute 5-3. The command is sent to the OIS Firmware through a bus such as I2C to drive the OIS Firmware to perform anti-shake processing.

[0125] During the image exposure process, the Sensor can output image data to the ISP. In order to instruct the Sensor to output image data, the Sensor Firmware can execute 6. Output a sensor SOF interrupt signal to the Scan driver. The sensor SOF interrupt signal is used to instruct the Sensor to start outputting the first line of image data.

[0126] The Scan driver executes 7. Monitors the sensor SOF interrupt signal sent by the Sensor Firmware, and then executes to send code to the Scan Firmware. After receiving the code, the Scan Firmware will not respond to the code immediately, but can monitor other interrupt signals sent by the Sensor Firmware. After monitoring the Sensor Firmware, it executes 8. Sends the sensor EOF interrupt signal, which is used to instruct the Sensor to output the last line of image data, and can execute 9-1 to 9-4. In some embodiments, the Scan driver sends code to the Scan Firmware in the following way: the Scan driver converts the code into register instructions, and the instructions are passed to the Scan Firmware through buses such as I2C.

[0127] Scan Firmware executes 9-1. Controls the brake firmware to unlock the brake, and after the rotation is completed, controls the brake firmware to lock the brake; after the brake is unlocked, Scan Firmware executes 9-2. Controls the rotation of the reflective prism; and Scan Firmware can also periodically obtain TMP data and execute 9-3. Updates TMR data to OIS Firmware in real time. In some embodiments, TMR data indicates the position of the reflective prism.

[0128] During the process of controlling the rotation of the reflective prism by Scan Firmware, 9-4 can also be executed. When controlling the rotation of the reflective prism, control the OIS Firmware to enter HOLD ON; after the rotation is completed, control the OIS Firmware to enter OIS ON.

[0129] The sensor firmware also controls the running state of the OIS firmware according to the interrupt signal. In some embodiments, the sensor firmware executes 10-1. sensor SOF interrupt signal to control the OIS firmware to be in the OIS ON state; and executes 10-2. sensor EOF interrupt signal to control the OIS firmware to switch to the HOLD ON state. Among them, HOLD ON can be understood as the anti-shake pause state, and OIS ON can be understood as the anti-shake on state.

[0130] The OIS Firmware responds to the control of the Scan Firmware and the Sensor Firmware, and executes 11-1. During the rotation, it is in HOLD ON, and after the rotation is completed, it switches to OIS ON; 11-2. Receive and update the anti-shake parameters according to the TMR data; 11-3. Synchronously receive the information driven by the OIS. When the OIS Firmware responds to the control of the Scan Firmware and enters HOLD ON, the OIS drive command is not executed for anti-shake.

[0131] In some embodiments, Scan Firmware executes 9-4, and Sensor Firmware executes 10-1 and 10-2, either of which can be executed alternatively.

[0132] Scan Firmware can execute 12-1. Report TMP data to Scan driver. In addition, OIS Firmware can also periodically obtain HALL data and execute 12-2. Report HALL data, which is used to indicate the position of OIS motor.

[0133] The Scan driver executes 13-1. Reports TMR data to the Scan control module via the V4L2 bus and the CSL bus. The OIS driver executes 13-2. Reports HALL data to the OIS control module via the V4L2 bus and the CSL bus.

[0134] The Scan control module executes 14-1. Reports TMR data to the NCS. The OIS control module executes 14-2. Reports HALL data to the NCS.

[0135] The NCS executes 15. to summarize the data reported by the Scan control module and the data reported by the OIS control module, and reports the summarized data to the algorithm module. The algorithm module stores the summarized data.

[0136] The Actuator control module can also monitor the AF motor code written into the Metadata Pool and send it to the AF motor; the AF motor executes 16. The AF motor code is passed to the OIS Firmware of the OISIC, and the OIS Firmware performs position compensation to obtain a new code. The new code is sent to the AF Firmware of the AFIC to execute focusing to ensure clear imaging.

[0137] The following combination Figure 5 , the process of continuously tracking objects provided in an embodiment of the present application is introduced.

[0138] like Figure 5 As shown, the process of continuously tracking an object provided by the embodiment of the present application includes:

[0139] S501: The user inputs a continuous tracking object operation in the camera preview interface.

[0140] In some embodiments, the user starts a camera application, and the camera application displays a camera preview interface on the display screen. The user can select a camera operating mode on the camera preview interface, input a continuous tracking object operation, and the like. Figure 1 (a) shows that the user clicks on an object in the camera preview interface to input a continuous tracking object operation.

[0141] In some embodiments, the camera application starts shooting, and during the video shooting process, the camera application displays a video shooting interface, and the user can also input a continuous tracking object operation in the video shooting interface. Exemplarily, the user can input the continuous tracking object operation in the video shooting interface by clicking the object.

[0142] In some embodiments, the camera application receives an input operation from a user and may also obtain an operation position of the user.

[0143] like Figure 1 As shown in (a), the user clicks on the object in the camera preview interface to input the continuous tracking object operation, and the camera application receives the user's operation and obtains the user's operation position. In some embodiments, the user's operation position can be understood as the coordinates of the operation position input by the user in the image displayed on the camera preview interface.

[0144] S502: The camera application sends a request to the camera service, where the request carries the coordinates of the operation position of the continuously tracked object.

[0145] After receiving the continuous object tracking operation input by the user and obtaining the user's operation position, the camera application can send a request to the camera service. The request carries the user's operation position, that is, carries the coordinates of the user's operation position. The request sent by the camera application to the camera service is used to request the execution of the continuous object tracking process.

[0146] S503: The camera service sends a request to the algorithm module, where the request carries the coordinates of the operation position of the continuous tracking object.

[0147] In some embodiments, the camera service sends a request to the algorithm module in the following manner:

[0148] The camera service sends a request to CamxHal3, which carries the coordinates of the operation position of the continuous tracking object. After receiving the request sent by the camera application, the camera service sends the request to CamxHal3. After receiving the request sent by the camera service, CamxHal3 sends the request to Pipeline. After receiving the request, Pipeline sends the request to the algorithm module to call the algorithm module to process the request.

[0149] In some embodiments, the user inputs a continuous tracking object operation, and after CamxHal3 receives the request sent by the camera service, it sends the request to the Pipeline corresponding to the camera module 190.

[0150] It should be noted that when the user turns on the camera, the camera 130 of the electronic device (which can be understood as the main camera) runs to collect images, and the functional modules of the Pipeline corresponding to the camera 130 run to display the camera preview interface on the display screen, which displays the image taken by the camera 130.

[0151] The user inputs a continuous tracking object operation in the camera preview interface, and the function module of the Pipeline corresponding to the camera module 190 is driven to run, and the image captured by the camera module 190 can be displayed in the tracking screen window. In some embodiments, the camera module 190 is a telephoto module, and the subject in the image captured by the camera module 190 is large, which can achieve the zooming of the object screen. In some embodiments, the Pipeline corresponding to the camera 130 can continue to run to maintain the camera preview interface.

[0152] S504, the algorithm module analyzes the image, coordinates and position information of the reflecting prism to obtain a code.

[0153] In some embodiments, the algorithm module stores data such as the position information of the reflective prism and the state of the brake component. In some embodiments, the position information of the reflective prism can be the detection value of the TMR sensor, referred to as TMR data.

[0154] After receiving the request, the algorithm module obtains the coordinates and position information of the reflective prism carried in the request, and uses the coordinates, position information of the reflective prism and image data carried in the request to obtain the position reached by the reflective prism after rotation for the current frame image, that is, the target position. In some embodiments, the target position may be a code.

[0155] It should be noted that when the camera application is in recording mode, the sensor of the camera module 190 continuously captures images and sends them to the algorithm module through the ISP control module. The algorithm module can calculate the target position of the reflection prism corresponding to each frame of the image based on the frame of the image.

[0156] The user inputs the operation of continuously tracking the object. After receiving the request, the algorithm module obtains the coordinates and position information of the reflective prism carried in the request, and also obtains the image data collected by the camera, and uses the image data, coordinates and position information of the reflective prism to obtain the target position of the reflective prism.

[0157] It should be noted that the user inputs a continuous tracking object operation, and the electronic device needs to continuously track and shoot the object specified by the operation. Therefore, the algorithm module can continuously obtain the target position of the reflective prism to drive the reflective prism to continuously rotate and track the object. The algorithm module uses the coordinates of the user's operation position to determine the user's operation object, and determines the moving direction of the object in the image data collected by the camera. The algorithm module combines the moving direction of the object and the current position information of the reflective prism to determine the target position of the next rotation of the reflective prism. The target position of each rotation of the reflective prism can be understood as a position that can ensure that the camera can capture the user's operation object. In this way, the electronic device can automatically follow the moving object for shooting. Since no user operation is required, the clarity of the captured video can also be guaranteed.

[0158] The camera for collecting images can be the main camera, that is, camera 130, or it can be the camera module 190; the camera can also be the main camera for the user input of the continuous tracking object operation moment, and after the algorithm module obtains the target position of the reflecting prism corresponding to the moment, the camera is the camera module 190.

[0159] S505: The algorithm module writes code into the Metadata Pool.

[0160] In some embodiments, the target position of the reflective prism obtained by the algorithm module, i.e., the code, can be written into the data pool Metadata Pool for other modules to monitor the changes of data in the data pool. In some embodiments, the Metadata Pool includes multiple metadata, each metadata is used to store the value of a different function. The algorithm module writes the code into the metadata corresponding to the continuous tracking object operation function.

[0161] In some embodiments, the algorithm module also stores the anti-shake parameters of the OIS motor. In some embodiments, the anti-shake parameters of the OIS motor are the detection values ​​of the Hall sensor, referred to as HALL data. After receiving the request, the algorithm module can also obtain the indication value of the anti-shake strategy of the OIS motor according to the coordinates carried in the request and the anti-shake parameters of the OIS motor. In some embodiments, the anti-shake strategy includes the operating mode and operating parameters of the OIS.

[0162] In some embodiments, the indicator value of the anti-shake strategy of the OIS motor obtained by the algorithm module can also be written into the metadata pool so that other modules can monitor the changes of data in the data pool. In some embodiments, the indicator value of the anti-shake strategy of the OIS motor obtained by the algorithm module is written into the metadata corresponding to the anti-shake function.

[0163] S506. The Scan control module monitors the data of the Metadata Pool.

[0164] S507, the Scan control module monitors that the Metadata Pool writes the code, verifies the code, and then sends a control command to the Scan driver, where the control command carries the code.

[0165] In some embodiments, before the Scan control module sends the code to the Scan driver, it verifies the code, which can be understood as verifying whether the target position of the reflective prism indicated by the code is a legal position. If the verification passes, the code is sent to the Scan driver.

[0166] In some embodiments, after the algorithm module obtains the target position of the reflective prism, it may also send it directly to the Scan control module.

[0167] S508. The Scan driver stores the code in the memory cache queue.

[0168] After receiving the control command carrying the code, the Scan driver will not respond to the control command immediately, but will wait for the moment when the camera module 190 produces an image before pushing the instruction to the Scan Firmware.

[0169] S509, the Sensor driver configures the Sensor Firmware to output an exposure interrupt.

[0170] like Figure 4As described in the corresponding embodiment, the Sensor driver receives the exposure sequence and exposure interrupt control sequence, and can configure the Sensor Firmware to output the exposure interrupt as the interrupt input of the Scan driver, Scan Firmware, and OISFirmware.

[0171] Figure 6 The timing diagram shows how the Sensor Firmware outputs an exposure interrupt signal to control the operation of the Scan driver, Scan Firmware, OIS Firmware, and brake Firmware.

[0172] like Figure 6 As shown, the exposure interrupt output by the Sensor Firmware can refer to the OIS-EN signal, where a high level refers to the exposure time and a low level refers to the non-exposure time. The OIS-EN signal is the sensor SOE interrupt signal, which indicates the start of exposure, that is, the start time of exposure. The OIS-EN signal is the sensor SOF interrupt signal, which indicates that the Sensor starts to output the Nth frame image, that is, starts to output the first line of data of the Nth frame image data. The OIS-EN signal is the sensor EOE or EOF interrupt signal, which indicates that the Sensor has finished framing, and indicates the end of exposure and entering the non-exposure time.

[0173] The AP monitors the OIS-EN signal, and the AP executes 1. If it monitors the sensor SOE interrupt signal, it executes 2. Exposure processing according to the sensorSOE interrupt signal. The AP monitors the sensor SOF interrupt signal, and executes 3. Sending code to ScanFirmware through the Scan driver. The Scan Firmware receives the code and executes 4. Parsing and calculating the code to determine the target position of the reflective prism. In some embodiments, the AP also monitors the sensor EOF interrupt signal, and executes 3. Sending code to Scan Firmware through the Scan driver.

[0174] Scan Firmware monitors the OIS-EN signal and executes 5. It monitors the sensor EOF interrupt signal and executes 6-1. After processing the sensor EOF interrupt signal, it executes 7 to 11. Scan Firmware executes 7. It sends a brake unlock command to the brake firmware, and the brake firmware responds to the command and executes 8. It executes brake unlock; Scan Firmware executes 9. It controls the rotation of the reflective prism; after the rotation is completed, Scan Firmware executes 10. It sends a brake lock command to the brake firmware, and the brake firmware responds to the command and executes 11. It executes brake lock.

[0175] The OIS Firmware monitors the OIS-EN signal and executes 6-2. It monitors the sensor EOF interrupt signal and enters the HOLD ON state in response to the interrupt signal.

[0176] Scan Firmware executes 12. to periodically read TMR data, and transmits the TMR data to OIS Firmware via 13. After receiving the TMR data, OIS Firmware executes 14. to update the anti-shake parameters according to the TMR data.

[0177] The OIS Firmware executes step 15. If it detects the sensor SOE interrupt signal, it executes step 16. In response to the interrupt signal, it enters the OIS ON state.

[0178] S510, Scan driver monitors the sensor firmware output sensor SOF interrupt.

[0179] The sensor SOF interrupt signal indicates that the sensor starts to output frames, that is, starts to output the first line of image data. The Scan driver monitors the sensor firmware output sensor SOF interrupt, and then executes step S513.

[0180] S511, OIS Firmware monitors the sensor Firmware output sensor SOF interrupt.

[0181] If the OIS Firmware monitors the sensor SOF interrupt output by the Sensor Firmware, step S512 is executed.

[0182] In some embodiments, step S510 and step S511 may be understood to be performed in parallel.

[0183] S512, OIS Firmware switches to OIS ON.

[0184] OIS Firmware monitors the sensor SOF interrupt output by Sensor Firmware, indicating that the sensor starts to output frames. OIS Firmware can perform anti-shake processing to ensure that the clarity of the image output by the sensor is high. Therefore, OIS Firmware monitors the sensor SOF interrupt output by Sensor Firmware, and enters the OIS ON state. The OIS ON state can be understood as the anti-shake on state.

[0185] S513, Scan driver reads the code of the kernel cache queue.

[0186] The Scan driver monitors the sensor SOF interrupt output by the Sensor Firmware, reads the code in the kernel cache queue, and prepares to control the Scan Firmware to execute the reflective prism rotation process according to the code.

[0187] S514, the Scan driver sends a code to the Scan Firmware to control the execution of brake unlocking, reflective prism rotation and brake locking.

[0188] In some embodiments, the Scan driver encodes the target position of the reflective prism into a register instruction, and the register instruction is transmitted to the Scan Firmware via a bus such as I2C.

[0189] S515, Scan Firmware monitors sensor EOF interrupt.

[0190] The sensor EOE interrupt signal indicates that the sensor frame output has ended, and that the exposure has ended, and the non-exposure time has begun. If the Scan Firmware monitors the sensor EOF interrupt, steps S518 to S524 are executed.

[0191] During the sensor frame output and exposure process, the rotation of the reflective prism will affect the clarity of the image. Therefore, during the sensor frame output and exposure process, the Scan Firmware needs to wait for the sensor frame output and exposure to end before controlling the rotation of the reflective prism.

[0192] S516, OIS Firmware monitors sensor EOF interrupt.

[0193] If the OIS Firmware monitors the sensor EOF interrupt, step S517 is executed.

[0194] In some embodiments, step S515 and step S516 may be understood to be performed in parallel.

[0195] S517, OIS Firmware enters HOLD ON state.

[0196] As described in step S515, the sensor EOE interrupt signal indicates that the sensor frame output and exposure are completed, and ScanFirmware can drive the reflective prism to rotate. During the rotation of the reflective prism, the electronic device performs anti-shake, which is invalid. Therefore, the electronic device may not perform anti-shake. After the rotation of the reflective prism is completed, the electronic device performs anti-shake. Based on this, OIS Firmware monitors the sensor EOF interrupt and enters HOLDON. HOLD ON can be understood as the anti-shake pause state.

[0197] S518. Scan Firmware sends a HOLD ON command to OIS Firmware.

[0198] Before the Scan Firmware controls the rotation of the reflective prism, the anti-shake function must be stopped first. Therefore, the Scan Firmware sends a HOLD ON command to the OIS Firmware to control the OIS Firmware to stop the anti-shake function.

[0199] In some embodiments, the OIS Firmware may execute step S516 and step S517 to stop anti-shake during the rotation of the reflective prism, and therefore, step S518 may not be executed.

[0200] In some other embodiments, the OIS Firmware may not execute step S516 and step S517. After the Scan Firmware sends a HOLD ON instruction to the OIS Firmware in step S518, the OIS Firmware responds to the instruction and enters the HOLD ON state to stop anti-shake.

[0201] S519, Scan Firmware controls the brake firmware to execute brake unlocking.

[0202] S520: The brake firmware executes brake unlocking.

[0203] The brake firmware controls the brake components to perform brake unlocking.

[0204] S521, Scan Firmware controls the reflective prism to rotate to the position indicated by the code.

[0205] In some embodiments, the Scan Firmware controls the Scan motor to rotate and drive the reflective prism to rotate until the reflective prism rotates to the target position, that is, the position indicated by the code.

[0206] against Figure 1 The continuous tracking object operation is demonstrated. After the reflective prism rotates to the target position, the display shows Figure 1 (b) or Figure 1 In the interface shown in (c), the user can view the tracking screen through the display screen.

[0207] In the embodiment of the present application, the algorithm module analyzes the image, coordinates and position information of the reflective prism through step S504 to obtain the target position of the reflective prism. The target position can be understood as ensuring that the camera module 190 can track the object. Therefore, the Scan Firmware controls the Scan motor to rotate and drive the reflective prism to rotate until the reflective prism rotates to the target position. After the user performs a continuous tracking operation on an object, the tracking screen window of the display screen can display the tracked object without the need for manual tracking by the user. In addition, the user does not need to track manually, and the problem of blurred video caused by the user moving the device for tracking by hand is avoided.

[0208] In some embodiments, the camera module 190 is a telephoto module, and the image captured by the camera module 190 forms a tracking picture, which can also ensure that the tracking picture has a high definition.

[0209] S522, Scan Firmware controls the brake firmware to execute brake locking.

[0210] S523, the brake firmware executes brake locking.

[0211] Brake Firmware controls the brake components to perform brake locking.

[0212] S524. Scan Firmware sends an OIS ON command to OIS Firmware.

[0213] After the brake is locked, Scan Firmware sends an OIS ON command to OIS Firmware to control the execution of anti-shake. In response to the command, OIS Firmware enters the OIS ON state.

[0214] S525. The Scan control module sends a query command to the Scan driver to query TMR data.

[0215] The Scan control module has a timer that regularly reads TMR data and brake status to pass them to NCS, which then passes them to the algorithm module for Scan's next operating mode and code decision, forming a Scan closed-loop control.

[0216] In some embodiments, a timer may also exist in the Scan driver. The Scan driver periodically reads the TMR data and the brake status and reports them to the algorithm module through the NCS.

[0217] In some embodiments, the OIS control module may also have a timer to periodically read HALL data from the OIS driver, and the HALL data is passed to the NCS, which then passes the data to the algorithm module for anti-shake.

[0218] S526. The Scan driver sends a query command to the Scan Firmware to query the TMR data.

[0219] The Scan driver receives the query command from the Scan control module, or is triggered by its own timer to send a query command to the ScanFirmware to query the TMR data.

[0220] S527, Scan Firmware reads TMR data.

[0221] The Scan Firmware receives the query command and reads the TMR data from the TMR sensor.

[0222] S528. Scan Firmware sends TMR data to OIS Firmware.

[0223] S529, OIS Firmware updates anti-shake parameters according to TMR data.

[0224] In some embodiments, before anti-shake is turned on, the OIS Firmware may update anti-shake parameters according to TMR data to ensure the anti-shake effect when the Scan motor is in different positions.

[0225] S530, Scan Firmware reads the brake status value.

[0226] The brake status value indicates the operating status of the brake components.

[0227] S531, Scan Firmware reports TMR data and brake status value to Scan driver.

[0228] After the Scan Firmware reads the TMR data and brake status value, it can report the TMR data and brake status value to the Scan driver.

[0229] S532, the Scan driver reports the TMR data and brake status value to the Scan control module.

[0230] After receiving the TMR data and the brake status value, the Scan driver reports the TMR data and the brake status value to the Scan control module.

[0231] S533, the Scan control module reports the TMR data and brake status value to the NCS.

[0232] After receiving the TMR data and the brake status value, the Scan control module reports the TMR data and the brake status value to the NCS.

[0233] NCS (Non Camera Sensor Service) Figure 4 As shown in FIG. 1 , it is located in the hardware abstraction layer and is used to manage the data of other auxiliary devices except the Sensor.

[0234] The NCS may pass the TMR data to the algorithm module. In some embodiments, the algorithm module registers a TMR data subscription, after which the TMR data may be passed from the Scan control module to the NCS, which then passes the data to the algorithm module.

[0235] S534, NCS sends TMR data and brake status value to the algorithm module.

[0236] The algorithm module receives the TMR data and the brake status value, and can update and store the TMR data and the brake status value.

[0237] In some embodiments, the Scan control module may send TMR data and brake status values ​​directly to the algorithm module, or send TMR data and brake status values ​​to the algorithm module through other modules, and is not limited to the methods of steps S533 and S534 of this embodiment.

[0238] In response to the continuous tracking object operation input by the user, the electronic device can also perform a focusing process to ensure the clarity of the captured image. Figure 7 , the focusing process of the electronic device is introduced.

[0239] It is understood that the electronic device may execute Figure 5 The continuous object tracking process shown and the focusing process shown in the following embodiments.

[0240] like Figure 7 As shown, the focusing process of the electronic device provided in the embodiment of the present application includes:

[0241] S701: The user inputs a continuous tracking object operation in the camera preview interface.

[0242] The specific implementation of step S701 can refer to the content of step S501 in the above embodiment, which will not be repeated here.

[0243] S702: The camera application sends a request to the camera service, where the request carries the coordinates of the operation position of the continuously tracked object.

[0244] The specific implementation of step S702 can refer to the content of step S502 in the above embodiment, which will not be repeated here.

[0245] S703: The camera service sends a request to the algorithm module, where the request carries the coordinates of the operation position of the continuous tracking object.

[0246] The specific implementation of step S703 can refer to the content of step S503 in the above embodiment, which will not be repeated here.

[0247] S704: The algorithm module obtains a focus parameter code.

[0248] The focus parameters can be understood as the operating parameters of the AF motor. In some embodiments, the algorithm module can use a contrast focus algorithm or a phase focus algorithm to obtain the focus parameters. In other embodiments, the algorithm module uses a contrast focus algorithm to obtain one focus parameter and uses a phase focus algorithm to obtain another focus parameter. The algorithm module combines the two focus parameters to obtain the focus parameters to be issued.

[0249] S705: The algorithm module writes code into the Metadata Pool.

[0250] The algorithm module obtains the focus parameters and writes them into the Metadata Pool corresponding to the focus function's metadata.

[0251] S706. The Actuator control module monitors the data of the Metadata Pool.

[0252] S707, the Actuator control module monitors that the Metadata Pool writes the code, verifies the code and then sends a control command to the AF driver, the control command carrying the code.

[0253] Before the Actuator control module sends the code to the AF driver, it verifies the code, which can be understood as verifying whether the target position of the AF motor indicated by the code is a legal position. If the verification passes, the code is sent to the AF driver.

[0254] In some embodiments, after obtaining the focus parameters, the algorithm module may directly send the focus parameters to the Actuator control module.

[0255] S707. The AF driver writes code into the CRM.

[0256] After receiving the control command carrying the code, the AF driver will not respond to the control command immediately, but will wait for the moment when the camera module 170 outputs the image before controlling the focus, so as to achieve synchronization between the camera module 190 outputting the image and the operation of the focus motor.

[0257] CRM (camera request manager) is a module for managing camera application requests. The requirements issued by the hardware abstraction layer module can be first transmitted to CRM, which matches according to the sensor frame timing, and then sends the configuration to the corresponding module in the N+1 or N+2 effective mechanism. The data interruption of the underlying sensor frame can also be reported to the hardware abstraction layer module through CRM.

[0258] S707. IFE receives an interrupt signal corresponding to the image frame.

[0259] IFE (Image front-end engine) can be understood as a hardware processing unit. The image frame output by the sensor of the camera module 190 can be pre-processed by IFE. If IFE receives an interrupt signal corresponding to the image frame, it means that the sensor of the camera module 190 outputs the image frame.

[0260] In some embodiments, the interrupt signal corresponding to the image frame is a sensor SOF interrupt signal.

[0261] S710. IFE schedules CRM to send code to AF driver.

[0262] IFE receives the interrupt signal corresponding to the image frame, which can indicate that the sensor of the camera module 190 outputs the image frame. Therefore, IFE schedules CRM to send code to the Scan driver to control the moment when the sensor of the camera module 190 outputs the image frame. The Scan driver uses the code to drive the rotation of the reflective prism to realize the rotation of the reflective prism following the action of the sensor of the camera module 190 outputting the image frame, and the two are executed synchronously.

[0263] S711, the AF driver controls the OIS Firmware to perform focusing according to the code.

[0264] S712, OIS Firmware performs algorithmic compensation on the focus parameters according to the target position of the reflective prism.

[0265] In some embodiments, the OIS Firmware stores the correspondence between the target positions of multiple reflective prisms and the compensation values ​​of the focus parameters. The OIS Firmware can use the correspondence to determine the compensation value corresponding to the target position of the reflective prism and use the compensation value to compensate the focus parameter.

[0266] In some embodiments, the AF motor includes two, and the OIS Firmware compensates the focus parameters by the compensation values ​​of the two AF motors according to the target position of the reflective prism to obtain the compensated focus parameters of the two AF motors.

[0267] In some embodiments, the Scan Firmware periodically sends the position information of the reflective prism to the OIS Firmware, and the OIS Firmware may use the position information of the reflective prism sent by the Scan Firmware at the latest moment as the target position of the reflective prism.

[0268] S713. The OIS Firmware controls the AF Firmware to perform focusing according to the compensated focus parameters.

[0269] In some embodiments, when the OIS Firmware receives the focus parameters, step S712 may not be executed, and the focus parameters may be transparently transmitted to the AF Firmware, and the AF Firmware performs focusing according to the focus parameters.

[0270] S714, AF Firmware performs focusing according to the compensated focus parameters.

[0271] The AF Firmware controls the AF motor to operate and achieve focus according to the compensated focus parameters. In some embodiments, there are two AF motors, and the AF Firmware controls the two AF motors to operate and achieve focus according to the compensated focus parameters.

[0272] In some embodiments, the AF target code value may be passed to the Scan IC, which has TMR data inside and may also support compensation operations.

[0273] In some embodiments, the AF code may also be directly transmitted to the AF motor for execution without compensation based on the TMR data.

[0274] Another embodiment of the present application further provides a computer-readable storage medium, which stores instructions, and when the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0275] The computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0276] Another embodiment of the present application further provides a computer program product including instructions. When the computer program product is run on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

Claims

1. A focus tracking method, characterized in that: Applied to an electronic device, the electronic device includes a first camera module and a second camera module, the second camera module includes a rotatable reflective prism, and the tracking focus method includes: Displaying a first interface based on data collected by the first camera module, the first interface being a camera preview interface or a camera shooting interface in a video recording mode, and the first interface displays a first object; Receiving a first operation of the user on the first object in the first interface; In response to the first operation, controlling the second camera module to capture images and sending the images captured by the second camera module to the algorithm module through the ISP control module; The algorithm module determines the target position of the reflective prism at a second moment by using the position information based on the first operation, the position information of the reflective prism, and the image data collected by the second camera module at the first moment, obtaining the target position of the reflective prism corresponding to each frame of image received by the algorithm module, and writing the target position of the reflective prism corresponding to each frame of image into a data pool, wherein the second moment is the next moment of the first moment; After the control module for rotating the reflecting prism monitors that the data pool is written to the target position of the reflecting prism corresponding to a frame of image captured by the second camera module, a control command is issued to the driver for rotating the reflecting prism, wherein the control command carries the target position of the reflecting prism corresponding to the frame of image, and the driver for rotating the reflecting prism stores the target position of the reflecting prism corresponding to the frame of image into a memory cache queue; When a sensor SOE interrupt signal is monitored, the application processor performs exposure processing according to the sensor SOE interrupt signal, where the sensor SOE interrupt signal is used to indicate that a frame of image captured by the second camera module starts to be exposed; In the case of monitoring the sensor SOF interrupt signal, the application processor sends the target position of the reflective prism corresponding to the one frame of image to the Scan Firmware through the driving of the reflective prism to rotate, and the Scan Firmware analyzes and calculates the target position of the reflective prism corresponding to the one frame of image to determine the target position of the reflective prism, and the sensor SOF interrupt signal is used to instruct the second camera module to start outputting the one frame of image; In the case of monitoring a sensor EOE or EOF interrupt signal, the Scan Firmware controls the reflective prism to rotate to a target position of the reflective prism corresponding to the one frame of image, and the sensor EOE or EOF interrupt signal is used to indicate that the exposure of the one frame of image is finished; After the reflecting prism rotates to the target position of the reflecting prism corresponding to the frame of image, a tracking picture window is displayed on the first interface, and the tracking picture window displays the first object. The first object displayed in the tracking picture window is obtained based on the data collected by the second camera module.

2. The focus tracking method according to claim 1, characterized in that: The step of controlling the reflective prism to rotate to a target position of the reflective prism corresponding to the one frame of image by the Scan Firmware includes: The Scan Firmware controls the reflection prism to rotate in the first plane and / or the second plane until it rotates to the target position of the reflection prism corresponding to the one frame of image, and the first plane is perpendicular to the second plane.

3. The focus tracking method according to claim 1 or 2, characterized in that: The second camera module also includes an anti-shake component, and further includes: A first interrupt signal is monitored, and the anti-shake component is driven to stop anti-shake, wherein the first interrupt signal is used to indicate that the exposure of the first image is ended, and the first image is collected by the second camera module at the first operation time; The second interrupt signal is monitored and the anti-shake component is driven to perform anti-shake. The second interrupt signal is used to instruct the second camera module to start outputting the first image.

4. The focus tracking method according to claim 1 or 2, characterized in that: The second camera module also includes a brake component, wherein: Before driving the reflective prism to rotate to the target position of the reflective prism corresponding to the one frame of image, the method further includes: driving the brake component to unlock; After driving the reflective prism to rotate to the target position of the reflective prism corresponding to the one frame of image, the method further includes: driving the brake component to lock.

5. The focus tracking method according to claim 3, characterized in that: The second camera module also includes a brake component, wherein: Before driving the reflective prism to rotate to the target position of the reflective prism corresponding to the one frame of image, the method further includes: driving the brake component to unlock; After driving the reflective prism to rotate to the target position of the reflective prism corresponding to the one frame of image, the method further includes: driving the brake component to lock.

6. The focus tracking method according to claim 1 or 2, characterized in that: The second camera module further includes a focusing component, and the method further includes: During the process of the reflection prism rotating to the target position of the reflection prism corresponding to the one frame of image, the focusing component is driven to focus on the first object.

7. The focus tracking method according to claim 6, characterized in that: Before driving the focusing component to perform focusing, the method further includes: Compensating the focus parameter using the target position of the reflection prism corresponding to the one frame of image; In the process of rotating the reflection prism to the target position of the reflection prism corresponding to the one frame of image, driving the focusing component to perform focusing includes: During the process of the reflection prism rotating to the target position of the reflection prism corresponding to the one frame of image, the focusing component is driven to focus on the first object with the compensated focusing parameters.

8. The focus tracking method according to claim 3, characterized in that: Also includes: The position information of the reflecting prism is periodically acquired and stored.

9. The focus tracking method according to claim 8, characterized in that: The anti-shake component updates anti-shake parameters using the position information of the reflective prism and operates with the updated anti-shake parameters.

10. The focus tracking method according to claim 3, characterized in that: The hardware abstraction layer of the operating system of the electronic device includes: an anti-shake control module, and the kernel layer of the operating system includes an anti-shake driver; The algorithm module is also used to generate anti-shake parameters, and the electronic device controls the anti-shake component to perform optical anti-shake through the anti-shake control module and the anti-shake driver.

11. An electronic device, characterized in that: include: One or more processors, memories, camera modules and display screens; the camera modules include a rotatable reflective prism; The memory, the camera module and the display screen are coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the focus tracking method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed, is specifically used to implement the focus tracking method as described in any one of claims 1 to 10.

Images