Automatic focusing method, electronic equipment, storage medium and program product

Abstract

The invention provides an automatic focusing method, electronic equipment, a storage medium and a program product, and belongs to the technical field of electronic equipment. The method comprises the following steps: starting an RGB image sensor and a depth image sensor; the RGB image sensor performs exposure processing on the RGB data of the Nth frame, and the depth image sensor processes the depth data corresponding to the Nth frame; the RGB image sensor outputs an RGB image of the Nth frame; in response to all output of the RGB image of the Nth frame, sending a control instruction to a camera motor; controlling a camera motor to work based on the control instruction so as to complete automatic focusing of the (N + 1) th frame; the RGB image sensor performs exposure processing on the RGB data of the (N + 1) th frame, and the depth image sensor processes the depth data corresponding to the (N + 1) th frame. According to the invention, automatic focusing can be rapidly and accurately carried out, and the quality of the shot image is improved.

Description

Technical Field

[0001] The present application relates to the technical field of electronic equipment, and in particular to an autofocus method, electronic equipment, storage medium, and program product. Background Art

[0002] Automatic Focus (AF) automatically controls the camera's motor to move the lens based on the selected object in the scene, adjusting the distance between the lens and the photosensitive element. AF accuracy directly impacts the clarity and three-dimensionality of the captured image. Therefore, achieving fast and accurate autofocus is crucial to improving image quality.

[0003] In the related art, the RGB data of any frame is obtained based on the RGB image sensor, and the PD (Phase Detection) focusing algorithm is used to calculate the RGB data of the frame to obtain the PD value of the frame. Based on the PD value, the movement distance of the camera lens when automatically focusing on the frame is calculated. Based on the movement distance, a control instruction is generated and sent to the camera motor to control the operation of the camera motor, thereby achieving automatic focus on the frame image.

[0004] However, the PD focus algorithm is a serial algorithm, and its ability to follow the focus of moving objects, especially those moving in the direction perpendicular to the direction of the camera lens, is weak. It cannot automatically focus quickly and accurately, resulting in poor quality of the captured image. Summary of the Invention

[0005] The embodiments of the present application provide an autofocus method, electronic device, storage medium, and program product, which can quickly and accurately perform autofocus and improve the quality of captured images. The technical solution is as follows:

[0006] In a first aspect, an autofocus method is provided, which is applied to an electronic device and includes:

[0007] At the first moment, the RGB image sensor and the depth image sensor are started;

[0008] At the second moment, the RGB image sensor performs exposure processing on the RGB data of the Nth frame, and the depth image sensor processes the depth data corresponding to the Nth frame;

[0009] At a third moment, the RGB image sensor outputs an RGB image of the Nth frame;

[0010] At a fourth moment, in response to all RGB images of the Nth frame being output, a control instruction is sent to a camera motor;

[0011] At a fifth moment, the camera motor is controlled to operate based on the control instruction to complete autofocus on the N+1th frame;

[0012] At the sixth moment, the RGB image sensor performs exposure processing on the RGB data of the N+1th frame, and the depth image sensor processes the depth data corresponding to the N+1th frame.

[0013] In a first possible implementation manner of the first aspect, at the third moment, the RGB image sensor outputting the RGB image of the Nth frame includes:

[0014] From the third moment on, the RGB image sensor performs row-by-row statistics on the RGB image of the Nth frame obtained after the exposure process, corrects the quality of the RGB image of the Nth frame after the statistics, and outputs the RGB image of the Nth frame.

[0015] In a first possible implementation manner of the first aspect, the method further includes:

[0016] From the third moment on, the depth image sensor performs statistics on the depth data corresponding to the Nth frame to obtain a histogram corresponding to the Nth frame, and the histogram is used to generate the control instruction.

[0017] In a third possible implementation of the first aspect, the hardware abstraction layer of the electronic device includes a camera control module, and the sending of a control instruction to a camera motor at the fourth moment in response to all RGB images of the Nth frame being output includes:

[0018] In response to the RGB image of the Nth frame being fully output, the camera control module acquires a first distance of the camera lens and an exposure time of the N+1th frame, where the first distance is a movement distance of the camera lens when automatically focusing on the N+1th frame, predicted based on the focus frame of the Nth frame;

[0019] The camera control module determines a second distance of the camera lens based on the first distance and the exposure time of the N+1th frame;

[0020] The camera control module sends the control instruction to the camera motor, where the control instruction includes the second distance.

[0021] In a fourth possible implementation manner of the first aspect, before the camera control module acquires the first distance of the camera lens, the method further includes:

[0022] In response to all focus frames of the Nth frame being output, the RGB image sensor sends a first interrupt message to the camera control module;

[0023] In response to the first interrupt message, the camera control module acquires a target RGB image of the Nth frame and a depth image corresponding to the Nth frame, wherein the target RGB image is an RGB image including the focus frame;

[0024] The camera control module determines the detection frame of the Nth frame and the prediction frame of the N+1th frame based on the target RGB image and the depth image;

[0025] The camera control module determines the first distance based on the detection frame of the Nth frame, the prediction frame of the N+1th frame, and the depth image.

[0026] In a fifth possible implementation of the first aspect, the camera control module includes a first pipeline, a second pipeline, and a client, the first pipeline is used to control the RGB image sensor, the second pipeline is used to control the depth image sensor, and the client is used to implement communication between the first pipeline and the second pipeline. In response to the first interrupt message, the camera control module obtains the depth image corresponding to the Nth frame, including:

[0027] In response to the first interrupt message, the first pipeline sends an acquisition request for the depth image to the client;

[0028] The client sends the acquisition request to the second pipeline;

[0029] In response to the acquisition request, the second pipeline acquires a histogram corresponding to the Nth frame from the depth image sensor, processes the histogram to obtain the depth image, and sends the depth image to the client;

[0030] The client sends the depth image to the first pipeline.

[0031] In a sixth possible implementation of the first aspect, the framework layer of the electronic device includes a camera service module, the driver layer of the electronic device includes a sensor driver, and at the first moment, before starting the RGB image sensor and the depth image sensor, the method further includes:

[0032] After the camera application is started, the camera service module sends a camera opening request to the camera control module;

[0033] In response to the camera open request, the camera control module creates the first pipeline, the client, and the second pipeline, configures image output resources for the RGB image sensor and the depth image sensor, and then controls the sensor driver to start the RGB image sensor and the depth image sensor.

[0034] In a seventh possible implementation of the first aspect, in response to all RGB images of the Nth frame being output, the camera control module acquiring the first distance of the camera lens and the exposure time of the (N+1)th frame includes:

[0035] In response to all RGB images of the Nth frame being output, the RGB image sensor sends a second interrupt message to the camera control module;

[0036] In response to the second interrupt message, the camera control module obtains the first distance and the exposure time.

[0037] In an eighth possible implementation of the first aspect, the camera control module determines the second distance of the camera lens based on the first distance and the exposure time of the (N+1)th frame, including:

[0038] The camera control module determines, based on the first distance, a driving time of the camera motor when the camera lens moves the first distance;

[0039] When the pushing time does not intrude into the exposure time, the camera control module uses the first distance as the second distance;

[0040] When the pushing time intrudes into the exposure time, the camera control module modifies the first distance and uses the modified first distance as the second distance.

[0041] In a second aspect, an electronic device is provided, comprising a processor and a memory; the memory stores at least one program code; the at least one program code is used to be called and executed by the processor to implement the autofocus method described in the first aspect.

[0042] In a third aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and when the at least one computer program is executed by a processor, the autofocus method described in the first aspect can be implemented.

[0043] In a fourth aspect, a computer program product is provided, comprising a computer program, and when the computer program is executed by a processor, the automatic focusing method described in the first aspect can be implemented.

[0044] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0045] The electronic device of the embodiment of the present application is equipped with an RGB image sensor and a depth image sensor. The RGB image sensor can output an RGB image of an object, and the depth image sensor can output a depth image of an object. When the electronic device is used to shoot the Nth frame, a control instruction for automatically focusing the N+1th frame can be generated in advance based on the RGB image and the corresponding depth image of the Nth frame. After the shooting of the Nth frame is completed and before the shooting of the N+1 frame, the camera motor is controlled based on the control instruction to achieve automatic focusing of the N+1th frame. In this way, when shooting the N+1th frame, since the focus has been completed, there is no need to perform the focus operation again, so the focusing speed is faster, and the control instruction of the N+1th frame is generated based on the depth image corresponding to the Nth frame, which can predict the position of the object in advance, so the focusing result is accurate. Furthermore, the image obtained by shooting the N+1th frame based on the focal length after focusing is clearer and has better image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 1 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application;

[0048] Figure 2 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application;

[0049] Figure 3 is a flow chart of an auto-focusing method according to an embodiment of the present application;

[0050] Figure 4 is a timing diagram of an autofocus process according to an embodiment of the present application;

[0051] Figure 5 is a flow chart of an auto-focusing method provided in an embodiment of the present application;

[0052] Figure 6 This is a logic block diagram of an auto-focus provided in an embodiment of the present application;

[0053] Figure 7 This is a block diagram of a timing for sending a code corresponding to the N+1th frame provided by an embodiment of the present application;

[0054] Figure 8 This is a flowchart of an autofocus process according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0056] It should be understood that the terms "each," "plurality," and "any" used in the embodiments of this application include two or more, "each" refers to each of the corresponding plurality, and "any" refers to any one of the corresponding plurality. For example, if a plurality of words includes 10 words, "each" refers to each of the 10 words, and "any" refers to any one of the 10 words.

[0057] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0058] To improve the imaging quality of moving objects, an embodiment of the present application provides an autofocus method, which is applied to an electronic device that is equipped with not only an RGB image sensor but also a depth image sensor, such as a dTOF (direct Time-Of-Flight) sensor. When the electronic device is used to capture an image, the RGB image sensor of the electronic device outputs an RGB image of the current frame, and at the same time, the depth image sensor outputs a depth image corresponding to the current frame. Based on the focus frame and depth image on the RGB image, the detection frame of the current frame and the predicted frame of the next frame can be determined. Furthermore, based on the detection frame of the current frame, the predicted frame of the next frame, and the depth image corresponding to the current frame, the distance that the camera lens moves when autofocusing on the next frame can be predicted. Based on this distance, a control instruction for the camera motor can be generated, and then the control instruction is sent to the camera motor. The camera motor operates based on the control instruction to drive the camera lens to move, thereby achieving autofocus on the next frame.

[0059] Figure 1 The figure shows a schematic diagram of the structure of an electronic device 100 used in the present embodiment. The electronic device 100 can be a mobile phone, a personal computer (PC), a tablet computer, an AR (Augmented Reality) device, a VR (Virtual Reality) device, an in-vehicle computer, a wearable device, a smart home device, etc.

[0060] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. Among them, the sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0061] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0063] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0064] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0065] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module interface, and / or a Universal Serial Bus (USB) interface.

[0066] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can be coupled to the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0067] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0068] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0069] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, implementing the function of playing music through a Bluetooth headset.

[0070] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. MIPI interfaces include the Camera Serial Interface (CSI) and the Display Serial Interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0071] The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0072] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, transfer data between the electronic device 100 and peripheral devices, or connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices.

[0073] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0074] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0075] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193 and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0076] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, the baseband processor, and the like.

[0077] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0078] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0079] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0080] The wireless communication module 160 can provide wireless communication solutions including Wireless Local Area Networks (WLAN) (such as Wireless Fidelity (Wi-Fi) network), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate and amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0081] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a Global Positioning System (GPS), a Global Navigation Satellite System (GNASS), a Beidou Navigation Satellite System (BDS), a Quasi-Zenith Satellite System (QZSS) and / or a Satellite Based Augmentation System (SBAS).

[0082] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0083] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini-LED, a micro-LED, a micro-o-LED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0084] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0085] The ISP processes data fed back by camera 193. For example, when the shutter is opened to take a photo, light passes through the lens and is transmitted to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise and brightness. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0086] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0087] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0088] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0089] The NPU is a neural network (NN) computing processor that rapidly processes input information and continuously self-learns by drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0090] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0091] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, for example, at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0092] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0093] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0094] The electronic device 100 is installed with a software system that can run on the processor 110. The software system can be an Android system, a Windows system, an IOS system, a Hongmeng system, etc. The architecture adopted by the software system can be a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device 100. Figure 2 ,The Android system can be divided into four layers, from top to bottom: application layer, framework layer (Framework, FWK), hardware abstraction layer (HAL) and kernel layer.

[0095] The application layer can include a series of application packages. Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message. These applications can be built-in system applications or non-system applications. These applications can have icons and application interfaces, or they can have application interfaces without icons, or they can have neither icons nor application interfaces.

[0096] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2As shown, the application programming interface provided by the framework layer may include interfaces related to camera services, interfaces related to sensor services, and interfaces related to other services. Interfaces related to different services can be defined as different service modules. For example, interfaces related to sensor services can be defined as sensor service modules, interfaces related to camera services can be defined as camera service modules, and interfaces related to other services can be defined as other service modules (not shown in the figure), etc. Among them, the camera service module may include sub-modules such as ICameraSerivce and CameraSerivce. Optionally, the camera service module can be an independent process, and ICameraSerivce and CameraSerivce can be two threads in the process. Of course, the application framework layer may also include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0097] The hardware abstraction layer (HAL) lies between the framework layer and the kernel layer, aiming to abstract the hardware. The HAL hides the platform-specific hardware interface details, providing a virtual hardware platform for the software system, making it hardware-independent and portable across multiple platforms. Depending on the functionality implemented, the HAL can be further refined, such as into a camera-specific HAL. This camera-specific HAL can be defined as a camera control module. Alternatively, the camera control module can be a standalone process. The camera control module supports the Camx-CHI architecture and includes multiple instances, such as Usecase:MultiCamera and NISUsecase. Each instance can include multiple features, each of which can create multiple pipelines, and each pipeline can include multiple nodes. For example, the pipeline created by the Usecase:MultiCamera feature includes an RGB SensorNode, an IFE Node, an IPE Node, a SIT Node, and an AF Node; the pipeline created by the NISUsecase feature includes a dTOF SensorNode, an IFElite Node, and a Depth Node. The Usecase is responsible for business processing and resource management within the CHI. A Feature represents a specific function. A Usecase selects the corresponding Feature and then associates it with a set of Pipelines. When the upper layer sends a request, the Hardware Abstraction Layer selects the corresponding Feature based on the request. A Pipeline is a collection of all resources that provide a single, specific function. It manages all hardware resources and data flow, and is responsible for maintaining the software and hardware resources and processing business logic for the entire Pipeline. A Node is an abstract unit with independent processing capabilities, which can be either software or hardware.

[0098] The kernel layer is the layer between hardware and software. The kernel layer may include display drivers, camera drivers, ISP drivers, sensor drivers, etc., through which corresponding functions can be realized.

[0099] The present invention provides an automatic focusing method. Figure 2 The electronic device 100 of the software system shown in FIG. 1 executes the embodiment of the present application as an example, see Figure 3 , the method process provided in the embodiment of the present application includes:

[0100] 301. At a first moment, start an RGB image sensor and a depth image sensor.

[0101] The RGB image sensor converts light signals into electrical signals, thereby generating a digital image, also known as an RGB image. To better implement the camera function, the system also provides ISP components, including the CSID component, SFE component, IFE component, and IPE component. The CSID component receives the RAW image output by the RGB image sensor so that the RAW image can be input to the ISP component. The SFE component performs some format processing on the RAW image, such as remosaic processing. The IFE component can perform statistical processing on the image to obtain the PD value of the current frame, the brightness value of the next frame, and so on. The IPE component can correct the image quality.

[0102] The depth image sensor, which can measure the distance between an object and the camera lens by emitting and receiving light signals, can be a dTOF sensor, for example. The depth image sensor can include multiple functional components, such as a CSID component and an IFE Lite component. The IFE Lite component transparently transmits the histogram output by the depth image sensor.

[0103] After the camera application is activated, the electronic device initializes the camera. During the initialization process, image output resources such as the RGB image sensor and the depth image sensor are configured. Then, at a first moment, the RGB image sensor and the depth image sensor are activated to enable the electronic device's photo-taking function. This first moment is the start of the photo-taking operation, which occurs after the allocation moment.

[0104] 302. At the second moment, the RGB image sensor performs exposure processing on the RGB data of the Nth frame, and the depth image sensor processes the depth data corresponding to the Nth frame.

[0105] Wherein, N is a natural number, and the value of N is greater than or equal to 1. In an embodiment of the present application, the Nth frame represents the current frame, which is the image being captured; the N+1th frame represents the next frame of the current frame, which is the image not captured. When a user touch operation on the camera shutter control is detected, the electronic device generates a shooting instruction. In response to the shooting instruction, the RGB image sensor performs exposure processing on the RGB data of the Nth frame to convert the RGB data from a light signal to an electrical signal, thereby generating an image that can be displayed on the display screen of the electronic device. In response to the shooting instruction, the depth image sensor processes the depth data corresponding to the Nth frame to measure the distance between the object in the shooting scene and the camera lens. The second moment is the exposure start moment, and the second moment is after the first moment. Since the photosensitive element in the RGB image sensor requires a period of reaction time when the exposure starts, the RGB image sensor will not output an image at the start of the exposure, that is, the second moment.

[0106] 303. At a third moment, the RGB image sensor outputs an RGB image of the Nth frame.

[0107] The third moment is the moment when the RGB image sensor starts to output the RGB image of the Nth frame, that is, the moment corresponding to the SOF (Start of Frame) of the RGB image of the Nth frame. The third moment is after the second moment.

[0108] 304. At a fourth moment, in response to the output of all RGB images of the Nth frame, a control instruction is sent to the camera motor.

[0109] The fourth moment is the moment when the entire RGB image of the Nth frame is output, that is, the moment corresponding to the EOF (End of Frame) of the Nth RGB image. The fourth moment is located after the third moment. In response to the complete output of the Nth RGB image, the electronic device may send a control instruction to the camera motor, including, for example, a distance for the camera lens to move.

[0110] 305. At the fifth moment, the camera motor is controlled to operate based on the control instruction to complete the autofocus of the N+1th frame.

[0111] The fifth moment is when the camera motor starts operating, and this fifth moment is after the fourth moment. Upon receiving the control command, the camera motor starts operating in response to the control command, driving the camera lens to move, thereby adjusting the distance between the camera lens and the photosensitive element, and completing autofocus on the N+1th frame.

[0112] 306. At a sixth moment, the RGB image sensor performs exposure processing on the RGB data of the N+1th frame, and the depth image sensor processes the depth data corresponding to the N+1th frame.

[0113] The sixth moment is the exposure moment of the N+1th frame, which is after the fifth moment. At the sixth moment, the autofocus of the N+1th frame has been completed, and the focus is now on the object to be photographed, so the captured image is clearer and has better image quality.

[0114] Figure 4 A timing diagram of an autofocus process is shown in FIG. Figure 4, TI time is the start time of the RGB image sensor and the depth image sensor, T2 time is the time corresponding to the SOE of the Nth frame, T3 time is the time corresponding to the SOF of the Nth frame, T4 time is the time corresponding to the EOF of the Nth frame, T2 time is the time corresponding to the SOE of the N+1th frame, T3 time is the time corresponding to the SOF of the N+1th frame, T4 time is the time corresponding to the EOF of the N+1th frame, and the order from first to last is: T1 <T2<T3<T4<T5<T6。

[0115] The present invention provides an automatic focusing method. Figure 2 The electronic device 100 of the software system shown in FIG. 1 executes the embodiment of the present application as an example, see Figure 5 , the method process provided in the embodiment of the present application includes:

[0116] 501. After the camera application is started, the camera service module sends a camera opening request to the camera control module.

[0117] Among them, the camera application is any camera application in the application layer, which can be a system camera application or a third-party camera application. The camera service module (i.e., CameraService) can be an independent process, which can be defined as a camera service process. The camera service module can include multiple sub-modules, which can be different threads in the camera service process. The camera control module (i.e., Camera HAL) can be an independent process, which can be defined as a camera control process. The sub-modules included in the camera control module (such as CHK-CDK, CAMX, etc.) can be different threads in the camera control process.

[0118] After the camera application is started, it generates a camera open request and sends it to the camera service module in the framework layer. The camera open request may include the type of camera requested to be opened (front or rear), the package name, etc. Upon receiving the camera open request, the camera service module sends it to the camera control module.

[0119] 502. In response to the camera open request, the camera control module creates a first pipeline, a client, and a second pipeline, configures image output resources for the RGB image sensor and the depth image sensor, and then controls the sensor driver to start the RGB image sensor and the depth image sensor.

[0120] After receiving the camera open request, in response to the camera open request, the camera control module selects the target Feature in the first instance based on the distribution information indicated by the camera open request and creates the first pipeline. Typically, the camera open request requires that the camera to be opened be a logical camera, so the first instance can be Usecase:MultiCamera, and the target Feature can be RealTime in Feature2 of Usecase:MultiCamera. The first pipeline (i.e., Pipeline) can be used to manage the RGB image sensor and can also be used to generate control instructions for autofocusing the next frame. The first pipeline can include multiple nodes, such as RGB SensorNode, IFE Node, IPE Node, SIT Node, AFNode, etc. The RGB SensorNode is used to configure the image output resources of the RGB image sensor to trigger the driver layer to start the RGB image sensor. The IFE Node is used to perform statistics and processing on the image to obtain image information, including the PD value of the current frame, the brightness value of the next frame, etc. The IPE Node is used to correct the image quality to obtain a higher-quality image. SITNode is used to calculate the detection frame of the current frame and the prediction frame of the next frame. The SIT Node may include HwCamcfgSvr, a computing sub-node (including computing resources, SOT algorithm), a SIT Adapter, a decision engine, a processing engine, etc. The computing sub-node is used to determine the detection frame of the current frame based on the processing engine, and predict the prediction frame of the next frame based on the decision engine. SITAdapter is used to send the calculated detection frame of the current frame and the prediction frame of the next frame to the AF Node. The AF Node is used to calculate the movement distance of the camera lens when automatically focusing on the next frame, and send control instructions to the camera motor. Each Node can correspond to a device or functional component in the hardware. For example, the RGB SensorNode can correspond to an RGB image sensor, the IFE Node can correspond to an IFE component, and the IPE Node can correspond to an IPE component.

[0121] During the creation of the first pipeline, the camera control module will initialize each Node in the first pipeline in turn. When initializing the AF Node, the AF Node can register a client and then create a second pipeline (i.e., Pipeline) in the second instance through the client. The client is a port used to establish communication between the first pipeline and the second pipeline, and the client is not perceived by the upper layer. The client can be NCS, and the client can include sub-interfaces, such as NCSService and CSSService. When the AF Node creates the second pipeline through the client, it can send a creation request to the client. The creation request may include the stream type (such as depth stream) and resolution to be returned. The client receives the creation request and sends the creation request to the second instance, thereby creating the second pipeline in the second instance. During the initialization of the SITNode, the SIT Node can also be registered on the client to communicate with the second pipeline through the client.

[0122] Among them, the second instance can be NISUsecase. The second pipeline can be used to manage the depth image sensor, and the second pipeline can also be used to generate depth images. The second pipeline includes multiple Nodes, such as dTOFSensorNode, IFE liteNode, Depth Node, etc. The dTOF SensorNode is used to configure the output resources of the depth image sensor to trigger the driver layer to start the depth image sensor. The IFE liteNode is used to pass the histogram of the time of flight output by the depth image sensor to the Depth Node. The Depth Node is used to generate a corresponding depth image based on the histogram of the time of flight. Each Node can correspond to a device or functional component in the hardware. For example, dTOFSensorNode can correspond to a depth image sensor, and IFE liteNode can correspond to an IFE lite component.

[0123] After configuring image output resources for the RGB image sensor and the depth image sensor, the camera service module can control the driver layer to start the RGB image sensor and the depth image sensor to realize the camera streaming.

[0124] It should be noted that when the camera application is closed, the first pipeline, the client and the second pipeline in the camera control module will be deregistered, thereby releasing the allocated resources.

[0125] 503. In response to the shooting instruction for the Nth frame, the RGB image sensor performs exposure processing on the RGB data of the Nth frame, and then outputs the RGB image of the Nth frame.

[0126] When the RGB image sensor and the depth sensor are started, the RGB image sensor can collect RGB image data of the Nth frame, and the depth image sensor can collect depth data corresponding to the Nth frame. The data collection frequencies of the RGB image sensor and the depth image sensor can be the same or different. If the data collection frequencies of the RGB image sensor and the depth image sensor are the same, the RGB image sensor collects one frame of RGB image data and the depth image sensor also collects one frame of depth data, and the one frame of RGB image data corresponds to one frame of depth data. If the data collection frequencies of the RGB image sensor and the depth image sensor are different, for example, the data collection frequency of the depth image sensor is twice the data collection frequency of the RGB image sensor, the RGB image sensor collects one frame of RGB image data and the depth image sensor collects two frames of depth data, and the one frame of RGB image data corresponds to two frames of depth data. To better represent the correspondence between the images collected by the RGB image sensor and the depth image sensor, the embodiment of the present application represents the depth data collected by the depth image sensor as depth data corresponding to each frame. The depth data corresponding to each frame can generate one frame of depth image or multiple frames of depth image.

[0127] When a touch operation on the shutter control in the camera application is detected, the camera application can generate a shooting instruction for the Nth frame and send the shooting instruction to the camera service module in the framework layer, which is then sent to the camera control module in the hardware abstraction layer. After receiving the shooting instruction, the first pipeline in the camera service module obtains the shooting request of each Node and sends the shooting instruction containing the shooting request of each Node to the RGB image sensor. When receiving the shooting instruction, the photosensitive element in the RGB image sensor begins to react, thereby exposing the RGB data of the Nth frame. The RGB image sensor can expose the RGB data of the Nth frame line by line, so that as the exposure time increases, an RGB image is gradually generated.

[0128] Furthermore, after each row of RGB data is exposed and processed, the RGB image sensor can sequentially pass the resulting RGB image to various functional components. For example, the CSID component receives the RAW image output by the RGB image sensor, then inputs this RAW image into the SFE component for some formatting processing, then into the IFE component for statistics, and finally into the IPE component for image quality correction. Because RGB images are generated by exposure row by row, post-processing using the RGB image sensor's functional components can also be performed row by row.

[0129] 504 . In response to the shooting instruction for the Nth frame, the depth image sensor performs statistics on the depth data corresponding to the Nth frame to obtain a histogram corresponding to the Nth frame.

[0130] In response to the shooting instruction for frame N, the second pipeline in the camera service module can also obtain the shooting request of each Node, and send the shooting instruction containing the shooting request of each Node to the depth image sensor. After receiving the shooting instruction, the depth image sensor performs statistics on the depth data corresponding to the Nth frame to obtain the histogram corresponding to the Nth frame. If the depth data corresponding to the Nth frame can generate a frame of depth image, the depth data is counted to generate the histogram corresponding to the Nth frame; if the depth data corresponding to the Nth frame can generate multiple frames of depth image, the moving speed of the object can be estimated based on the depth data of multiple frames, and then the depth data of the last frame is counted to generate the histogram corresponding to the Nth frame. Since the depth image sensor does not need to perform exposure processing on the depth data, the processing speed of the depth image sensor for the depth data is shorter than the processing time of the RGB image sensor for RGB data, and the histogram corresponding to the Nth frame can be generated faster.

[0131] 505. In response to all focus frames of the Nth frame being output, the RGB image sensor sends a first interrupt message to the camera control module.

[0132] During the process of the RGB image sensor outputting the RGB image of the Nth frame line by line, when the IFE component of the RGB image sensor detects that the focus frame of the Nth frame is completely output, the IFE component will generate a first interrupt message, which may include the RGB image output by the IFE component when the first interrupt message is generated, and the RGB image includes the complete focus frame of the Nth frame, and then send the first interrupt message to the camera control module.

[0133] It should be noted that the first interrupt message and the second interrupt message in the subsequent steps are used to trigger the scheduling of the AF Node in the first pipeline. When the AF Node is scheduled for the first time, the distance the camera lens moves when automatically focusing on the N+1 frame can be calculated. When the AF Node is scheduled for the second time, the distance the camera lens moves calculated in the first scheduling of the AF Node can be safety checked. The two interrupt messages do not interrupt the processing of the RGB image by the RGB image sensor.

[0134] 506 . In response to the first interrupt message, the camera control module obtains the target RGB image of the Nth frame and the depth image corresponding to the Nth frame.

[0135] The IFE Node in the first pipeline of the camera control module receives the first interrupt message, triggering the first scheduling of the AF Node. Specifically, the IFE Node in the first pipeline can send the RGB image including the focus frame of the Nth frame to the IPE Node. The IPE Node corrects the quality of the RGB image including the focus frame of the Nth frame to obtain the target RGB image of the Nth frame. The target RGB image is the RGB image output by the image sensor when the first interrupt message is generated (because the operation of the RGB image sensor is not interrupted, the image output by the IPE Node is the same as the image output after processing by the IPE component). The target RGB image includes the complete focus frame. The target RGB image is then sent to the SIT Node. The IFE Node in the first pipeline can also send a scheduling instruction to the AF Node. In response to the scheduling instruction, the AF Node sends a request to obtain a depth image to the client. The client sends the acquisition request to the second pipeline. After receiving the acquisition request, the IFE liteNode of the second pipeline obtains the histogram image corresponding to the Nth frame transmitted by the IFE lite component from the depth image sensor, and inputs the histogram image corresponding to the Nth frame to the Depth Node. The Depth Node processes the histogram image corresponding to the Nth frame to obtain the depth image corresponding to the Nth frame, and sends the depth image corresponding to the Nth frame to the client. The client sends the depth image corresponding to the Nth frame to the AF Node of the first pipeline. In addition, after receiving the target RGB image, the SIT Node will also send a request to obtain the depth image of the Nth frame to the client. Therefore, when the depth image corresponding to the Nth frame is received, the client will also send the depth image corresponding to the Nth frame to the SIT Node. Furthermore, if the depth image sensor calculates information such as the object's motion speed based on the depth data corresponding to the Nth frame, the object's motion speed information may be sent to the client, which then sends it to the AF Node and the SIT Node.

[0136] 507. The camera control module determines a detection frame of the Nth frame and a prediction frame of the N+1th frame based on the target RGB image and the depth image, and determines a first distance based on the detection frame of the Nth frame, the prediction frame of the N+1th frame, and the depth image.

[0137] When the target RGB image of the Nth frame and the depth image corresponding to the Nth frame are obtained, the SIT Node in the first pipeline of the camera control module can use the SOT (Single Object Tracking) algorithm based on the target RGB image and the depth image to select the target object of the N+1th frame. The target object can be the same as or different from the subject of the Nth frame. Based on the target object, the SIT Node can determine a detection frame of the Nth frame that can surround the target object, and based on the target RGB image, the depth image, and the motion speed of the target object, etc., determine the prediction frame of the N+1th frame by performing motion estimation and motion compensation. The prediction frame can surround the target object of the N+1th frame.

[0138] When the detection frame of the Nth frame and the predicted frame of the N+1th frame are obtained, the SIT Node sends the detection frame of the Nth frame and the predicted frame of the N+1th frame to the AF Node. The AF Node determines a first distance based on the detection frame of the Nth frame, the predicted frame of the N+1th frame, and the depth image corresponding to the Nth frame. The first distance is the distance that the camera lens moves when automatically focusing on the N+1th frame, which is predicted based on the focus frame of the Nth frame.

[0139] 508. In response to the RGB image of the Nth frame being completely output, the RGB image sensor sends a second interrupt message to the camera control module.

[0140] During the process of the RGB image sensor outputting the RGB image of the Nth frame line by line, when the IFE component of the RGB image sensor detects that the RGB image of the Nth frame is completely output, the IFE component will generate a second interrupt message and then send the second interrupt message to the camera control module.

[0141] 509. In response to the second interrupt message, the camera control module obtains the first distance and the exposure time of the N+1th frame, determines the second distance of the camera lens based on the first distance and the exposure time of the N+1th frame, and then sends a control instruction to the camera motor.

[0142] The IFE Node in the first pipeline of the camera control module receives the second interrupt message, triggering a second dispatch of the AF Node. Specifically, the AF Node obtains the exposure time of the N+1th frame and determines the second distance of the camera lens based on the first distance and the exposure time of the N+1th frame.

[0143] When the AF Node determines the second distance of the camera lens based on the first distance and the exposure time of the N+1th frame, it specifically includes:

[0144] 5091. The AF Node determines, based on the first distance, a driving time of a camera motor when the camera lens moves the first distance.

[0145] This embodiment of the present application maintains a camera motor push time list, which includes a correspondence between the distance the camera lens moves and the camera motor push time. Based on the push time list and the first distance, the AFNode can obtain the camera motor push time when the camera lens moves the first distance.

[0146] 5092. When the push time does not intrude into the exposure time, the camera control module uses the first distance as the second distance.

[0147] The AF Node compares the camera motor's push time when the camera lens moves the first distance with the exposure time of the N+1th frame. If the push time does not intrude into the exposure time, the AF Node uses the first distance of the camera lens as the second distance. The push time not intruding into the exposure time means that the push time and the intruding exposure time do not intersect. For example, if the push time is [1, 5] and the exposure time is [7, 10], the push time does not intrude into the exposure time.

[0148] 5093. When the push time invades the exposure time, the camera control module modifies the first distance and uses the modified first distance as the second distance.

[0149] This embodiment of the present application maintains a correction list for camera lens movement distances, including camera lens movement distances and corresponding correction values. Based on this correction list and a first distance, the AF node obtains the correction value corresponding to the first distance, subtracts the correction value from the first distance, and uses the modified first distance as the second distance.

[0150] When determining a second distance that the camera lens moves when performing autofocus on the N+1th frame, the AF Node generates a control instruction including the second distance, and then sends the control instruction to the camera motor.

[0151] It should be noted that although the first distance is generated during the first AF Node scheduling in this embodiment, the RGB image of the Nth frame has not yet been fully output. If a control instruction is sent to the camera motor at this time, the imaging effect of the RGB image of the Nth frame will be affected. However, if the control instruction is sent after the RGB image of the Nth frame has been fully output, the imaging effect of the RGB image of the Nth frame will not be affected. In other words, the control instruction is sent after the second AF Node scheduling.

[0152] In addition, since signaling transmission requires a certain amount of time, the camera motor receives the control instruction after a period of time (for example, 1 ms) has passed since the control instruction was sent.

[0153] Figure 6The process of the camera control module generating the code corresponding to the next frame (i.e., the second distance) in the embodiment of the present application is shown, wherein the pre-processing unit can be the functional unit of the SIT Node in the first pipeline, and the processing unit and the output module can be the functional units of the AF Node in the first pipeline. Figure 6 The pre-processing unit determines the detection frame of the Nth frame (corresponding to the PDAF in the figure) and the prediction frame (Depth AF) of the N+1th frame based on the focus frame of the Nth frame, the depth image corresponding to the Nth frame, and the speed information based on the depth. The processing unit can calculate the distance the camera lens moves when autofocusing on the N+1th frame based on the detection frame of the Nth frame and the prediction frame of the N+1th frame. The output unit can calculate the driving time of the camera motor based on the distance the camera lens moves when autofocusing on the N+1th frame, and obtain the exposure time of the N+1th frame. Then, based on the driving time of the camera motor and the exposure time of the N+1th frame, it is determined whether the driving time of the camera motor intrudes into the exposure time of the N+1th frame. If so, the distance the camera lens moves when autofocusing on the N+1th frame is adjusted to obtain the adjusted distance (code), and then a control instruction including the adjusted distance is sent to the camera motor.

[0154] Figure 7 A block diagram showing the timing of sending the code corresponding to the N+1 frame is shown. Figure 7 After the Nth frame is exposed, the RGB image sensor outputs the RGB image of the Nth frame line by line after exposure processing and processing by other functional components. At the time corresponding to the SOF of the Nth frame, it starts to output the RGB image of the Nth frame. When the focus frame of the RGB image of the Nth frame is completely output, the first AF Node scheduling is performed to generate a code. When the RGB image of the Nth frame is completely output, the second AF Node scheduling is performed, the code is adjusted, and then a control instruction containing the adjusted code is sent to the camera motor. After a period of time (for example, 1ms), the camera motor receives the control instruction and operates based on the adjusted code. During the operation, it drives the camera lens to move together, thereby adjusting the distance between the camera lens and the photosensitive element.

[0155] 510. The camera motor controls operation based on the control command to complete autofocus on the N+1th frame.

[0156] When the control command is received, the camera motor operates based on the control command, causing the camera lens to move. Before the SOE of the N+1th frame, the camera motor stops operating, thereby completing the autofocus of the N+1th frame. Using the method provided in the embodiment of the present application, based on the RGB image and the corresponding depth image of the Nth frame, the N+1th frame can be quickly and accurately autofocused in advance after the Nth frame is captured and before the N+1th frame is captured, thereby improving the imaging effect of the N+1th frame image.

[0157] The method provided in the embodiments of the present application can be applied to any image and video shooting scenarios, and has good imaging effects in these scenarios. In particular, the effect is more significant for shooting scenarios of objects moving in opposite directions.

[0158] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0159] Figure 8 A flowchart of an automatic focusing process according to an embodiment of the present application is shown. Figure 8When the camera application in the application layer or a third-party camera application is detected to be launched, the launched camera application generates a camera open request and sends the camera open request to the CameraService in the Framework layer. After receiving the camera open request, the camera open request is sent to the Camera HAL. After receiving the camera open request, the Camera HAL selects the Usecase based on the stream allocation information: Feature2 in MultiCamera. RealTime in Feature2 creates a Pipeline. This Pipeline includes multiple nodes, such as the RGB SensorNode, IFE Node, IPE Node, SIT Node, and AF Node. It then initializes each node in the Pipeline. During the AF Node initialization process, the AF Node can register with the NCS and use the NCS to create a Pipeline in the NISUsecase. This Pipeline includes multiple nodes, such as the dTOF SensorNode, IFE liteNode, and Depth Node. When initializing the SIT Node, the SIT Node can communicate with the Pipeline of the NISUsecase by registering on the NCS. In response to the shooting instruction for the Nth frame, the RGBSensor performs exposure processing on the RGB data of the Nth frame, and then inputs the exposed RGB image into the CSID component, SFE component, IFE component, and IPE component in sequence for processing, and then outputs it. In response to the shooting instruction for the Nth frame, the dTOFSensor processes the depth data corresponding to the Nth frame, generates a histogram based on the time of flight, and inputs the processed histogram into the CSID component and the IFE lite component in sequence, and then outputs it. When it is detected that the focus frame of the Nth frame is fully output, the FE component will generate a first interrupt message and send the first interrupt message to the camera control module. The IFE Node in the Pipeline of the camera control module receives the first interrupt message and sends the RGB image including the focus frame of the Nth frame to the IPE Node. The IPE Node corrects the quality of the RGB image including the focus frame of the Nth frame to obtain the target RGB image of the Nth frame, and then sends the target RGB image to the SIT Node.The IFE Node will also send a scheduling instruction to the AF Node. In response to the scheduling instruction, the AF Node sends a request for obtaining the depth image to the NCS. The NCS sends the acquisition request to the Pipeline created in the NISUsecase. The IFE liteNode of the Pipeline obtains the histogram image corresponding to the Nth frame transmitted by the IFElite component from the depth image sensor, and inputs the histogram image corresponding to the Nth frame to the Depth Node. The Depth Node processes the histogram image corresponding to the Nth frame to obtain the depth image corresponding to the Nth frame, and sends the depth image corresponding to the Nth frame to the NCS. The NCS sends the depth image corresponding to the Nth frame to the AF Node and SIT Node of the Pipeline created in the Usecase: MultiCamera. Based on the target RGB image and depth image, the SIT Node determines the detection frame of the Nth frame and the predicted frame of the N+1th frame, and sends the detection frame of the Nth frame and the predicted frame of the N+1th frame to the AF Node via the SOTAdpater. The AF Node determines the first distance the camera lens should move when autofocusing on the N+1th frame based on the detection frame of the Nth frame, the predicted frame of the N+1th frame, and the depth image. When the IFE component of the RGB image sensor detects that the RGB image of the Nth frame has been fully output, the IFE component generates a second interrupt message and then sends it to the camera control module. The IFE Node in the NISUsecase of the camera control module receives this second interrupt message and schedules the AF Node a second time. The AF Node obtains the exposure time of the N+1th frame and determines the second distance of the camera lens based on the first distance and the exposure time of the N+1th frame. It then generates a control instruction based on the second distance and sends the control instruction to the VCM Driver through the driver layer to control the operation of the camera motor, thereby adjusting the distance between the camera lens and the photosensitive element to complete the autofocus of the N+1th frame.

[0160] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer program, and when the at least one computer program is executed by a processor, the above-mentioned auto-focusing method can be implemented.

[0161] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it can implement the above-mentioned auto-focus method.

[0162] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An automatic focusing method, characterized in that: The method is applied to an electronic device, and includes: At the first moment, the RGB image sensor and the depth image sensor are started; At the second moment, the RGB image sensor performs exposure processing on the RGB data of the Nth frame, and the depth image sensor processes the depth data corresponding to the Nth frame; At a third moment, the RGB image sensor outputs an RGB image of the Nth frame; At a fourth moment, in response to all RGB images of the Nth frame being output, sending a control instruction to a camera motor; At a fifth moment, the camera motor is controlled to operate based on the control instruction to complete autofocus on the N+1th frame; At the sixth moment, the RGB image sensor performs exposure processing on the RGB data of the N+1th frame, and the depth image sensor processes the depth data corresponding to the N+1th frame.

2. The method according to claim 1, characterized in that At the third moment, the RGB image sensor outputs the RGB image of the Nth frame, including: From the third moment on, the RGB image sensor performs row-by-row statistics on the RGB image of the Nth frame obtained after the exposure process, corrects the quality of the RGB image of the Nth frame after the statistics, and outputs the RGB image of the Nth frame row-by-row.

3. The method according to claim 1, characterized in that The method further comprises: From the third moment on, the depth image sensor performs statistics on the depth data corresponding to the Nth frame to obtain a histogram corresponding to the Nth frame, and the histogram is used to generate the control instruction.

4. The method according to claim 1, wherein The hardware abstraction layer of the electronic device includes a camera control module, and the sending of a control instruction to a camera motor in response to all RGB images of the Nth frame being output at the fourth moment includes: In response to the RGB image of the Nth frame being fully output, the camera control module acquires a first distance of the camera lens and an exposure time of the N+1th frame, where the first distance is a movement distance of the camera lens when automatically focusing on the N+1th frame, predicted based on the focus frame of the Nth frame; The camera control module determines a second distance of the camera lens based on the first distance and the exposure time of the N+1th frame; The camera control module sends the control instruction to the camera motor, where the control instruction includes the second distance.

5. The method according to claim 4, characterized in that Before the camera control module obtains the first distance of the camera lens, the method further includes: In response to all focus frames of the Nth frame being output, the RGB image sensor sends a first interrupt message to the camera control module; In response to the first interrupt message, the camera control module acquires a target RGB image of the Nth frame and a depth image corresponding to the Nth frame, where the target RGB image is the RGB image output by the RGB image sensor when the first interrupt message is generated; The camera control module determines the detection frame of the Nth frame and the prediction frame of the N+1th frame based on the target RGB image and the depth image; The camera control module determines the first distance based on the detection frame of the Nth frame, the prediction frame of the N+1th frame, and the depth image.

6. The method according to claim 5, characterized in that The camera control module includes a first pipeline, a second pipeline, and a client, wherein the first pipeline is used to control the RGB image sensor, the second pipeline is used to control the depth image sensor, and the client is used to implement communication between the first pipeline and the second pipeline. In response to the first interrupt message, the camera control module obtains the depth image corresponding to the Nth frame, including: In response to the first interrupt message, the first pipeline sends an acquisition request for the depth image to the client; The client sends the acquisition request to the second pipeline; In response to the acquisition request, the second pipeline acquires a histogram corresponding to the Nth frame from the depth image sensor, processes the histogram to obtain the depth image, and sends the depth image to the client; The client sends the depth image to the first pipeline.

7. The method according to claim 6, characterized in that The framework layer of the electronic device includes a camera service module, the driving layer of the electronic device includes a sensor driver, and at the first moment, the RGB image sensor and the depth image sensor are started, including After the camera application is started, the camera service module sends a camera opening request to the camera control module; In response to the camera open request, the camera control module creates the first pipeline, the client, and the second pipeline, configures image output resources for the RGB image sensor and the depth image sensor, and then controls the sensor driver to start the RGB image sensor and the depth image sensor.

8. The method according to claim 4, characterized in that In response to all RGB images of the Nth frame being output, the camera control module acquires a first distance of the camera lens and an exposure time of the N+1th frame, including: In response to all RGB images of the Nth frame being output, the RGB image sensor sends a second interrupt message to the camera control module; In response to the second interrupt message, the camera control module obtains the first distance and the exposure time.

9. The method according to claim 4, characterized in that The camera control module determines a second distance of the camera lens based on the first distance and the exposure time of the (N+1)th frame, including: The camera control module determines, based on the first distance, a driving time of the camera motor when the camera lens moves the first distance; When the pushing time does not intrude into the exposure time, the camera control module uses the first distance as the second distance; When the pushing time intrudes into the exposure time, the camera control module modifies the first distance and uses the modified first distance as the second distance.

10. An electronic device, characterized in that: The system comprises a processor and a memory; the memory stores at least one program code; the at least one program code is used to be called and executed by the processor to implement the autofocus method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and when the at least one computer program is executed by a processor, it can implement the auto-focusing method according to any one of claims 1 to 9.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program can implement the auto-focus method according to any one of claims 1 to 9.

Images