A focusing optimization method for terminal device, electronic device and storage medium

Abstract

The present application provides a focus optimization method for a terminal device, an electronic device and a storage medium, and relates to the field of terminal technology. The method includes: the terminal device first obtains a target image to be optimized; then determines a center frame or a face frame from the target image, and detects a significant object frame from the target image; then performs phase detection on the center frame or the face frame and the significant object frame of the target image to determine the PD value corresponding to the center frame or the face frame and the significant object frame of the target image. After calculating the first saliency confidence of the center frame or the face frame of the target image and the second saliency confidence corresponding to the significant object frame using a saliency algorithm, the PD value of the center frame or the face frame of the target image, the comparison result with the PD value of the significant object frame, the first saliency confidence and the second saliency confidence can be used to more accurately determine the focus position of the terminal device, so as to focus on the subject that the user wants to focus on.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a focus optimization method for a terminal device, an electronic device, and a storage medium. Background Art

[0002] With the continuous development of science and technology, terminal devices represented by mobile phones and tablet computers are increasingly used in people's lives and work, bringing great convenience to people. For example, people can take photos, make video calls, record videos or live broadcast videos through terminal devices.

[0003] However, currently in scenarios such as user recording or live video, in order to ensure the clarity of the focused subject, the face is usually pre-set as the highest priority for focusing. Therefore, the face is used for focusing in any situation in scenarios such as video recording or live video. As a result, in these scenarios, if the user wants to focus on an object in his or her hand or other objects, the terminal will still focus on the face and will not be able to find the subject that the user really wants to focus on, resulting in out-of-focus photos or videos, reducing the user experience. Summary of the invention

[0004] In order to solve the above problems, the present application provides a focus optimization method for a terminal device, an electronic device and a storage medium, the purpose of which is to focus on the subject that the user really wants to focus on, optimize the focus result, so as to avoid out-of-focus photos or videos taken, thereby improving the user experience.

[0005] In the first aspect, the present application provides a focus optimization method for a terminal device, the method comprising: the terminal device first acquires a target image to be optimized; then determines a center frame or a face frame of the target image from the target image, and detects a significant object frame (such as a handheld small object frame) from the target image; then, phase detection is performed on the center frame or the face frame of the target image to determine a phase difference PD value corresponding to the center frame or the face frame of the target image; and phase detection is performed on the significant object frame to determine a PD value corresponding to the significant object frame (such as a handheld small object frame). Furthermore, after calculating a first saliency confidence of the center frame or the face frame of the target image and a second saliency confidence corresponding to the saliency object frame using a saliency algorithm, the PD value of the center frame or the face frame of the target image, the comparison result with the PD value of the saliency object frame, as well as the first saliency confidence and the second saliency confidence can be used to more accurately determine the focus position of the terminal device.

[0006] It can be seen that in the focus optimization method of the above-mentioned terminal device, when the user takes the terminal device to take pictures, make video calls, record videos or live broadcast videos, the terminal device in this embodiment no longer uses a focus window with a fixed position and size and a fixed focus priority for focusing. Instead, after determining the significant object frame based on a preset AI algorithm and / or a high-density laser or other saliency detection method, it determines whether the PD value of the center frame or face frame of the target image and the significant object frame (such as a handheld small object frame) is available according to the saliency confidence of the center frame or face frame of the target image. This can more accurately determine the subject that the user really wants to focus on, dynamically optimize the focus area, accurately identify the user's intention, and accurately focus on the area of ​​interest, thereby improving the user experience.

[0007] In a possible implementation, the face frame may be determined by detecting the face frame from the target image using a face recognition algorithm.

[0008] In one possible implementation, detecting a significant object frame from a target image may include: detecting a significant object frame from the target image using a high-density laser, and / or detecting a significant object frame from the target image using a preset AI algorithm to improve the accuracy of the detection result.

[0009] In a possible implementation, using a high-density laser to detect a significant object frame from a target image may include: using a high-density laser to obtain high-density laser data corresponding to the target image, and filtering out depth data with a confidence level lower than a preset threshold from the high-density laser data to obtain filtered depth data; clustering the filtered depth data, and calculating the average depth value of the clustered depth data to form depth map information; performing saliency detection on the downsampled image and depth map information of the target image to obtain a significant object frame. This can help eliminate interference from the depth image information, thereby improving the detection accuracy of the significant object frame.

[0010] In a possible implementation, performing saliency detection on the downsampled image and depth map information of the target image to obtain the salient object frame may include: performing foreground segmentation on the downsampled image and depth map information of the target image to obtain a foreground segmentation result; performing saliency detection on the red, green and blue (RGB) three color channels of the downsampled image and foreground segmentation result of the target image to obtain a salient object frame corresponding to the salient subject as a detection result. This can help improve the detection accuracy of the depth image information.

[0011] In a possible implementation, after detecting the salient object frame from the target image, the salient object frame can also be input into the salient state machine to determine whether the area where the salient object frame is located is in a stable state and within a limited range, so as to ensure that the change of the salient object frame is not too drastic.

[0012] In one possible implementation, phase detection is performed on a significant object frame to determine a PD value corresponding to the significant object frame, which may include: deleting sub-windows in a preset multi-matched window whose confidence level is lower than a preset threshold value based on the coordinate value of the significant object frame, and using the PD values ​​of the remaining sub-windows in the preset multi-matched window to more accurately determine the PD value corresponding to the significant object frame.

[0013] In a possible implementation, the focus position of the terminal device is determined by comparing the PD value of the center frame or face frame of the target image with the PD value of the significant object frame, as well as the first significance confidence and the second significance confidence. This may include: when it is determined through comparison that the PD value of the center frame or face frame of the target image is not lower than the PD value of the significant object frame, and the second significance confidence is not lower than a preset confidence threshold, determining the significant object frame as the focus position of the terminal device; or, when it is determined through comparison that the PD value of the center frame or face frame of the target image is lower than the PD value of the significant object frame, and the first significance confidence is not lower than the preset confidence threshold, determining the center frame or face frame of the target image as the focus position of the terminal device, so as to ensure that the main body of the image is in a focused state, which is convenient for users to view and improves the user experience.

[0014] In a possible implementation, after determining the focus position of the terminal device by comparing the PD value of the center frame or face frame of the target image with the PD value of the salient object frame, as well as the first saliency confidence and the second saliency confidence, the method further includes: obtaining the running state of the salient object, wherein the running state includes a motion state, a stable state, an appearance or disappearance state; and when the salient object is in a stable state and meets the conditions for salient focusing, focusing is performed using the PD value of the salient object frame. Thus, the effect of not triggering prematurely, fast speed (such as being able to take a photo immediately after pressing the photo button), and no missed triggering, false triggering, or multiple triggering can be achieved.

[0015] In a second aspect, the present application provides an electronic device, which includes a memory and a processor, the memory storing a computer program, and the processor being used to call and execute the computer program to implement the focus optimization method of the terminal device described in any one of the first aspects above.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, is used to implement the focus optimization method of a terminal device as described in any one of the first aspects above.

[0017] In a fourth aspect, the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the focus optimization method of a terminal device as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a scenario provided for an embodiment of the present application;

[0019] Figure 2 A schematic diagram of an electronic device provided in an embodiment of the present application;

[0020] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0021] Figure 4 An overall implementation process diagram of focus optimization of a terminal device provided in an embodiment of the present application;

[0022] Figure 5 An example diagram of a center frame or face frame, a notable object frame, and multiple frames of a target image provided in an embodiment of the present application;

[0023] Figure 6 A flowchart of a focus optimization method for a terminal device provided in an embodiment of the present application;

[0024] Figure 7 An example diagram of a scene with a small handheld object as the foreground provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of a multi-window configuration provided in an embodiment of the present application;

[0026] Fig. 9 This is an example diagram of a focus optimization scenario for a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be used as limitations on the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear indication to the contrary in the context.

[0028] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0030] In order to enable those skilled in the art to more clearly understand the solution of the present application, the application scenario of the technical solution of the present application is first explained below.

[0031] See also Figure 1 , which shows several scenario schematic diagrams provided in the embodiments of the present application.

[0032] like Figure 1 As shown in (a), in this example scene, a photo of a user holding a small object "lipstick" is shown. It can be seen that the subject area that the user actually wants to focus on is Figure 1 In (a), the focus is on the lipstick in the background, but the focus is not on the small object held by the user, causing it to be out of focus.

[0033] like Figure 1 As shown in (b), in this example scene, the "sunflower" as the main subject of the picture deviates from the center of the picture. When the depth of field cannot cover it, it will easily cause the main subject to be out of focus.

[0034] like Figure 1 As shown in (c), in this example scene, a user who is selling goods live is holding an "apple" in his hand for a live video introduction. It is understandable that the main area that the user actually wants to focus on is the "apple" in his hand, but the focus is actually on the user's face area, and the small foreground object "apple" in the user's hand is not focused, causing it to be out of focus.

[0035] like Figure 1 As shown in (d), in this example scenario, a user who is selling goods during a live broadcast turns around and moves to display the goods he is wearing. At this time, the focus of the terminal device may be unstable.

[0036] In addition, in other example scenarios, the focus window position and size are fixed, and the face window and center window have different priorities, which may result in an inability to provide accurate focus on the subject in certain scenarios. For example, in live video broadcast scenarios, there are problems such as slow switching of focus targets. In face scene detection scenarios, unstable detection results are also prone to occur. For example, the face detection result of 11100000111, where 1 represents that the face is detected in the frame image, and 0 represents that the face is not detected in the frame image. When the detection value is 1, the focus will be on close distance, and when the detection value is 0, the focus will be on long distance. There will be obvious focus pulling, etc., which will not be elaborated here.

[0037] Therefore, for depth of field scenes, small object scenes, video live broadcasts and other similar scenes shot by cameras, due to the limitations of traditional focus strategies (i.e., the focus window size and position are fixed and cannot be adjusted), the camera may focus on something other than the subject that the user actually wants to focus on, resulting in out-of-focus photos or videos, reducing the user experience.

[0038] In order to overcome the above technical problems, the present application provides a focus optimization method for a terminal device, an electronic device and a storage medium, which can focus on the subject that the user really wants to focus on, achieve a more accurate and stable image focus effect, avoid out-of-focus photos or videos, and thus improve the user experience.

[0039] The focus optimization method of the terminal device provided in the embodiment of the present application can be applicable to electronic devices (i.e., terminal devices) such as mobile phones, tablet computers, personal digital assistants (PDAs), desktops, laptops, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices. And the present application does not limit the specific mode modules and resolution parameters of the terminal device. For example, the terminal device implementing the focus optimization method provided in the embodiment of the present application can be a device that supports a high-density depth device, and supports mode modules such as ordinary photography, ordinary video recording, and main photography, as well as resolutions, frame rates, and magnifications such as 720p, 1080p, 4K, 30FPS, 60FPS, and 1x.

[0040] In order to enable those skilled in the art to more clearly understand the focus optimization method of the terminal device provided in the present application, the hardware architecture and software system architecture of the electronic device are first introduced in detail below.

[0041] See also Figure 2 , which shows a schematic diagram of an electronic device provided in an embodiment of the present application.

[0042] like Figure 2 As shown, the electronic device 200 may include a processor 210, an internal memory 220, a gyroscope sensor 230, a button 240, a motor 250, a touch sensor 260, a camera 270, a display screen 280, an audio module 290, a speaker 290A, a receiver 290B, a microphone 290C, and a headphone jack 290D.

[0043] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0044] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0045] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0046] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0047] The internal memory 220 can be used to store computer executable program codes, which include instructions. The internal memory 220 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound collection function, an image shooting function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, image data, etc.), etc. In addition, the internal memory 220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 210 executes various functional applications (such as face image recognition, smart code recognition, breath awakening, etc.) and data processing of the electronic device 200 by running instructions stored in the internal memory 220 and / or instructions stored in a memory provided in the processor.

[0048] The gyro sensor 230 can be used to determine the motion posture of the electronic device 200. In some embodiments, the angular velocity of the electronic device 200 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 230. The gyro sensor 230 can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyro sensor 230 detects the angle of the shaking of the electronic device 200, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 200 through reverse movement to achieve anti-shake. The gyro sensor 230 can also be used for navigation and somatosensory game scenes. The present application can determine the focus state of the electronic device 200 by the operating state of the electronic device 200 fed back by the gyro sensor 230.

[0049] The button 240 includes a power button, a volume button, etc. The button 240 may be a mechanical button. It may also be a touch button. The electronic device 200 may receive a button input and generate a key signal input related to the user settings and function control of the electronic device 200. For example, if a user triggers a touch button at any position of an image or video on the screen of the electronic device 200, the display of the corresponding function on the electronic device 200 (such as changing the focus position, etc.) may be realized.

[0050] Motor 250 can generate vibration prompts. Motor 250 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization. For example, after the new focus position is determined using the focus optimization method provided in the present application, the focus optimization effect can be achieved by pushing and pulling the position of the motor 250 to improve the focus result and improve the image or video quality.

[0051] The touch sensor 260 is also called a "touch control device". The touch sensor 260 can be set on the display screen 280. The touch sensor 260 and the display screen 280 form a touch screen, also called a "touch control screen". The touch sensor 260 is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 280. In other embodiments, the touch sensor 260 can also be set on the surface of the electronic device 200, which is different from the position of the display screen 280.

[0052] The display screen 280 is used to display images, videos, etc. For example, when a user takes the electronic device 200 to take pictures, make video calls, record videos, or live broadcast videos, it can be used to display the images or videos taken, recorded, or broadcast by the electronic device 200. The display screen 280 may include a display panel (the specific composition is not limited), and the number of display screens 280 included in the electronic device 200 is not limited. The electronic device 200 implements the display function through a GPU, a display screen 280, and an application processor. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0053] The camera 270 is used to capture still images or videos. For example, when a user takes the electronic device 200 to take photos, make video calls, record videos, or live stream videos, it can be used to capture images or video streams containing the user's facial information. The number of cameras 270 included in the electronic device 200 is not limited.

[0054] The electronic device 200 can implement audio functions such as music playback, recording, and voice input and output through the audio module 290, the speaker 290A, the receiver 290B, the microphone 290C, the headphone jack 290D, and the application processor.

[0055] The audio module 290 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 290 can also be used to encode and decode audio signals. In some embodiments, the audio module 290 can be arranged in the processor 210, or some functional modules of the audio module 290 can be arranged in the processor 210.

[0056] The speaker 290A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 200 can listen to music or listen to a hands-free call through the speaker 290A.

[0057] The receiver 290B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 200 receives a call or voice message, the voice can be received by placing the receiver 290B close to the human ear.

[0058] Microphone 290C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 290C to input the sound signal into the microphone 290C. The electronic device 200 can be provided with at least one microphone 290C. In other embodiments, the electronic device 200 can be provided with two microphones 290C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 200 can also be provided with three, four or more microphones 290C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0059] The earphone interface 290D is used to connect a wired earphone, and the standard attributes of the interface are not limited.

[0060] It should be understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is merely a schematic illustration and does not constitute a structural limitation on the electronic device 200 .

[0061] In addition, on top of the above components, the electronic device 200 runs an operating system, such as an iOS operating system, an Android operating system, a Windows operating system, etc. Application programs can be installed and run on the operating system.

[0062] See also Figure 3 , which shows a schematic diagram of the software structure of the electronic device provided in an embodiment of the present application.

[0063] The software system of the electronic device 200 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 200.

[0064] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer (APK), the application framework layer (Framework), the hardware abstraction layer (HAL), and the driver layer.

[0065] The application layer can include a series of application packages (APP). Figure 3 As shown, the application package may include applications such as camera, call, music, navigation, WLAN, Bluetooth, map, etc. When the user takes the electronic device 200 to take pictures, make video calls, record videos, or live stream videos, the camera application may communicate with the framework layer to request camera functions and obtain image data.

[0066] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 3 As shown, the application framework layer may include a window manager, a notification manager, a view system, a phone manager, a resource manager, a camera service layer (CameraService), etc. The application framework layer may be used to implement the interaction between the camera service and the camera API. That is, it provides a unified interface so that different camera hardware can interact with different camera applications. The framework layer is also used to handle many common aspects of camera functions, such as autofocus, exposure control, etc.

[0067] In this embodiment, the hardware abstraction layer (HAL) can be regarded as a layer of abstraction provided by the camera hardware manufacturer, which connects the framework layer with the actual hardware device. HAL provides a set of standard interfaces so that the framework layer can communicate with camera hardware from various manufacturers without knowing the underlying hardware details. The hardware abstraction layer (HAL) stores a focus optimization module for implementing focus optimization, etc. The focus optimization algorithm may include but is not limited to sensors, image front end (IFE), image processing engine (IPE), image detection algorithm (face detection, saliency detection algorithm), focus algorithm, motor (VCM), etc.

[0068] In this way, the interoperability of different camera applications and hardware devices on the Android platform can be achieved through the application layer, framework layer, and HAL layer, thereby achieving the purpose of focusing optimization of terminal devices.

[0069] Next, the main data interactions among the application layer, framework layer, and HAL layer in the process of implementing focus optimization in this application will be introduced:

[0070] (1) Camera API calls: The camera application communicates with the Framework layer through Android's camera API. These API calls are used to request camera functions, set parameters (such as exposure, focus, white balance, etc.), and capture image data.

[0071] (2) Callback mechanism: The Framework layer provides callback interfaces to the camera application. These callbacks are used to notify the camera application about changes in camera status, completion of image capture, and other events. The camera application can register these callbacks to handle these events.

[0072] (3) Camera parameter passing: The camera application can pass settings to the Framework layer through parameter objects. For example, the application can create a parameter object containing settings such as exposure time and ISO, and pass it to the Framework layer to apply these settings.

[0073] (4) Image data transfer: When the camera application initiates an image capture request, the Framework layer is responsible for calling the HAL layer to obtain the actual image data from the camera hardware. This image data can be passed to the camera application through callbacks or other means.

[0074] In addition, the Framework layer is also used to handle possible errors, such as hardware failure, inability to open the camera, etc. These error messages are usually passed to the camera application through callback or exception mechanisms. In short, the Framework layer can be regarded as an intermediary between the camera application and the hardware, providing a standard interface and some basic data transmission mechanisms so that they can interact effectively. This structure enables different camera applications to communicate with a variety of different hardware devices without having to understand the specific implementation of the underlying hardware.

[0075] Next, the various main information and parameters involved in the data interaction between the application layer, framework layer, and HAL layer in the process of implementing focus optimization in this application will be introduced:

[0076] (1) Camera parameters: The Framework layer passes parameters related to camera settings to the HAL layer, such as exposure time, ISO, focus mode, white balance, etc. These parameters are set by the camera application through the camera API, and the Framework layer is used to pass them to the underlying hardware.

[0077] (2) Image buffer information: When the camera application initiates an image capture request, the Framework layer passes information about the image buffer to the HAL layer, such as the image resolution and format. This information helps the HAL layer correctly configure the hardware to generate image data that meets the request.

[0078] (3) Image capture request: The Framework layer sends an image capture request to the HAL layer, which contains information about the capture operation, such as whether to perform continuous autofocus and the number of images captured.

[0079] (4) Callback interface: The Framework layer provides a callback interface that enables the HAL layer to notify the Framework layer of events such as camera status changes and image capture completion. In this way, the Framework layer can promptly learn the camera status and take appropriate actions.

[0080] In addition, the HAL layer can pass information about hardware failures, operating errors, etc. to the Framework layer so that the camera application can handle these errors appropriately. The transmission of this data and information ensures effective communication between the Framework layer and the underlying hardware, allowing different hardware devices to work with the camera application in the Android system. The design of this abstract layer makes it unnecessary for the camera application to interact directly with the underlying hardware, but to interact with the Framework layer through a unified interface. The data provided by the HAL layer to the Framework layer are as follows:

[0081] (1) Image data: When the HAL receives an image capture request from the Framework layer, it is used to interact with the actual camera hardware, obtain image data, and pass it back to the Framework layer. This may involve reading raw image data from the camera sensor or performing some image processing before passing it on.

[0082] (2) Camera status information: HAL may provide the Framework layer with information about the camera status, such as whether it is ready to capture an image, whether a focus change has occurred, etc. This helps the Framework layer to understand the camera status in a timely manner.

[0083] (3) Callback interface: The HAL layer notifies the Framework layer of asynchronous events such as image capture completion, focus completion, etc. through the callback interface. In this way, the Framework layer can take corresponding measures when necessary.

[0084] (4) Hardware parameters: The HAL layer may provide the Framework layer with information about hardware capabilities and features, such as supported exposure range, supported focus modes, supported image formats, etc. This helps the Framework layer adapt to the characteristics of different hardware devices.

[0085] And, if an error occurs at the hardware level, the HAL layer will pass the error information to the Framework layer so that the camera application can handle these errors appropriately. This can include but is not limited to hardware failure, inability to perform specific operations, etc. In short, the HAL layer ensures that the Framework layer can communicate with different hardware devices without having to care about the specific implementation details of the underlying hardware. By providing this data and interface, the HAL layer promotes the collaborative work between the Framework layer and various different hardware devices.

[0086] The driver layer is the layer between hardware and software. The driver layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0087] The technical solutions involved in the following embodiments can all be implemented in electronic devices having the above-mentioned hardware architecture and software architecture.

[0088] Next, the overall implementation process of the focus optimization of the terminal device provided by this application will be introduced:

[0089] like Figure 4 As shown, when implementing the focus optimization of the terminal device, firstly, multiple focus areas in the image to be optimized are determined, including but not limited to the following Figure 5 The default focus area (i.e., the center frame or face frame of the target image), the focus area and the salient object frame (the area in the image that is most interesting to the user and best represents the image content) shown in the figure. For these focus areas, on the one hand, they will be passed to the algorithm library (including but not limited to the saliency detection algorithm, PD algorithm, etc.), and then the algorithm library will pass the detection results and PD value (the distance and change between pixels to determine the focus offset (i.e., the phase difference PD value) calculation results to the focus (Automatic Focus, AF) algorithm at the back end of the device; on the other hand, these focus areas will also be passed to the saliency algorithm to calculate the corresponding saliency confidence level, which is used to evaluate the saliency of each focus area, and the obtained saliency confidence level will be passed to the focus algorithm at the back end of the device. In order to use the back-end algorithm, combined with the detection results, PD value calculation results and saliency confidence level corresponding to each focus area, the optimized quasi-focus position can be finally confirmed.

[0090] Next, the specific implementation process of the focus optimization method of the terminal device provided by this application will be described in detail:

[0091] like Figure 6 As shown, the specific implementation process of the focus optimization method of the terminal device may include the following steps S601-S605:

[0092] S601: Acquire a target image to be optimized.

[0093] In this embodiment, when a user uses a mobile phone or other terminal device to take photos, make video calls, record videos, or live stream videos, the terminal device can immediately start the camera APP and start the camera in the background (such as a camera pre-loaded on the terminal device) to capture images around the terminal device as the target image to be optimized for executing subsequent step S602.

[0094] It should be noted that the present embodiment does not limit the type of the target image. For example, the target image may be a color image composed of three primary colors of red (R), green (G), and blue (B), or a grayscale image, etc. It should be noted that the subsequent embodiments will be introduced with the target image being an RGB image.

[0095] S602: Determine a center frame or a face frame from the target image, and detect a salient object frame from the target image.

[0096] After the terminal device acquires the target image to be optimized by using the camera in step S601, it can further directly determine the center frame of the target image, or use an existing or future face recognition algorithm to perform image recognition processing on the target image to detect the face frame from the target image. The target image is detected by using an existing or future image detection method to detect a significant object frame from the target image to perform the subsequent step S603.

[0097] It should be noted that the embodiments of the present application do not limit the method for recognizing a user's facial image, nor do they limit the specific recognition process. An appropriate recognition method may be selected and corresponding recognition operations may be performed according to actual conditions.

[0098] It should be noted that the present application does not limit the detection method of the significant object frame, nor does it limit the specific detection process. The appropriate detection method can be selected according to the actual situation, and the corresponding detection operation can be performed. For example, the target image can be directly downsampled, and then RGB saliency detection can be performed to obtain the detection result. However, the detection result will contain significant subjects and interference. Therefore, in order to improve the detection accuracy of the significant object frame, an optional implementation method is to use a high-density laser to detect the significant object frame from the target image, and / or use a preset AI algorithm to detect the significant object frame from the target image.

[0099] In this implementation, the reason why this application uses high-density laser to detect significant object frames from the target image is that the applicant found that in addition to the problem of the detection effect of the saliency network itself, various interferences also have a great impact on the saliency detection results. For this reason, this application uses depth data information to filter interference to avoid erroneous saliency output and improve the detection effect. Specifically, first, the laser device on the terminal device can be used to obtain the corresponding high-density laser data, that is, the high-density laser data corresponding to the target image is obtained by using a high-density laser, and the depth data with a confidence lower than a preset threshold (the specific value is not limited, and can be set according to the actual situation and the empirical value, such as 0.7, etc.) is filtered out from the high-density laser data to obtain the filtered depth data. Then, the existing or future data clustering algorithm (such as the k-means clustering algorithm) can be used to cluster the filtered depth data, and the average value of the depth value of the clustered depth data (including the depth data of the near cluster and the far cluster) is calculated to form the depth map information. Next, saliency detection can be performed on the downsampled image and depth map information of the target image to obtain a salient object frame.

[0100] Specifically, an optional implementation method is that after obtaining the depth map information, the downsampled image of the target image and the depth map information can be further segmented to obtain the foreground segmentation result (i.e., the foreground object in the target image is identified). Then, the saliency detection of the three RGB color channels is performed on the downsampled image of the target image and the foreground segmentation result to obtain the salient object frame corresponding to the salient subject as the detection result. In this way, after using the depth map information to eliminate background interference, the saliency detection result tends to estimate the subject target separately rather than the whole. The advantage of this detection method is that it can work when some actual targets are very close (i.e., the depth of field is very small).

[0101] However, the above implementation method uses the depth image information as a hard constraint, and there may be interference problems in the same distance interval in practical applications. Therefore, another optional implementation method is that after obtaining the depth map information, the downsampled image of the target image and the depth map information can be further matched through RGBD alignment. In this way, any RGB coordinate can be addressed in the depth map (Depth map) to obtain the corresponding depth value. At this time, the aligned downsampled image and depth map information are preprocessed (including but not limited to matching and adjusting the image size) to obtain a preprocessed image, and then the preprocessed image is subjected to saliency detection of the four channels of RGBD three colors and depth information to obtain a salient object frame corresponding to the salient subject as the detection result, thereby further solving interference problems such as background and avoiding the limitations of the above implementation method.

[0102] In addition, for terminal devices that cannot emit lasers, such as terminal devices that are not equipped with laser devices, although they cannot use high-density lasers to detect significant object frames from the target image, they can use preset AI algorithms to detect significant object frames from the target image. Among them, the specific content of the preset AI algorithm is not limited and can be selected according to actual conditions and experience. For example, a pre-trained classification model based on a deep residual network (ResNet) can be used to integrate high- and low-level features to achieve saliency detection; or, an end-to-end mode of encoding-decoding (encoder-decoder) can be directly used to directly predict saliency; or, the u2-net model structure can be used to predict saliency, in which the RSU module is similar to the traditional residual, but due to multiple downsampling operations, the computational cost can be reduced.

[0103] It is understandable that for a terminal device that can emit laser, both high-density laser and preset AI algorithm can be used to detect significant object frames from the target image.

[0104] On this basis, after detecting the salient object frame from the target image, the salient object frame can also be input into the salient state machine to determine whether the area where the salient object frame is located is in a stable state and within a limited range (the specific value is not limited and is set according to actual conditions and experience values) to ensure that the change of the salient object frame is not too drastic and that the picture in the salient object frame is within the limited range.

[0105] S603: performing phase detection on the center frame or face frame of the target image to determine the phase difference PD value corresponding to the center frame or face frame of the target image; and performing phase detection on the salient object frame to determine the PD value corresponding to the salient object frame.

[0106] After the terminal device determines the center frame or face frame from the target image and detects the significant object frame through step S602, it can further use the existing or future image phase detection method (i.e., any PD algorithm) to perform phase detection on the center frame or face frame of the target image to determine the phase difference PD value corresponding to the center frame or face frame of the target image; and perform phase detection on the significant object frame to determine the PD value corresponding to the significant object frame, so as to execute the subsequent step S605.

[0107] It should be noted that when detecting the salient object frame, after obtaining the salient image information (saliency map), further processing is required to obtain the focus frame. To this end, existing or future algorithms and search strategies with lower complexity can be used to narrow the focus target (i.e., salient object) to a certain range. For example, a fixed threshold binarization processing function (cv.threshold) can be used to perform saliency The map is processed, and the focus targets that meet the binarization threshold (the specific value is not limited, such as the value can be 127, that is, threshold=127) are not processed, and the focus targets that do not meet the binarization threshold are processed. For example, the positions of various types of handheld hollow small targets in the saliency detection results need to be processed. For example, a multi-window method (other methods such as segmentation can also be adopted, the specific method is not limited) is used to more accurately determine the PD value corresponding to the salient object frame. The specific determination process may include: according to the coordinate value of the salient object frame, deleting the sub-windows with confidence lower than the preset threshold in the preset multi-window (the specific setting method is not limited), and using the PD values ​​of the remaining sub-windows in the preset multi-window to determine the PD value corresponding to the salient object frame.

[0108] In this implementation, Figure 7 In scenes where the foreground is a small handheld object, even if the saliency detection algorithm can accurately detect the area of ​​the small object (such as Figure 7 The salient object is located in the box area marked by the PD algorithm, but it is difficult to give a focus position. In this case, the salient focus method can be used to accurately identify the small handheld object, so that its focus position is more in line with the user's expectations.

[0109] Among them, the schematic diagram of multiple windows can be as follows Figure 8 As shown in FIG. 1 , the calculation area can be customized according to the PD value corresponding to each block output by the system. For example, a back-shaped window can be configured, and the PD value of each sub-window is the average of the PD values ​​of the remaining blocks after screening in the window. The sub-window with similar PD value is selected to determine the PD value corresponding to the salient object box.

[0110] S604: Calculate a first saliency confidence of a center frame or a face frame of the target image and a second saliency confidence corresponding to a salient object frame using a saliency algorithm.

[0111] After the terminal device determines the center frame or the face frame and detects the salient object frame from the target image in step S602, it can further use the existing or future saliency algorithm (such as Figure 4As shown in FIG. 6 , the saliency confidence of the center frame or the face frame of the target image (defined as the first saliency confidence here) and the saliency confidence corresponding to the saliency object frame (defined as the second saliency confidence here) are calculated to execute the subsequent step S605.

[0112] S605: Determine the focus position of the terminal device by comparing the PD value of the center frame or the face frame of the target image with the PD value of the salient object frame, as well as the first saliency confidence and the second saliency confidence.

[0113] After the terminal device determines the PD value corresponding to the center frame or face frame of the target image and the PD value corresponding to the salient object frame through step S603, and calculates the first saliency confidence of the center frame or face frame of the target image and the second saliency confidence corresponding to the salient object frame through step S604, the PD value of the center frame or face frame of the target image can be further compared with the PD value of the salient object frame to obtain a comparison result, and after determining whether the first saliency confidence and the second saliency confidence are higher than a preset confidence threshold, the final focus position of the terminal device is determined.

[0114] Specifically, it should be noted that in actual applications, in order to improve the user experience, it is necessary to keep the main body of the screen in a quasi-focus state, and it can also be smooth and natural when the main body of the screen switches. Therefore, when using the focus algorithm to optimize the significant focus, it is necessary to clarify the entry and exit timing of the significant focus. In addition, based on the existing focus priority Touch (i.e., the user touches to determine the intersection) > Face (face focus) > Track (tracking focus on moving objects) > Default (Center, the default center focus of the screen), when considering the significant focus, it is necessary to determine the priority without affecting the existing state as much as possible. Therefore, the smoothness and naturalness when switching between focus targets includes both changes in the focus area inside the significant focus and the smoothness between the significant focus and other focus methods, such as between the center focus and significant focus of ordinary scenes, between face focus and significant focus, and between significant focus and some features such as tracking focus of all things. It is necessary to ensure that the entry and exit of the significant focus is smooth and natural, without the phenomenon of pulling focus or focus jitter, and the switching timing of the focus target should be accurate. For some special scenes, it is necessary to reconsider the appropriate focus interface. For example, in moon mode, it is necessary to prioritize center (focus on the center of the picture) and focus according to the existing moon-gazing logic.

[0115] Therefore, the architecture of the saliency focus optimization algorithm used in this application is as follows Figure 3As shown, when the terminal device detects a focus request, the integrated side of the device will call the AF algorithm. The algorithm side will first obtain various device information and judge the validity, and then enter the focus trigger judgment, that is, determine whether to start the focus based on the terminal device (such as a mobile phone) and the scene status, and finally perform a specific focus process, that is, adopt different focus strategies according to different scenes.

[0116] The entry conditions of the saliency focus optimization algorithm include but are not limited to four points: one is in photo / video mode, two is in auto focus, three is in center focus scene or small face focus scene, and four is that the saliency object frame deviates from the center point within a certain range. If the above conditions are met, the saliency focus process is carried out to realize the overall logic of the saliency focus optimization algorithm.

[0117] It is understandable that, in order to improve the user experience, it is necessary to keep the main body of the picture in a focused state. Therefore, an optional implementation method is that when it is determined by comparison that the PD value of the center frame or the face frame of the target image is not lower than the PD value of the significant object frame, it can only be determined that the significant object is closer to the front than other objects such as the face, that is, the foreground. At this time, it is also necessary to determine whether the corresponding second significance confidence is not lower than the preset confidence threshold (the specific value is not limited and can be set according to actual conditions and experience values). If the second significance confidence is not lower than the preset confidence threshold, it means that the significant object is not located at a position that is too marginal in the target image. At this time, the significant object frame can be determined as the focus position of the terminal device. On the contrary, if the second significance confidence is lower than the preset confidence threshold, it means that the significant object is located at a position that is too marginal in the target image. Even if it is focused and displayed clearly, it may be easy for the user to ignore it. Therefore, it is not necessary to focus on the significant object at this time, that is, the significant object frame cannot be used as the focus position of the terminal device.

[0118] Alternatively, when it is determined through comparison that the PD value of the center frame or face frame of the target image is lower than the PD value of the significant object frame, and the first significant confidence is not lower than a preset confidence threshold, the center frame or face frame of the target image can be directly determined as the focus position of the terminal device.

[0119] Furthermore, after determining the focus position of the terminal device, the terminal device can perform a specific focus process, that is, adopt different focus strategies according to different scenarios, adjust the focus position of the motor, and optimize the focus of the terminal device. An optional implementation method is to use sensors such as an accelerometer (gsensor) or a gyroscope (gyro) on the terminal device to obtain the operating data of the terminal device to determine the operating state of the significant object (including motion state, stable state, appearance or disappearance state), so as to determine when to trigger the focus and the corresponding focus type, and ensure that it is not triggered prematurely, but the speed must be fast (such as being able to take a photo immediately after pressing the photo button), and it must not be missed, triggered by mistake, or triggered multiple times.

[0120] Specifically, after obtaining the operation data of the terminal device (such as laser data), these data can be filtered to more accurately determine the motion state of the significant object. When it is determined that the significant object is in a stable state and meets the conditions for significant focusing (the specific content is not limited and can be set according to actual conditions and experience values), the PD value of the significant object frame is used for focusing, so that the integration side of the terminal can achieve correct data interaction with the AF algorithm side, thereby ensuring that the focus area obtained is dynamically changing, rather than fixed, so that the user's intention can be accurately identified and the area of ​​interest can be accurately focused.

[0121] In this way, when the user uses the terminal device to take photos, make video calls, record videos or live stream videos, in order to be able to focus on the subject that the user really wants to focus on and optimize the focus result to avoid out-of-focus photos or videos, thereby improving the user experience, the terminal device can execute the above steps S601-S605, determine the significant object frame based on the preset AI algorithm and / or high-density laser, and then judge whether the PD values ​​of the center frame or face frame and the significant object frame of the target image are available based on the significance confidence of the center frame or face frame and the significant object frame of the target image, thereby being able to more accurately determine the subject that the user really wants to focus on, optimize the focus result, and improve the user experience.

[0122] To facilitate understanding of the focus optimization method of the terminal device provided by this application, this application will be explained through an example scenario of live video broadcasting, such as Fig. 9 As shown, in the example scenario of the live video broadcast, the face of the user who is conducting the live video broadcast is far away, while the displayed object "tablet computer" is closer, that is, when the camera is used to shoot the video for live broadcast, the "tablet computer" is closer to the front (that is, the foreground) than the face. It is understandable that the subject that the user actually wants to focus on is the "tablet computer". If the existing focusing method is used, the face of the user at a farther position will still be focused on to clearly display the face, and the "tablet computer" will not be focused, resulting in blurring and loss of focus of the "tablet computer", as shown in FIG. Fig. 9 However, after the focus optimization method provided by the present application is used to optimize the focus, the image in the preview stream or video stream obtained will focus on the "tablet computer" and clearly display the content of the tablet computer, but will not focus on the face of the user at a farther position, that is, the user's face will be blurred, such as Fig. 9 As shown in (b), the subject that the user actually wants to focus on can be determined more accurately, which optimizes the focusing result and improves the user experience.

[0123] In addition, the present application embodiment also provides an electronic device (i.e., a terminal device). For the hardware structure and software framework of the electronic device, please refer to Figure 2 and Figure 3 Corresponding description. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor is used to call and execute the computer program to implement the focus optimization method of the terminal device provided in the above description.

[0124] The present application also provides a computer-readable storage medium in an embodiment, on which a computer program is stored. When the computer program is executed by a processor of a terminal device, it is used to implement the focus optimization method of the terminal device provided in the above description.

[0125] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A focus optimization method for a terminal device, characterized in that: Applied to a terminal device, the method comprises: Acquire a target image to be optimized; Determine a center frame or a face frame from the target image, and detect a significant object frame from the target image; Performing phase detection on the center frame or the face frame of the target image to determine a phase difference PD value corresponding to the center frame or the face frame of the target image; and performing phase detection on the salient object frame to determine a PD value corresponding to the salient object frame; Calculating a first saliency confidence of a center frame or a face frame of the target image and a second saliency confidence corresponding to the saliency object frame by using a saliency algorithm; Determining a focus position of the terminal device by using a comparison result of a PD value of a center frame or a face frame of the target image and a PD value of the salient object frame, as well as the first saliency confidence and the second saliency confidence; The detecting a significant object frame from the target image comprises: Detecting a significant object frame from the target image using a high-density laser, and / or detecting a significant object frame from the target image using a preset AI algorithm; The detecting a significant object frame from the target image using a high-density laser comprises: Acquire high-density laser data corresponding to the target image using a high-density laser, and filter out depth data with a confidence level lower than a preset threshold from the high-density laser data to obtain filtered depth data; Clustering the filtered depth data, and calculating an average of depth values ​​of the clustered depth data to form depth map information; Performing saliency detection on the downsampled image of the target image and the depth map information to obtain the salient object frame; The performing phase detection on the salient object frame to determine the PD value corresponding to the salient object frame includes: According to the coordinate values ​​of the salient object frame, the sub-windows with confidence values ​​lower than a preset threshold in the preset multi-matched window are deleted, and the PD values ​​of the remaining sub-windows in the preset multi-matched window are used to determine the PD value corresponding to the salient object frame; The determining the focus position of the terminal device by comparing the PD value of the center frame or the face frame of the target image with the PD value of the salient object frame, as well as the first saliency confidence and the second saliency confidence, includes: When it is determined through comparison that the PD value of the center frame or face frame of the target image is not lower than the PD value of the significant object frame, and the second significance confidence is not lower than a preset confidence threshold, the significant object frame is determined to be the focus position of the terminal device; or, when it is determined through comparison that the PD value of the center frame or face frame of the target image is lower than the PD value of the significant object frame, and the first significance confidence is not lower than a preset confidence threshold, the center frame or face frame of the target image is determined to be the focus position of the terminal device.

2. The method according to claim 1, characterized in that The face frame is determined as follows: A face frame is detected from the target image using a face recognition algorithm.

3. The method according to claim 1, characterized in that: The performing saliency detection on the downsampled image of the target image and the depth map information to obtain the salient object frame includes: Performing foreground segmentation on the downsampled image of the target image and the depth map information to obtain a foreground segmentation result; The downsampled image of the target image and the foreground segmentation result are subjected to saliency detection of the three color channels of red, green and blue (RGB), and a salient object frame corresponding to the salient subject is obtained as a detection result.

4. The method according to claim 1, characterized in that: After detecting the salient object frame from the target image, the method further includes: The salient object frame is input into a salient state machine to determine whether the area where the salient object frame is located is in a stable state and within a limited range.

5. The method according to any one of claims 1 to 4, characterized in that: After determining the focus position of the terminal device by comparing the PD value of the center frame or the face frame of the target image with the PD value of the salient object frame, and the first saliency confidence and the second saliency confidence, the method further includes: Acquiring the running state of the significant object, where the running state includes a moving state, a stable state, and an appearing or disappearing state; When the salient object is in a stable state and satisfies the conditions for salient focusing, focusing is performed using the PD value of the salient object frame.

6. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor is used to call and execute the computer program to implement the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by an electronic device, the method according to any one of claims 1 to 5 is implemented.

Images