Photographing preview interface display method and terminal equipment

Abstract

The invention discloses a photographing preview interface display method and terminal equipment, and relates to the technical field of photographing. When the moving main body exists in the photographing preview image, the position of the moving main body in the photographing preview image is determined by combining the position of the moving main body detected in the two-dimensional image collected by the camera and the depth value of the moving main body in the depth image collected by the depth sensor, and the moving main body is marked by adopting a marking box; and the marking box continuously tracks the moving main body. Because the motion detection calibration area and the depth value are combined, the motion subject can be accurately marked in the photographing preview image.

Description

Technical Field

[0001] The present application relates to the field of photographing technology, and in particular to a photographing preview interface display method and terminal device. Background Art

[0002] With the rapid development of terminal equipment software and hardware technology, terminal equipment can provide photo functions in various scenarios. For example, the portrait mode focuses on the face of the person and takes a clear picture of the face; the smile capture automatically takes a picture when a smile is detected; the wonderful capture automatically takes a picture when a person smiles, jumps, runs, or a cat or dog is recognized.

[0003] One camera function that brings a good camera experience to users is to automatically take a photo when a moving object is detected. For example, this function can be called motion capture. After the terminal device starts the motion capture function, the moving object can be detected, the moving object can be marked with a marking box in the camera preview interface, and the moving object can be automatically captured.

[0004] If the moving object cannot be accurately tracked, the moving object may be inaccurately marked or even not marked at all in the photo preview interface, which will further affect the automatic capture of the moving object. Summary of the invention

[0005] The embodiments of the present application provide a method and a terminal device for displaying a photo preview interface, which can accurately mark moving objects in the photo preview interface, thereby improving the user's photo taking experience when using a motion capture function.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, a method for displaying a photo preview interface is provided, wherein a terminal device displays a first interface (photo preview interface), wherein a first display area in the first interface includes a first image (a preview image of an image captured by a camera); and a first subject position on the first image includes a first annotation frame. The first annotation frame annotates a moving object, that is, the first subject position is the position of the moving object. The first annotation frame is generated based on a first calibrated area in the first image (a moving object area determined by motion detection of the image captured by the camera) and depth information of the first image.

[0008] In this method, when there is a moving subject (e.g., a moving object, a moving person, a moving animal, etc.) in the photo preview image, the position of the moving subject in the photo preview image is determined by combining the position of the moving subject detected in the two-dimensional image (calibration area) and the depth value of the moving subject in the depth image, and the moving subject is annotated with a labeling box, which continuously tracks the moving subject. Due to the combination of the calibration area and the depth value, the moving subject can be accurately annotated in the photo preview image.

[0009] In combination with the first aspect, in a possible implementation, the terminal device includes a matrix laser array device, the matrix laser array device includes a plurality of groups of laser emitters, and the depth information of the first image is collected by the matrix laser array device.

[0010] The matrix laser array device includes multiple groups of laser emitters, which can emit multiple groups of lasers at the same time. The laser covers a wider detection area, can accurately capture moving objects, and obtain depth information of moving objects.

[0011] In combination with the first aspect, in a possible implementation, the position of the first annotation box is determined according to depth information of each pixel in a first calibrated area in the first image.

[0012] In this method, after the first marked area is determined by using a motion detection algorithm, the position of the marking frame is determined according to the depth information in the first marked area. At this time, the depth information in the first marked area is a valid value.

[0013] Wherein, determining the position of the first annotation box according to the depth information of each pixel point in the first calibration area in the first image includes: determining the position of the first annotation box according to the pixel points whose depth values ​​meet the preset conditions in the first calibration area in the first image.

[0014] In a possible implementation, the preset condition includes: the depth value is the smallest, or the difference between the depth value and the minimum depth value in the first calibration area in the first image is less than or equal to the preset value. That is, the pixel point with the smallest depth value, or the pixel point with the difference between the depth value and the minimum depth value in the first calibration area in the first image is less than or equal to the preset value, is determined as the position of the moving object.

[0015] In some other possible implementations, if the depth information in the first marked area is an invalid value, the depth information of the area surrounding the first marked area is used to determine the position of the first marked box, thereby increasing the probability of determining the marked box.

[0016] In combination with the first aspect, in a possible implementation, the camera focus position is also determined based on the phase difference data obtained when the camera captures the first image. If it is determined that the first calibration area includes the camera focus position, the first annotation box is displayed on the first image in the photo preview interface.

[0017] This ensures that the moving object marked by the annotation box is the focus position of the camera, so that the moving object is clear in the taken photo.

[0018] In combination with the first aspect, in a possible implementation, in response to a user's zoom operation, a second image is displayed in a first display area in a first interface; the image of a first subject in the second image is magnified by a first multiple compared to the image of the first subject in the first image, and the position of the first subject on the second image includes a second annotation box; wherein the second annotation box is generated based on a second calibrated area in the second image and depth information of the second image, and the second calibrated area is a moving object area determined by motion detection of an image captured by a camera.

[0019] In this method, in response to the user's zoom operation, the camera zooms accordingly, and the object in the preview image in the preview area is magnified accordingly according to the same proportion as the zoom focal length. After the zoom operation, it is still supported to use a labeling frame to label the moving object.

[0020] In combination with the first aspect, in a possible implementation, the second annotation frame is enlarged by a first multiple compared to the first annotation frame, that is, the annotation frame and the main object are enlarged at the same ratio.

[0021] In this embodiment, the terminal device includes multiple cameras, the multiple cameras include a main camera and a telephoto camera, and the first magnification is less than the first value (the optical zoom magnification of the telephoto camera). That is, when the zoom magnification is less than the optical zoom magnification of the telephoto camera, the annotation box and the main object are enlarged at the same ratio.

[0022] When the digital zoom focal length of the main camera is m times zoom, the FOV of the preview image becomes 1 / m of the FOV of the 1x zoom focal length. Correspondingly, the number of pixels in the depth image that can be used to calculate the reported points in the small box is 1 / m of the 1x zoom focal length. The greater the digital zoom multiple of the main camera, the fewer pixels in the depth image that can be used to calculate the reported points in the small box. When the zoom focal length of the main camera is greater than or equal to 1x zoom and less than the first value, the number of pixels in the depth image that can be used to calculate the reported points in the small box is sufficient to accurately calculate the position of the annotation box.

[0023] In combination with the first aspect, in a possible implementation, the second annotation frame has the same display ratio as the first annotation frame, that is, the main object is magnified according to the zoom factor, and the annotation frame is displayed at the same ratio of 1x zoom.

[0024] In this embodiment, the terminal device includes multiple cameras, the multiple cameras include a main camera and a telephoto camera, and the first magnification is equal to the first value (the optical zoom magnification of the telephoto camera). That is, when the zoom magnification is equal to the optical zoom magnification of the telephoto camera, the main object is magnified according to the zoom magnification, and the annotation frame is displayed at the same ratio of 1x zoom.

[0025] When the camera zoom focal length is equal to the first value, the camera switches to the telephoto camera, which is optical zoom. The FOV of the preview image is the FOV of the telephoto camera, and the pixel range in the depth image that can be used to calculate the reported points of the small box is consistent with the FOV of the telephoto camera. The pixel points in the depth image that can be used to calculate the reported points of the small box are sufficient to accurately calculate the position of the annotation box.

[0026] In combination with the first aspect, in a possible implementation, the terminal device includes multiple cameras, the multiple cameras include a main camera and a telephoto camera, the second annotation frame is magnified by a second multiple than the first annotation frame, the second multiple is the first multiple divided by a first value, and the first value is the optical zoom multiple of the telephoto camera. That is, the main object is magnified according to the zoom multiple, and the annotation frame is magnified based on the size at 2.5x zoom.

[0027] In this embodiment, the first magnification is greater than the first value. That is, when the zoom magnification is greater than the optical zoom magnification of the telephoto camera, the subject is magnified according to the zoom magnification, and the annotation frame is magnified based on the size at 2.5x zoom.

[0028] In a second aspect, a terminal device is provided, which has the function of implementing the method described in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0029] In a third aspect, a terminal device is provided, comprising: a processor and a memory; the memory is used to store computer execution instructions, and when the terminal device is running, the processor executes the computer execution instructions stored in the memory to enable the terminal device to perform a method as described in any one of the above-mentioned first aspects.

[0030] In a fourth aspect, a terminal device is provided, comprising: a processor; the processor is used to couple with a memory, and after reading instructions in the memory, execute a method as described in any one of the above-mentioned first aspects according to the instructions.

[0031] In a fifth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on a computer, the computer can execute any of the methods described in the first aspect.

[0032] In a sixth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect.

[0033] In a seventh aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting a terminal device to implement the functions involved in the first aspect. In one possible design, the device also includes a memory for storing program instructions and data necessary for the terminal device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.

[0034] Among them, the technical effects brought about by any design method in the second to seventh aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a scenario applicable to the photo preview interface display method provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of the structure of a matrix laser array device applicable to the photo preview interface display method provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of the software architecture of a terminal device provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of a flow chart of a method for displaying a photo preview interface provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0043] Fig. 9 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0044] Fig.10A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0045] Fig.11 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0046] Fig.12 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0047] Fig.13 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0048] Fig.14 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0049] Fig.15 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0050] Fig.16 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0051] Fig.17 A schematic diagram of a scenario example of the method for displaying a photo preview interface provided in an embodiment of the present application;

[0052] Fig.18 A schematic diagram of the structural composition of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include such expressions as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0054] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0055] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0056] Mobile phones, tablet computers and other terminal devices all support camera applications. Users can open the camera application, and the terminal device can collect images through the camera to take photos and videos. Figure 1 , mobile phone 100 displays a desktop interface, which includes icons of multiple applications. Users can click on the application icons to start the corresponding applications. For example, the desktop interface includes icons for the "clock" application, the "calendar" application, the "gallery" application, the "notes" application, the "camera" application, and the like. In response to the user clicking on the "camera" application icon, mobile phone 100 starts the camera application. Exemplarily, mobile phone 100 displays a photo preview interface 101. Photo preview interface 101 includes a preview area, for example, the preview area is a preview box 102, and preview box 102 is used to display a photo preview image. It can be understood that, Figure 1 The middle dotted frame is only used to indicate the area range of the preview frame 102 and is not displayed in the actual photo preview interface.

[0057] The embodiment of the present application provides a method for displaying a photo preview interface. When there is a moving subject (e.g., a moving object, a moving person, a moving animal, etc.) in the photo preview image, a labeling frame is used to label the moving subject in each frame of the photo preview image, and the labeling frame continuously tracks the moving subject. In one example, the labeling frame is composed of a group of small squares, and the shape of the group of small squares can be a regular shape or an irregular shape.

[0058] Exemplary, reference Figure 2 The preview frame 102 displays the photo preview image including the moving subject, that is, the sphere A in the image. During the movement of the sphere A, the sphere A is annotated with a labeling frame in each frame of the photo preview image. For example, the labeling frame is Figure 2 An irregular shaped figure composed of small red squares.

[0059] If the exact position of the moving object cannot be identified, the annotation frame cannot be accurately marked in the photo preview image. For example, the annotation frame is not marked on the moving object, or the annotation frame is not displayed. In some embodiments, the annotation frame is the focus position of the camera when taking a photo. If the position of the annotation frame is inaccurate, the focus position will not be on the moving object, and the moving object may be blurred in the captured photo.

[0060] An embodiment of the present application provides a method for displaying a photo preview interface, which combines the position of a moving subject detected in a two-dimensional image and the depth value of the moving subject in a depth image to determine the position of the moving subject in the photo preview image, and can accurately mark the moving subject in the photo preview image.

[0061] The photo preview interface display method provided in the embodiment of the present application can be applied to a terminal device including a camera and a display screen. The above-mentioned terminal devices may include mobile phones, tablet computers, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, netbooks, smart home devices (e.g., smart TVs, smart screens, large screens, smart speakers, smart air conditioners, etc.), personal digital assistants (PDAs), wearable devices (e.g., smart watches, smart bracelets, etc.), vehicle-mounted devices, virtual reality devices, etc., and the embodiment of the present application does not impose any restrictions on this.

[0062] Figure 3 Schematic diagram of the structure of a terminal device 100 provided in an embodiment of the present application. Figure 3As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a camera 190, and a display screen 191. The sensor module 180 may include a temperature sensor, an ambient light sensor, a touch sensor, a depth sensor, and the like.

[0063] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0064] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0065] The controller may be the nerve center and command center of the terminal device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0066] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0067] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0068] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0069] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0070] The charging management module 140 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 is charging the battery 142, it can also power the terminal device through the power management module 141. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and powers the processor 110, the internal memory 121, the external memory, the display screen 191, the camera 190, and the wireless communication module 160.

[0071] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0072] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0073] The terminal device 100 implements the display function through a GPU, a display screen 191, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 191 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0074] The display screen 191 is used to display images, videos, etc. The display screen 191 is also referred to as a screen 191. The display screen 191 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 191, where N is a positive integer greater than 1.

[0075] The terminal device 100 can realize the shooting function through the ISP, the camera 190, the video codec, the GPU, the display screen 191 and the application processor.

[0076] Camera 190 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.

[0077] The ISP is used to process the data fed back by the camera 190. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 190.

[0078] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0079] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0080] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0081] In some embodiments, the terminal device 100 may include 1 or N cameras 190, where N is a positive integer greater than 1. For example, the terminal device 100 may include a main camera, a wide-angle camera, a telephoto camera, etc. The main camera, the wide-angle camera, the telephoto camera, etc. are used to capture two-dimensional images, such as RGB images.

[0082] In some embodiments, a depth sensor is provided in the camera module of the terminal device 100, and the depth sensor is used to collect depth information of an image, for example, to generate a depth image. The depth sensor may include a structured light sensor, a time of flight (TOF) component, a matrix laser array device, etc.

[0083] Structured light sensors are used to project light of a specific wavelength emitted by a laser onto the surface of a structure. The light of the specific wavelength is reflected or scattered by the surface of the object, and then filtered out by a neutral density film or a filter to remove stray light and retain the specific wavelength emitted by the structured light source. The light of the specific wavelength enters a camera arranged at a certain angle to the structured light source (laser) for imaging, and the position, depth and other information of the object can be calculated based on the changes in the light signal caused by the object.

[0084] The ToF component is used to measure the time or phase difference between the light pulse and the object to be measured to obtain the depth information of the object to be measured, thereby obtaining a three-dimensional model of the object to be measured. The ToF component can be called by other names, such as ToF sensor, ToF camera, etc.

[0085] In one example, reference Figure 4, the matrix laser array device includes a transmitting unit and a receiving unit, the transmitting unit includes multiple groups of laser emitters for emitting laser pulses, for example, multiple groups of laser emitters form a matrix array; the laser emitters can be laser diodes, vertical-cavity surface-emitter lasers (vertical-cavity surface-emitter laser, VCSEL), edge emitting lasers (edge ​​emitting laser, EEL), etc. Optionally, the transmitting unit also includes a scattering plate for converting the laser emitted by the laser emitter into a parallel laser beam. The receiving unit includes a lens and an imaging module. The lens is an optical lens for converging the received laser and converging the received laser to the imaging module. The imaging module can be an imaging sensor, including 2 or more shutters for sampling the reflected laser beam at different times. For example, the imaging module can be a CMOS. The matrix laser array device can also include a calculation unit (not shown in the figure), which is used to calculate the distance between the object to be measured and the matrix laser array device according to the time difference or phase difference between the laser received by the receiving unit and the laser emitted by the transmitting unit, and obtain a depth map of the object to be measured. The depth map is usually a grayscale image, in which the value of each pixel represents the distance between a pixel of the object being measured and the matrix laser array device, that is, the depth information.

[0086] In the photo preview interface display method provided in the embodiment of the present application, the position of the moving object is detected in the two-dimensional image captured by the camera, for example, the calibrated region of interest (ROI) of the moving object is obtained, and the accurate position of the moving object is comprehensively determined in combination with the depth information captured by the depth sensor, so as to be marked in the photo preview image; the moving object can be accurately marked in the photo preview interface.

[0087] In the embodiment of the present application, the terminal device is a terminal device that can run an operating system and install application programs. Optionally, the operating system running on the terminal device can be system, system, System, etc.

[0088] In some embodiments, the software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture or a cloud architecture. Taking the system as an example, the software structure of the terminal device 100 is exemplarily described.

[0089] Figure 5 A software architecture diagram of an electronic device provided in an embodiment of the present application.

[0090] It is understandable that the layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The system can include the application (App) layer, the application framework (FWK) layer, the hardware abstraction layer (HAL) and the kernel layer. Figure 5 As shown, The system may also include the Android runtime and system libraries.

[0091] The application layer can include a series of application packages. Figure 5 As shown, the application package may include applications such as camera, gallery, calendar, call, music, short message, etc. The camera application is used to take photos and videos.

[0092] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. The application framework layer provides programming services to the application layer through the API interface. Figure 5 As shown, the application framework layer includes the media framework, media codec services, etc. The media framework is used to manage multimedia resources such as photos, pictures, videos, and audio. The media codec services are used to manage the codecs of multimedia resources such as photos, pictures, videos, and audio.

[0093] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0094] The core library consists of two parts: one part is the functional functions that Java voice needs to call, and the other part is the Android core library.

[0095] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0096] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0097] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0098] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0099] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc.

[0100] A 2D graphics engine is a drawing engine for 2D drawings.

[0101] The hardware abstraction layer (HAL) is used to abstract the hardware, encapsulate the kernel layer driver, and provide an interface to the upper layer. Figure 5 As shown, the hardware abstraction layer includes camera HAL, display HAL, audio HAL, etc. The camera HAL includes an AF processing module, a motion detection module, etc. The AF processing module is used for auto focus (AF) processing, and the motion detection module is used for detecting moving objects in a two-dimensional image.

[0102] The kernel layer provides low-level drivers for various hardware of electronic devices. For example, Figure 5 As shown, the kernel layer may include a camera driver, a depth sensor driver, a display driver, an audio driver, etc.

[0103] The camera captures a two-dimensional image, and the two-dimensional image is sent to the motion detection module through the camera driver. The motion detection module calculates the ROI area of ​​the moving object in the two-dimensional image according to the motion detection algorithm; the motion detection module sends the calculated ROI area information to the AF processing module. The depth sensor captures the depth information, and sends the depth information to the AF processing module through the depth sensor driver. The AF processing module determines the coordinate point set of the moving object in the two-dimensional image based on the ROI area information and the depth information. The AF processing module sends the coordinate point set to the camera application, so that the camera application can generate a preview image based on the two-dimensional image and determine the position of the annotation box based on the coordinate point set.

[0104] The following describes in detail the photo preview interface display method provided by the embodiment of the present application in conjunction with the accompanying drawings.

[0105] In some embodiments, the user can turn on or off the motion capture function on the terminal device. Figure 6, the photo preview interface 101 of the mobile phone 100 includes a pull-up button 103. In response to the user clicking the pull-up button 103, the photo preview interface 101 of the mobile phone 100 displays a plurality of buttons, including a "motion focus" button 104. In response to the user clicking the "motion focus" button 104, the photo preview interface 101 displays a "motion focus priority" button 105. The user can click the "motion focus priority" button 105 to turn on the motion capture function.

[0106] In one implementation, after the motion capture function is turned on, or the continuous shooting function is turned on, when there is a moving subject (for example, a moving object, a moving person, a moving animal, etc.) in the photo preview image, a labeling box is used to label the moving subject in each frame of the photo preview image, and the labeling box continuously tracks the moving subject.

[0107] For example, Figure 7 A flow chart of a method for displaying a photo preview interface provided in an embodiment of the present application is shown.

[0108] like Figure 7 As shown, the method may include:

[0109] S710: The camera collects a two-dimensional image.

[0110] In one example, the two-dimensional image is an RGB image.

[0111] S711 . The motion detection module acquires a two-dimensional image, and detects the two-dimensional image using a motion detection algorithm to acquire information of the ROI region.

[0112] The motion detection module can use a conventional motion detection algorithm to detect the RGB image and obtain the position information of the ROI area of ​​the moving object. In an example, the ROI area is a rectangular area where the moving object is located, and the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2) of the rectangular area can be used to represent the position information of the ROI area. For example, Figure 8 As shown, the moving object detected in the RGB image is a sphere, and the coordinates of the upper left corner of the rectangular area where the sphere is located are (x1, y1), and the coordinates of the lower right corner are (x2, y2); that is, the position information of the ROI area is (x1, y1), (x2, y2).

[0113] S712: The motion detection module sends the position information of the ROI area to the AF processing module.

[0114] S713: The AF processing module determines that the ROI area includes the focus position.

[0115] In some cameras, phase detection auto focus (PDAF) is used for auto focus, that is, auto focus is achieved through phase detection. In the PDAF method, some shielded pixels (SP) are reserved on the left and right sides of the camera photosensitive device for phase detection. The distance and change of the image of the object through the shielded pixels on the left and right sides are detected to determine the focus offset, so as to achieve accurate focus. Specifically, the control unit of the camera (such as the camera driver) obtains the phase difference of the image of the object through the left and right sides of the SP on the photosensitive device, and maps the object to a certain position on the lens moving axis according to the phase difference of the image of the object through the left and right sides of the SP; that is, the phase difference (PD) of the detected object through the left and right sides of the SP on the photosensitive device is converted into a defocus value (DV). In one implementation, DV = PD*DCC; wherein DCC is the defocus conversion coefficient, which is a fixed parameter of the camera (camera internal parameter). The camera control unit can obtain the DV value based on the detected PD value and the value of the camera internal reference DCC. The camera control unit transmits the DV value to the camera focus motor. The camera focus motor drives the camera lens forward or backward according to the DV value, so that the camera lens adjusts the focal length according to the DV value to achieve automatic focus.

[0116] In some embodiments, for each frame of two-dimensional image captured by the camera, the AF processing module calculates the focus position based on the PD value; thereby pushing the motor to the focus position when displaying each frame of preview image; and achieving frame-by-frame focus tracking.

[0117] Optionally, in one implementation, after the AF processing module obtains the ROI area, it determines whether the ROI area contains the focus position; if the ROI area contains the focus position, it means that the ROI area is in focus, that is, the focus position determined by the AF processing module is consistent with the ROI area determined by the motion detection module. If the ROI area is in focus, the AF processing module further performs subsequent steps to determine the position of the annotation box; this ensures that the moving object marked by the annotation box is the focus position of the camera, so that the moving object in the taken photo is clear.

[0118] S720: The depth sensor collects a depth image.

[0119] The value of each pixel in the depth image represents the depth information (depth value) of the pixel, that is, the distance value between the pixel and the depth sensor.

[0120] In one implementation, the depth sensor uses a matrix laser array device. The matrix laser array device includes multiple groups of laser emitters, which can emit multiple groups of lasers at the same time. The laser covers a wider detection area, and even fast-moving objects are unlikely to exceed the laser coverage area. Therefore, it can accurately capture moving objects and obtain depth information of moving objects.

[0121] Exemplary, reference Fig. 9 , a matrix laser array device with a resolution of 40*30 is used to obtain a depth image. The depth image includes 40*30 pixels, that is, the resolution of the depth image is 40*30; the value of each pixel represents the depth value of the pixel. For example, Fig. 9 As shown, the depth value of a pixel can be 2.4, 2.3, 2.7, etc.

[0122] S721: The depth sensor sends the depth image to the AF processing module, and the AF processing module obtains data information of the depth image.

[0123] In one example, the data information of the depth image can be represented by a matrix series. For example, the data information of the depth image with a resolution of 40*30 can be represented by a 40*30 matrix series, and each value in the matrix series represents the depth value of the pixel at the corresponding position. For example, the matrix series can be represented as XY[40*30].

[0124] S730: The AF processing module synchronizes the depth image and the two-dimensional image in time domain and space domain.

[0125] In some embodiments, the camera periodically captures two-dimensional images at a preset frequency, and each two-dimensional image corresponds to a capture time; the depth sensor periodically captures depth images at a preset frequency, and each depth image corresponds to a capture time.

[0126] The two-dimensional image and the depth image corresponding to the same acquisition time are associated with each other, that is, the depth image and the two-dimensional image are synchronized in the time domain.

[0127] In some embodiments, the positions of the camera and the depth sensor on the terminal device are not overlapped, the angles are also offset, the camera imaging plane and the depth sensor imaging plane may not be in the same plane, or the camera imaging area and the depth sensor imaging area are offset. Fig.10As shown, the camera imaging area and the matrix laser array device imaging area are not in the same plane, and the position of the same sphere in the camera imaging area is offset from its position in the matrix laser array device imaging area. For example, the position of the sphere in the matrix laser array device imaging area is the yellow area, and the position of the sphere in the camera imaging area is the area in the black dotted box, and the yellow area and the area in the black dotted box do not overlap.

[0128] In one implementation, the pixels in the depth image and the pixels in the two-dimensional image are matched according to the position, angle, field of view and other parameters of the camera and the position, angle, field of view and other parameters of the depth sensor, that is, the depth image and the two-dimensional image are synchronized in the spatial domain. Fig.11 As shown, 40*30 pixels in the depth image are associated with pixels in the RGB image, the position of the sphere in the depth image is the yellow area, the position of the sphere in the RGB image is the area in the black dotted box, the pixels in the yellow area are associated with the pixels in the black dotted box, and the yellow area basically overlaps with the area in the black dotted box.

[0129] S731. If the depth value of each pixel point in the ROI area is a valid value, execute S732; if the depth value of each pixel point in the ROI area is an invalid value, execute S733-S734.

[0130] After the depth image and the two-dimensional image are synchronized in time domain and space domain, the depth value corresponding to each pixel can be determined. In some embodiments, the depth value corresponding to each pixel in the ROI area is a valid value, that is, the depth value corresponding to all the pixels in the ROI area can be determined. For example, the depth values ​​corresponding to all the pixels in the ROI area are as follows: Fig.12 In some embodiments, due to the depth sensor device or the surrounding conditions, the depth values ​​corresponding to the pixels in the ROI area are invalid values. For example, the depth values ​​of some pixels are preset values ​​such as 255, 65535, etc.

[0131] S732: The AF processing module determines the position information of the annotation box in the two-dimensional image according to the depth value of each pixel point in the ROI area.

[0132] In one implementation, the position of the annotation box in the two-dimensional image is composed of the coordinate points of the small boxes that constitute the annotation box. For example, the coordinate point of the center point of the small box represents the coordinate point of the small box, and the coordinate points of multiple small boxes constitute the position information of the annotation box.

[0133] In one implementation, the position of the annotation box is determined based on the pixel points in the ROI region whose depth values ​​meet the preset conditions. In one example, the preset condition is that the depth value is the smallest (ie, the distance is the shortest); for example, Fig.12 In the example, the minimum depth value is 1.5, and the pixel points that meet the preset condition are the pixel points whose depth value is equal to 1.5. In one example, the preset condition is that the difference between the minimum depth value and the minimum depth value is less than or equal to the preset value (for example, the preset value is 0.4), the minimum depth value is 1.5, and the pixel points that meet the preset condition are the pixel points whose depth value is less than or equal to 1.9. The coordinate points corresponding to the pixel points that meet the preset condition constitute the position information of the annotation box.

[0134] In some embodiments, the size of the small boxes that make up the annotation box and the spacing between the small boxes are pre-set. The size of the small boxes and the spacing between the small boxes are related to the density of reporting the coordinate points of the small boxes. In one example, when the photo ratio is 4:3 for the full field of view (FOV), the density of reporting the coordinate points of the small boxes is 24:18, that is, in the photo (for example, Figure 1 In one example, when the photo ratio is 1:1, the density of the small square coordinate points reported is 18:18, that is, in the photo (within the preview box area), at most 18 rows and 18 columns of small squares can be arranged. In one example, when the photo ratio is full screen, the density of the small square coordinate points reported is 24:11, that is, in the photo (within the preview box area), at most 24 rows and 11 columns of small squares can be arranged. In one example, reference Figure 6 The photo preview interface 101 includes a “ratio” button, and the user can click the “ratio” button to modify the photo ratio.

[0135] For example, Fig.13 A schematic diagram showing the arrangement of small boxes corresponding to different reporting densities. Fig.13 The denser the reporting density, the smaller the small boxes and the smaller the spacing between the small boxes; the sparser the reporting density of the small boxes, the larger the small boxes and the larger the spacing between the small boxes. When the photo ratio is full FOV 4:3, the reporting density of the small box coordinate points is 24:18, and the display effect is more beautiful.

[0136] In one implementation, when the depth image resolution is 40*30 and the reporting density of the small box coordinate points is 24:18, a small box coordinate point corresponds to multiple pixels of the depth image, and the depth value corresponding to the small box coordinate point can be determined according to the average value of the depth values ​​of the multiple pixels. For example, Fig.14 As shown, the depth value corresponding to a small square coordinate point can be determined according to the average value of the depth values ​​of multiple (for example, 5) pixels in the depth image.

[0137] The small box coordinate points that meet the preset conditions constitute the position information of the annotation box. Exemplarily, the preset condition is that the depth value is the smallest (i.e., the distance is the shortest), and the small box coordinate points that meet the preset condition are the small box coordinate points with the smallest depth value; Exemplarily, the preset condition is that the difference between the minimum depth value and the small box coordinate point is less than or equal to the preset value (for example, the preset value is 0.4), and the small box coordinate points that meet the preset condition are the small box coordinate points whose difference between the minimum depth value and the small box coordinate point is less than or equal to the preset value. In one example, reference Fig.14 , the minimum depth value is 1.7, and the small square coordinate point that meets the preset conditions is the small square coordinate point with a depth value of 1.7.

[0138] The position information of the annotation box in the two-dimensional image is a set of small square coordinate points that meet preset conditions.

[0139] S733. The AF processing module obtains depth values ​​of a corresponding number of pixels in the area surrounding the ROI area.

[0140] In one example, a rule for searching surrounding areas can be preset. For example, first search for an area on the right side of the ROI area that is the same area and shape as the ROI area; if the depth value of each pixel in the right area is an invalid value, then search for an area on the left side of the ROI area that is the same area and shape as the ROI area; if the depth value of each pixel in the left area is an invalid value, then search for an area on the top side of the ROI area that is the same area and shape as the ROI area; if the depth value of each pixel in the top area is an invalid value, then search for an area on the bottom side of the ROI area that is the same area and shape as the ROI area.

[0141] S734: If a valid depth value is obtained in an area around the ROI area, the AF processing module determines the position information of the annotation box in the two-dimensional image according to the depth value of each pixel in the area.

[0142] S735. The AF processing module reports the position information of the annotation box in the two-dimensional image to the camera application.

[0143] S736: The camera application generates a preview image according to the two-dimensional image and the position information of the annotation box.

[0144] The camera application calls the image synthesis module in the operating system to generate a preview image, and the preview image is used to be displayed in the preview area of ​​the photo preview interface.

[0145] The preview image includes a two-dimensional image and a label frame. The label frame is generated according to the label frame position information reported by the AF processing module. In one implementation, the label frame is composed of at least one small square frame. In one example, the shape composed of multiple small square frames can be an irregular shape. Fig.14, each small square coordinate point with a depth value equal to 1.7 generates a corresponding red small square. 14 small square coordinate points that meet the preset conditions generate 14 red small squares. The irregular shape formed by these 14 red small squares is the annotation box of the moving object.

[0146] In one implementation, a ratio threshold between the annotation box and the preview area in the photo preview interface is preset, and the ratio of the annotation box generated by the camera application to the preview area is less than or equal to the preset ratio threshold. For example, the preset ratio threshold is 1 / 16, that is, the ratio of the annotation box to the preview area is less than or equal to 1 / 16. In one implementation, the threshold for the number of small squares is determined according to the ratio threshold, and the threshold for the number of small squares represents the maximum number of small squares displayed in the preview area. Exemplarily, the reporting density of the small square coordinate points is 24:18, and the preset ratio threshold is 1 / 16, then the number of small squares displayed in the preview area is less than or equal to 27. In some embodiments, the AF processing module reports all the small square coordinate points that meet the preset conditions. If the number of small squares determined according to the annotation box position information reported by the AF processing module is greater than the small square number threshold, the number of small squares determined to be displayed when the camera application generates the annotation box is the small square number threshold. In other embodiments, when the AF processing module reports the position information of the annotation box in the two-dimensional image to the camera application, the number of small box coordinate points reported is less than or equal to the small box number threshold.

[0147] In one implementation, when the annotation box is generated according to the position information of the annotation box, each small box in the annotation box satisfies the connectivity and center convergence principles. In one example, each small box in the annotation box satisfies the connectivity, that is, each part of the annotation box is interconnected. In one example, the center convergence principle, that is, each small box in the annotation box gathers toward the center of the annotation box. For example, when the number of small boxes determined according to the annotation box position information reported by the AF processing module is greater than the small box number threshold, 27 small boxes close to the center of the annotation box are displayed.

[0148] In one implementation, if the camera has turned on functions such as subject tracking and intelligent tracking, multiple focus frames such as face frame, human frame, and subject frame may be generated on a two-dimensional image according to various preset strategies. Under the condition that multiple focus frames are generated, only the annotation frame generated by the method provided in the embodiment of the present application is displayed. In other words, the display priority of the annotation frame composed of small squares is higher than that of the face frame, human frame, subject frame, etc.

[0149] The photo preview interface display method provided in the embodiment of the present application calculates the focus position in real time for each frame of the image based on the depth information collected by the depth sensor and the ROI area determined by the motion detection, and optionally combines the PD value, and uses a labeling frame to mark the moving object at the focus position. Since the position of the moving object is determined by combining the depth information and the ROI area, the determined position of the labeling frame is more accurate. In addition, the position of the labeling frame is the position of the moving object, which is also the focus position, which can ensure that the moving object in the captured photo is clear.

[0150] In some embodiments, the camera supports optical zoom and digital zoom. Exemplarily, the user can perform a zoom operation in the preview interface. For example, in the preview interface, the user can reduce the zoom factor by pinching two fingers together, and can increase the zoom factor by spreading two fingers outward. For example, the preview interface includes a zoom factor option, and the user can select the corresponding zoom factor by clicking the zoom factor option.

[0151] In one example, the terminal device includes a main camera (e.g., a wide-angle camera), a telephoto camera, etc. The optical zoom range of the main camera is 1x zoom, and the optical zoom range of the telephoto camera is a first value, for example, the first value is 2.5x zoom; taking the camera supporting 1x (1x)-5x (5x) zoom as an example, 1x-2.5x (excluding 2.5x) is the zoom range of the main camera, and 2.5x-5x is the zoom range of the telephoto camera. Among them, 1x zoom is the optical zoom range of the main camera, and zoom greater than 1x zoom and less than 2.5x zoom is the digital zoom range of the main camera; 2.5x zoom is the optical zoom range of the telephoto camera, and zoom greater than 2.5x is the digital zoom range of the telephoto camera.

[0152] In one implementation, in response to the user's zoom operation, the camera zooms accordingly, and the subject in the preview image in the preview area is magnified accordingly according to the same proportion as the zoom focal length. For example, if the camera zoom focal length is 2.3 times, the subject in the preview image in the preview area is magnified 2.3 times in proportion to the zoom of 1 times; if the camera zoom focal length is 2.5 times, the subject in the preview image in the preview area is magnified 2.5 times in proportion to the zoom of 1 times.

[0153] When the camera zoom range is greater than 1x (main camera optical zoom range) and less than 2.5x (telephoto camera optical zoom range), the annotation box displayed on the preview image is also enlarged in proportion to the zoom range. In one implementation, each small box in the annotation box is enlarged in proportion to the zoom range. For example, if the camera zoom range is m times, the small boxes are enlarged m times, that is, the reporting density of the small boxes becomes 1 / m. For example, when the camera zooms 1x, the reporting density of the small box coordinate points is 24:18, and when the camera zooms 2x, the reporting density of the small box coordinate points is 12:9. For example, Fig.15 As shown, the mobile phone 100 displays a photo preview interface 101, which includes a preview box 102; the photo preview interface 101 also includes multiple zoom multiple options, for example, a 0.6x zoom option 1061, a 1x zoom option 1062, a 2x zoom option 1063, a 2.5x zoom option 1064, etc. For example, Fig.15 As shown in (a), the mobile phone camera is at 1x zoom. In response to the user clicking the 2x zoom option 1063, the preview image displayed in the preview box 102 is as follows: Fig.15 As shown in (b), the subject (e.g., person, sphere A) in the preview image is enlarged to 2 times of its original size; accordingly, the annotation box is proportionally enlarged to 2 times of its original size, and each small box in the annotation box is proportionally enlarged to 2 times of its original size. In one example, after the annotation box is proportionally enlarged, if the proportion of the annotation box in the preview area is greater than the ratio threshold of the annotation box to the preview area, a portion of the small boxes are displayed (according to the connectivity and center convergence rules). For example, Fig.16 As shown in (a), the mobile phone 100 displays a photo preview interface 101, which includes a preview box 102. The photo preview interface 101 also includes multiple zoom multiple options, for example, a 0.6x zoom option 1061, a 1x zoom option 1062, a 2x zoom option 1063, a 2.5x zoom option 1064, etc. For example, Fig.16 As shown in (a), the mobile phone camera is at 1x zoom. In response to the user clicking the 2x zoom option 1063, the preview image displayed in the preview box 102 is as follows: Fig.16 As shown in (b), the subject (eg, person, sphere A) in the preview image is magnified to twice its original size. Fig.16 After all the small and medium boxes in (a) are enlarged proportionally to twice their original sizes, the proportion of the label box in the preview area will be greater than the ratio threshold of the label box to the preview area; according to the connectivity and center convergence rules, only 14 of the small boxes are displayed. In this way, the proportion of the label box composed of 14 small boxes in the preview area is less than or equal to the ratio threshold of the label box to the preview area.

[0154] The camera zoom focal length is greater than 1x and less than 2.5x, which is the main camera digital zoom focal length. When the main camera digital zoom focal length is m times zoom, the FOV of the preview image becomes 1 / m of the 1x zoom focal length FOV. Correspondingly, the number of pixels in the depth image that can be used to calculate the reported points of the small box is 1 / m of the 1x zoom focal length. The greater the digital zoom multiple of the main camera, the fewer pixels in the depth image can be used to calculate the reported points of the small box. In the photo preview interface display method provided in the embodiment of the present application, the main camera zoom focal length is greater than or equal to 1x zoom and less than 2.5x zoom, and the number of pixels in the depth image that can be used to calculate the reported points of the small box is sufficient, and the position of the annotation box can be accurately calculated.

[0155] When the camera zoom focal length is equal to 2.5x, the camera switches to a telephoto camera, 2.5x is the optical zoom focal length of the telephoto camera, the FOV of the preview image is the FOV of the telephoto camera, and the pixel range in the depth image that can be used to calculate the reported points in the small square is consistent with the FOV of the telephoto camera. The pixel points in the depth image that can be used to calculate the reported points in the small square are sufficient to accurately calculate the position of the annotation box. In one implementation, when the camera zoom focal length is equal to 2.5x, the annotation box displayed on the preview image is displayed in the same ratio as 1x zoom. For example, when the camera zooms 1x, the reporting density of the small square coordinate points is 24:18, and when the camera zooms 2.5x, the reporting density of the small square coordinate points is also 24:18. For example, Fig.17 As shown in (a), the mobile phone camera is at 1x zoom. In response to the user clicking the 2.5x zoom option 1064, the preview image displayed in the preview box 102 is as follows: Fig.17 As shown in (b), the subject (eg, person, sphere A) in the preview image is magnified 2.5 times of the original; the marking frame has the same proportion as that when the zoom is 1x, and the small squares in the marking frame have the same proportion as that when the zoom is 1x.

[0156] When the camera zoom focal length is greater than 2.5x, the annotation box displayed on the preview image is enlarged in proportion to the ratio of (zoom focal length / 2.5). In one implementation, each small box in the annotation box is enlarged in proportion to the ratio of (zoom focal length / 2.5). For example, if the camera zoom focal length is n times, the small boxes are enlarged in proportion to n / 2.5 times, that is, the reporting density of the small boxes becomes 1 / (n / 2.5). For example, when the camera zooms 2.5x, the reporting density of the small box coordinate points is 24:18, and when the camera zooms 5x, the reporting density of the small box coordinate points is 12:9.

[0157] The photo preview interface display method provided in the embodiment of the present application supports the use of annotation boxes to mark moving objects when the camera is zoomed 1x-5x, and can also accurately mark moving objects in camera zoom scenes.

[0158] It should be noted that the present embodiment of the application takes the example of the annotation box consisting of a small square box as an example. In actual application, it can also be displayed as other shapes instead of a small square box; for example, the annotation box consists of a small rectangular box, the annotation box consists of a small circular box, the annotation box consists of a small triangular box, etc. The present embodiment of the application does not limit the specific form of the annotation box.

[0159] It is understandable that the terminal device provided in the embodiment of the present application includes a hardware structure and / or software module corresponding to each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiment of the present application.

[0160] The embodiment of the present application can divide the functional modules of the above-mentioned terminal device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0161] In one example, see Fig.18 , which shows a possible structural diagram of the terminal device involved in the above embodiment. The terminal device 900 includes: a processing unit 910, a storage unit 920, an image acquisition unit 930, a sensor unit 940 and a display unit 950.

[0162] The processing unit 910 is used to control and manage the actions of the terminal device 900. The storage unit 920 is used to store the program code and data of the terminal device 900. The image acquisition unit 930 is used to acquire images. The sensor unit 940 is used to acquire depth information. The display unit 950 is used to display the interface of the terminal device 900; for example, displaying a photo preview interface, etc.

[0163] Of course, the unit modules in the above-mentioned terminal device 900 include but are not limited to the above-mentioned processing unit 910, storage unit 920, image acquisition unit 930, sensor unit 940 and display unit 950.

[0164] Optionally, the terminal device 900 may further include an audio unit, a communication unit, etc. The audio unit is used to collect audio, play audio, etc. The communication unit is used to support the terminal device 900 to communicate with other devices.

[0165] Among them, the processing unit 910 can be a processor or a controller, for example, a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The storage unit 920 can be a memory. The image acquisition unit 930 can be a camera, etc. The sensor unit 940 can include a depth sensor. The display unit 950 can be a display screen, etc. The audio unit can include a microphone, a speaker, etc. The communication unit can include a mobile communication unit and / or a wireless communication unit.

[0166] For example, the processing unit 910 is a processor (such as Figure 3 The processor 110 shown in FIG. 1 ), the storage unit 920 may be a memory (eg, Figure 3 The internal memory 121 shown in FIG. 1 ), the image acquisition unit 930 may be a camera (eg Figure 3 The camera 190 shown in FIG. 1 ), the sensor unit 940 may be a sensor module (eg, Figure 3 The sensor module 180 shown in the figure) display unit 950 can be a display screen (such as Figure 3 The audio unit may be an audio module (such as Figure 3 The communication unit may include a mobile communication unit (such as Figure 3 The mobile communication module 150 shown) and the wireless communication unit (such as Figure 3 The wireless communication module 160 shown in FIG. 1 ). The terminal device 900 provided in the embodiment of the present application may be Figure 3 The terminal device 100 shown in FIG. The processor, memory, display screen, camera, sensor module, audio module, mobile communication unit, wireless communication unit, etc. may be connected together, for example, via a bus.

[0167] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected by lines. For example, the interface circuit can be used to receive signals from other devices (such as the memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as processors). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the terminal device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiment of the present application.

[0168] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned terminal device, the terminal device executes each function or step executed by the terminal device in the above-mentioned method embodiment.

[0169] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step executed by the mobile phone in the above method embodiment.

[0170] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0171] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0172] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0173] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0174] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0175] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for displaying a photo preview interface, applied to a terminal device, characterized in that: The terminal device includes a camera, and the method includes: The terminal device displays a first interface, wherein a first display area in the first interface includes a first image; the first image is a preview image of an image captured by the camera; The first subject position on the first image includes a first annotation box; the first annotation box is generated according to a first calibrated area in the first image and depth information of the first image, and the first calibrated area is a moving object area determined by motion detection of the image captured by the camera.

2. The method according to claim 1, characterized in that The terminal device includes a matrix laser array device, which includes a plurality of groups of laser emitters, and the depth information of the first image is collected by the matrix laser array device.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The position of the first annotation box is determined according to the depth information of each pixel point in the first calibration area in the first image.

4. The method according to claim 3, characterized in that: Determining the position of the first annotation box according to the depth information of each pixel point in the first calibration area in the first image includes: The position of the first annotation box is determined according to the pixel points whose depth values ​​meet a preset condition in the first calibration area in the first image.

5. The method according to claim 4, characterized in that The preset condition includes: the depth value is minimum, or the difference with the minimum depth value in the first calibration area in the first image is less than or equal to a preset value.

6. The method according to any one of claims 3 to 5, characterized in that: The method further comprises: Determine that the first calibration area includes a camera focus position; the camera focus position is determined based on phase difference data obtained when the camera captures the first image.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: In response to the user's zoom operation, the first display area in the first interface displays a second image; the image of the first subject in the second image is magnified by a first multiple compared to the image of the first subject in the first image, and the position of the first subject on the second image includes a second annotation box, which is generated based on a second calibrated area in the second image and depth information of the second image, and the second calibrated area is a moving object area determined by motion detection of the image captured by the camera.

8. The method according to claim 7, characterized in that The second annotation box is larger than the first annotation box by the first factor.

9. The method according to claim 8, characterized in that The terminal device includes multiple cameras, the multiple cameras include a main camera and a telephoto camera, the first magnification is less than a first value, and the first value is the optical zoom magnification of the telephoto camera.

10. The method according to claim 7, characterized in that The display ratio of the second annotation frame is consistent with that of the first annotation frame.

11. The method according to claim 10, characterized in that The terminal device includes multiple cameras, the multiple cameras include a main camera and a telephoto camera, the first magnification is equal to a first value, and the first value is the optical zoom magnification of the telephoto camera.

12. The method according to claim 7, characterized in that The terminal device includes multiple cameras, including a main camera and a telephoto camera. The second annotation box is magnified by a second multiple than the first annotation box. The second multiple is the first multiple divided by a first value. The first value is the optical zoom multiple of the telephoto camera.

13. The method according to claim 12, characterized in that The first multiple is greater than the first value.

14. An electronic device, characterized in that: include: One or more processors, displays, cameras, depth sensors, and memory; Wherein, the display screen is used to display a user interface; the camera is used to capture images; the depth sensor is used to capture depth information of images; the memory stores one or more computer programs, and the one or more computer programs include instructions, which, when executed by the processor, enable the electronic device to execute the method described in any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images