A photographing method, a terminal device, and a computer readable storage medium

Abstract

A shooting method, terminal device, and computer-readable storage medium, relating to the field of terminal technology, solves the problem of missing the recording of exciting moments in an image due to response time, ensuring the output of excellent images and improving user experience. In this method, after receiving a user's operation to open the camera application, the terminal device containing a camera module captures a preview stream of the current scene. After determining that the current scene contains multiple subjects, including children, or that the current scene contains multiple subjects, including people and animals, the target image captured from the preview stream of the current scene is saved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a shooting method, terminal device, and computer-readable storage medium. Background Technology

[0002] Terminal devices, such as the cameras on mobile phones, can often capture beautiful scenes. However, when a user wants to freeze a beautiful moment, there is a certain reaction time between pressing the shutter button and actually capturing the image. Often, within this reaction time, the wonderful moment has already passed. This is especially true when shooting moving subjects, often resulting in missing the perfect instant or even capturing a distorted or blurry image, leading to a poor user experience. Summary of the Invention

[0003] Based on this, this application provides a shooting method, a terminal device, and a computer-readable storage medium, which solves the problem of missing the recording of wonderful moments of an image due to reaction time, ensures the output of wonderful images, and improves the user experience.

[0004] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0005] Firstly, this application provides a shooting method applicable to a terminal device including a camera module. In this method, the terminal device receives an operation from a user to open a camera application, and based on this operation, captures a preview stream of the current scene through the camera module. If the current scene meets a first condition, the target image captured in the preview stream of the current scene is saved. The first condition includes either the current scene including multiple subjects, and a child being among the multiple subjects; or the first condition includes the current scene including multiple subjects, and a person and an animal being among the multiple subjects.

[0006] The method described above automatically captures images after identifying a scene that meets certain criteria, such as including multiple subjects and children, or including multiple subjects and people and animals. This eliminates the need to consider reaction time and avoid missing crucial moments. It also caters to current user needs for recording human-pet scenes and parent-child scenes, ensuring a positive user experience.

[0007] Additionally, preview stream capture is achieved through a camera module installed on the terminal device, enabling automatic image capture. This camera module can include the terminal device's front-facing camera and / or rear-facing camera.

[0008] This implementation method can meet users' needs for automatic snapshots using the front and / or rear cameras, freeing up their hands and allowing them to capture wonderful moments with their children or animals.

[0009] In one possible implementation of the first aspect, determining whether the current scene is a scene that meets the conditions mainly involves determining the number of subjects included in the current scene and whether the current scene includes people. If the number of subjects is greater than a first threshold and the current scene includes people, it is further determined whether the current scene includes children or animals. The first threshold can be an integer greater than 1.

[0010] The method of determining whether a scene meets the criteria by checking whether the number of subjects exceeds a first threshold and whether the current scene includes people, and by checking whether the current scene includes children or animals if the number of subjects meets the first threshold and whether the current scene includes people, is more accurate.

[0011] Determining whether the current scene includes a person can be achieved through subject detection on the images in the preview stream, while determining whether the current scene includes a child or animal can be achieved through face detection on the images in the preview stream. As an example, the subject detection and face detection can be performed on every frame of the preview stream, or they can be performed on images every few frames, thus saving power. Furthermore, the number of frames for subject detection and face detection can be the same or different. For example, subject detection can be performed every four frames, and face detection can be performed every five frames.

[0012] In one possible implementation of the first aspect, information about the current scene can be determined based on a deep learning network. For example, images included in the preview stream can be input into the deep learning network for processing, and then image feature information can be output. Image feature information characterizes the information of the current scene. In one implementation, the image feature information can be used to indicate the number and type of subjects included in the current scene, with the subject type indicating whether the subjects included in the current scene are children or animals. Thus, the number of subjects included in the current scene, whether the current scene includes people, and whether the current scene includes children or animals can be determined based on the image feature information output by the deep learning network. In another implementation, the image feature information can be used to indicate the number of subjects included in the current scene. If the current scene includes people, the image feature information can also be used to indicate the age characteristics of the people included in the current scene to determine whether the current scene includes children. If the current scene includes animals, the image feature information can also be used to indicate whether the subjects included in the current scene are animals to determine whether the current scene includes animals.

[0013] This implementation method only requires inputting the image into the deep learning network to output information about the current scene, which is used to determine whether the current scene meets the conditions, making it more efficient.

[0014] In one possible implementation of the first aspect, when the current scene is a scene that meets the conditions, that is, when the current scene includes multiple subjects and there are children among the multiple subjects; or when the current scene includes multiple subjects and there are people and animals among the multiple subjects, before saving the target image in the preview stream of the current scene, the faces of the subjects in the current scene can also be matched, that is, it is determined whether the face angle of the subjects in the image included in the preview stream is within a preset range.

[0015] This implementation method, when the current scene meets the conditions, detects the facial angle of the subject in the current scene, and performs the operation of saving the target image in the preview stream when the facial angle of the subject is within the preset range. This ensures that the captured image includes the subject's face, guaranteeing a wonderful image presentation.

[0016] In one possible implementation of the first aspect, if the current scene is a scene that meets the conditions, that is, the current scene includes multiple subjects and there are children among the multiple subjects; or if the current scene includes multiple subjects and there are people and animals among the multiple subjects, the operation of saving the target image in the preview stream can be performed if the exposure time and / or image sharpness of the image included in the preview stream are determined, that is, the exposure time of the image is less than a second threshold and / or the image sharpness of the image is greater than a third threshold.

[0017] This implementation method ensures that the captured image is clear and accurate by triggering the photo-taking decision logic when the exposure time and image sharpness meet the conditions.

[0018] In one possible implementation of the first aspect, upon triggering the photo-taking decision logic, multiple frames of images can be cached, and a target image can be selected from the cached multiple frames of images based on the facial angle of the subject in the image being within a preset range; and / or the image of the subject in the image being located at a preset position in the image.

[0019] In this implementation, the image parameters in multiple frames need to meet preset angles and preset positions to ensure that the proportion of the main image to the image in the captured image is appropriate, ensuring the quality of the target image and thus meeting the user's needs.

[0020] Secondly, this application provides a shooting device that has the function of implementing the terminal device behavior in the method described in the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function, such as an input unit or module, a display unit or module, and a processing unit or module.

[0021] Thirdly, a terminal device is provided, the electronic device including: a processor; a memory; a camera module; and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, it causes the terminal device to perform the shooting method as described in the first aspect and any implementation thereof.

[0022] Fourthly, a computer-readable storage medium is provided, comprising a computer program that, when executed on an electronic device, enables the electronic device to perform the method described in any one of the first aspects.

[0023] Fifthly, a computer program product containing instructions is provided, which, when run on an electronic device, enables the electronic device to execute the shooting method described in the first aspect and any implementation thereof.

[0024] In a sixth aspect, embodiments of this application provide a chip, the chip including a processor, the processor being configured to invoke a computer program in memory to execute the imaging method as described in the first aspect and any implementation thereof.

[0025] Understandably, the beneficial effects that the apparatus described in the second aspect, the terminal device described in the third aspect, the computer-readable storage medium described in the fourth aspect, the computer program product described in the fifth aspect, and the chip described in the sixth aspect can achieve can be referred to the beneficial effects of the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0026] Figure 1 An illustration of automatic image capture provided for related technologies. Figure 1 ;

[0027] Figure 2 An illustration of automatic image capture provided for related technologies. Figure 2 ;

[0028] Figure 3 An illustration of automatic image capture provided for related technologies. Figure 3 ;

[0029] Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of a shooting method provided in an embodiment of this application;

[0031] Figure 6 A schematic diagram of a shooting interface provided in an embodiment of this application;

[0032] Figure 7This is a schematic diagram of the overall process of a shooting method provided in an embodiment of this application;

[0033] Figure 8 This is a flowchart illustrating a process for determining whether a current scenario meets certain conditions, as provided in an embodiment of this application.

[0034] Figure 9A A scene illustration of a shooting method provided in an embodiment of this application. Figure 1 ;

[0035] Figure 9B A scene illustration of a shooting method provided in an embodiment of this application. Figure 2 ;

[0036] Figure 10 A flowchart illustrating another method for determining whether the current scenario meets the conditions, provided in an embodiment of this application;

[0037] Figure 11 A flowchart illustrating the process of determining a target image provided in an embodiment of this application;

[0038] Figure 12 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0039] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0040] Since its inception, photography has created countless wonderful moments and brought people diverse visual enjoyment. With the development of technology, camera modules can be installed on a variety of terminal devices, providing convenience for people. Initially, users could use camera modules on terminal devices to freeze wonderful moments, such as shooting some static scenes. However, as users' needs have increased, there is also a need to shoot some moving scenes and record beautiful moments. When capturing moving scenes, users often need to operate the shutter button on the terminal device. However, when users see a wonderful moment they want to freeze, there is often a reaction time between pressing the shutter button and the moment, and since the wonderful moment is fleeting, it is often difficult to freeze it. For example... Figure 1 As shown, the user wants to capture the subject of the shot (such as...). Figure 1 The state of the person shown in the picture as they stretch to the left and right. Figure 1The third image in the image. During this process, if the shutter button is pressed when the scene to be captured is seen by the human eye, the actual captured image is often not what the human eye sees due to the human reaction time, which is usually 50ms. For example, the captured image may be the one that stretches to the left.

[0041] like Figure 2 As shown, current solutions for capturing images on terminal devices primarily involve using the rear camera to identify moving subjects within the shooting range and automatically capturing their dynamic movements. However, the scenarios for capturing motion scenes using the rear camera are currently limited, only applicable to scenes with significant movement, such as playing ball. This is insufficient for scenes with less movement, and current automatic capture methods only work with the rear camera; the front camera cannot perform automatic capture. Furthermore, when using the front camera for selfies, it may fail to capture certain memorable moments. For example… Figure 3 In the human-pet scene shown, the animal is quite active, and the adult needs to control the animal while also pressing the shutter button to take a selfie, making it difficult to ensure the image quality.

[0042] Based on the above, this application provides a shooting method. After receiving a user's instruction to start shooting, if the terminal device determines that the current scene is a human-pet scene or a parent-child scene, it can use the image in the preview stream of the current scene that meets the judgment conditions as the final output target image. This shooting method can automatically capture high-quality images during the shooting process, thereby meeting the user's needs for capturing various operating scenarios, including motion scenes with relatively small movements, and improving the user experience.

[0043] For example, the terminal device in this application embodiment may be a mobile phone, tablet computer, smartwatch, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc., including a camera module. This application embodiment does not impose any special restrictions on the specific form of the electronic device.

[0044] For example, taking a mobile phone as an example of the aforementioned terminal device 400, Figure 4 A schematic diagram of the mobile phone's structure is shown. (For example...) Figure 4As shown, a mobile phone may include a processor 410, an external memory interface 420, an internal memory 421, a mobile communication module 430, a wireless communication module 440, a charging management module 450, a power management module 460, a battery 470, an antenna 1, an antenna 2, an audio module 480, a speaker 480A, a receiver 480B, a microphone 480C, a headphone jack 480D, a camera 490, a display screen 491, etc.

[0045] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile phone. In other embodiments of this application, the mobile phone may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. For example, the mobile phone may also include: a subscriber identification module (SIM) card interface, a sensor module, etc. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0046] Processor 410 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0047] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0048] The processor 410 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 410 is a cache memory. This memory can store instructions or data that the processor 410 has just used or that are used repeatedly. If the processor 410 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 410, and thus improves the efficiency of the system. In some embodiments, the processor 410 may include one or more interfaces.

[0049] The charging management module 450 receives charging input from a charger, which can be either a wireless charger or a wired charger. While charging the battery 470, the charging management module 450 can also supply power to the terminal device via the power management module 450.

[0050] The power management module 460 is used to connect the battery 470, the charging management module 460, and the processor 410. The power management module 460 receives input from the battery 470 and / or the charging management module 450, and supplies power to the processor 410, internal memory 421, display screen 400, camera 490, and wireless communication module 440, etc. The power management module 460 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 460 may also be located within the processor 410. In other embodiments, the power management module 460 and the charging management module 450 may be located in the same device.

[0051] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, mobile communication module 430, wireless communication module 440, modem processor, and baseband processor.

[0052] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0053] The mobile communication module 430 can provide solutions for wireless communication applications in mobile phones, including 2G / 3G / 4G / 5G. The mobile communication module 430 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 430 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation.

[0054] The wireless communication module 440 can provide solutions for wireless communication applications in mobile phones, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0055] In some embodiments, the mobile phone's antenna 1 is coupled to the mobile communication module 430, and the antenna 2 is coupled to the wireless communication module 440, enabling the terminal device 400 to communicate with the network and other devices via wireless communication technology.

[0056] The terminal device 200 implements display functions through a GPU, a display screen 491, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 491 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 410 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0057] Display screen 491 is used to display images, videos, etc. Display screen 491 includes a display panel. For example, display screen 491 may be a touch screen. In some embodiments of this application, after a user opens a camera application, display screen 491 may be used to display a preview stream captured by camera 490.

[0058] Camera 490 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the mobile phone may include one or N cameras 490, where N is a positive integer greater than 1. In some embodiments of this application, after the mobile phone receives an operation from the user to open the camera application, the mobile phone can control the camera 490 to turn on. After the camera 490 is turned on, it can be used to capture a preview stream of the current scene. In addition, in some embodiments of this application, when the mobile phone includes multiple cameras 490, these multiple cameras 490 may include a front-facing camera and / or a rear-facing camera.

[0059] Internal memory 421 can be used to store computer executable program code, which includes instructions. Internal memory 421 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during mobile phone use (such as audio data, phonebook, etc.). Furthermore, internal memory 421 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 410 enables the mobile phone to implement audio functions through audio module 480 and processor 410 by executing instructions stored in internal memory 421 and / or instructions stored in memory located in the processor. Examples include music playback and recording. Audio module 480 includes speaker 480A, receiver 480B, microphone 480C, and headphone jack 480D.

[0060] The audio module 480 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 480 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 480 may be located in the processor 410, or some functional modules of the audio module 480 may be located in the processor 410.

[0061] The speaker 480A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 400 can listen to music or make hands-free calls through the speaker 480A.

[0062] The receiver 480B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 400 answers a phone call or voice message, the receiver 480B can be brought close to the listener's ear to hear the voice.

[0063] Microphone 480C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 480C, inputting the sound signal into microphone 480C. A mobile phone can have at least one microphone 480C. In some embodiments, a mobile phone can have two microphones 480C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, a mobile phone can have three, four, or more microphones 480C, enabling sound signal collection, noise reduction, sound source identification, and directional recording, among other functions.

[0064] The 480D headphone jack is used to connect wired headphones. The 480D headphone jack can be a USB 410 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0065] The following describes the specific process of shooting using a terminal device. For example... Figure 5 As shown, Figure 5 This application provides a schematic flowchart of a shooting method. The method is applied to a terminal device including a camera module. Taking a mobile phone as an example, the method includes steps S501-S503.

[0066] S501: The mobile phone receives the user's command to open the camera application.

[0067] The phone can have a camera app installed for users to take photos. In some embodiments, with automatic capture mode enabled by default, opening the camera app can be done by interacting with the camera app icon. For example, the phone's home screen may include a camera app icon, which users can interact with, such as by clicking. Correspondingly, the phone can receive user interactions with the camera app icon.

[0068] In some other embodiments, when the automatic capture mode is not enabled by default, opening the camera application may include operating on the camera application icon and enabling the automatic capture mode. For example, a user can operate on the camera application icon displayed on the desktop. Correspondingly, the mobile phone can receive the user's operation on the camera application icon. In response, the mobile phone can access the camera application interface. The camera application interface may include a control for enabling the automatic capture mode. The user can operate on this control to enable the automatic capture mode. It should be noted that this embodiment is described using the example of the control for enabling the automatic capture mode being included in the camera application interface. The control for enabling the automatic capture mode can also be set in other interfaces, such as the settings application interface; this application embodiment does not impose specific limitations on this. It is understood that in this embodiment, the photo-taking decision logic can be triggered after the user enables the automatic capture mode, which can save device power consumption.

[0069] S502, in response to an operation, the phone captures a preview stream of the current scene through the camera module.

[0070] For example, after receiving a user's command to open the camera app, the phone can respond by displaying the camera app's interface, such as a preview screen. Additionally, in response, the phone can also activate its camera module to capture a preview stream of the current scene. For instance, after receiving the user's command to open the camera, the camera module activates, capturing the light in the current scene to create a real-time preview stream of the current scene that the user can see. The phone can also display the preview stream of the current scene captured by the camera module in the aforementioned preview screen.

[0071] The preview stream is a video stream composed of multiple preview images. It's essentially a real-time recording of the current scene by the camera module, displayed on the phone's preview screen as a video stream of the current scene, awaiting capture. The camera module typically possesses basic functions such as video recording / transmission and still image capture. It captures images through a lens, and then the image is processed by the internal photosensitive components, circuitry, and control components, converting the image into a digital signal that the terminal device can recognize. This internal processing then reconstructs the image. The reconstructed image can be displayed to the user on the phone's screen. For example, combined with... Figure 6 The current scene is a puppy walking forward. The preview stream, captured by the phone's camera module, is a video stream composed of multiple frames depicting the puppy walking forward. The phone's screen can display, for example... Figure 6 The preview interface shown includes a video stream captured by the camera module showing the dog walking forward, for the user to view. It's understandable that the preview interface is the viewfinder before the phone begins automatic image capture. The preview stream included in this interface is real-time footage of the actual scene.

[0072] In the above example, the preview stream captured by the camera module can be obtained through the rear camera of the terminal device or through the front camera of the terminal device.

[0073] S503. In response to the current scene satisfying the first condition, save the target image in the preview stream of the current scene; wherein the first condition includes: the current scene includes multiple subjects, and children are among the multiple subjects; or the first condition includes: the current scene includes multiple subjects, and people and animals are among the multiple subjects.

[0074] For example, users often want to capture memorable moments with their children, but most children are naturally active and restless. Even with a good camera module, it's possible to not get the photos they want. Furthermore, when using the front-facing camera for selfies or rear-facing cameras for group photos, users need to control the child's movements. Therefore, the device's automatic snapshot function is necessary to free up adults' hands and meet user needs. Similarly, animals, like children, are also active and restless, and sometimes even more agile than children, making them difficult to control. To take selfies or group photos with their animals using the front or rear cameras, the device's automatic snapshot function is also needed. To allow users to capture photos with their animals or children during shooting, the device needs to select suitable scenes from a wide range of options.

[0075] Among numerous scenarios, determining which ones meet the criteria primarily involves two conditions. In some embodiments, one condition is whether the current scene contains multiple subjects, and the other is whether a child is among those subjects. If both conditions are met, the current scene is considered a qualifying scenario, such as a parent-child scene (adult and child), a photo of two children, or a pet scene (child and animal). In other embodiments, one condition is whether the current scene contains multiple subjects, and the other is whether both a person and an animal are among those subjects. If both conditions are met, the current scene is considered a qualifying scenario, such as a pet scene (person and animal). Here, the person can be either a child or an adult.

[0076] In this embodiment, the subject refers to a person or animal in the current scene captured by the camera module. For example, an adult, a child, a cat, a dog, etc. For instance, assuming the current scene is "a user squatting down outdoors to pet their pet dog," the preview stream acquired by the camera module includes not only the person and the pet dog, but also the road and roadside scenery. In this case, the subject refers to the person and the pet dog in that scene.

[0077] like Figure 7As shown, after capturing the preview stream, the mobile phone can perform scene-aware calculations on the images included in the preview stream to determine the number of subjects in the current scene and whether the current scene includes people. If the number of subjects is greater than a first threshold and the current scene includes people, it can further determine whether the current scene includes children or animals. The first threshold can be an integer greater than 1. As an example, subject detection can be performed on the preview stream to determine the number of subjects in the current scene and whether the current scene includes people. The specific judgment process is as follows... Figure 8 As shown, subject detection can be performed on the preview stream to determine whether the current scene includes multiple subjects and whether it includes people. For example, it can determine if the number of subjects in the current scene is greater than a first threshold (e.g., whether the number of subjects in the current scene is greater than or equal to 2) to determine if the condition of multiple subjects is met. After the phone determines that the condition of multiple subjects is met and that the current scene also includes people, it can further determine whether the current scene includes children or animals. Specifically, the phone can first determine whether children are included; if not, it then further determines whether animals are included. That is, in the logical judgment, the priority of judging whether a subject is a child is higher than the priority of judging whether a subject is an animal. For example, continuing to combine... Figure 8 The system can perform face detection on the preview stream to determine if the current scene includes children (e.g., check if the number of children in the current scene is greater than 0). If children are included (e.g., the number of children in the current scene is greater than 0), the scene is considered a valid scenario, such as a parent-child scene. If children are not included, the system can check if animals are included (e.g., check if the number of animals in the current scene is greater than 0). If animals are included (e.g., the number of animals in the current scene is greater than 0), the scene is considered a valid scenario, such as a pet scene. If the number of subjects is less than 2, or the number of subjects is greater than or equal to 2 but no people are present, or the number of subjects is greater than or equal to 2 and people are present but no children or animals are present in the current scene, the current process can be terminated.

[0078] It should be noted that in the above example, children have a higher priority than animals, but it can also be set so that animals have a higher priority than children; the order is not limited here.

[0079] The following example demonstrates how to determine whether the current scenario is a parent-child scenario or a pet-human scenario.

[0080] In some examples, the current scenario is taken as a parent-child scenario. Let's assume that the current parent-child scenario includes one adult and one child. Combined with... Figure 8As shown, after the phone captures the preview stream through its camera module, it performs subject detection on the images included in the preview stream to obtain a result showing two subjects and that the current scene includes people. Then, the phone continues to perform face detection based on the images in the preview stream to determine if a child is present among the subjects in the current scene. Face detection refers to acquiring facial feature information of the subjects, which can be used to determine if a child is present. Therefore, the current scene can be determined to be a parent-child scene, thus qualifying it as a suitable scene. The phone can then automatically save the target image from the preview stream of the current scene, enabling automatic snapshot capture. Figure 9A As shown, Figure 9A The current scene is shown to be a parent-child scene.

[0081] In other examples, taking a human-pet scenario as an example, let's assume that the current human-pet scenario includes an adult and a pet. Combined with... Figure 8 As shown, after the phone captures the preview stream through its camera module, it performs subject detection on the images included in the preview stream, obtaining a result of 2 subjects and the presence of people in the current scene. Next, the phone continues to perform face detection based on the images in the preview stream. If it determines that there are no children among the subjects in the current scene, it then determines that there are animals among the subjects, thus classifying the current scene as a human-pet scene, and thus confirming that the current scene meets the criteria. The phone can then automatically save the captured images from the preview stream of the current scene, achieving automatic snapshot capture. Figure 9B As shown, Figure 9B The current scene is a human-pet scenario.

[0082] Furthermore, the determination of the number of subjects and whether the current scene includes people, children, or animals is based on the output of the deep learning network. Specifically, after the mobile phone captures the preview stream through the camera module, it can process the preview stream through a deep learning network, outputting image feature information of the images in the preview stream. This image feature information indicates the number and type of subjects included in the current scene. The subject type indicates whether the subjects included in the current scene are children or animals; whether the current scene includes people or children / animals is determined based on the subject type. Based on the output of the deep learning network, the mobile phone can determine whether the current scene meets the criteria. For example, the camera module captures the current scene, obtaining a preview stream. The images included in the preview stream are input into a deep learning network for processing, yielding the processed output. The output shows that the number of subjects is 2, including animals and children. Based on the above two conditions, the current scene is determined to be a qualified scene.

[0083] In other embodiments, the output of the deep learning network, besides maintaining the same number of subjects, may not directly indicate whether the subject includes animals or children. Instead, when the subject is a human, the subject type can be the subject's age characteristic, indirectly indicating whether the subject is a child. The final output can be the number of subjects and the subject's age characteristic. The mobile phone can then match the subject's age characteristic with the corresponding characteristics of the target scene to determine if the current scene meets the criteria. For example, taking a child as an example, the mobile phone can match the subject's age characteristic with the age characteristics of subjects in a parent-child scene to determine if the current scene includes children, thereby determining whether the current scene is a parent-child scene that meets the criteria. Taking the above scenario as an example, assuming the number of subjects detected is two, and one of the subjects is a human with an output age characteristic of 8 years old, the age characteristic is matched with the characteristics of the target scene. The age characteristic corresponding to children in the target scene is less than or equal to 13 years old. The match shows that 8 years old falls within the range of the target characteristic of 13 years old, therefore, the scene is determined to be a parent-child scene that meets the criteria. Additionally, if the current scene includes animals, you can also input an indication of whether the current scene includes animals to determine whether the current scene includes animals.

[0084] Furthermore, the above embodiments illustrate how the mobile phone determines the current scene to be a suitable scene when it meets the conditions of the number and type of subjects, thus triggering automatic capture. In other embodiments of this application, in addition to using the above two conditions as triggering automatic capture, the condition of whether the subject in the current scene is looking at the camera module can also be used as a triggering condition, so as to capture images including the subject's face. Figure 10 As shown, as an example, before saving the target image in the preview stream of the captured current scene, it is possible to further determine whether the facial angle of each of the multiple subjects in the current scene is within a preset range, i.e., whether the subject in the image included in the preview stream is a frontal face. Automatic capture will only be triggered if the facial angle of each of the multiple subjects is determined to be within the preset range. The preset range can be [0°, 45°]. Here, 0° means that the subject's eyes are looking directly at the center point of the camera module; 45° means that the angle between the subject's eyes and the camera module is 45 degrees. It should be noted that the 45° deviation angle includes the angles of 45 degrees up, down, left, and right to the center point of the camera module. It should be noted that the above example is based on first determining whether the current scene meets the conditions, and then determining whether the facial angle of each of the multiple subjects is within the preset range. However, the determination of the above conditions is not sequential, or can be performed simultaneously. This embodiment of the application does not impose any restrictions here.

[0085] In some examples, taking a parent-child scene as an example, suppose the scene includes one adult and one child, and both the adult's and child's faces are at an angle of 30°. Referring to Figure 10, after the phone captures the preview stream through the camera module, it can perform subject detection on the images in the preview stream to obtain the number of subjects (2) and whether the current scene includes people. Then, the phone continues to perform face detection based on the images in the preview stream to further determine that a child is present among the subjects in the current scene. It further determines whether the face angle of each subject in the current scene is within a preset range (e.g., each subject's face angle is 30°, within the preset range). Therefore, the current scene can be determined to be a parent-child scene, thus confirming that the current scene meets the criteria. The phone can then automatically save the captured images from the preview stream of the current scene, achieving automatic snapshot capture.

[0086] In other examples, taking a human-pet scene as an example, suppose the scene includes an adult and an animal, and the facial angles of both the adult and the animal are at 40°. Referring to Figure 10, after the phone captures the preview stream through the camera module, it can perform subject detection on the images in the preview stream to obtain the number of subjects (2) and whether the current scene includes a human. Then, the phone continues to perform face detection based on the images in the preview stream, determining that there are no children among the subjects in the current scene, further determining that there are animals among the subjects, and then determining whether the facial angle of each subject in the current scene is within a preset range (e.g., each subject's facial angle is 40°, within the preset range). This determines that the current scene is a human-pet scene, thus confirming that the current scene meets the conditions. The phone can then automatically save the captured images from the preview stream of the current scene, achieving automatic snapshot capture.

[0087] It should be noted that when determining whether the current scene meets the criteria based on the images included in the preview stream, detection can be performed on every frame of the preview stream or on a subset of frames. When detecting each frame, a fusion algorithm is used to process multiple frames into a single frame for subject identification. For partial frame detection, a subset of frames with a consistent pattern is selected. For example, when determining the number of subjects in the current scene, assuming there are sixteen frames, a fusion algorithm can be used to process four frames to obtain four more frames for subject identification; alternatively, frames four, eight, twelfth, and sixteenth can be selected systematically for subject identification. After determining the number of subjects in the current scene, when further determining the subject type through face detection, detection can be performed every five frames, thus saving power without affecting scene determination. Furthermore, the number of frames detected for subject detection and face detection can be the same or different. For example, subject detection and face detection can also be performed every four frames.

[0088] Continue to combine Figure 7 As shown, once the terminal device detects that the current scene meets the conditions, it can trigger the photo-taking decision logic, or in other words, trigger automatic capture, that is, automatically save the target image in the preview stream of the current scene. Furthermore, the timing of triggering the photo-taking decision logic is determined based on at least one of image sharpness or exposure time. This is because when the camera module captures an image of the current scene, its internal processing mainly occurs at the moment of capture, where the camera module receives the light reflected from the object and focuses it onto the film, forming an inverted, reduced real image. During this process, factors such as the effectiveness of light reception affect image sharpness; in other words, image sharpness is a crucial indicator of image quality. If the image is not sharp enough, even if the photo-taking logic is triggered and the captured image is obtained, the resulting image will still not meet the user's expectations and will still be considered a failed image. Therefore, image sharpness becomes a condition for determining whether to trigger the photo-taking decision logic. In addition, since exposure time is related to image sharpness—that is, the longer the exposure time, the greater the possibility of motion blur and shaky blur, and the more difficult it is to capture a sharp image. Therefore, after recognizing the current scene as meeting the conditions, meaning that triggering the photo-taking decision logic also requires a certain exposure time, it's necessary to keep the exposure time as short as possible to ensure normal photo taking. In summary, the photo-taking decision logic can be triggered if at least one of the above conditions is met.

[0089] Specifically, regarding image sharpness, the image capture decision logic is triggered only if the image sharpness exceeds the third threshold. Regarding image exposure time, the image exposure time must be less than the second threshold; that is, the image capture decision logic is triggered to take a picture only if the exposure time meets the second threshold.

[0090] In some instances, after determining that the current scene meets the criteria, the phone further assesses the image within that scene to trigger a photo-taking decision. Specifically, it determines whether the image sharpness in the preview stream, including the image itself, exceeds a third threshold (represented by L, where L can be set to a probability value greater than 0.75). If the sharpness exceeds the third threshold, the photo-taking decision is triggered. This third threshold is determined based on multiple parameter values, such as edge texture intensity.

[0091] In other instances, after determining that the current scene meets the criteria, the phone further analyzes the image within that scene to trigger a photo-taking decision. Specifically, it checks whether the exposure time of the image in the preview stream exceeds a second threshold (represented by H, which can be set to less than 1 / 60ms). If the exposure time exceeds the second threshold, the photo-taking decision is triggered. This second threshold is also determined based on multiple parameters, such as the subject's speed and the distance between the subject and the camera module.

[0092] In some other embodiments, exposure time and image sharpness can be limited simultaneously to ensure that the final output target image presents a better result. Combined with Figure 11 As shown, after determining that the current scene meets the criteria, the phone further considers whether the image sharpness is greater than a second threshold and whether the exposure time is less than a second threshold when making the decision to trigger a photo. In other words, when determining that the current scene meets the criteria, it checks whether the image exposure time is less than the second threshold (e.g., 1 / 60ms). If the image exposure time is less than the second threshold, it further checks whether the image sharpness is greater than the second threshold (e.g., N). If the image sharpness is greater than the second threshold, the photo-taking decision is triggered; if the exposure time is greater than the second threshold or the image sharpness is less than the second threshold, the decision to trigger the photo-taking decision is re-evaluated.

[0093] It should be noted that the above example determines the exposure time first, then the image sharpness. Alternatively, the image sharpness can be determined first, followed by the exposure time; the order is not strictly limited. Furthermore, the specific values ​​for exposure time and image sharpness mentioned above are merely examples; other values ​​for exposure time and image sharpness are also possible and are not the only ones specified here.

[0094] Furthermore, continue to combine Figure 7As shown, after triggering the photo-taking decision logic, the current scene can be automatically captured. Since moments of brilliance often occur in a fleeting instant, it's difficult to guarantee that the frame triggering the decision logic will be the most stunning image. Therefore, to further improve the quality of the final captured image, the highest quality image within a certain range can be selected as the target image. This can be achieved through frame selection, choosing the most outstanding frame as the target image.

[0095] In some embodiments, before the phone triggers the photo-taking decision logic, the cached image frames can be each frame in the cached preview stream. In other embodiments, before triggering the photo-taking decision logic, the phone automatically caches a certain number of image frames, and continuously updates the cached image frames while maintaining a constant number. That is, the number of image frames cached by the phone before the photo-taking decision logic is triggered is fixed and does not accumulate indefinitely. For example, suppose the phone is set to cache 10 image frames, i.e., the cached image frames are frames 1 through 10. At the eleventh frame, the photo-taking decision logic has not been triggered, so the cached image frames are updated, i.e., frames 2 through 11 are now cached. Specifically, when the photo-taking decision logic is triggered, the phone can determine a certain number of image frames within the range before and after the target frame from the cached image frames. The target frame can be the image frame that triggers the photo-taking decision logic. For example, assuming there are 10 images within 5 seconds (frames A, B, C, D, E, F, G, H, I, and J), the photo-taking decision logic is triggered in the 5th frame, which is frame E, and thus the target frame. Afterwards, image frames within 1.5 seconds before and after the target frame (frame E) that triggers the photo-taking decision logic can be cached. This results in the final cached image frames corresponding to 1 to 4 seconds, i.e., frames B to H. The phone can then select "highlighted frames" from the cached image frames. "Highlighted frame selection" means that the image parameters must meet any one or both of the following second conditions to be output as the final target image. The second conditions include: the facial angle of the subject in the image is within a preset range, and the subject's image is located at a preset position in the image. The restriction on the subject's image being located at a preset position in the image is because the position of the subject's image within the image needs to be guaranteed when outputting an image. In other words, it determines the relative position of the image size to the size of the subject image in the image. If the subject size is too large or too small, or if the position is too far off-center, it will not be aesthetically pleasing and cannot be defined as the target image.

[0096] In some examples, after the phone captures the preview stream, it identifies the images included in the preview stream to determine if the current scene meets the criteria. Then, based on the images in the preview stream for the current scene, the phone determines when the criteria are met and triggers a photo-taking decision logic. It also determines a certain number of image frames within a range before and after the target frame corresponding to the triggering decision logic; that is, it can cache image frames within 1.5 seconds before and after the frame corresponding to the triggering decision logic. The final cached image frames are the target frames corresponding to 1 to 4 seconds, namely, frame B, frame C, frame D, frame E, frame F, frame G, and frame H. Then, a highlight frame is selected from these seven target frames as the final target image. For example, it checks whether the facial angle of the subject in these seven target frames is within a preset range, i.e., whether the facial angle is within [0°, 45°], thus meeting the selection requirements for a highlight frame. If not, the process ends, and the phone re-executes the preview stream capture operation for the current scene.

[0097] In other examples, after the phone captures the preview stream, it identifies the images included in the preview stream to determine if the current scene meets the criteria. Then, based on the images in the preview stream within the current scene, the phone can determine if the criteria are met and trigger a photo-taking decision logic, identifying a certain number of target frames within a range before and after the target frame that triggers the decision logic. Then, it selects a highlight frame from these target frames as the final target image. For example, it checks if the subject image in these target frames is located at a preset position in the image, i.e., whether it is centered in the image. If it is, it is identified as a highlight frame and output as the target image. If it is not, the processing ends, and the phone re-executes the preview stream capture operation for the current scene. For example, suppose the goal is to output a 6-inch image, with a 6-inch x 4-inch aspect ratio. In this case, the preset range can be set so that the ratio of the subject image to the image is 2:3, meaning that within a 3-inch x 2-inch range, the subject image is centered in the image, thus ensuring that the image corresponding to that frame is presented as a highlight image, i.e., the target image.

[0098] In some examples, the facial angle of the subject in the image can be limited within a preset range, and the subject's image can be positioned at a preset location within the image, to ensure a better final output image. For instance, after the phone captures the preview stream, it identifies the images included in the preview stream and determines that the current scene meets the criteria. Then, based on the images in the preview stream within the current scene, the phone determines when the conditions are met and triggers a photo-taking decision logic. It then determines a certain number of image frames within the range before and after the target frame corresponding to the triggering decision logic; that is, it can cache image frames within 1.5 seconds before and after the frame corresponding to the triggering decision logic. The final cached image frames are the target frames corresponding to 1 to 4 seconds, namely, frame B, frame C, frame D, frame E, frame F, frame G, and frame H. Then, a highlight frame is selected from these seven target frames as the final target image. First, determine if the facial angle of the subject in the 7 target frames is within a preset range, i.e., whether the facial angle is within [0°, 45°]. Assuming that both frame A and frame B meet the preset range, further determine if the subject image is located at a preset position in the image. If only frame A meets the preset range, then frame A is output as the final target image. If the facial angle of the subject in frames A and B is not within the preset range, or if the subject image is not located at the preset position in frames A and B, then no target image has been acquired, the selection of highlights ends, and the phone re-captures the preview stream.

[0099] It should be noted that the above example first determines whether the subject's facial angle is within a preset range, and then further determines whether the subject's image is located in a preset position of the image. Alternatively, it can first determine whether the subject's image is located in a preset position of the image, and then determine whether the subject's facial angle is within a preset range. The specific order is not limited here.

[0100] In summary, after receiving a user's request to start taking a picture, if the terminal device determines that the current scene is a human-pet scene or a parent-child scene, it can use the image in the preview stream of the current scene that meets the judgment criteria as the final output target image. Whether using the terminal device's front or rear camera, it can meet the user's needs in real-world scenarios. Especially for group selfies using the front camera, it can free the user's hands, providing convenience and improving the user experience.

[0101] Other embodiments of this application provide a terminal device, which may include: the aforementioned display screen, camera, memory, and one or more processors. The display screen, camera, memory, and processor are coupled. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the mobile phone in the above method embodiments. The structure of the electronic device can be referred to... Figure 4 The structure of the mobile phone is shown.

[0102] This application also provides a chip system, such as... Figure 12 As shown, the chip system 1200 includes at least one processor 1201 and at least one interface circuit 1202. The processor 1201 and the interface circuit 1202 are interconnected via lines. For example, the interface circuit 1202 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1202 can be used to send signals to other devices (e.g., the processor 1201). Exemplarily, the interface circuit 1202 can read instructions stored in memory and send those instructions to the processor 1201. When the instructions are executed by the processor 1201, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0103] This application also provides a computer storage medium that includes computer instructions. When the computer instructions are executed on the electronic device, the electronic device causes the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiment.

[0104] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the mobile phone in the above method embodiments.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0107] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A photographing method characterized by comprising: Applied to a terminal device including a camera module, the method includes: Receive user input to open the camera application; In response to the operation, a preview stream of the current scene is captured by the camera module; Perform subject detection on the preview stream to obtain the number and type of subjects in the current scene, wherein the subject type is a person or an animal; When there are multiple people in the current scene, including children, or when there are people and animals in the current scene, automatic snapshot is triggered to save the target image in the preview stream; The step of triggering automatic image capture and saving the target image in the preview stream includes: Acquire multiple cached image frames within a preset time period before and after the target image frame; wherein, the target image frame is the cached image frame triggered when automatic capture is activated; The cached image frame that meets the preset conditions among the plurality of cached image frames is taken as the target image; wherein the preset conditions are: the angle of the subject's face is within a preset range, and / or the subject is located at a preset position in the image frame.

2. The method according to claim 1, characterized in that, The step of performing subject detection on the preview stream to obtain the number and type of subjects in the current scene includes: Determine the number of subjects included in the current scene and whether the current scene includes people; If the number of subjects is greater than a first threshold and the current scene includes people, determine whether the current scene includes children or animals, where the first threshold is an integer greater than 1.

3. The method according to claim 1 or 2, characterized in that, Before saving the target image in the preview stream of the current scene, the method further includes: Based on the images included in the preview stream, it is determined that the facial angle of each of the plurality of subjects is within a preset range.

4. The method according to claim 1 or 2, characterized in that, Before saving the target image in the preview stream of the current scene, the method further includes: Determine that the exposure time of the images included in the preview stream is less than a second threshold; and / or, The image sharpness of the images included in the preview stream is determined to be greater than a third threshold.

5. The method according to claim 2, characterized in that, Determining the number of subjects included in the current scene and whether the current scene includes people includes: The preview stream is processed through a deep learning network to output image feature information. The image feature information is used to indicate the number and type of subjects included in the current scene. The subject type is used to indicate whether the subjects included in the current scene are children or animals. Whether the current scene includes people is determined based on the subject type, and whether the current scene includes children or animals is determined based on the subject type.

6. The method according to claim 1 or 2, characterized in that, The camera module includes the front-facing camera or the rear-facing camera of the terminal device.

7. A terminal device, characterized in that, The terminal device includes: a memory, a camera module, and one or more processors; the memory and the camera module are coupled to the processors. The memory is used to store computer program code, which includes computer instructions; when the computer instructions are executed by the processor, the terminal device performs the method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, Includes computer instructions; When the computer instructions are executed on a terminal device, the terminal device causes the terminal device to perform the method as described in any one of claims 1-6.

Images