Shooting method and electronic equipment

Abstract

The invention provides a shooting method and electronic equipment, and relates to the technical field of artificial intelligence. According to the electronic equipment, the display picture of the camera can be adjusted in a self-adaptive manner by analyzing the face image. The method comprises the steps that the electronic equipment detects a first operation of opening a camera application, a shooting interface is displayed, the interface displays a first picture collected through a first camera, and the first camera is located on the reverse side of a display screen for displaying the shooting interface; collecting a first face image through a second camera, wherein the second camera is located on the same side of a display screen displaying a shooting interface; acquiring position information of the first face image corresponding to the first picture, wherein the position information comprises a first viewpoint position and a first relative distance; according to the position information, a second picture collected by the first camera is displayed on the shooting interface, the picture center of the second picture corresponds to a first object in the first picture indicated by the first viewpoint position, and / or the zoom multiple corresponding to the second picture is matched with the first relative distance.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a shooting method and an electronic device. Background Art

[0002] With the development of photography technology, users use the cameras in electronic devices (such as mobile phones, tablets, etc.) more and more frequently, and users can control the cameras through various interaction methods. Among them, the interaction methods include touch methods, voice interaction methods, etc.

[0003] For example, the camera shooting interface displayed on the electronic device includes several controls. The electronic device responds to the user's operation on the controls on the camera shooting interface and controls the camera to execute corresponding response events. For example, in response to the user's operation on the shooting control, the electronic device controls the camera to take a photo. However, in the touch method, generally, the user's hand needs to operate the display screen while holding the electronic device, which is likely to cause the display screen to shake, resulting in an abnormal display of the shooting interface.

[0004] For another example, during the operation of the camera application on the electronic device, in response to the keywords (such as "take a photo") included in the acquired user voice, the electronic device controls the corresponding response events of the camera. However, the response events that can be executed by controlling the camera through the voice interaction method are relatively limited, and the user needs to remember the keywords, increasing the user's memory difficulty. Summary of the Invention

[0005] To solve the above technical problems, the embodiments of the present application provide a shooting method and an electronic device. According to the technical solution provided by the embodiments of the present application, the electronic device can analyze the face image and adaptively adjust the camera display screen, so as to solve the technical problem of poor interaction effect during the shooting process of the current electronic device.

[0006] To achieve the above technical purpose, the present application provides the following technical solutions:

[0007] In a first aspect, a shooting method is provided, which is applied to an electronic device and includes: detecting a first operation to open a camera application; in response to the first operation, displaying a shooting interface, where the shooting interface displays a first image captured by a first camera, and the first camera is located on the side opposite to the display screen where the shooting interface is displayed; capturing a first face image through a second camera, where the second camera is located on the same side as the display screen where the shooting interface is displayed; obtaining position information of the first face image corresponding to the first image, where the position information includes a first viewing point position and a first relative distance; according to the position information, displaying a second image captured by the first camera on the shooting interface, where the center of the second image corresponds to a first object in the first image indicated by the first viewing point position, and / or, the zoom factor corresponding to the second image matches the first relative distance.

[0008] In some examples, the first operation is, for example, an operation of clicking on the camera application icon, a voice command indicating to start the camera application, etc. The first viewpoint position is used to indicate the position indicated by the user's line of sight in the first screen, and the first relative distance is used to indicate the distance of the user's face (or eyes) relative to the display screen.

[0009] In this way, the electronic device obtains the position information (the position information includes the first viewpoint position and the first relative distance) corresponding to the first face image of the first screen, and can recognize the position change actions of the user relative to the display screen (the viewpoint position change action and / or the relative position change action of the user and the display screen). The electronic device collects and displays a second screen corresponding to the position information according to the position information, and adaptively adjusts the camera display screen, solving the technical problem of poor interaction effect during the shooting process of the current electronic device.

[0010] According to the first aspect, obtaining the position information corresponding to the first face image of the first screen includes: obtaining the eye image, eye position, and face position corresponding to the first face image, where the eye position includes the left eye position and / or the right eye position; inputting the eye image into the first model to obtain the eye feature vector output by the first model; inputting the eye feature vector, eye position, and face position into the second model to obtain the position information output by the second model.

[0011] In some examples, the eye image, eye position, and face position are the eye image, eye position, and face position of the user operating the electronic device.

[0012] In this way, the eye feature vector obtained by the electronic device from the first model includes eye features, and the position information obtained by the electronic device from the second model based on the eye feature vector including eye features, eye position, and face position can accurately reflect the eye movement adjustment of the user relative to the display screen, and the eye movement includes viewpoint position change, distance adjustment of the eyes relative to the display screen, etc.

[0013] According to the first aspect, or any implementation manner of the above first aspect, displaying the second screen captured by the first camera on the shooting interface according to the position information includes: obtaining the position deviation of the first viewpoint position relative to the center point of the display screen; obtaining the angle to be offset of the shooting screen view angle of the first camera according to the position deviation; controlling the offset of the shooting screen view angle according to the angle to be offset to obtain the second screen; and displaying the second screen on the shooting interface.

[0014] In some examples, the first object in the first screen is located at the first viewpoint position in the first screen. After the electronic device controls the offset of the shooting screen view angle, the first object is located at the center of the screen in the second screen.

[0015] In this way, the electronic device can control the perspective shift of the shooting screen of the first camera in response to the adjustment of the user's eye movement, so that the first object that the user's eyes focus on moves to the center of the screen.

[0016] According to the first aspect, or any one of the above implementation manners of the first aspect, controlling the perspective shift of the shooting screen according to the angle to be shifted to obtain a second screen includes: acquiring a panoramic image collected by the first camera; and acquiring the second screen with a perspective shift from the panoramic image according to the angle to be shifted and the first screen.

[0017] In this way, the electronic device can acquire the second screen with a perspective shift from the panoramic image according to the angle to be shifted. Without other complex operations by the user, the operation steps of the user are simplified, and the accuracy of acquiring the second screen is improved.

[0018] According to the first aspect, or any one of the above implementation manners of the first aspect, if the first camera includes a deflectable lens, controlling the perspective shift of the shooting screen according to the angle to be shifted to obtain a second screen includes: adjusting the deflectable lens according to the angle to be shifted, and then acquiring the second screen through the first camera.

[0019] In this way, the electronic device can acquire the second screen collected by the first camera only by deflecting the deflectable lens of the first camera. Without other complex operations by the user, the operation steps of the user are simplified, and the accuracy of acquiring the second screen is improved.

[0020] According to the first aspect, or any one of the above implementation manners of the first aspect, the first screen corresponds to a first zoom ratio. Displaying the second screen collected by the first camera according to the position information includes: in the case where the first relative distance is less than or equal to a first threshold, acquiring, through the first camera, a second screen corresponding to a second zoom ratio, where the second zoom ratio is greater than the first zoom ratio; or, in the case where the first relative distance is greater than or equal to a second threshold, acquiring, through the first camera, a second screen corresponding to a third zoom ratio, where the third zoom ratio is less than the first zoom ratio, and the first threshold is less than the second threshold.

[0021] In this way, the electronic device can acquire second screens with different zoom ratios according to the change of the first relative distance between the user's eyes and the display screen. So that the user can control the zoom of the camera shooting screen only by changing the relative position between the user's eyes and the display screen. Without other complex operations by the user, the operation steps of the user are simplified, and the accuracy of controlling the zoom of the camera shooting screen is improved.

[0022] According to the first aspect, or any one of the above implementation manners of the first aspect, the shooting interface is a shooting preview interface or a video recording interface.

[0023] In some examples, the shooting preview interface is an interface displayed on the display screen of the electronic device before taking a photo or recording a video, and the video recording interface is an interface displayed on the display screen of the electronic device after starting to record a video.

[0024] In this way, the electronic device can control the camera to implement corresponding shooting functions in response to the adjustment of the user's eye movement before taking a photo or recording a video. The electronic device can also control the camera to implement corresponding shooting functions in response to the adjustment of the user's eye movement during the video recording process. This enables the user to control the electronic device to implement shooting-related functions in taking a photo or recording a video without complex operations, improving the intelligence level of the electronic device shooting and optimizing the user interaction experience.

[0025] In a second aspect, an electronic device is provided. The electronic device includes: a processor, a display screen, and a memory. The memory and the display screen are coupled to the processor. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device performs: detecting a first operation to open the camera application; in response to the first operation, displaying a shooting interface, where the shooting interface displays a first picture captured by a first camera, and the first camera is located on the opposite side of the display screen where the shooting interface is displayed; capturing a first face image through a second camera, and the second camera is located on the same side of the display screen where the shooting interface is displayed; obtaining position information of the first face image corresponding to the first picture, where the position information includes a first viewing point position and a first relative distance; according to the position information, displaying a second picture captured by the first camera on the shooting interface, where the center of the second picture corresponds to a first object in the first picture indicated by the first viewing point position, and / or, the zoom factor corresponding to the second picture matches the first relative distance.

[0026] According to the second aspect, obtaining the position information of the first face image corresponding to the first picture includes: obtaining an eye image, an eye position, and a face position corresponding to the first face image, where the eye position includes a left eye position and / or a right eye position; inputting the eye image into a first model to obtain an eye feature vector output by the first model; inputting the eye feature vector, the eye position, and the face position into a second model to obtain the position information output by the second model.

[0027] According to the second aspect, or any one of the implementation manners of the above second aspect, displaying the second picture captured by the first camera on the shooting interface according to the position information includes: obtaining a position deviation of the first viewing point position relative to the center point of the display screen; according to the position deviation, obtaining a to-be-offset angle of the shooting picture view angle of the first camera; controlling the shooting picture view angle to be offset according to the to-be-offset angle to obtain the second picture; and displaying the second picture on the shooting interface.

[0028] According to a second aspect, or any implementation manner of the above second aspect, controlling the perspective shift of the captured image according to the angle to be shifted to obtain a second image includes: acquiring a panoramic image captured by a first camera; and acquiring the second image with a perspective shift from the panoramic image according to the angle to be shifted and a first image.

[0029] According to a second aspect, or any implementation manner of the above second aspect, if the first camera includes a deflectable lens, controlling the perspective shift of the captured image according to the angle to be shifted to obtain a second image includes: after adjusting the deflectable lens according to the angle to be deflected, capturing the second image through the first camera.

[0030] According to a second aspect, or any implementation manner of the above second aspect, the first image corresponds to a first zoom ratio. Displaying the second image captured by the first camera according to the position information includes: in a case where a first relative distance is less than or equal to a first threshold, acquiring, through the first camera, a second image corresponding to a second zoom ratio, where the second zoom ratio is greater than the first zoom ratio; in a case where the first relative distance is greater than or equal to a second threshold, acquiring, through the first camera, a second image corresponding to a third zoom ratio, where the third zoom ratio is less than the first zoom ratio, and the first threshold is less than the second threshold.

[0031] According to a second aspect, or any implementation manner of the above second aspect, the shooting interface is a shooting preview interface or a video recording interface.

[0032] In a third aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program (which may also be referred to as an instruction or code). When the computer program is executed by an electronic device, the electronic device executes the method according to the first aspect or any implementation manner of the first aspect.

[0033] In a fourth aspect, a computer program product is provided. When the computer program product runs on an electronic device, the electronic device executes the method according to the first aspect or any implementation manner of the first aspect.

[0034] In a fifth aspect, a chip system is provided, including at least one processor and at least one interface circuit. The at least one interface circuit is configured to perform a transceiver function and send instructions to the at least one processor. When the at least one processor executes the instructions, the at least one processor executes the method according to the first aspect or any implementation manner of the first aspect.

[0035] In a sixth aspect, a circuit system is provided. The circuit system includes a processing circuit configured to execute the method according to the first aspect or any implementation manner of the first aspect.

[0036] The technical effects of the foregoing aspects may be referred to each other, and details are not described herein again. Brief Description of the Drawings

[0037] Figure 1 Schematic diagram of the interface of the shooting method provided by the embodiment of the present application Figure 1 ;

[0038] Figure 2 Schematic diagram of the interface of the shooting method provided by the embodiment of the present application Figure 2 ;

[0039] Figure 3 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present application;

[0040] Figure 4 Schematic diagram of the product of the electronic device provided by the embodiment of the present application;

[0041] Figure 5 Schematic diagram of the software structure of the electronic device provided by the embodiment of the present application;

[0042] Figure 6 Flowchart of the shooting method provided by the embodiment of the present application;

[0043] Figure 7 Flowchart of obtaining the eye image, eye position and face position corresponding to the first face image provided by the embodiment of the present application;

[0044] Figure 8 Schematic diagram of obtaining the eye image, eye position and face position corresponding to the first face image provided by the embodiment of the present application;

[0045] Figure 9 Flowchart of training the image encoder provided by the embodiment of the present application;

[0046] Figure 10 Flowchart of obtaining the position information corresponding to the first face image provided by the embodiment of the present application;

[0047] Figure 11 Schematic diagram of the pixel point offset of the first viewpoint position relative to the center point position of the display screen provided by the embodiment of the present application;

[0048] Figure 12 Schematic diagram of tracking the first viewpoint position provided by the embodiment of the present application;

[0049] Figure 13 Flowchart of controlling the shooting screen view angle offset of the first camera provided by the embodiment of the present application;

[0050] Figure 14 Schematic diagram of controlling the shooting screen view angle offset of the first camera provided by the embodiment of the present application;

[0051] Figure 15 Schematic diagram of realizing perspective shift of the shooting screen of the first camera by the electronic device provided in the embodiment of the present application through image calculation;

[0052] Figure 16 Schematic diagram of obtaining an image after perspective shift of the shooting screen of the first camera provided in the embodiment of the present application Figure 1 ;

[0053] Figure 17 Schematic diagram of obtaining an image after perspective shift of the shooting screen of the first camera provided in the embodiment of the present application Figure 2 ;

[0054] Figure 18 Flowchart of adjusting the screen displayed on the shooting interface provided in the embodiment of the present application;

[0055] Figure 19 Schematic diagram of adjusting the screen displayed on the shooting interface provided in the embodiment of the present application;

[0056] Figure 20 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Detailed implementation manners

[0057] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an", "the above", "the", and "this" are intended to include, for example, the expression "one or more", unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of the present application, "at least one" and "one or more" mean one or more than two (including two).

[0058] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" etc. that appear in different places in this specification are not necessarily all referring to the same embodiment, but rather mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connection" includes direct connection and indirect connection, unless otherwise stated. "First", "second" are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features.

[0059] In the embodiments of the present application, words such as "exemplarily" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplarily" or "for example" in the embodiments of the present application should not be construed as being more preferred or more advantageous than other embodiments or design solutions. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0060] In some embodiments, a camera is installed in the electronic device, and the user generally controls the camera through a touch display screen or voice interaction to implement the shooting function.

[0061] In some examples, the electronic device controls the camera to implement the shooting function in response to the user's touch operation on the display screen. Exemplarily, as Figure 1 shown, the electronic device displays a shooting preview interface 11 in response to the user's operation of starting the camera. A preview image captured and function controls for adjusting the shooting are displayed on the shooting preview interface 11. The electronic device controls the camera to execute the function corresponding to the function control in response to the user's touch operation on the above function control. For example, the electronic device executes the shooting function in response to the user's operation on the shooting control 12. Again, for example, the electronic device executes the zooming of the preview image in response to the user's two-finger zoom operation on the shooting preview interface 11, or the user's operation on the zoom control 13. The electronic device can also respond to the user's physical position rotation operation of the electronic device, align the electronic device to the desired direction, and make the shooting preview interface 11 display a preview image including the subject to be photographed corresponding to this direction.

[0062] However, in the actual operation process, due to the limited accuracy of the user's limb movements, the camera may not take good pictures. For example, the interaction between the hands and the display screen will inevitably cause the electronic device to shake, ultimately resulting in a jittery and blurred preview image displayed on the shooting preview interface, or the camera being unable to align with the subject to be photographed. Although anti-shake algorithms are built into electronic devices, the problem of poor picture quality caused by picture shaking still cannot be completely eliminated.

[0063] Another example is that since the camera of the electronic device is fixed on the back of the electronic device. Therefore, when the subject to be photographed in the shooting preview interface moves, the electronic device needs to respond to the operation of the user rotating the wrist or moving the body to align the camera with the subject to be photographed. Since the operation of the user rotating the wrist or moving the body has a time delay compared to the movement of the subject to be photographed, during the process of the electronic device controlling the camera to align with the subject in response to the above actions of the user, it is impossible to always ensure the target position of the subject in the picture, which will lead to technical problems such as poor picture quality.

[0064] In some other examples, the electronic device controls the camera to implement the shooting function in response to the user's voice interaction. Exemplarily, Figure 2 shows the setting interface for controlling the electronic device camera in response to the user's voice interaction method. As Figure 2As shown in the figure, the electronic device determines the keyword for voice - controlled photographing in response to the user's operation on the recognition keyword control 21. After the setting is completed, during the operation of the camera, when the electronic device detects that the user makes sounds of keywords such as "take a photo", "cheese", "Cheese", etc., it automatically takes a photo. The electronic device can also respond to the user's operation on the recognition volume control 22 and set to automatically take a photo when the user's voice is greater than a preset value. For example, during a group photo, when everyone says "cheese" together, when the electronic device detects that the user's volume or the ambient volume reaches a certain level, it automatically takes a photo. However, when the electronic device controls the camera to implement the photographing function in response to the user's voice interaction method, there is a technical problem of poor real - time interaction ability. For example, when the electronic device controls the camera in response to the user's voice interaction method, different from the real - time interaction method such as touch on the display screen, the electronic device needs to wait until the user finishes saying the keyword before it can control the camera to execute the corresponding function. Therefore, there is an obvious time interval between the time point when the electronic device controls the camera in response to the user's voice interaction method and the time point when the user starts to issue the voice command. Moreover, the keywords corresponding to each camera function are different, and the electronic device needs to execute different functions of the camera in response to different voice keywords. Since the number of keywords that the user can actually remember is limited, the camera functions that the electronic device can implement in response to the user's voice interaction operation are also relatively limited. And because the voice recognition technology has a low accuracy rate in a noisy environment, it is also difficult for the electronic device to respond to the user's voice interaction operation to implement the camera function in a noisy environment.

[0065] It can be seen that when the electronic device controls the camera in response to the user's touch method or voice interaction method, there are technical problems of poor control effects.

[0066] Therefore, an embodiment of the present application proposes a photographing method and an electronic device. In the embodiment of the present application, the electronic device can adaptively adjust the camera display screen by analyzing the face image, so as to solve the technical problem of poor interaction effect during the photographing process of the current electronic device.

[0067] Optionally, the photographing method provided by the embodiment of the present application can be applied to the electronic device 100. For example, the electronic device 100 can specifically be a terminal device such as a mobile phone, a digital camera, a computer, a tablet computer, a laptop computer, a wearable device, an artificial intelligence (AI) device, etc., which has a front - facing image sensor, a rear - facing image sensor, and a display screen. The operating systems installed on the above - mentioned electronic device 100 include but are not limited to or other operating systems. The present application does not limit the specific type of the above - mentioned electronic device 100 and the operating system installed thereon.

[0068] Exemplarily, Figure 3Shows a schematic structural diagram of the electronic device 100.

[0069] As Figure 3 shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a distance sensor 180F, a touch sensor 180K, etc.

[0070] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0071] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0072] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0073] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface to implement the touch function of the electronic device 100.

[0074] The MIPI interface may be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

[0075] The USB interface 130 is an interface that conforms to the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0076] It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only illustrative and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0077] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0078] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives the inputs of the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.

[0079] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0080] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: The antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0081] The mobile communication module 150 may provide solutions for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves through the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 may be provided in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be provided in the same device.

[0082] The wireless communication module 160 may provide solutions for wireless communication including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the electronic device 100. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves through the antenna 2, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive the signal to be transmitted from the processor 110, frequency-modulate and amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

[0083] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with a network and other devices via wireless communication technologies. The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0084] Electronic device 100 implements a display function via a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0085] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), for example, an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-led, a Micro-led, a Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0086] The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc.

[0087] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera photosensitive element, where the optical signal is converted into an electrical signal. The camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0088] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats.

[0089] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1. Among them, the camera 193 may be located in the edge area of the electronic device, and may be an under-screen camera or a liftable camera. The camera 193 may include a rear camera and may also include a front camera. The specific position and form of the camera 193 in the embodiments of the present application are not limited.

[0090] Exemplarily, for the layout of the cameras on the electronic device 100, reference can be made to Figure 4 , where the front of the electronic device 100 is the plane where the display screen 194 is located. As Figure 4 shown in (a) therein, if the camera 1931 is located on the front of the electronic device 100, the camera is a front camera. As Figure 4 shown in (b) therein, if the camera 1932 is located on the back of the electronic device 100, the camera is a rear camera.

[0091] Optionally, the solution in the embodiments of the present application can be applied to the electronic device 100 with a foldable screen having multiple display screens (i.e., the display screen 194 can be folded). Such as the foldable screen electronic device 100 shown in (c) of Figure 4 . In response to the user's operation, as shown in (d) of Figure 4 , the display screen is folded inward (or outward) along the folding edge, so that the display screen forms at least two screens (such as screen A and screen B). As Figure 4 shown in (e) of Figure 4 , there is a display screen (such as screen C) on the outer side of the fold. If the electronic device 100 is provided with a camera on the surface where the screen C is located. Then, in the non-folded scenario of the electronic device 100 shown in (c) of Figure 4 , the camera on the screen C is on the back of the electronic device 100 and can be regarded as a rear camera. In the folded scenario of the electronic device 100 shown in (e) of Figure 4 , the camera on the screen C becomes on the front of the electronic device 100 and can be regarded as a front camera. That is to say, in the present application, the front camera and the rear camera do not limit the nature of the camera itself, but only illustrate a positional relationship.

[0092] Thus, the electronic device 100 can determine whether the camera is a front camera or a rear camera according to whether the camera is on the same side as the display screen showing the shooting interface. For example, currently, the electronic device 100 captures an image through a rear camera located on the back of the electronic device 100, and displays the captured image through the camera shooting interface displayed on the display screen 194. And, the electronic device 100 captures a face image through a front camera on the same side as the display screen 194, and then realizes the control of the picture displayed on the shooting interface through face image analysis.

[0093] In the following text, the rear camera is uniformly referred to as the first camera, and the front camera is uniformly referred to as the second camera.

[0094] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0095] The NPU is a neural-network (NN) computing processor. By drawing on the structure of the biological neural network, such as the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.

[0096] In some embodiments, the NPU uses image recognition technology to identify whether the image collected by the camera 193 contains a face image. Further, the NPU can also obtain data such as the face position, eye position, and eye image based on the face image, for further identifying user eye features, etc. Subsequently, the processor 110 determines whether to adjust the captured image based on the eye features.

[0097] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external memory card.

[0098] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, and / or the instructions stored in the memory provided in the processor.

[0099] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, such as playing camera shutter sound effects, music playing, and recording.

[0100] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A.

[0101] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, such as when a camera is shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focus. In some embodiments, the electronic device uses the distance sensor to obtain the distance between the user's eyes and the electronic device display screen. Then, the electronic device can control the camera to achieve functions such as zoom based on the distance between the user's eyes and the electronic device display screen.

[0102] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0103] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0104] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting or removing the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0105] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0106] Figure 5 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0107] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer, the application framework layer, the Android runtime, and the system library, and the kernel layer.

[0108] The application layer may include a series of application packages.

[0109] Such as Figure 5 shown, the application packages may include applications such as camera, contacts, memo, calendar, music, gallery, map, call, video, etc.

[0110] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0111] Such as Figure 5 shown, the application framework layer may include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc.

[0112] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the display screen, capture the display screen, etc.

[0113] The content provider is used to store and obtain data, and make this data accessible to applications. The data may include video, image, audio, incoming and outgoing calls, browsing history and bookmarks, phone book, etc.

[0114] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

[0115] The telephone manager is used to provide the communication function of the electronic device 100. For example, the management of call states (including answering, hanging up, etc.).

[0116] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0117] The notification manager enables an application to display notification information in the status bar. It can be used to convey messages of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that a download is complete, a message reminder, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of a background-running application, or a notification that appears in the form of a dialog window on the display screen. For example, it can prompt text information in the status bar, emit a prompt tone, vibrate the electronic device, blink the indicator light, etc.

[0118] The Android runtime includes the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0119] The core libraries contain two parts: one part is the functional functions that the Java language needs to call, and the other part is the core libraries of Android.

[0120] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0121] The system libraries can include multiple functional modules. For example: surface manager, Media Libraries, 3D graphics processing library (e.g., OpenGL ES), 2D graphics engine (e.g., SGL), etc.

[0122] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0123] The Media Libraries support the playback and recording of multiple common audio and video formats, as well as static image files, etc. The Media Libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0124] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0125] The 2D graphics engine is the graphics engine for 2D drawing.

[0126] The kernel layer is the layer between the hardware and the software. The kernel layer contains at least a display driver, a camera driver, an audio driver, and a sensor driver.

[0127] The following uses the scenario of an eye movement control camera taking a photo as an example to exemplarily illustrate the working processes of the software and hardware of the electronic device 100.

[0128] The camera 193 responds to the operation of the user's eye movement to control the camera, and the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the operation of the user's eye movement to control the camera into raw input events (including information such as changes in the user's viewing point and changes in the user's position relative to the display screen). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input events from the kernel layer. The camera application calls the interface of the application framework layer to perform operations such as zooming the image in the camera shooting preview screen, migrating the shooting perspective, etc., and starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0129] Figure 6 It is a schematic flowchart of a shooting method provided by an embodiment of the present application. It should be noted that this method is not limited to Figure 6 and the following specific order. It should be understood that in other embodiments, the order of some steps of this method can be interchanged according to actual needs, or some of the steps can also be omitted or deleted. The method includes the following steps:

[0130] S601. The electronic device detects a first operation to open the camera application.

[0131] S602. In response to the first operation, the electronic device displays a shooting interface, and the shooting interface displays a first image captured by the first camera. The first camera is located on the side opposite to the display screen of the shooting interface.

[0132] S603. The electronic device captures a first face image through the second camera. The second camera is located on the same side as the display screen of the shooting interface.

[0133] In some embodiments, in response to a user operation, the electronic device starts the camera application. Among them, the user operation is, for example, an operation of clicking on the camera application icon, a voice command indicating the start of the camera application, etc. Then, the electronic device displays the shooting interface of the camera application and performs image capture through the first camera to display the image captured by the first camera on the shooting interface. In some examples, during the process of the electronic device displaying the shooting interface, the electronic device captures the face image of the user currently operating the electronic device through the second camera.

[0134] Among them, referring to the above Figure 4 example, the first camera is located on the side opposite to the display screen of the shooting interface, and the second camera is located on the same side as the display screen of the shooting interface.

[0135] In some examples, the shooting interface is a shooting preview interface or a video recording interface. For example, during the process of taking a photo, the electronic device displays a shooting preview interface. During this process, the user can control the change of the displayed image of the shooting preview interface of the camera through eye movement. Another example is that during the video recording process, the electronic device displays a video recording interface. During this process, the user can control the change of the displayed image of the video recording interface of the camera through eye movement.

[0136] In some embodiments, the eye movement tracking technology is a method for the electronic device to track the gaze position of a person's eyes to perform human-computer interaction. When the eye movement tracking technology is applied during the human-computer interaction process, the movement of the eyes can be used as input. When the user's eyes move, by estimating the viewpoint position of the human eye gaze on the display screen of the electronic device, the interaction between the person and the electronic device is realized.

[0137] In some examples, to implement eye movement tracking, the electronic device needs to recognize the user's face image to obtain the eye image included therein, and then subsequently, eye movement tracking can be implemented based on the eye image. Therefore, during the process of displaying the image of the shooting camera application, the electronic device also needs to collect the user's face image to implement eye movement control of the camera application.

[0138] S604: The electronic device obtains the position information of the first face image corresponding to the first screen, and the position information includes the first viewpoint position and the first relative distance.

[0139] Among them, the first viewpoint position is used to indicate the position indicated by the user's gaze in the first screen, and the first relative distance is used to indicate the distance between the user's face (or eyes) and the display screen. The first viewpoint position and the first relative distance will be introduced in detail below and will not be elaborated here.

[0140] In some embodiments, after the electronic device obtains the first face image, it can obtain the corresponding eye image, eye position, and face position of the first face image through image analysis. Exemplarily, as Figure 7 shown, this process includes S701 - S702.

[0141] S701: The electronic device obtains the first face image.

[0142] In some examples, the electronic device directly collects the first face image through the camera on the same side as the display screen of the electronic device. Or, after the electronic device collects an image through the camera on the same side as the display screen of the electronic device, it obtains the face image included therein through image recognition technology.

[0143] S702: The electronic device obtains the corresponding eye image, eye position, and face position of the first face image, and the eye position includes the left eye position and / or the right eye position.

[0144] Among them, the eye image includes the left eye image and / or the right eye image, and the face position includes the face frame position, which is identified as the position where the human face is located by means of framing.

[0145] Optionally, the electronic device processes the first face image. For example, the electronic device performs face detection processing on the first face image to obtain the eye image, eye position, and face position corresponding to the first face image. Exemplarily, a face detection model is preset in the electronic device. After the electronic device inputs the obtained first face image (or a portrait including a face image) into the face detection model, the face detection model can output at least one of the face position, left eye position, right eye position, left eye original image, right eye original image, etc. For example, the electronic device performs face detection processing on the face image shown in (a) below, and obtains the face position of the user's face in the face image shown in (b) below, the eye position of the user's eyes in the face image shown in (c) below, the left eye image shown in (d) below, and the right eye image shown in (e) below. Figure 8 the face image shown in (a) as follows, and obtains Figure 8 the face position of the user's face in the face image shown in (b) as follows, Figure 8 the eye position of the user's eyes in the face image shown in (c) as follows, Figure 8 the left eye image shown in (d) as follows, and Figure 8 the right eye image shown in (e) as follows.

[0146] Optionally, the face detection model can be a convolutional neural network, etc. For example, the electronic device inputs the face image into the convolutional neural network. After the convolutional neural network reads the face image and performs face detection processing, it outputs at least one of the face position, left eye position, right eye position, left eye image, and right eye image.

[0147] Optionally, after the face detection model performs face detection processing on the face image, it can first obtain the face position, further obtain the eye position based on the face position, and further obtain the eye image based on the eye position.

[0148] In this way, the electronic device obtains the user's eye position information and eye image information by performing face detection processing on the user's face image, which is convenient for the subsequent electronic device to identify the user's eye movements based on this data.

[0149] In some embodiments, the electronic device periodically obtains the user's eye position information and eye image information.

[0150] In some embodiments, after the electronic device obtains the eye image, eye position, and face position corresponding to the first face image, it can obtain the position information of the first face image relative to the first screen according to the eye image, eye position, and face position corresponding to the first face image.

[0151] For example, the electronic device inputs the eye image into the first model and obtains the eye feature vector output by the first model. Then, the electronic device inputs the eye feature vector, the eye position, and the face position into the second model and obtains the position information output by the second model.

[0152] Among them, the first model is, for example, an image encoder, and the second model is, for example, a fully connected layer.

[0153] In some examples, if the image encoder needs to have the ability to recognize the eye feature vector, a training process needs to be performed on the image encoder in advance. Then, the trained image encoder can be configured in the electronic device. In this way, during the operation of the camera application in the electronic device, the electronic device can output the eye feature vector through the trained image encoder.

[0154] Optionally, the server or the electronic device performs the training process of the image encoder. Exemplarily, as Figure 9 shown, taking the server training the image encoder as an example, the training process of the image encoder will be introduced.

[0155] Exemplarily, as Figure 9 shown, this training process includes S901 - S903.

[0156] S901: The server obtains training data.

[0157] In some embodiments, the training data obtained by the server is the initial eye image, including paired left - eye initial images and right - eye initial images. Among them, multiple left - eye initial images form a left - eye initial image sequence, and multiple right - eye initial images form a right - eye initial image sequence.

[0158] S902: The server processes the training data to obtain incomplete training data.

[0159] In some embodiments, the incomplete training data is the incomplete eye image obtained by the server after processing the initial eye image. In order to enable the image encoder to have the eye feature extraction ability, the server needs to use the incomplete eye image to train the image encoder and the image decoder. During the process of the image encoder and the image decoder repairing the incomplete eye image, it is necessary to extract the eye features from the incomplete eye image and repair the complete eye image according to the extracted eye features. When the repaired eye image is similar to the initial eye image, it is determined that the image encoder has the eye feature extraction ability.

[0160] Exemplarily, the server masks the paired left-eye initial images and right-eye initial images to obtain paired incomplete left-eye images and incomplete right-eye images. Multiple incomplete left-eye images form a left-eye incomplete image sequence, and multiple incomplete right-eye images form a right-eye incomplete image sequence. For example, the server divides each image in the left-eye initial image sequence and the right-eye initial image sequence into 16*16 image blocks, and masks a preset proportion (such as a random value in 90%-90%) of the image blocks to obtain paired incomplete left-eye images and incomplete right-eye images.

[0161] Optionally, during each round of training, the masking proportion of each image in the left-eye image sequence and the right-eye image sequence is the same. For example, the server masks 90% of the image blocks for each image. Optionally, to achieve better training results, as the training process progresses, this masking proportion can be continuously increased, thereby continuously improving the accuracy of the image encoder in extracting eye features.

[0162] S903: The server inputs the incomplete training data into the image encoder and the image decoder to obtain the restored training data.

[0163] In some embodiments, the restored training data is the restored eye images obtained by the server after restoring the incomplete eye images, including paired restored left-eye images and restored right-eye images.

[0164] In some examples, the server inputs the incomplete eye images into the image encoder to obtain the eye feature vectors containing eye features corresponding to the incomplete eye images output by the image encoder. The eye feature vectors include information such as the position and size of the user's pupil in the eye image. The server inputs the eye feature vectors into the image decoder to obtain the restored (restored) eye images output by the image decoder after decoding the eye feature vectors. For example, the server masks 50% of the image blocks of a pair of left-eye original images and right-eye original images to obtain paired incomplete left-eye images and incomplete right-eye images. The server inputs the incomplete left-eye images and incomplete right-eye images into the image encoder. The image encoder obtains the eye feature vectors corresponding to the incomplete images based on the incomplete images, and the eye feature vectors include the eye features corresponding to the unmasked image blocks. The image encoder inputs the eye feature vectors into the image decoder, and the image decoder restores (restores) the masked image blocks in the incomplete left-eye images and incomplete right-eye images based on the unmasked parts of the incomplete left-eye images and incomplete right-eye images to obtain complete restored left-eye images and complete restored right-eye images.

[0165] Optionally, the image encoder may include a multi-layer structure, and each layer structure of the image encoder may output a vector. The eye feature vector may be a multi-layer vector.

[0166] In some examples, when the server determines that the repaired eye image is similar to the initial eye image, it stops the training process. Exemplarily, the server uses a loss function to evaluate the similarity between the repaired eye image and the initial eye image. When the loss function no longer significantly decreases as the number of training rounds increases, the server determines that the repaired eye image is similar to the initial eye image, and the server stops the training process. Optionally, when the server determines the similarity between the repaired eye image and the initial eye image, it may be based on the similarity of each pixel point of the repaired eye image and the initial eye image to obtain the overall similarity between the repaired eye image and the initial eye image.

[0167] If the server obtains a repaired eye image that is similar to the initial eye image, it indicates that the image encoder and the image decoder have the ability to repair eye images. Among them, the image encoder has the ability to understand and extract eye features, that is, the eye feature recognition ability.

[0168] In this way, the server trains an image encoder with eye feature recognition ability. The image encoder with eye feature recognition ability can be configured in the electronic device, and the electronic device can obtain the user's eye features from the user's eye image based on this image encoder. In this way, based on the eye features, the electronic device can more accurately recognize the user's eye movements to confirm the user's intention, and thus realize precise control of the camera in response to the user's eye movements.

[0169] The following details the process by which an electronic device obtains the position information corresponding to a face image through a trained first model (such as an image encoder) and a second model (such as a fully connected layer). Exemplarily, this process includes Figure 10 S1001 - S1002 as shown.

[0170] S1001: The electronic device inputs the eye image into the first model and obtains the eye feature vector output by the first model.

[0171] In some examples, the electronic device obtains the user's first face image through the second camera, and Figure 7 obtains the user's eye image from the user's first face image through the method shown. The electronic device inputs the user's eye image into the trained first model to obtain the eye feature vector output by the first model. The eye feature vector includes the eye features extracted by the first model from the user's eye image.

[0172] S1002: The electronic device inputs the eye feature vector, the eye position, and the face position into the second model to obtain the position information output by the second model.

[0173] Among them, the position information indicates the user's intention, and the user's intention includes moving the perspective of the shooting screen, zooming in or out the shooting screen, etc.

[0174] Exemplarily, during the operation of the camera, the electronic device displays the shooting screen. Generally, the focus of the user's line of sight is the main object in the shooting screen, and this main object is generally located at the center point of the display screen. The position of the focus of the user's line of sight in the shooting screen is described as the viewpoint position. If the viewpoint position does not coincide with the center point of the display screen, that is, the user does not pay attention to the object displayed at the center point of the display screen. The electronic device can determine that the user's intention is to move the perspective of the shooting screen so as to move the object that the user is concerned about to the center point of the display screen for display. For example, if the first viewpoint position is offset to the upper right direction relative to the center point of the display screen, then the electronic device determines that the user's intention is to move the perspective of the shooting screen to the lower left. The electronic device moves the perspective of the shooting screen according to the offset of the first viewpoint position relative to the center point of the display screen, so that the first viewpoint position moves to the center point of the display screen.

[0175] For another example, during the operation of the camera, if the electronic device determines that the position change of the user's eyes relative to the display screen is approaching the display screen, it can determine that the user's intention is to zoom in the display of the shooting screen so that the user can view the local details of the shooting screen more clearly; or, if the electronic device determines that the position change of the user's eyes relative to the display screen is moving away from the display screen, it can determine that the user's intention is to zoom out the display of the shooting screen so that the user can view the shooting screen globally.

[0176] In some embodiments, after the electronic device obtains the eye feature vector and the eye position information, it can obtain the viewpoint position and the relative distance information of the eyes relative to the display screen based on the eye feature vector and the eye position information, so as to confirm the user's intention.

[0177] In some examples, a trained second model (such as a fully connected layer) is configured in the electronic device. After the electronic device obtains the eye feature vector, the eye position, and the face position, it inputs the eye feature vector, the eye position, and the face position into the second model to obtain the position information output by the second model.

[0178] Optionally, the training process of the second model is executed by the server or the electronic device to configure the trained second model to the electronic device used by the user. Among them, the training process of the second model can refer to the prior art, and the embodiments of the present application do not limit this.

[0179] Exemplarily, such as Figure 10As shown, the electronic device inputs the eye feature vector, face position, and eye position into the fully connected layer of the electronic device. The fully connected layer of the electronic device calculates parameters such as the first viewing point position and the relative distance information of the eyes relative to the display screen based on the eye feature vector, face position, and eye position.

[0180] Optionally, in some actions where the user moves the viewing point position, the user's eye position may remain unchanged, but the user's face position changes. For example, when the user looks up at the viewing point position above the center point of the display screen, the user's eye position may not change, but the user's face position changes. Therefore, inputting the user's face position into the fully connected layer can make the obtained parameters such as the first viewing point position and the relative distance information of the eyes relative to the display screen more accurate, and thus more accurately identify the user's eye movements.

[0181] Among them, the first viewing point position can be represented by the pixel point offset of the first viewing point position relative to the pixel point corresponding to the origin position of the display screen. Exemplarily, the electronic device takes the upper left vertex of the display screen as the origin, establishes a coordinate system, and obtains the xy-axis offsets of the center point of the display screen and the first viewing point position relative to the origin of the display screen (such as the origin of the coordinate system). As needed, the origin of the display screen can also be other vertices of the display screen, and the embodiments of the present application do not limit this. For example, as Figure 11 shown, the reference numeral 112 represents the origin of the display screen, and the reference numeral 111 represents the first viewing point position. The first viewing point position 111 is represented by (x1, x2), where x1 is the horizontal pixel point offset of the viewing point position 111 relative to the origin 112 of the display screen, and x2 is the vertical pixel point offset of the first viewing point position 111 relative to the origin 112 of the display screen.

[0182] Exemplarily, as Figure 12 shown, the electronic device can, through the Figure 10 method shown, achieve tracking of the first viewing point position and obtain the first viewing point position 121 and the relative distance information of the user relative to the display screen (such as the distance 122 of the user relative to the display screen) as Figure 12 shown.

[0183] In this way, the electronic device can obtain the first viewing point position and the relative distance information of the user's eyes relative to the display screen according to the acquired user face image.

[0184] In some examples, the electronic device can acquire the user face image according to a preset period, and thus periodically obtain the first viewing point position and the relative distance information of the user relative to the display screen. The electronic device can identify the user's eye movements according to the changes in the first viewing point position and / or the relative distance information of the user relative to the display screen, determine the user's intention, and control the camera to implement the shooting function.

[0185] S605. The electronic device displays, on the shooting interface, a second picture captured by the first camera according to the location information. The center of the second picture corresponds to the first object in the first picture indicated by the first viewing point position, and / or the zoom factor corresponding to the second picture matches the first relative distance.

[0186] In some embodiments, after obtaining the location information, the electronic device can adjust the picture displayed on the shooting interface according to the first viewing point position and / or the first relative distance included in the location information, and display the adjusted second picture on the shooting interface so that the second picture meets the user's requirements.

[0187] The process of the electronic device adjusting the picture displayed on the shooting interface according to the first viewing point position included in the location information will be introduced in detail below.

[0188] In some scenarios, the electronic device recognizes that the viewing point position of the user's eyes looking at the display screen has moved, and determines that the user's intention is to control the deviation of the shooting picture angle of the first camera. In response to the movement of the viewing point position, the electronic device controls the deviation of the shooting picture angle of the first camera. For example, when the main body in the shooting picture moves, the user's viewing point will move from the center of the picture to other positions on the display screen along with the movement of the main body in the picture. Or when the user needs to align the center of the shooting picture with another main body in the picture, when the viewing point position of the user moves from the center of the picture to the position where the other main body is displayed on the display screen. The electronic device needs to control the deviation of the shooting picture angle of the first camera to ensure that the shooting picture angle is aligned with the main body in the picture that the user's viewing point is looking at. In some embodiments, after obtaining the user's intention to control the deviation of the shooting picture angle of the first camera, the electronic device controls the deviation of the shooting picture angle of the first camera according to the first viewing point position.

[0189] The process of the electronic device controlling the deviation of the shooting picture angle of the first camera according to the first viewing point position will be introduced below. Exemplarily, as Figure 13 shown, this process includes S1301 - S1303.

[0190] S1301: The electronic device obtains the position deviation of the first viewing point position relative to the center point of the display screen.

[0191] In some embodiments, the position deviation is the proportion of the pixel point offset between the first viewing point position and the center point of the display screen in the horizontal and vertical pixel points of the entire display screen. The electronic device can obtain the angle to be deflected of the shooting picture angle of the first camera based on the obtained position deviation. The electronic device controls the deviation of the shooting picture angle of the first camera according to this angle to be deflected.

[0192] Exemplarily, as Figure 11As shown in the figure, the number of pixels of the display screen of the electronic device in the horizontal direction is w, and the number of pixels in the vertical direction is h. The electronic device obtains that the pixel offset of the center point 113 of the display screen relative to the origin 112 of the display screen is (w / 2, h / 2), and the pixel offset of the first viewing point position 111 relative to the origin 112 of the display screen is (x1, x2). Based on these two offsets, the electronic device obtains that the pixel offset of the first viewing point position 111 relative to the center point 113 of the display screen is (x1 - w / 2, x2 - h / 2). The electronic device obtains the position deviation of the first viewing point position 111 relative to the center point 113 of the display screen according to the proportion of the pixel offset of the first viewing point position 111 relative to the center point 113 of the display screen in the number of pixels in the horizontal or vertical direction of the display screen. For example, the position deviation of the first viewing point position 111 relative to the center point 112 of the display screen is represented by (dx, dy). Wherein, dx = (x1 - w / 2) / w = x1 / w - 0.5, dy = (x2 - h / 2) / h = x2 / h - 0.5.

[0193] S1302: The electronic device obtains the angle to be offset of the shooting view of the first camera according to the position deviation.

[0194] In some embodiments, since the image captured by the first camera (the first camera) of the electronic device has a perspective effect, the actual angle to be offset for the electronic device to control the main body of the image at the first viewing point position to move to the center point of the display screen does not exactly correspond to the position deviation of the first viewing point position relative to the center point of the display screen. Therefore, if the electronic device controls the offset of the shooting view of the first camera according to the position deviation of the first viewing point position relative to the center point of the display screen, there will be an error. Therefore, when the electronic device actually controls the offset of the shooting view of the first camera, it needs to obtain the angle to be offset of the shooting view of the first camera according to the position deviation of the first viewing point position relative to the center point of the display screen.

[0195] Optionally, the angle to be offset of the shooting view of the first camera includes the horizontal angle to be offset and / or the vertical angle to be offset. Optionally, this angle to be offset can be used for the electronic device to control the offset of the first camera, and can also be used for the electronic device to calculate and obtain the image corresponding to the shooting view of the first camera after the viewpoint offset from the panoramic perspective image collected by the first camera using this angle to be offset, etc.

[0196] In some embodiments, the electronic device obtains the angle to be offset of the shooting view angle of the first camera based on the position deviation of the first viewpoint position relative to the center point of the display screen and the upper limit of the offsettable amount of the shooting view angle of the first camera. The upper limit of the offsettable amount is the upper limit value of the offsettable angle of the first camera of the electronic device, or the upper limit value of the offsettable angle of the perspective points in the panoramic perspective image collected by the first camera. The horizontal upper limit of the offsettable amount refers to the upper limit of the angle that can be offset to the left or right. The vertical upper limit of the offsettable amount refers to the upper limit of the angle that can be offset upward or downward.

[0197] Exemplarily, the first camera of the electronic device is a fish-eye camera, and the field of view range of the fish-eye camera is 180°. The field of view range of the shooting view angle displayed on the display interface of the electronic device is 70°. Then, the upper limit of the offsettable angle of the shooting view angle of the first camera of the electronic device in the up, down, left, and right directions is (180° - 70°) / 2 = 55°, that is, both the horizontal upper limit of the offsettable amount and the vertical upper limit of the offsettable amount are 55°.

[0198] In some examples, the electronic device uses the product of the position deviation of the viewpoint position relative to the center point of the display screen and the upper limit of the offsettable amount of the shooting view angle of the first camera as the angle to be offset of the shooting view angle of the first camera.

[0199] Exemplarily, if the horizontal upper limit of the offsettable amount of the shooting view angle of the first camera is represented by mx, the vertical upper limit of the offsettable amount is represented by my, and the relative deviation of the first viewpoint position 71 relative to the center point 72 of the display screen is represented by (dx, dy). The electronic device determines that the horizontal angle to be offset is mx × dx, and the vertical angle to be offset is my × dy.

[0200] S1303: The electronic device controls the offset of the shooting view angle according to the angle to be offset, obtains and displays the second picture.

[0201] In some embodiments, after the electronic device obtains the angle to be offset of the shooting view angle of the first camera (the first camera), it controls the shooting view angle of the first camera to be offset according to the angle to be offset, obtains the shooting picture corresponding to the first viewpoint position, and displays the shooting picture corresponding to the first viewpoint position in the shooting interface of the electronic device.

[0202] In some embodiments, the electronic device can control the perspective shift of the shooting screen of the first camera in various ways. Two implementation methods are provided below. The first implementation method is that the electronic device realizes the perspective shift of the shooting screen of the first camera through image calculation. The second implementation method is that the electronic device controls the perspective shift of the shooting screen of the first camera by controlling the deflection of the lens of the camera. Optionally, the electronic device can, according to needs, simultaneously use the methods of image calculation and controlling the deflection of the camera lens to control the perspective shift of the shooting screen of the first camera.

[0203] In the first implementation method, the electronic device acquires a panoramic image collected by the first camera. After that, the electronic device obtains a second screen with a perspective shift from the panoramic image according to the angle to be shifted and the first screen.

[0204] For example, if the first camera of the electronic device includes an ultra-wide-angle camera or a fish-eye camera, since the image collected by the first camera is larger than the image displayed on the shooting interface of the electronic device, that is, the image displayed on the shooting interface is a partial image of the image collected by the first camera. Therefore, the electronic device can perform image calculation on the image collected by the first camera according to the angle to be shifted of the shooting screen perspective of the first camera, obtain the image corresponding to the user's viewing point perspective, and display the image corresponding to the user's viewing point perspective on the shooting interface of the electronic device to realize the perspective shift of the shooting screen of the first camera (hereinafter, an example is given where the first camera of the electronic device is a fish-eye camera).

[0205] Exemplarily, as Figure 14 shown in (a) below, the spherical image collected by the electronic device through the fish-eye camera can cover a viewing angle range of nearly 180°. The spherical image covers the image corresponding to the shooting screen perspective currently displayed on the shooting interface of the first camera and the image outside this perspective. When the electronic device controls the perspective shift of the shooting screen of the first camera, it only needs to obtain the image corresponding to the user's viewing point perspective from the spherical image according to the angle to be shifted of the shooting screen perspective of the first camera, and display the image corresponding to the user's viewing point perspective on the camera shooting interface, then the perspective shift of the shooting screen of the first camera can be realized. For example, Figure 14 in (a) below shows the image after the perspective of the shooting screen of the first camera is shifted to the left and the image after the perspective of the shooting screen of the first camera is shifted to the right.

[0206] Exemplarily, Figure 15The spherical image captured by the fisheye camera of the electronic device shown can cover a viewing angle range of 180° in both the horizontal and vertical directions. The image corresponding to the viewing angle of the first camera displayed on the current camera shooting interface is Image 1, and the angle to be offset for the viewing angle of the first camera's shooting screen is to be offset upward by 50°. Based on this angle to be offset, the electronic device obtains, from the spherical image, Image 2 corresponding to the user's viewing angle after the viewpoint corresponding to Image 1 is offset upward by 50°. The electronic device displays this Image 2 on the camera shooting interface, and thus can achieve the offset of the viewing angle of the first camera's shooting screen.

[0207] In some examples, such as Figure 14 shown in (b) therein, the electronic device tracks the user's viewpoint. When the first viewpoint position moves to the right side of the center point of the display screen, the electronic device determines that the user's intention is to control the viewing angle of the first camera's shooting screen to be offset to the right. The electronic device calculates and obtains, from the 180° spherical image captured by the camera, the image after the user's viewpoint in the spherical image is offset to the right according to the angle to be offset for the viewing angle of the first camera's shooting screen in this example. The image after the viewing angle of the first camera's shooting screen is offset to the right, as shown in the lower left corner of (b) therein, is displayed on the camera shooting interface. Figure 14 in (b) therein.

[0208] In other examples, such as Figure 14 shown in (c) therein, the electronic device tracks the user's viewpoint. When the first viewpoint position moves to the left side of the center point of the display screen, the electronic device determines that the user's intention is to control the viewing angle of the first camera's shooting screen to be offset to the left. The electronic device calculates and obtains, from the 180° spherical image captured by the camera, the image after the user's viewpoint in the spherical image is offset to the left according to the angle to be offset for the viewing angle of the first camera's shooting screen in this example. The image after the viewing angle of the first camera's shooting screen is offset to the left, as shown in the lower left corner of (c) therein, is displayed on the camera shooting interface. Figure 14 in (c) therein.

[0209] In still other examples, such as Figure 14 shown in (d) therein, the electronic device tracks the user's viewpoint. When the first viewpoint position moves to the upper side of the center point of the display screen (such as when the user tilts the head up towards the sky), the electronic device determines that the user's intention is to control the viewing angle of the first camera's shooting screen to be offset upward. The electronic device calculates and obtains, from the 180° spherical image captured by the camera, the image after the user's viewpoint in the spherical image is offset upward according to the angle to be offset for the viewing angle of the first camera's shooting screen in this example. The image after the viewing angle of the first camera's shooting screen is offset upward, as shown in the lower left corner of (d) therein, is displayed on the camera shooting interface. Figure 14 in (d) therein.

[0210] In this way, in response to a user's viewpoint movement operation on the display screen of the electronic device, the electronic device can adjust the shooting screen angle of the first camera through image calculation, thereby helping the user capture the required image. This simplifies the user operation steps and has higher accuracy and stronger real-time performance compared to the user controlling the shooting screen angle offset of the first camera through body movements.

[0211] In the second implementation manner, after the electronic device adjusts the deflectable lens according to the angle to be deflected, it acquires a second image through the first camera.

[0212] For example, when the first camera of the electronic device has a narrow field of view, the viewing angle range of the image captured by the first camera of the electronic device is narrow. In this case, the electronic device cannot obtain an image with the shooting screen angle of the first camera offset through image calculation. Then, the electronic device can obtain an image with the shooting screen angle of the first camera offset by adjusting the shooting angle of the first camera, so that the shooting screen angle of the first camera is offset.

[0213] Exemplarily, as Figure 16 shown, when the first camera of the electronic device has a narrow field of view (such as a camera with a field of view in the range of 60 - 100°), a deflectable lens 1601 can be added at the light incident position of the first camera of the electronic device, so that the electronic device can realize the offset of the shooting screen angle of the first camera by controlling the rotation of the deflectable lens 1601.

[0214] Exemplarily, after the electronic device obtains the angle to be offset of the shooting screen of the first camera, it controls the deflectable lens 1601 to deflect according to the angle to be offset. As Figure 16 shown, the deflectable lens 1601 added by the electronic device includes two reflecting lenses. The first reflecting lens can be a biaxial rotating mirror 1602, and the biaxial rotating mirror 1602 can rotate in the horizontal and vertical directions. There are two motors 1703 at the rear end of the biaxial rotating mirror 1602 as Figure 17 shown. As Figure 17 shown in 1704 or 1705, the electronic device can make the biaxial rotating mirror 1602 rotate in the horizontal and vertical directions through the operation of the motor. By controlling the biaxial rotating mirror 1602 to rotate in different directions, images with different shooting screen angles can be obtained.

[0215] In some examples, as Figure 16 shown, the current display of the shooting interface of the electronic device is as Figure 16As shown in Image 1, the electronic device tracks the user's viewing point. When the first viewing point position moves towards the lower left, the electronic device determines that the viewing angle of the shooting screen of the first camera deflects towards the lower left. Based on the angle to be deflected of the viewing angle of the shooting screen of the first camera obtained, after controlling the biaxial rotating mirror 1602 to deflect towards the lower left, the electronic device captures in real time to obtain an image such as Figure 16 Image 2 shown in. The electronic device displays this Image 2 on the camera shooting interface. Optionally, the electronic device controls the biaxial rotating mirror 1602 to rotate counterclockwise along Figure 17 as shown in 1705 to achieve a leftward rotation. The electronic device controls the biaxial rotating mirror 1602 to rotate counterclockwise along Figure 17 as shown in 1704 to achieve a downward rotation. Based on this, the electronic device realizes controlling the biaxial rotating mirror 1602 to deflect towards the lower left.

[0216] Optionally, the motor at the rear end of the biaxial rotating mirror can also be a coil or a piezoelectric ceramic driver, etc., and the embodiments of the present application do not limit this.

[0217] In some embodiments, even if the first camera (the first camera) of the electronic device can capture an image larger than the image displayed on the shooting interface, such as a panoramic image (for example, the first camera is a fish-eye camera), the electronic device can also adopt the above second implementation method to obtain the second image after adjusting the viewing angle of the shooting screen of the first camera by adjusting the deflection angle of the camera.

[0218] In this way, even if the main body of the image moves, as long as the user's eyes look at the target main body of the image and ensure that the viewing point is located at the position of the target main body of the image, the electronic device can control the viewing angle of the shooting screen of the first camera to deflect through the above process, ensuring that the target main body of the image is located at the center of the image in the camera shooting screen, without the need for complex operations such as hand operations or body rotation and movement to adjust the orientation of the camera. In some other examples, if the user's viewing point moves to another main body of the image in the viewing angle of the shooting screen of the first camera, the electronic device can also control the viewing angle of the shooting screen of the first camera to deflect towards the other main body in response to the user's viewing point movement operation. Until when the user's viewing point focuses on the other main body, the electronic device adjusts the other main body to the center of the image in the camera shooting screen.

[0219] In this way, the electronic device can display the main body of the image concerned by the user near the center position of the image in the camera shooting screen in a pure visual fixation manner, without the need for complex operations such as hand operations or body deflection and movement to align the camera, simplifying the user operation steps and having higher accuracy and stronger real-time performance in camera control.

[0220] The following details the process by which an electronic device adjusts the displayed image on the shooting interface based on the first relative distance included in the location information. In some scenarios, the user needs to zoom in or out on the image displayed in the camera shooting interface, and the electronic device can control the camera to automatically zoom in response to the user's eye movement. In some examples, the electronic device can control the camera zoom based on the change in the distance between the user's eyes and the display screen.

[0221] In some embodiments, as Figure 18 shown, the electronic device obtains a first image with a first zoom ratio (i.e., S1801). Then, when the first relative distance is less than or equal to the first threshold, the electronic device obtains a second image corresponding to a second zoom ratio through the first camera, that is, as shown in S1802a, the electronic device obtains a first image with the second zoom ratio, and the second zoom ratio is greater than the first zoom ratio. Alternatively, when the first relative distance is greater than or equal to the second threshold, the electronic device obtains a second image corresponding to a third zoom ratio through the first camera, that is, as shown in S1802b, the electronic device obtains a second image with the third zoom ratio, and the third zoom ratio is less than the first zoom ratio. And the first threshold is less than the second threshold.

[0222] The following details the process by which the electronic device adjusts the zoom ratio according to the first relative distance.

[0223] In some embodiments, the electronic device obtains the relative distance information between the user's eyes and the display screen of the electronic device. When the distance between the user's eyes and the display screen is greater than or equal to the second threshold (i.e., when the user's eyes are far from the display screen until the distance between the user's eyes and the display screen reaches the second threshold), the electronic device determines that the camera is to perform a camera zoom-out operation. That is, when the user's eyes are relatively far from the display screen, the electronic device automatically reduces the zoom ratio to shrink the image displayed on the shooting interface to help the user obtain more shooting scenes. When the distance between the user's eyes and the display screen is less than the preset distance condition (i.e., when the user's eyes are close to the display screen until the distance between the user's eyes and the display screen reaches the first threshold), the electronic device determines that the camera is to perform a camera zoom-in operation. That is, when the user's eyes are relatively close to the display screen, the electronic device automatically increases the zoom ratio to magnify the image displayed on the shooting interface to help the user obtain the details of the shooting scene.

[0224] Optionally, the distance between the user's eyes and the display screen can be the distance between the user's eyes and the operator's viewing point position, or the distance between the user's eyes and the center point of the display screen. The embodiments of the present application do not limit this.

[0225] Optionally, the preset distance condition can be a set value, or a value iteratively obtained by the electronic device according to the user's usage habits. The preset distance condition can be a numerical value or a numerical range. Exemplarily, asFigure 19 As shown, the preset distance condition is a numerical range from L1 to L2, where L1 is the first threshold and L2 is the second threshold. For example, L1 can be 20 cm and L2 can be 40 cm. When the preset distance condition is a numerical range, when the user's distance from the display screen exceeds this numerical range, the electronic device determines that the user intends to control the electronic device to perform a zoom operation by changing the distance between the eyes and the display screen. When the distance between the user's eyes and the display screen is within this numerical range, the electronic device determines that the user does not intend to control the electronic device to perform a zoom operation by changing the distance between the eyes and the display screen, and the electronic device does not perform a zoom operation according to the change in the distance between the user's eyes and the display screen.

[0226] Exemplarily, when the distance between the user's eyes and the display screen changes, when the distance between the user's eyes and the display screen is less than 20 cm, the electronic device determines that the camera is to perform a camera zoom-in operation. When the distance between the user's eyes and the display screen is greater than 40 cm, the electronic device determines that the camera is to perform a camera zoom-out operation. When the distance between the user's eyes and the display screen is between 20 cm and 40 cm, the camera focal length remains unchanged.

[0227] Exemplarily, the camera zoom-out operation can be that the electronic device adjusts the camera zoom ratio to reduce the zoom magnification, so that the shooting view angle of the first camera approaches the main subject of the shot. When the shooting view angle of the first camera approaches the main subject of the shot, the electronic device can update the first image in the camera shooting interface to the second image of the shot. The camera zoom-in operation is that the electronic device adjusts the camera zoom ratio to increase the zoom magnification, so that the shooting view angle of the first camera moves away from the main subject of the shot. When the shooting view angle of the first camera moves away from the main subject of the shot, the electronic device can update the camera shooting image in the camera shooting interface to the second image. The proportion of the main subject in the second image is smaller than that in the first image.

[0228] In this way, the electronic device can automatically control the camera to perform a zoom operation in response to the change in the distance between the user's eyes and the display screen. Without the user having to touch complex operations such as Figure 1 the display screen zoom control or the electronic device buttons shown, the operation steps are simplified and the user interaction experience is improved.

[0229] In some embodiments, in the eye movement control camera mode, the electronic device detects the user's eye movements, determines the user's intention, and adjusts the display of the shooting screen. In some examples, the electronic device starts or closes the eye movement control camera mode in response to the user's operation, so as to meet the user's personalized needs.

[0230] Combined above with Figures 6 - 19The shooting method provided by the embodiments of the present application is described in detail. The following will be combined with Figure 20 The electronic device provided by the embodiments of the present application will be described in detail.

[0231] In a possible design, Figure 20 It is a schematic structural diagram of the electronic device provided by the embodiments of the present application. As Figure 20 shown, the electronic device 2000 may include: a transceiver unit 2001, a processing unit 2002, and a display unit 2003. The electronic device 2000 can be used to implement the functions of the electronic device involved in the above method embodiments.

[0232] Optionally, the transceiver unit 2001 is used to support the electronic device 2000 to execute Figure 6 S601 in

[0233] Optionally, the processing unit 2002 is used to support the electronic device 2000 to execute Figure 6 S602 - S605 in Figure 7 ; and / or, support the electronic device 2000 to execute Figure 10 S701, S702 in Figure 13 ; and / or, support the electronic device 2000 to execute Figure 18 S1001, S1002 in

[0234] Optionally, the display unit 2003 is used to support the electronic device 2000 to display the interface content. And / or, support the electronic device 2000 to execute Figure 6 S605 in Figure 13 ; and / or, support the electronic device 2000 to execute

[0235] Among them, the transceiver unit may include a receiving unit and a sending unit, and can be implemented by a transceiver or transceiver-related circuit components, and can be a transceiver or a transceiver module. The operations and / or functions of each unit in the electronic device 2000 are respectively for implementing the corresponding processes of the shooting method in the above method embodiments. All relevant contents of each step involved in the above method embodiments can be cited in the function descriptions of the corresponding functional units. For the sake of brevity, they will not be repeated here.

[0236] Optionally, Figure 20 The shown electronic device 2000 may further include a storage unit ( Figure 20 not shown in Figure 20The electronic device 2000 shown can execute the shooting method in the above method embodiments.

[0237] Figure 20 For the technical effects of the electronic device 2000 shown, reference can be made to the technical effects of the shooting method in the above method embodiments, which will not be elaborated here. In addition to being in the form of the electronic device 2000, the technical solution provided in this application can also be a functional unit or chip in the electronic device, or a device used in conjunction with the electronic device.

[0238] This application embodiment also provides a chip system, including: a processor, the processor is coupled to a memory, and the memory is used to store programs or instructions. When the programs or instructions are executed by the processor, the chip system implements the method in any of the above method embodiments.

[0239] Optionally, the processor in the chip system can be one or more. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor that implements by reading the software code stored in the memory.

[0240] Optionally, the memory in the chip system can also be one or more. The memory can be integrated with the processor or separately provided from the processor, which is not limited in this application embodiment. Exemplarily, the memory can be a non-transitory processor, such as a read-only memory ROM, which can be integrated with the processor on the same chip or separately provided on different chips. This application embodiment does not specifically limit the type of the memory and the setting manner of the memory and the processor.

[0241] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processing circuit (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0242] It should be understood that each step in the above method embodiments can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The method steps disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware processor, or executed and completed by the combination of the hardware and software modules in the processor.

[0243] The embodiments of the present application further provide a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, the computer is enabled to execute the above-related steps to implement the shooting method in the above embodiments.

[0244] The embodiments of the present application further provide a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above-related steps to implement the shooting method in the above embodiments.

[0245] In addition, the embodiments of the present application further provide a device. The device may specifically be a component or a module, and the device may include one or more processors and a memory connected to each other. The memory is used to store a computer program. When the computer program is executed by one or more processors, the device is enabled to execute the shooting method in each of the above method embodiments.

[0246] Among them, the device, the computer-readable storage medium, the computer program product or the chip provided by the embodiments of the present application are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved by them can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0247] The steps of the methods or algorithms described in connection with the disclosed embodiments of the present application may be implemented in hardware or by a processor executing software instructions. The software instructions may be composed of corresponding software modules, and the software modules may be stored in a random access memory (RAM), flash memory, read only memory (ROM), erasable programmable ROM (EPROM), electrically EPROM (EEPROM), registers, hard disk, removable hard disk, CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may be located in an application specific integrated circuit (ASIC).

[0248] From the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions may be allocated to different functional modules as needed; that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working processes of the systems, devices, and units described above, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be elaborated herein again.

[0249] In the several embodiments provided in the present application, it should be understood that the disclosed methods may be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the couplings or direct couplings or communication connections shown or discussed with each other may be through some interfaces, and the indirect couplings or communication connections of modules or units may be in electrical, mechanical, or other forms.

[0250] In addition, in each embodiment of the present application, the functional units may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated units may be implemented in the form of hardware or in the form of software functional units.

[0251] Computer-readable storage media include, but are not limited to, any of the following: USB flash drives, external hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs, and other media that can store program code.

[0252] The above are only specific implementation manners of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A shooting method, applied to an electronic device, characterized in that The method includes: Detecting a first operation to open a camera application; In response to the first operation, displaying a shooting interface, where the shooting interface displays a first image captured by a first camera, and the first camera is located on the opposite side of the display screen that displays the shooting interface; Collecting a first face image through a second camera, where the second camera is located on the same side of the display screen that displays the shooting interface; Obtaining position information of the first face image corresponding to the first image, where the position information includes a first viewing point position and a first relative distance; According to the position information, displaying a second image captured by the first camera on the shooting interface, where the center of the second image corresponds to a first object in the first image indicated by the first viewing point position, and / or the zoom factor corresponding to the second image matches the first relative distance.

2. The method according to claim 1, wherein The obtaining the position information of the first face image corresponding to the first image includes: Obtaining an eye image, an eye position, and a face position corresponding to the first face image, where the eye position includes a left eye position and / or a right eye position; Inputting the eye image into a first model to obtain an eye feature vector output by the first model; Inputting the eye feature vector, the eye position, and the face position into a second model to obtain the position information output by the second model.

3. The method according to claim 1 or 2, characterized in that, The displaying the second image captured by the first camera on the shooting interface according to the position information includes: Obtaining a position deviation of the first viewing point position relative to the center point of the display screen; According to the position deviation, obtaining a to-be-offset angle of the shooting view angle of the first camera; According to the to-be-offset angle, controlling the shooting view angle to be offset to obtain the second image; Displaying the second image on the shooting interface.

4. The method according to claim 3, wherein The controlling the shooting view angle to be offset according to the to-be-offset angle to obtain the second image includes: Obtaining a panoramic image captured by the first camera; According to the to-be-offset angle and the first image, obtaining the second image with the view angle offset from the panoramic image.

5. The method according to claim 3, characterized in that, If the first camera includes a deflectable lens, the controlling the shooting view angle to be offset according to the to-be-offset angle to obtain the second image includes: After adjusting the deflectable lens according to the to-be-deflected angle, collecting the second image through the first camera.

6. The method according to any one of claims 1-5, characterized in that, The first image corresponds to a first zoom ratio, and the displaying the second image captured by the first camera according to the position information includes: In the case where the first relative distance is less than or equal to a first threshold, obtaining a second image corresponding to a second zoom ratio through the first camera, where the second zoom ratio is greater than the first zoom ratio; Or, In the case where the first relative distance is greater than or equal to a second threshold, obtaining a second image corresponding to a third zoom ratio through the first camera, where the third zoom ratio is less than the first zoom ratio, and the first threshold is less than the second threshold.

7. The method according to any one of claims 1 to 6, characterized in that, The shooting interface is a shooting preview interface or a video recording interface.

8. An electronic device, characterized in that, Including: A processor, a memory, and a display screen, where the memory and the display screen are coupled to the processor, the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor reads the computer instructions from the memory, the electronic device is caused to execute the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program, which, when running on an electronic device, causes the electronic device to execute the method according to any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1-7.

Images