TOF camera sensor control method and electronic equipment

Abstract

The embodiment of the invention provides a TOF camera sensor control method and electronic equipment, and relates to the technical field of intelligent interaction. The problem of energy consumption of the electronic equipment when the TOF camera sensor is started is solved. According to the specific scheme, the electronic equipment comprises an application processor AP and a sensor management assembly SensorHub, the AP and the SensorHub are mutually independent chip systems, the electronic equipment further comprises a TOF camera sensor, the TOF camera sensor is connected with the SensorHub, and the method comprises the steps that in response to a first starting request from the AP, the SensorHub controls the TOF camera sensor to be powered on, and the TOF camera sensor is powered on; the first starting request comprises a camera identifier of the TOF camera sensor; after the TOF camera sensor is powered on, the SensorHub receives first image data from the TOF camera sensor.

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction technology, and in particular to a control method and electronic device for a TOF camera sensor. Background Art

[0002] The TOF camera sensor includes a transmitter (TX) and a receiver (RX), where TX is used to transmit infrared light or laser pulses and RX is used to receive reflected light and form an image. Currently, TOF camera sensors are widely used in the field of human-computer interaction.

[0003] However, after electronic devices are equipped with TOF camera sensors, energy consumption increases significantly, significantly shortening the battery life of the electronic devices. Summary of the invention

[0004] The embodiments of the present application provide a control method for a TOF camera sensor and an electronic device, which are used to solve the energy consumption problem of the electronic device.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a control method for a TOF camera sensor, which is applied to an electronic device, wherein the electronic device includes an application processor AP and a sensor management component SensorHub, wherein the AP and the SensorHub are independent chip systems, and the electronic device also includes a TOF camera sensor, which is connected to the SensorHub. The method includes: in response to a first startup request from the AP, the SensorHub controls the TOF camera sensor to power on, wherein the first startup request includes a camera identifier of the TOF camera sensor; after the TOF camera sensor is powered on, the SensorHub receives first image data from the TOF camera sensor.

[0007] In the above embodiment, the TOF camera sensor is controlled and managed by the SensorHub, which is independent of the AP and has low energy consumption. After the AP instructs the SensorHub to start the TOF camera sensor, the AP does not need to participate in the management and control of the TOF camera sensor. In the absence of other tasks, the AP can also enter a dormant state to reduce the overall energy consumption of the electronic device.

[0008] In some embodiments, an always-on camera (AON) client is running on the AP side, and the AON client can query the camera ID of the camera sensor managed by the SensorHub. When the AP identifies the scenario where the TOF camera sensor needs to be enabled, the AON client can generate the above-mentioned first start request and send it to the SensorHub.

[0009] In some embodiments, before the SensorHub receives the first image data from the TOF camera sensor, the SensorHub writes the first camera parameter into the TOF camera sensor. Correspondingly, after the TOF camera sensor loads the first camera parameter, it starts to collect data. The data collected by the TOF camera sensor may include a RAW image (e.g., the second image data) and corresponding meta information (e.g., the first metadata corresponding to the second image data).

[0010] After the TOF camera sensor acquires the second image data, the SensorHub reads first metadata of the second image data from the TOF camera sensor through the integrated circuit I2C bus, wherein the first metadata includes status information of the TOF camera sensor during the process of acquiring the second image data.

[0011] When the first metadata indicates that the TOF camera sensor operates normally, the SensorHub writes the second camera parameters to the TOF camera sensor. Correspondingly, the TOF camera sensor loads the second camera parameters. After loading the second camera parameters, the TOF camera sensor continues to collect data, and the collected data may include the first image data and the corresponding second metadata.

[0012] In addition, the first current value in the first camera parameter is less than the second current value in the second camera parameter, and the first current value and the second current value are driving currents for the emitter in the TOF camera sensor. In this way, when the TOF camera sensor loads the first camera parameter, the current actually driving TX is not greater than the first current value, and when the TOF camera sensor loads the second camera parameter, the current actually driving TX is not greater than the second current value.

[0013] In the above embodiment, before the TOF camera sensor collects the first image data, the SensorHub can write the first camera parameter into the TOF camera sensor, instructing the TOF camera sensor to use a relatively small current value to drive TX, and determine whether the TOF camera sensor can work normally while reducing the impact of the infrared rays emitted by TX on the user's eyes. When it is determined that the TOF camera sensor can work normally, the second camera parameter is written into the TOF camera sensor, and the current driving TX is increased without affecting the user's eyes to ensure the image quality of the first image data. In this way, the safety of using the TOF camera sensor is improved.

[0014] Exemplarily, a digital signal processor DSP is configured in the SensorHub. The AON client can send a first startup request to the DSP in the SensorHub, and the DSP can control the TOF camera sensor to power on in response to the first startup request, and write the first camera parameter into the TOF camera sensor. After determining that the TOF camera sensor can work normally, the DSP also writes the second camera parameter into the TOF camera sensor, so that the TOF camera sensor can collect the first image data whose image quality meets the processing requirements.

[0015] In this example, the low power consumption characteristics of DSP are used to reduce the energy consumption generated in the process of controlling the TOF camera sensor.

[0016] In some embodiments, when the TOF camera sensor is configured with the first camera parameter, the second image data collected by the TOF camera sensor is not transmitted to the SensorHub.

[0017] In the above embodiment, the SensorHub writes the first camera parameter into the TOF camera sensor, indicating that the TOF camera sensor enters the eye safety detection mode. After the TOF camera sensor configures (or loads) the first camera sensor, the collected data includes two parts: the RAW image part (the second image data) and the meta information (the first metadata). The TOF camera sensor does not transmit the second image data to the SensorHub, which reduces the image processing amount of the DSP after the TOF camera sensor enters the eye safety detection mode, thereby reducing the energy consumption of the device.

[0018] Correspondingly, in other embodiments, after the TOF camera sensor acquires the second image data, it may send a first notification to the SensorHub, instructing the SensorHub to read the first metadata of the second image data to determine whether the TOF camera sensor can work properly. In some examples, the first notification may be sent to the DSP in the SensorHub through the target image signal processor ISP, and the DSP reads the first metadata from the TOF camera sensor and determines whether the TOF camera sensor can work properly. In other embodiments, the first notification may be sent to the target ISP, and the target ISP reads the first metadata from the TOF camera sensor and determines whether the TOF camera sensor can work properly.

[0019] In some embodiments, the SensorHub includes a digital signal processor DSP and a target image signal processor ISP. Before the SensorHub receives the first image data from the TOF camera sensor, the method also includes: the TOF camera sensor divides the collected data into first image data and corresponding second metadata; the SensorHub receives the first image data from the TOF camera sensor, including: the DSP receives the first image data from the TOF camera sensor through the target ISP; after the SensorHub receives the first image data from the TOF camera sensor, the method also includes: the DSP reads the corresponding second metadata from the TOF camera sensor through the I2C bus, and the second metadata includes status information of the TOF camera sensor during the process of collecting the first image data.

[0020] In the above embodiment, the TOF camera sensor transmits the collected data to the DSP through different channels, thereby reducing the data processing requirements for the target ISP, allowing the target ISP to adopt a lighter ISP and reduce energy consumption.

[0021] In some embodiments, the electronic device also includes a display screen, and before the Sensor Hub controls the TOF camera sensor to power on, the method also includes: after the display screen displays the first information or the first interface, the AP sends a first startup request to the DSP in the Sensor Hub, the first information is an interactive display control, and the first interface is an application interface provided by a preset application; after the Sensor Hub receives the first image data from the TOF camera sensor, the method also includes: the Sensor Hub determines the corresponding target eye movement data based on the first image data, the target eye movement data includes the position coordinates of the user's gaze point on the display screen, and the dwell time; when the target eye movement data indicates that the user's gaze point is located in the display area of ​​the first information, the AP responds to the target eye movement data to display a second interface, and the second interface is the interface indicated by the first information; or, when the target eye movement data indicates that the user's gaze point is located in a specified area in the first interface, the AP responds to the target eye movement data to display a third interface, and the third interface is another interface provided by the preset application, and the display content of the third interface is different from that of the first interface.

[0022] Exemplarily, the first information includes notification push information and a reminder capsule control.

[0023] In addition, the preset application may be an application with eye movement recognition permission, which may be pre-configured by the user or a default configured application. The first interface may be an interface with a page jump entry in an application interface provided by the preset application.

[0024] In the above embodiment, displaying the first information and the first interface can be regarded as a scenario where the eye movement recognition service is required. In a scenario where the eye movement recognition service is required, the AP determines that the TOF camera sensor needs to be enabled, and uses the low-power SensorHub to control the TOF camera sensor, and calculates the corresponding eye movement recognition result.

[0025] In the above process, the high-power AP only needs to instruct the SensorHub to turn on the TOF camera sensor, and then it can enter a sleep state and wait for the SensorHub to feedback the eye movement recognition results. Then, according to the eye movement recognition results, it executes the corresponding response to reduce the energy consumption generated by the AP during the eye movement recognition process.

[0026] In some embodiments, the SensorHub determines corresponding target eye movement data based on the first image data, including: the DSP in the SensorHub converts the first image data into an infrared IR image; the DSP in the SensorHub determines the target eye movement data based on the IR image and a preconfigured eye movement recognition model.

[0027] In some embodiments, the electronic device further includes a display screen, and before the SensorHub controls the TOF camera sensor to power on, the method further includes: after the display screen displays the fourth interface, the AP sends a first startup request to the SensorHub.

[0028] Exemplarily, the fourth interface may be any display interface. Also exemplarily, the fourth interface may be an interface that can be displayed in a horizontal display form or in a vertical display form. That is, when the electronic device displays the fourth interface, the display form of the fourth interface may be switched between a horizontal screen and a vertical screen.

[0029] In the above embodiment, in a scenario where the screen can be switched between horizontal and vertical orientations, the TOF camera sensor can be enabled in a low-power manner.

[0030] In some embodiments, after the SensorHub receives the first image data from the TOF camera sensor, the method also includes: the DSP in the SensorHub determines the target face data based on the first image data, and the target face data includes the position of the face area rectangular frame and the face key point array; the DSP sends the target face data to the AP; when the target face data indicates that the relative position between the user's face and the display screen satisfies the first condition, the AP responds to the target face data and displays the fourth interface in horizontal screen; when the target face data indicates that the relative position between the user's face and the display screen satisfies the second condition, the AP responds to the target face data and displays the fourth interface in vertical screen, and the first condition and the second condition respectively indicate different positional relationships.

[0031] In some embodiments, the first condition includes: the angle between the first direction axis of the user's face and the second direction axis of the electronic device is greater than a preset angle; the second condition includes: the angle between the first direction axis and the second direction axis is not greater than a preset angle.

[0032] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising one or more camera sensors, a memory and one or more processors; the one or more camera sensors comprise a TOF camera sensor, the one or more processors comprise an application processor AP and a sensor management component SensorHub, the AP and the SensorHub are independent chip systems; the TOF camera sensor is connected to the SensorHub; the camera sensor is used to collect image data, the memory is used to store computer program code, the SensorHub is used to manage and control the TOF camera sensor, the computer program code comprises computer instructions, when one or more processors execute the computer instructions, in response to a first startup request from the AP, the SensorHub controls the TOF camera sensor to power on, the first startup request comprises a camera identifier of the TOF camera sensor; after the TOF camera sensor is powered on, the SensorHub receives the first image data from the TOF camera sensor.

[0033] In some embodiments, before the SensorHub receives the first image data from the TOF camera sensor, the SensorHub writes the first camera parameter into the TOF camera sensor; after the TOF camera sensor acquires the second image data, the SensorHub reads the first metadata of the second image data from the TOF camera sensor through the integrated circuit I2C bus, wherein the first metadata includes the status information of the TOF camera sensor during the process of acquiring the second image data; when the first metadata indicates that the TOF camera sensor is operating normally, the SensorHub writes the second camera parameter into the TOF camera sensor; the first current value in the first camera parameter is less than the second current value in the second camera parameter, and the first current value and the second current value are driving currents for the emitter in the TOF camera sensor.

[0034] In some embodiments, when the TOF camera sensor is configured with the first camera parameter, the second image data collected by the TOF camera sensor is not transmitted to the SensorHub.

[0035] In some embodiments, the SensorHub includes a digital signal processor DSP and a target image signal processor ISP. Before the SensorHub receives the first image data from the TOF camera sensor, the TOF camera sensor divides the collected data into the first image data and corresponding second metadata. The SensorHub receives the first image data from the TOF camera sensor, including: the DSP receives the first image data from the TOF camera sensor through the target ISP. After the SensorHub receives the first image data from the TOF camera sensor, the method further includes: the DSP reads the corresponding second metadata from the TOF camera sensor through the I2C bus, and the second metadata includes status information of the TOF camera sensor during the process of collecting the first image data.

[0036] In some embodiments, the electronic device also includes a display screen. Before the SensorHub controls the TOF camera sensor to power on, after the display screen displays the first information or the first interface, the AP sends the first startup request to the DSP in the SensorHub, where the first information is an interactive display control, and the first interface is an application interface provided by a preset application. After the SensorHub receives the first image data from the TOF camera sensor, the SensorHub determines the corresponding target eye movement data based on the first image data, and the target eye movement data includes the position coordinates of the user's gaze point on the display screen, and the duration of the stay. When the target eye movement data indicates that the user's gaze point is located in the display area of ​​the first information, the AP displays a second interface in response to the target eye movement data, and the second interface is the interface indicated by the first information. Alternatively, when the target eye movement data indicates that the user's gaze point is located in a specified area in the first interface, the AP displays a third interface in response to the target eye movement data, and the third interface is another interface provided by the preset application, and the display content of the third interface is different from that of the first interface.

[0037] In some embodiments, the SensorHub determines the corresponding target eye movement data based on the first image data, including: the DSP in the SensorHub converts the first image data into an infrared IR image; the DSP in the SensorHub determines the target eye movement data based on the IR image and a preconfigured eye movement recognition model.

[0038] In some embodiments, the first information includes a notification push message and a reminder capsule control.

[0039] In some embodiments, the electronic device further includes a display screen, and before the SensorHub controls the TOF camera sensor to power on and after the display screen displays the fourth interface, the AP sends the first start request to the SensorHub.

[0040] In some embodiments, after the SensorHub receives the first image data from the TOF camera sensor, the DSP in the SensorHub determines the target face data based on the first image data, and the target face data includes the position of the rectangular frame of the face area and an array of face key points; the DSP sends the target face data to the AP; when the target face data indicates that the relative position between the user's face and the display screen satisfies a first condition, the AP displays the fourth interface in horizontal mode in response to the target face data; when the target face data indicates that the relative position between the user's face and the display screen satisfies a second condition, the AP displays the fourth interface in vertical mode in response to the target face data, and the first condition and the second condition respectively indicate different positional relationships.

[0041] In some embodiments, the first condition includes: the angle between the first direction axis of the user's face and the second direction axis of the electronic device is greater than a preset angle; the second condition includes: the angle between the first direction axis and the second direction axis is not greater than the preset angle.

[0042] In a third aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method in the above-mentioned first aspect and possible embodiments thereof.

[0043] In a fourth aspect, the present application provides a computer program product. When the computer program product is run on the above-mentioned electronic device, the electronic device executes the method in the above-mentioned first aspect and possible embodiments thereof.

[0044] It can be understood that the electronic devices, computer storage media and computer program products provided in the above-mentioned aspects are all applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 An example diagram of a scenario of using an electronic device with one hand provided in an embodiment of the present application;

[0046] Figure 2 An example diagram of the X-axis and Y-axis of an electronic device and a human face provided in an embodiment of the present application;

[0047] Figure 3An example diagram of the structure of an electronic device provided in an embodiment of the present application;

[0048] Figure 4 An example diagram of the software and hardware structure of an electronic device provided in an embodiment of the present application;

[0049] Figure 5 One of the example diagrams of the path for controlling the TOF camera sensor provided in the embodiment of the present application;

[0050] Figure 6 The second example diagram of the path for controlling the TOF camera sensor provided in the embodiment of the present application;

[0051] Figure 7 A flowchart of an information processing method provided in an embodiment of the present application;

[0052] Fig. 8A For application Figure 7 One of the example diagrams of the scenario of the method shown;

[0053] Figure 8B For application Figure 7 The second example of the scenario of the method shown;

[0054] Figure 8C For application Figure 7 Figure 3 of the scenario example of the method shown;

[0055] Fig.9A A flow chart of a display control method provided in an embodiment of the present application;

[0056] Fig. 9B For application Fig.9A One of the example diagrams of the scenario of the method shown;

[0057] Fig. 9C For application Fig.9A The second example of the scenario of the method shown;

[0058] Fig.10 To achieve Figure 7 The method shown is a signaling interaction diagram of various software and hardware modules in an electronic device. DETAILED DESCRIPTION

[0059] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0060] In order to make the following embodiments clear and concise, a brief introduction to related concepts or technologies is first given:

[0061] Eye movement recognition refers to the tracking of eye movements by detecting the position of the eye's gaze point or the movement of the eyeball relative to the head. At present, the application effect of eye movement recognition in the field of human-computer interaction is very good. Eye movement recognition can effectively simplify the difficulty of operation in the process of human-computer interaction. For example, Figure 1 As shown, when the user gazes at the display area a of the electronic device, the electronic device can detect that the user's gaze point is located in the display area a through eye movement recognition. In this scenario, it can be determined that the user clicks the display area a.

[0062] In some embodiments, the electronic device needs to collect one or more frames of continuous image data containing the user's eyes through a time of flight (TOF) camera sensor and / or an RGB camera sensor. Based on the above image data, eye movement data can be calculated, and the eye movement data can include the user's eye gaze point, gaze duration, and eye movement trajectory. The electronic device identifies the user's eye movement interaction operation based on the above eye movement data.

[0063] The TOF camera sensor includes a transmitter (TX) and a receiver (RX). TX is used to transmit infrared light or laser pulses, and RX is used to receive reflected light and form an image. After TX emits infrared light or laser pulses and RX receives the corresponding reflected light, the TOF camera can use TOF imaging technology to form an image.

[0064] In a dark environment, the imaging effect of the TOF camera sensor is better than that of the RGB camera sensor. The eye movement data calculated based on the image data collected by the TOF camera sensor is also more accurate. In other words, in a dark environment, the effect of performing eye movement recognition using the TOF camera sensor is also better.

[0065] The horizontal and vertical screen switching relative to the face refers to the electronic device controlling the display screen to adopt the horizontal screen display mode or the vertical screen display mode according to the relative position between the face and the display screen. Figure 2 As shown, the electronic device has an X-axis and a Y-axis. When the plane where the display screen of the electronic device is located is perpendicular to the horizontal plane, the X-axis (second direction axis) of the electronic device is perpendicular to the horizontal plane, and the Y-axis of the electronic device is parallel to the horizontal plane. Figure 2 As shown, the face also corresponds to an X-axis and a Y-axis. The X-axis of the face is the longitudinal axis of the face plane image (such as called the first direction axis), and the Y-axis of the face is the transverse axis of the face plane image. Exemplarily, when the X-axis of the electronic device is perpendicular to the X-axis of the face, the electronic device is controlled to adopt a horizontal screen display mode. Exemplarily, when the X-axis of the electronic device is parallel to the X-axis of the face, the electronic device is controlled to adopt a vertical screen display mode.

[0066] In the process of switching between horizontal and vertical screens relative to the face, the electronic device also needs to collect image data containing the face through the RGB camera sensor and / or the TOF camera sensor, and evaluate the relative position between the face and the display screen by analyzing the image data containing the face. Then, according to the relative position between the face and the display screen, the display screen is controlled to enable the horizontal screen display mode or the vertical screen display mode.

[0067] Similarly, in a dark environment, the relative position identified based on the image data collected by the TOF camera sensor is more accurate. In this way, in a dark environment, the effect of switching the horizontal and vertical screen relative to the face using the TOF camera sensor is also better.

[0068] However, whether using the TOF camera sensor for eye movement recognition or using the TOF camera sensor for horizontal and vertical screen switching relative to the face, the application processor (AP) in the electronic device cannot sleep during the above process, resulting in high energy consumption of the entire device.

[0069] In order to reduce energy consumption when an electronic device uses a TOF camera sensor, an embodiment of the present application provides a control method for a TOF camera sensor, which can be applied to the electronic device.

[0070] Exemplarily, after applying the above method, when the electronic device uses the TOF camera sensor to perform eye movement recognition, the digital signal processor (DSP) connected to the sensor management component (SensorHub) controls the TOF camera sensor and processes the image data collected by the TOF camera sensor. It can be understood that the SensorHub and the application processor (AP) in the electronic device are independent of each other. During the process of the DSP in the SensorHub performing eye movement recognition processing, the AP can sleep and wait, effectively reducing the overall energy consumption of the electronic device.

[0071] Exemplarily, after applying the above method, when the electronic device uses the TOF camera sensor to switch the screen between horizontal and vertical relative to the face, the DSP connected to the SensorHub controls the TOF camera sensor and processes the image data collected by the TOF camera sensor. When the DSP in the SensorHub performs eye movement recognition processing, the AP can sleep and wait, effectively reducing the overall energy consumption of the electronic device.

[0072] In some embodiments, the electronic device may be a mobile phone, a tablet computer, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, or other Android devices, etc. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0073] Please refer to Figure 3 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0074] like Figure 3 As shown, the electronic device (such as a mobile phone) may include: a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a SIM card interface 295, etc.

[0075] Among them, the above-mentioned sensor module 280 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0076] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0077] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a DSP, a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0078] The electronic device may include one or more DSPs, at least one of which is integrated in the SensorHub. Of course, the AP may also be integrated with a DSP. The operating power consumption of the SensorHub is lower than that of the AP and is not dependent on the AP. For example, when the AP is in sleep mode, the SensorHub can still operate normally.

[0079] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0080] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0081] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0082] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0083] The charging management module 240 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 240 charges the battery 242, it can also power the electronic device through the power management module 241.

[0084] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, and supplies power to the processor 210, the internal memory 221, the external memory, the display screen 294, the camera 293, and the wireless communication module 260. In some embodiments, the power management module 241 and the charging management module 240 can also be set in the same device.

[0085] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor, etc. In some embodiments, the antenna 1 of the electronic device is coupled with the mobile communication module 250, and the antenna 2 is coupled with the wireless communication module 260, so that the electronic device can communicate with the network and other devices through wireless communication technology, such as communication with wearable devices.

[0086] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0087] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to electronic devices. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation.

[0088] The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 250 can be set in the same device as at least some modules of the processor 210.

[0089] The wireless communication module 260 can provide wireless communication solutions for electronic devices including WLAN (such as (wirelessfidelity, Wi-Fi) network), Bluetooth (bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), nearfield communication technology (nearfield communication, NFC), infrared technology (infrared, IR), etc.

[0090] Among them, GNSS may include Beidou navigation satellite system (BDS), global positioning system (GPS), global navigation satellite system (GLONASS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation systems (SBAS).

[0091] The wireless communication module 260 may be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 210. The wireless communication module 260 may also receive signals to be sent from the processor 210, modulate the frequencies of the signals, amplify the signals, and convert the signals into electromagnetic waves for radiation via the antenna 2.

[0092] The electronic device implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0093] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel.

[0094] The electronic device can realize the shooting function through the ISP, the camera 293, the video codec, the GPU, the display screen 294 and the application processor, etc. The ISP is used to process the data fed back by the camera 293.

[0095] The camera 293 is used to capture static images or videos. In some embodiments, the electronic device may include 1 or N cameras 293, where N is a positive integer greater than 1. The N cameras 293 may include a camera configured with a TOF camera sensor and a camera configured with an RGB camera sensor.

[0096] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0097] The internal memory 221 may be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 221. For example, in an embodiment of the present application, the processor 210 may execute the instructions stored in the internal memory 221, and the internal memory 221 may include a program storage area and a data storage area.

[0098] The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0099] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0100] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes a system with a layered architecture as an example to exemplify the software structure of the electronic device.

[0101] Figure 4 It is a block diagram of the software and hardware structure of the electronic device according to an embodiment of the present invention.

[0102] The layered architecture of electronic devices divides software and hardware into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the electronic device may include, for example, an application layer, an application framework layer, a hardware abstraction layer (HAL), a kernel layer, a SensorHub, and a hardware layer. Of course, it may also include Figure 4 Layers not shown in the diagram, such as the runtime and system libraries.

[0103] Exemplarily, the application layer may include a series of applications.

[0104] like Figure 4 As shown, the application layer may include applications such as notifications, capsule reminders, short videos, etc. In some embodiments, the application layer may also include Figure 4 Applications not shown, such as call, camera, map, navigation, etc. applications.

[0105] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0106] like Figure 4 As shown, the application framework layer may include intelligent perception services, etc.

[0107] The intelligent perception service is used to process the eye movement recognition requests sent by the application and return the processing results to the application to implement the eye movement recognition function.

[0108] In other embodiments, the application framework layer may also include Figure 4 Modules not shown in the figure, such as window manager, content provider, view system, resource manager, notification manager, etc.

[0109] The hardware abstraction layer (HAL) can encapsulate the drivers in the kernel layer and provide a calling interface to the application framework layer, shielding the implementation details of the low-level hardware.

[0110] like Figure 4 As shown, the HAL of the electronic device includes at least an always-on camera (AON) service, an AON processing node, an AON client and an AON interface.

[0111] Among them, the AON service is used to interact with the application framework layer. The AON service includes an interface that can be called by the application framework layer. The interface can receive call requests from the application framework layer and call other software modules in the HAL (such as AON processing nodes, AON clients, and AON interfaces), triggering the awakening of the DSP in the SensorHub. The DSP executes the process of controlling the TOF camera sensor, instructs the above-mentioned TOF camera sensor to collect image data, and returns the corresponding result data (such as image data or eye movement data).

[0112] The AON processing node may also be referred to as AON svrsys, and the AON processing node may be used to process image data from a TOF camera sensor, for example, to filter the image data. The image data from the TOF camera sensor may be transmitted to an AON service via the AON processing node.

[0113] The AON client can be used to interact with the kernel layer, for example, to send control instructions to the SensorHub through the kernel layer. For example, the above-mentioned SensorHub includes a DSP and an ISP, and the AON client can instruct the SensorHub to manage the TOF camera sensor connected to the SensorHub through the DSP and the ISP.

[0114] The AON interface (eg, HISEE) is a communication interface for interaction between the AON client and the kernel layer, and can convert the instructions of the AON client into instruction information recognizable by the kernel layer.

[0115] like Figure 4 As shown in Figure 1, the kernel layer is the layer between hardware and software. The kernel layer at least includes the Qualcomm Message Interface (QMI). In addition, the kernel layer may also include Figure 4 Drivers not shown, such as camera sensor driver, display driver, sensor driver, etc.

[0116] Among them, QMI is a functional interface for multi-processor inter-process communication, for example, it can be used for communication between AP and SensorHub. It can be understood that the above-mentioned intelligent perception services, AON services, AON processing nodes, AON clients and AON interfaces all run on AP, and TOF camera sensors and DSPs are managed by SensorHub. AP and SensorHub belong to different chip systems and can run independently of each other. The software modules on the AP side (such as intelligent perception services, AON services, AON processing nodes, AON clients and AON interfaces) can instruct SensorHub to manage DSP and connected TOF camera sensors through QMI.

[0117] like Figure 4 As shown, the SensorHub includes a DSP and an AON ISP (target ISP).

[0118] Among them, the above-mentioned DSP includes an AON camera control module and multiple types of algorithms.

[0119] In some embodiments, the AON camera control module can control the working state and working mode of the TOF camera sensor.

[0120] For example, control the TOF camera sensor to power on, capture images, and power off.

[0121] For another example, determine the camera parameters that need to be written into the TOF camera sensor. Exemplarily, the AON camera control module can write the eye-safe mode parameters to the TOF camera sensor to control the TOF camera sensor to enter the eye-safe mode. Among them, in this eye-safe mode, the TX of the TOF camera sensor is driven by a current less than the preset threshold to ensure that the infrared light emitted by the TX of the TOF camera sensor does not cause harm to the user's eyes. In this mode, the device status detection is completed to confirm whether the device status is normal.

[0122] After the eye safety test is passed, the AON camera control module can also write working camera parameters to the TOF camera sensor to control the TOF camera sensor to enter the normal operation mode. In this normal operation mode, the TX of the TOF camera sensor can be driven with a current greater than the preset threshold, and the infrared light emitted by the TX of the TOF camera sensor is controlled to not cause harm to the user's eyes, while being able to collect image data that meets the requirements of eye movement recognition.

[0123] In other embodiments, the AON camera control module may control the working state and working mode of other types of camera sensors involved in AON, such as an RGB camera sensor.

[0124] In some embodiments, the above-mentioned multiple types of algorithms may include a TOF processing algorithm, a face recognition algorithm, an eye movement recognition algorithm (also referred to as an eye movement recognition model), etc. Among them, the above-mentioned TOF processing algorithm can be used to convert the image data collected by the TOF camera sensor into a TOF infrared (infrared radiation, IR) image, the above-mentioned eye movement recognition algorithm can calculate the corresponding eye movement data based on the image data, and the above-mentioned face detection algorithm can detect whether a face appears in the image data.

[0125] In addition, AON ISP is a lightweight ISP. Compared with conventional ISPs, AON ISP has fewer functions and a single business, and its operating energy consumption is also lower.

[0126] The aforementioned I2C bus is used to transfer data between the DSP and the TOF camera sensor.

[0127] In some embodiments, the image data collected by the TOF camera sensor may include raw data (RAWData, or RAW image) and metadata (meta Data, or meta information). Raw data refers to the data obtained after the TOF camera sensor converts the captured light source signal into a digital signal. Metadata refers to the information generated by the TOF camera sensor in the process of shooting raw data, for example, including information indicating the working mode of the TOF camera sensor, information indicating the size of the photocurrent value, information indicating the working status of the TOF camera sensor, information indicating the exposure value, etc.

[0128] In the embodiment of the present application, in the same frame of image data collected by the TOF camera sensor, the RAW Data is transmitted to the DSP via the AON ISP. The meta information is transmitted to the DSP via the I2C bus. In this way, the resolution of the data to be processed by the AON ISP is reduced, and correspondingly, the energy consumption can also be reduced.

[0129] like Figure 4As shown, the hardware layer includes a TOF camera sensor. It is understandable that the hardware layer of the electronic device may also include Figure 4 Hardware modules not shown, such as display screen, memory, AP, other types of camera sensors, etc.

[0130] Combine the following Figure 5 , the interaction between the software modules involved in the method provided in the embodiment of the present application is described.

[0131] like Figure 5 As shown in the figure, in the application layer, applications (such as notification applications, capsule prompt applications, and short videos) can call the smart perception service in the application framework layer according to business logic. For example, after the notification application detects the user's instruction to turn on the eye movement recognition function, it calls the smart perception service to turn on the eye movement recognition service, so as to respond to the user's instruction without contact and display the application interface corresponding to the notification.

[0132] For example, when displaying a message capsule provided by a capsule prompt application, or when the information displayed by a message capsule is updated, the intelligent perception service can also be called to turn on the eye movement recognition service to respond to user instructions without contact and display detailed information about the content displayed in the message capsule.

[0133] For example, after the short video detects the user's instruction to turn on the eye movement recognition function, it calls the intelligent perception service to turn on the eye movement recognition service, so as to respond to the user's instructions and switch the video being played without contact.

[0134] like Figure 5 As shown, the intelligent perception service in the application framework layer can interact with the AON service in the HAL layer in response to the call of the application.

[0135] In response to the call of the intelligent perception service, the AON service can instruct the AON client through the AON processing node to determine the camera sensor that performs eye movement recognition.

[0136] In some embodiments, the AON client can determine the camera sensor that performs eye movement recognition among the camera sensors connected to the SensorHub.

[0137] Exemplarily, the camera sensor of the front camera of the electronic device (such as a TOF camera sensor and an RGB camera sensor) can be connected to the SensorHub, so that the SensorHub can control and manage the camera sensor of the front camera through the DSP.

[0138] In this way, the AON client can determine the camera sensor that performs eye movement recognition between the TOF camera sensor and the RGB camera sensor according to preset rules.

[0139] Exemplarily, the above rule can be selected based on the ambient light brightness. For example, when the ambient light brightness value is greater than a preset brightness threshold, the RGB camera sensor is selected. When the ambient light brightness value is less than the preset brightness threshold, the TOF camera sensor is selected. Exemplarily, the above rule can also be selected based on whether the user's eyes are blocked by sunglasses. For example, if the user's eyes are blocked by sunglasses, the TOF camera sensor is selected.

[0140] In the subsequent embodiments, the AON client selects the TOF camera sensor as an example for description. It can be understood that the process of the AON client selecting the RGB camera sensor to perform the image acquisition task in the eye movement recognition can refer to the TOF camera sensor, and the embodiments of this application will not be repeated.

[0141] In some embodiments, after the AON client determines to enable the TOF camera sensor for eye movement recognition, it can interact with the kernel layer through the AON interface to instruct the DSP in the SensorHub to control the TOF camera sensor to power on and start image acquisition.

[0142] It can be understood that AON services, AON processing nodes, AON clients and AON interfaces all run on AP, occupying the computing resources of AP. AP and SensorHub are independent processing systems. AON interface can instruct SensorHub's DSP to control the TOF camera sensor to start image acquisition through QMI in the kernel layer, and instruct DSP to calculate the corresponding eye movement data based on the image data collected by the TOF camera sensor. The above process can be simply referred to as instructing DSP to actually perform eye movement recognition.

[0143] like Figure 5 As shown, the DSP includes an AON camera control module, and the AON interface notifies the AON camera control module through the QMI to control the TOF camera sensor to start working. In response to the notification from the QMI, the AON camera control module controls the TOF camera sensor to power on through the AON ISP and starts to collect image data.

[0144] In some embodiments, Figure 6 As shown in the figure, after the TOF camera sensor collects image data, the TOF camera sensor collects a frame of image data, the RAW image can be passed to the AON camera control module through the AON ISP, and the meta information is passed to the AON camera control module through the I2C bus. The AON ISP only needs to process the RAW image to reduce operating energy consumption.

[0145] After the AON camera control module receives the RAW image from the TOF camera sensor, a pre-configured multi-class algorithm can be called to calculate the corresponding eye movement data based on the image data. Exemplarily, the TOF algorithm can be used to convert the RAW image from the TOF camera sensor into a TOF IR image. Then, the face detection algorithm is used to detect the face area rectangular frame and the face key point array from the TOF IR image. According to the determined face area rectangular frame and the face key point array, combined with the pre-configured eye movement recognition algorithm, the gaze point position coordinates are determined, and the gaze point position coordinates can be determined with a specified position point in the display screen (which can be the upper left corner of the display screen) as the reference origin. The implementation principle of the above-mentioned eye movement recognition algorithm can refer to the relevant technology, which will not be repeated here.

[0146] After the AON camera control module calculates the eye movement data, the eye movement data can be sent back to the AON client through the QMI and AON interfaces. Afterwards, the AON processing node and AON service feed back to the smart perception service. The smart perception service feeds back the eye movement data to the application that requests the eye movement recognition service, such as a notification application, a capsule prompt application, or a short video application. In this way, the above-mentioned application can identify the user's contactless instructions based on the eye movement data, and respond accordingly, thereby providing contactless interactive services to the user.

[0147] It can be understood that after the AON interface instructs the DSP of the SensorHub to perform eye movement recognition, if there is no other task on the AP side, it can enter sleep mode and wait for the DSP of the SensorHub to feedback the corresponding eye movement recognition result (ie, eye movement data).

[0148] It can be seen that during the process of DSP performing eye movement recognition, the chip system to which the AP belongs does not need to participate and can even enter sleep mode. In addition, the operating energy consumption of the Sensor Hub and the DSP itself is also low. In short, compared with electronic devices in related technologies, the electronic device provided in the embodiment of the present application consumes less energy when performing eye movement recognition. In particular, in the scenario where the horizontal and vertical screen switching function relative to the face is turned on, the effect of reducing energy consumption is more obvious.

[0149] The following describes the implementation details of the method provided in the embodiment of the present application in combination with the accompanying drawings and actual scenarios.

[0150] like Figure 7As shown, based on the control method of the above-mentioned TOF camera sensor, an information processing method is also provided in an embodiment of the present application. After the electronic device applies the information processing method, it provides users with various types of contactless interactive services based on the eye movement recognition function (for example, application scenarios one to three mentioned in the following embodiments). The energy consumption generated is lower than that of other devices, thereby improving the battery life of the electronic device while ensuring the user experience.

[0151] One of the exemplary application scenarios: After the eye movement recognition function is turned on, if the notification application has the permission to enable the eye movement recognition function, the notification application can request the smart perception service to obtain the eye movement recognition service. The smart perception service instructs the DSP in the SensorHub to perform eye movement recognition and feedback the detected eye movement data through the AON service, AON processing node, AON interface, etc.

[0152] In this way, in the scenario where an application that has a business association with the notification application (for example, an application that enables the application message push service) needs to push application messages through the notification application, the notification application can judge the user's intention based on the eye movement data and respond accordingly to provide the user with contactless interactive services. In this process, compared with electronic devices of related technologies, since the main process of eye movement recognition is executed by a low-power chip system (SensorHub), energy consumption is also lower.

[0153] like Fig. 8A As shown, when the SMS application of the electronic device receives a SMS message, such as a verification code SMS message, the notification application of the electronic device can instruct the display screen to display a push banner 701 (first information, or notification push information) of the SMS message. While displaying the push banner 701, the notification application can determine whether the user intends to view the SMS message displayed by the push banner 701 based on the eye movement data fed back by the DSP in the SensorHub.

[0154] For example, when the user gazes at the display area of ​​the push banner 701, the eye movement data fed back by the DSP in the SensorHub may indicate that the user's gaze point is located in the display area of ​​the push banner 701. In the scenario where the eye movement data from the DSP in the SensorHub indicates that the user's gaze point is located in the display area of ​​the push banner 701 and the duration of the gaze reaches a preset duration, the notification application may determine that the user has the intention to view the SMS information displayed in the push banner 701.

[0155] As a corresponding response, the notification application can instruct the display screen to display the information interface 702 (second interface) provided by the SMS application, wherein the information interface 702 includes the complete content of the SMS message displayed on the push banner 701, that is, the complete content of the verification code SMS message.

[0156] Exemplary application scenario 2: After the eye movement recognition function is turned on in an electronic device, if the capsule prompt application has the permission to enable the eye movement recognition function, after the electronic device is turned on, the capsule prompt application can request the eye movement recognition service from the smart perception service. The smart perception service instructs the DSP in the SensorHub to perform eye movement recognition and feedback the detected eye movement data through the AON service, AON processing node, AON interface, etc.

[0157] In the above scenario, an application that has a business association with the capsule prompt application (for example, an application that authorizes the capsule prompt application to provide message reminders) can display application messages related to the business execution (for example, business progress messages) through the capsule prompt application when executing a preset type of business.

[0158] For example, the taxi app authorizes a capsule to prompt the application model message reminder. When the taxi app responds to user operations and performs the vehicle reservation service, the capsule prompt application can display application messages related to the vehicle reservation service. For example, an order acceptance message indicating that the reserved vehicle has been accepted.

[0159] like Figure 8B As shown, when the capsule reminder application receives an application message from the taxi application, such as an order acceptance message, the electronic device can display a brief order acceptance message on the reminder capsule 703 (first information, or called a reminder capsule control).

[0160] While content is displayed on the reminder capsule 703 , the capsule reminder application can determine whether the user intends to view the order acceptance message displayed on the reminder capsule 703 based on the eye movement data fed back by the DSP in the Sensor Hub.

[0161] For example, when the user is looking at the display area of ​​the reminder capsule 703, the eye movement data fed back by the DSP in the SensorHub can indicate that the user's gaze point is located in the display area of ​​the reminder capsule 703. In the scenario where the eye movement data from the DSP in the SensorHub indicates that the user's gaze point is located in the display area of ​​the reminder capsule 703 and the stay time reaches a preset time, the capsule reminder application can determine that the user has the intention to view the order acceptance message displayed by the reminder capsule 703.

[0162] After determining that the user has the intention to view, the capsule reminder application can display the reminder window 704 and hide the reminder capsule 703. The display size of the reminder window 704 is larger than the display size of the reminder capsule 703. Compared with the reminder capsule 703, the reminder window 704 can display more content. Figure 8B As shown, the reminder window 704 includes the detailed content of the order receiving message. In addition, the interface showing the reminder window 704 can be called the second interface.

[0163] Exemplary application scenario 3: After the eye movement recognition function is turned on in the electronic device, if the short video application has the permission to enable the eye movement recognition function, after the short video application is running, the short video application can request the smart perception service to obtain the eye movement recognition service. The smart perception service instructs the DSP in the SensorHub to perform eye movement recognition and feedback the detected eye movement data through the AON service, AON processing node, AON interface, etc.

[0164] like Figure 8C As shown, after the short video application is running, a video browsing interface 705 (first interface) can be displayed, in which video data 706 is played. The video browsing interface 705 includes an operation area 707 and an operation area 708 for indicating video switching, which are called designated areas.

[0165] For example, when it is detected that the user clicks on the operation area 707, the short video application can switch to play the previous video data (the video data before the video data 706 in the recommended order). When it is detected that the user clicks on the operation area 708, the short video application can switch to play the next video data (the video data after the video data 706 in the recommended order).

[0166] While the video browsing interface 705 is displayed, the short video application can determine whether the user intends to view the previous video data or the next video data based on the eye movement data fed back by the DSP in the SensorHub.

[0167] For example, when the user gazes at the display area of ​​the operating area 707 , the eye movement data fed back by the DSP in the Sensor Hub may indicate that the user's gaze point is located in the display area of ​​the operating area 707 .

[0168] In a scenario where the eye movement data indicates that the user's gaze point is located in the display area of ​​the operation area 707 and the duration of the gaze reaches a preset duration, the short video application can determine that the user intends to view the previous video data.

[0169] For another example, when the user gazes at the display area of ​​the operating area 708 , the eye movement data fed back by the DSP in the Sensor Hub may indicate that the user's gaze point is located in the display area of ​​the operating area 708 .

[0170] In a scenario where the eye movement data indicates that the user's gaze point is located in the display area of ​​the operation area 708 and the duration of the gaze reaches a preset duration, the short video application can determine that the user intends to view the next video data.

[0171] like Figure 8CAs shown, after determining that the user has the intention to view the next video, a video browsing interface 709 (third interface) is displayed, in which video data 710 is being played. The video data 710 is the next video data after the video data 706 in the recommended order.

[0172] In addition, applications that have permission to enable the eye tracking recognition function can also be called preset applications.

[0173] In some embodiments, Figure 7 As shown, the above information processing method may include the following steps:

[0174] S101, during display of a preset type of interactive object, instructing the TOF camera sensor to collect image data 1 through the DSP in the SensorHub.

[0175] Exemplarily, the above-mentioned interactive object may be a display control for displaying the information to be processed. For example, the above-mentioned interactive object may be Fig. 8A In the push banner 701, for example, the above interactive object can also be Figure 8B Reminder capsule 703 in.

[0176] For example, the interactive object may be a specific display control or a display area for performing a specific interactive operation. Figure 8C The operation area 707 and the operation area 708 in the figure.

[0177] It is understandable that the scene for displaying the above interactive objects can refer to the aforementioned Fig. 8A , Figure 8B and Figure 8C Of course, other types of interactive objects can also be displayed in other scenarios, and this embodiment of the present application does not specifically limit this.

[0178] In some embodiments, after the AP of the electronic device displays the above interactive object on the display screen, it can instruct the TOF camera sensor to collect images through the DSP in the SensorHub. The collected image data can be referred to as image data 1.

[0179] The implementation details of the above-mentioned instructing the TOF camera sensor to collect image data 1 through the DSP in the SensorHub can be referred to the description of the subsequent embodiments, which will not be repeated here.

[0180] S102, based on the image data 1, calculating the corresponding eye movement data.

[0181] In some embodiments, after the DSP in the SensorHub receives the image data 1 collected from the TOF camera sensor, the DSP can convert the image data 1 into a TOF IR image. Then, the DSP can also extract face detection data such as a face area rectangular frame and a face key point array from the TOF IR image, and calculate the corresponding eye movement data based on the face detection data. For specific implementation details, please refer to the implementation details of calculating eye movement data based on image data collected by the TOF camera sensor in the related technology, which will not be described here.

[0182] S103, when the eye movement data meets a preset condition, executing an interactive response related to the above-mentioned interactive object in response to the eye movement data.

[0183] Exemplarily, the above-mentioned preset condition may be that the eye movement data indicates that the user's gaze point is located in the display area of ​​the interactive object and the stay time reaches a preset time length.

[0184] In some embodiments, the AP may receive eye movement data from the DSP in the SensorHub. When the eye movement data meets preset conditions, the AP performs an interactive response related to the above-mentioned interactive object in response to the eye movement data.

[0185] Among them, different interactive objects may have different corresponding interactive responses. For example, when the interactive object is a push banner 701 provided by a notification application, the relevant interactive response may be an application interface corresponding to the content (text message) shown in the push banner 701. For another example, when the interactive object is a reminder capsule 703 provided by a capsule prompt application, the relevant interactive response may be a reminder window 704 that replaces the reminder capsule 703. The display size of the reminder window 704 is larger than the display size of the reminder capsule 703, and more information content can be displayed compared to the reminder capsule 703. For another example, when the interactive object is an operation area 707 (or an operation area 708) in a video browsing interface 705 provided by a short video application, the relevant interactive response may be switching to play the previous video data (or, switching to play the next video data).

[0186] In the above embodiments, electronic devices can provide users with various types of contactless interactive services based on the eye movement recognition function in various application scenarios, thereby improving the human-computer interaction efficiency of electronic devices. At the same time, most of the processing tasks corresponding to the eye movement recognition function are performed by the DSP in the SensorHub. Even if the eye movement recognition function is frequently enabled, it will not cause excessive energy consumption, thereby ensuring the battery life of the electronic device.

[0187] like Fig.9AAs shown, based on the control method of the TOF camera sensor, the embodiment of the present application also provides a display control method. After the electronic device applies the display control method, the energy consumption generated during the period when the electronic device provides the user with a horizontal or vertical screen switching service relative to the face (the fourth application scenario mentioned in the subsequent embodiments) is lower than that of other devices. In this way, in a dimly lit scene, the TOF camera sensor is used to accurately switch the horizontal and vertical screen display modes while effectively controlling the energy consumption generated to ensure the battery life of the electronic device.

[0188] Exemplary application scenario 4: After the electronic device turns on the horizontal and vertical screen switching function relative to the face, the horizontal and vertical screen switching application can request the eye movement recognition service from the smart perception service. The smart perception service instructs the DSP in the SensorHub to return the face detection data through the AON service, AON processing node, AON interface, etc., wherein the above-mentioned face detection data includes a face area rectangular frame and a face key point array, and the face detection data can be feature data extracted from the TOF IR image from the TOF camera sensor. In this way, the horizontal and vertical screen switching application can determine the relative position relationship between the user's face and the electronic device based on the face detection data, and control the display screen to switch between the horizontal screen display mode and the vertical screen display mode based on the above-mentioned relative position relationship.

[0189] like Fig. 9B As shown, when the user holds the electronic device so that the electronic device is in landscape mode relative to the user's face, the face detection data returned by the DSP in the SensorHub can indicate that the X-axis of the face is perpendicular to the X-axis of the electronic device. In this scenario, in response to the face detection data returned by the DSP, the display screen can be controlled to adopt landscape display mode.

[0190] like Fig. 9C As shown, when the user holds the electronic device in a vertical screen mode relative to the user's face, the face detection data returned by the DSP in the SensorHub can indicate that the X-axis of the face is parallel to the X-axis of the electronic device. In this scenario, in response to the face detection data returned by the DSP, the display screen can be controlled to adopt the vertical screen display mode.

[0191] In some embodiments, Fig.9A As shown, the above display control method may include the following steps:

[0192] S201, instructing the TOF camera sensor to collect image data 1 through the DSP in the SensorHub.

[0193] In some embodiments, after the electronic device is turned on, the DSP in the SensorHub may instruct the TOF camera sensor to periodically collect images, and the collected image data may be referred to as image data 1. The image data 1 may include a user's face image.

[0194] S202, based on the image data 1, determine the relative position relationship between the user's face and the electronic device.

[0195] In some embodiments, when the DSP determines that there is a face image in the image data 1, the DSP can extract the corresponding face detection data from the image data 1. Afterwards, the AP of the electronic device can determine the relative position relationship between the user's face and the electronic device based on the face detection data. For example, determine the relative position relationship between the X-axis of the face and the X-axis of the electronic device.

[0196] S203, when the relative position relationship between the user's face and the electronic device meets condition 1, control the display screen to adopt a horizontal screen display mode.

[0197] Exemplarily, condition 1 (the first condition) may be that the X-axis of the face is perpendicular to the X-axis of the electronic device. Exemplarily, condition 1 may also be that the angle between the X-axis of the face and the X-axis of the electronic device is greater than a preset angle, which may be a value less than 90 degrees.

[0198] In some embodiments, after the AP determines that the X-axis of the face is perpendicular to the X-axis of the electronic device, Fig. 9B As shown, the AP can control the display screen to adopt the horizontal screen display mode.

[0199] In some other embodiments, after the AP determines that the angle between the X-axis of the face and the X-axis of the electronic device is greater than a preset angle, such as Fig. 9B As shown, the AP can control the display screen to adopt the horizontal screen display mode.

[0200] S204: When the relative position relationship between the user's face and the electronic device meets condition 2, control the display screen to adopt a vertical display mode.

[0201] Among them, the above condition 2 (second condition) is the opposite condition to condition 1. In the case where condition 1 may be that the X-axis of the face is perpendicular to the X-axis of the electronic device, the above condition 2 may be that the X-axis of the face is parallel to or not perpendicular to the X-axis of the electronic device. In the case where condition 1 may be that the angle between the X-axis of the face and the X-axis of the electronic device is greater than a preset angle, the above condition 2 may be that the angle between the X-axis of the face and the X-axis of the electronic device is not greater than a preset angle.

[0202] In some embodiments, after the AP determines that the X-axis of the face and the X-axis of the electronic device meet condition 2, Fig. 9C As shown, the AP can control the display screen to adopt the vertical display mode.

[0203] In short, whether it is the information processing method or the display control method provided in the embodiment of the present application, both control the TOF camera sensor through the DSP in the SensorHub, and both use the DSP in the SensorHub to process the image data collected by the TOF camera sensor. Effectively utilize the low power consumption of the DSP to reduce the energy consumption of electronic devices. The following takes the information processing method as an example to introduce the implementation details of using the DSP in the SensorHub to control the TOF camera sensor:

[0204] like Fig.10 As shown, in the process of implementing the above method, the signaling interaction steps between the various software and hardware modules in the electronic device are as follows:

[0205] S301, the application sends an eye movement recognition service request to the intelligent perception service.

[0206] In some embodiments, the above-mentioned application may be a notification application, a capsule prompt application, a short video application, or other application that has permission to use the eye movement recognition service.

[0207] It is understandable that different applications may send eye movement recognition service requests to the smart perception service at different times. In addition, the above eye movement recognition service request can be used to register a callback corresponding to eye movement recognition on the smart perception service. After the smart perception service receives the eye movement data, the eye movement data can be fed back to the application through the above registered callback.

[0208] Taking the notification application as an example, after the notification application receives the application message that needs to be pushed (such as an application message from an application with business association), in addition to controlling the display screen to display the push information containing the application message (such as a push banner), it can also send an eye movement recognition service request to the smart perception service. In response to the eye movement recognition service request, the smart perception service can instruct the DSP to feedback real-time eye movement data through other software modules. The specific implementation process can refer to S302~S334. After the smart perception service receives the eye movement data, it can also send the eye movement data to the notification application.

[0209] In addition, in response to the display duration of the application message reaching duration 1, or in response to the user's instruction to cancel the display of the application message, the notification application may also send information to the smart perception service to instruct the cancellation of the eye movement recognition service. In this way, the smart perception service may stop sending eye movement data to the notification application. The duration 1 may be the maximum display duration corresponding to the pre-configured push information.

[0210] Taking the short video application as an example, after the short video application enters the foreground, it can send an eye movement recognition service request to the smart perception service. In response to the eye movement recognition service request, the smart perception service can instruct the DSP to feedback real-time eye movement data through other software modules. The specific implementation process can refer to S302~S334. After the smart perception service receives the eye movement data, it can also send the eye movement data to the short video application.

[0211] In addition, after the short video application enters the background or stops running, the short video application can also send information to the smart perception service to instruct the cancellation of the eye movement recognition service. In this way, the smart perception service can stop sending eye movement data to the short video application.

[0212] Taking the capsule prompt application as an example, after any content is displayed on the reminder capsule provided by the capsule prompt application, an eye movement recognition service request can be sent to the smart perception service. In response to the eye movement recognition service request, the smart perception service can instruct the DSP to feedback real-time eye movement data through other software modules. The specific implementation process can refer to S302~S334. After the smart perception service receives the eye movement data, it can also send the eye movement data to the capsule prompt application.

[0213] In addition, after canceling the displayed content on the reminder capsule provided by the capsule prompt application, information indicating the cancellation of the eye movement recognition service can be sent to the smart perception service, so that the smart perception service can stop sending eye movement data to the capsule prompt application.

[0214] S302, the intelligent perception service sends registration information 1 to the AON service, instructing to turn on eye movement recognition.

[0215] The above registration information 1 is used to register a callback corresponding to eye movement recognition on the AON service. After the AON service obtains the eye movement recognition result (such as eye movement data) from the DSP, the eye movement recognition result can be fed back to the intelligent perception service through the above registered callback.

[0216] In some embodiments, the intelligent perception service may send registration information 1 to the AON service in response to the eye movement recognition service request.

[0217] In other embodiments, during the operation of the electronic device, if the callback registration of eye movement recognition has been completed on the AON service before the intelligent perception service receives the eye movement recognition service request sent by the application, S302 to S327 can be skipped and the eye movement data from the DSP can be directly waited for.

[0218] For example, after the electronic device is turned on and before it is turned off, it first receives an eye movement recognition service request from a short video application, and then receives an eye movement recognition service request from a notification application. In this scenario, the smart perception service responds to the eye movement recognition service request from the short video application and sends registration information 1 to the AON service, indicating that eye movement recognition is turned on. After receiving the eye movement recognition service request from the notification application, registration information 1 may no longer be sent to the AON service.

[0219] S303, the AON service sends registration information 1 to the AON processing node.

[0220] S304, the AON processing node sends registration information 1 to the AON client.

[0221] S305, the AON client sends a start instruction 1 to the AON interface, including identification information indicating the TOF camera sensor.

[0222] In some embodiments, before S305, the AON client can select the camera sensor to be used according to a preset rule in response to the registration information 1. In this embodiment, the selection of the TOF camera sensor is mainly taken as an example, so that the start instruction 1 may include the identification information of the TOF camera sensor. The identification information may be the camera ID of the TOF camera sensor.

[0223] S306, the AON interface sends a start instruction 1 to the AON camera control module.

[0224] It can be understood that the AON interface is a communication interface between the AON client and the kernel layer, and the kernel layer includes the QMI for transferring data between different processors.

[0225] In some embodiments, the AON client can send a startup instruction 1 (such as a first startup request) to the QMI through the AON interface, and then the QMI passes it to the AON camera control module of the DSP in the SensorHub. In this way, the AON client instructs the AON camera control module to control the power-on of the TOF camera sensor by sending the startup instruction 1, and the process is as shown in S307 and S308.

[0226] S307, the AON camera control module instructs the AON ISP to control the TOF camera sensor to power on.

[0227] It can be understood that the AON camera control module can manage and control multiple camera sensors. The AON camera control module can respond to the startup instruction 1 and control the power-on of the TOF camera sensor through the AON ISP according to the identification information carried in the startup instruction 1.

[0228] S308, AON ISP controls the TOF camera sensor to power on.

[0229] In some embodiments, the AON camera control module may control the TOF camera sensor to power on through the AON ISP in response to the startup instruction 1. After the TOF camera sensor is powered on, the process proceeds to S309.

[0230] S309, after the TOF camera sensor is powered on, the TOF camera sensor sends a notification 1 to the AON ISP.

[0231] S310, the AON ISP sends a notification 1 to the AON camera control module.

[0232] S311, the AON camera control module sends notification 1 to the AON interface.

[0233] S312, the AON interface sends a notification 1 to the AON client.

[0234] S313, the AON client sends a notification 1 to the AON processing node.

[0235] Among them, notification 1 may indicate that the TOF camera sensor has been powered on. Through the above S309 to S313, after the TOF camera sensor is powered on, the AON processing node may search for camera parameters corresponding to the TOF camera sensor and pass them to the TOF camera sensor for configuration.

[0236] Exemplarily, the camera parameters corresponding to the TOF camera sensor may include calibrated eye-safe mode parameters and conventional working camera parameters. The above-mentioned eye-safe mode parameters and conventional working camera parameters may be data calibrated for electronic equipment during the production line testing phase. In subsequent embodiments, the above-mentioned camera parameters may also be referred to as TOF calibration data.

[0237] S314, the AON processing node sends TOF calibration data to the AON client, including calibrated eye-safe mode parameters.

[0238] Exemplarily, the AON processing node searches for the corresponding TOF calibration data in response to notification 1. The TOF calibration data includes eye safety mode parameters, and the eye safety mode parameters include a safe operating current value calibrated for the TOF camera sensor (e.g., referred to as a first current value). It can be understood that the TX of the TOF camera sensor is driven by the safe operating current value, and the infrared light generated has little effect on the user's eyes. The safe operating current value can be the preset threshold mentioned in the aforementioned embodiment.

[0239] Of course, the eye-safe mode parameters also include other types of attribute parameters required for the operation of the TOF camera sensor. It can be understood that after the TOF camera sensor loads the eye-safe mode parameters, it can enter the eye-safe mode to test whether the TOF camera sensor can be used normally without generating infrared light that harms the user's eyes.

[0240] S315, the AON client sends the TOF calibration data to the AON interface.

[0241] S316, the AON interface sends the TOF calibration data to the AON camera control module.

[0242] In some embodiments, the AON interface can send TOF calibration data to the AON camera control module through the QMI. Correspondingly, the AON camera control module can control the working mode of the TOF camera sensor according to the TOF calibration data. The working modes include at least an eye-safe mode and a normal operation mode.

[0243] Exemplarily, the AON camera control module can first control the TOF camera sensor to enable the eye-safe mode based on the TOF calibration data. In the eye-safe mode, the TOF camera sensor meets the pre-configured safety requirements, and then control the TOF camera sensor to enter the normal operation mode. The specific process can be seen in S317~S327.

[0244] S317, the AON camera control module parses the TOF calibration data and obtains eye-safe mode parameters.

[0245] It can be understood that the above TOF calibration data not only includes the eye-safe mode parameters (first camera parameters) required to enable the eye-safe mode, but also includes the working camera parameters (second camera parameters) required to enable the normal operation mode. Among them, the working camera parameters include the normal working current value (second current value) calibrated for the TOF camera sensor, which is not less than the safe working current value, and the normal working current value can ensure that the TOF camera sensor collects image data with image quality that meets the processing requirements.

[0246] Exemplarily, the AON camera control module parses the TOF calibration data, obtains the eye-safe mode parameters and the working camera parameters from the TOF calibration data, and then the process enters S318.

[0247] S318, the AON camera control module sends the eye safety mode parameters to the AON ISP.

[0248] S319, AON ISP sends eye-safe mode parameters to the TOF camera sensor.

[0249] In some embodiments, the AON camera control module writes the eye-safe mode parameters to the TOF camera sensor through the AON ISP, instructing the TOF camera sensor to load the eye-safe mode parameters and operate in the eye-safe mode.

[0250] S320: The TOF camera sensor loads eye-safe mode parameters and starts image acquisition.

[0251] In some embodiments, after receiving the eye-safe mode parameters, the TOF camera sensor can load the eye-safe mode parameters. For example, according to the eye-safe mode parameters, the attribute values ​​of different registers in the TOF camera sensor are configured so that the TOF camera sensor can operate according to the rules of the eye-safe mode.

[0252] For example, after the TOF camera sensor loads the eye-safe mode parameters, it drives the TX of the TOF camera sensor to illuminate according to the safe working current value specified by the eye-safe mode and starts to collect images.

[0253] S321, the TOF camera sensor sends a notification 2 to the AON ISP, indicating that image data a has been collected.

[0254] The image data a (second image data) is a portion of the image captured by the TOF camera sensor that does not contain meta information, and may also be referred to as a RAW image. The notification 2 may also be referred to as a first notification.

[0255] S322, AON ISP sends notification 2 to the AON camera control module.

[0256] In some embodiments, the TOF camera sensor may divide the captured image into two parts: a RAW image and meta information. In the eye-safe mode, the TOF camera sensor may not send the captured RAW image to the DSP, that is, the image data a is not sent to the DSP, thereby reducing the workload of the DSP. Correspondingly, the TOF camera sensor may send a notification 2 to the DSP via the ISP, notifying the DSP that the TOF camera sensor has captured a frame of image. In response to the notification 2, the DSP may read the meta information corresponding to the image data a via the I2C bus.

[0257] In other embodiments, the TOF camera sensor can also send the collected RAW image to the DSP in the eye-safe mode, which is not specifically limited in the embodiments of the present application.

[0258] S323, after receiving the notification 2 from the TOF camera sensor, the AON camera control module receives the meta information corresponding to the image data a.

[0259] In some embodiments, after the TOF camera sensor loads the eye-safe mode parameters, it can continuously capture multiple frames of images, or periodically capture images at preset time intervals.

[0260] Each time the TOF camera sensor captures a frame of image, the meta information of the image data a in the image can be transmitted to the AON camera control module through the AON ISP. Alternatively, the meta information (first metadata) of the image data a can be transmitted to the AON camera control module through the I2C bus.

[0261] S324, the AON camera control module determines whether the TOF camera sensor meets the preset safety requirements based on the obtained meta information.

[0262] It can be understood that the above-mentioned meta information records the information generated by the TOF camera sensor in the process of shooting image data a. The meta information may include status information 1, which may indicate whether the working state of TX of the TOF camera sensor is normal under the safe working current value. When the status information 1 indicates that the working state of TX is normal, the AON camera control module may determine that the TOF camera sensor meets the preset safety requirements. When the status information 1 indicates that the working state of TX is abnormal, the AON camera control module may determine that the TOF camera sensor does not meet the preset safety requirements.

[0263] In the case where it is determined that the TOF camera sensor does not meet the preset safety requirements, the AON camera control module can wait for the next frame of image data a and obtain the meta information of the next frame of image data a. In this way, it is possible to continue to determine whether the TOF camera sensor meets the preset safety requirements based on the meta information of the next frame of image data a.

[0264] When it is determined that the TOF camera sensor meets the preset safety requirements, the process proceeds to S325.

[0265] In other embodiments, S321 to S324 can be replaced by: each time the TOF camera sensor collects a frame of image data a, the meta information corresponding to the image data a can be sent to the AON ISP, and the AON ISP determines whether the TOF camera sensor meets the preset security requirements based on the meta information. After determining that the security requirements are met, the AON ISP can notify the AON camera control module, so that the process enters S325.

[0266] S325, after determining that the safety condition is met, the AON camera control module sends the working camera parameters to the AON ISP.

[0267] S326, AON ISP sends working camera parameters to the TOF camera sensor.

[0268] It can be understood that in S317, the AON camera control module can not only parse out the eye safety mode parameters, but also parse out the working camera parameters. After determining that the TOF camera sensor meets the safety conditions, the AON camera control module can write the working camera parameters to the TOF camera sensor through the AON ISP, instructing the TOF camera sensor to load the working camera parameters to enter the normal operation mode.

[0269] S327, the TOF camera sensor loads working camera parameters and starts image acquisition.

[0270] In some embodiments, after receiving the working camera parameters, the TOF camera sensor can load the working camera parameters. For example, according to the working camera parameters, the property values ​​of different registers in the TOF camera sensor are configured so that the TOF camera sensor can switch from the eye-safe mode to the normal operation mode.

[0271] S328, the TOF camera sensor sends the collected image data b to the AON ISP.

[0272] The above-mentioned image data b (such as referred to as the first image data) is the part of the image collected by the TOF camera sensor in the normal operation mode that does not contain meta information, and can also be referred to as a RAW image.

[0273] For example, the TOF camera sensor can periodically collect images at preset time intervals. After loading the working camera parameters, the RAW image corresponding to each frame of the collected image can be referred to as image data b.

[0274] S329, AON ISP sends the collected image data b to the AON camera control module.

[0275] S330, the AON camera control module calls a preconfigured multi-category algorithm to calculate the eye movement data corresponding to the image data b.

[0276] In some embodiments, in response to receiving the image data b, the AON camera control module may call a preconfigured multi-class algorithm based on the image data b to calculate the corresponding eye movement data (also referred to as target eye movement data). The process of calculating the eye movement data may refer to the aforementioned embodiment and will not be described in detail here.

[0277] In other embodiments, each time the AON camera control module receives a frame of image data b, it can read corresponding meta information (second metadata) from the TOF camera sensor through the I2C bus.

[0278] S331, the AON camera control module sends eye movement data to the AON interface.

[0279] For example, the eye movement data is transmitted to the AON interface on the AP side through QMI.

[0280] S332, the AON interface sends the eye movement data to the AON client.

[0281] In some embodiments, the TOF camera sensor can continuously collect images, and correspondingly, the AON camera control module can also continuously calculate the corresponding eye movement data. Each time the AON camera control module calculates an eye movement data, it can be sent to the AON client through the QMI and AON interfaces.

[0282] S333, the AON client sends the eye movement data to the AON processing node.

[0283] S334, the AON processing node sends the eye movement data to the AON service.

[0284] S335, the AON service sends eye movement data to the intelligent perception service.

[0285] In some embodiments, the AON client may send the received eye movement data to a smart perception service that is registered to obtain eye movement data.

[0286] S336, the intelligent perception service sends the eye movement data to the application.

[0287] In some embodiments, after receiving the eye movement data, the smart perception service may send the eye movement data to the application that has requested and has not canceled the eye movement recognition service, so that the application can execute a corresponding response based on the eye movement data.

[0288] The above S301 to S336 introduces the signaling interaction between various software and hardware modules in the electronic device in the process of implementing the information processing method. In other embodiments, in the process of the electronic device implementing the display control method, the signaling interaction between various software and hardware modules has the following differences: First, replace S301 with: after the application enters the foreground, it sends a horizontal and vertical screen switching service request to the smart perception service. Among them, the application may refer to an application that turns on the dynamic switching of the horizontal and vertical screen display mode. Secondly, replace S302 to S304 with: the smart perception service sends registration information 2 to the AON client, and the registration information 2 may indicate the relative position between the monitored face and the display screen. Then, replace S330 with: call the pre-configured multi-class algorithm to calculate the face detection data (target face data) corresponding to the image data b. Afterwards, replace S331 to S336 with: the face detection data is fed back to the application so that the application can determine whether to control the display screen to adopt the horizontal screen display mode (or the vertical screen display mode) and display the corresponding application interface (the fourth interface).

[0289] In other embodiments, after calculating the face detection data corresponding to the image data b, the AON camera control module may determine the relative position between the face and the electronic device based on the face detection result, and determine whether to adopt the horizontal display mode or the vertical display mode. Then, S331 to S336 are replaced by: sending information indicating the horizontal display mode (or, the vertical display mode) to the application, instructing the application to control the display screen to adopt the horizontal display mode (or, the vertical display mode) to display the corresponding application interface.

[0290] The embodiment of the present application also provides an electronic device, which may include: a camera, a display screen, a memory, and one or more processors. The memory is coupled to the processor. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various steps performed by the electronic device in the above embodiment. Of course, the electronic device includes but is not limited to the above memory and one or more processors.

[0291] The embodiment of the present application also provides a chip system, which can be applied to the electronic device in the aforementioned embodiment. The chip system includes at least one processor and at least one interface circuit. The processor can be the processor in the above-mentioned electronic device. The processor and the interface circuit can be interconnected by lines. The processor can receive and execute computer instructions from the memory of the above-mentioned electronic device through the interface circuit. When the computer instructions are executed by the processor, the electronic device can execute the various steps executed by the electronic device in the above-mentioned embodiment. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0292] In some embodiments, through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0293] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0294] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0295] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A control method for a TOF camera sensor, characterized in that: Applied to an electronic device, the electronic device includes an application processor AP and a sensor management component SensorHub, the AP and the SensorHub are independent chip systems, the electronic device also includes a TOF camera sensor, the TOF camera sensor is connected to the SensorHub, and the method includes: In response to a first startup request from the AP, the SensorHub controls the TOF camera sensor to power on, wherein the first startup request includes a camera identifier of the TOF camera sensor; After the TOF camera sensor is powered on, the SensorHub receives first image data from the TOF camera sensor.

2. The method according to claim 1, characterized in that Before the SensorHub receives the first image data from the TOF camera sensor, the method further includes: The SensorHub writes the first camera parameter into the TOF camera sensor; After the TOF camera sensor acquires the second image data, the SensorHub reads first metadata of the second image data from the TOF camera sensor through an integrated circuit I2C bus, wherein the first metadata includes state information of the TOF camera sensor in a process of acquiring the second image data; When the first metadata indicates that the TOF camera sensor is operating normally, the SensorHub writes second camera parameters into the TOF camera sensor; a first current value in the first camera parameter is less than a second current value in the second camera parameter, and the first current value and the second current value are driving currents for the emitter in the TOF camera sensor.

3. The method according to claim 2, characterized in that When the TOF camera sensor is configured with the first camera parameter, the second image data collected by the TOF camera sensor is not transmitted to the SensorHub.

4. The method according to any one of claims 1 to 3, characterized in that: The SensorHub includes a digital signal processor DSP and a target image signal processor ISP. Before the SensorHub receives the first image data from the TOF camera sensor, the method further includes: The TOF camera sensor divides the collected data into the first image data and the corresponding second metadata; The SensorHub receives the first image data from the TOF camera sensor, including: the DSP receives the first image data from the TOF camera sensor through the target ISP; After the SensorHub receives the first image data from the TOF camera sensor, the method further includes: the DSP reading the corresponding second metadata from the TOF camera sensor through the I2C bus, where the second metadata includes status information of the TOF camera sensor during the process of acquiring the first image data.

5. The method according to any one of claims 1 to 4, characterized in that: The electronic device further includes a display screen, and before the SensorHub controls the TOF camera sensor to power on, the method further includes: After the display screen displays the first information or the first interface, the AP sends the first start request to the DSP in the SensorHub, where the first information is an interactive display control, and the first interface is an application interface provided by a preset application; After the SensorHub receives the first image data from the TOF camera sensor, the method further includes: The SensorHub determines corresponding target eye movement data according to the first image data, where the target eye movement data includes the position coordinates of the user's gaze point on the display screen and the duration of the gaze; When the target eye movement data indicates that the user's gaze point is located within the display area of ​​the first information, the AP displays a second interface in response to the target eye movement data, where the second interface is the interface indicated by the first information; Alternatively, when the target eye movement data indicates that the user's gaze point is located in a specified area in the first interface, the AP displays a third interface in response to the target eye movement data, where the third interface is another interface provided by the preset application, and the display content of the third interface is different from that of the first interface.

6. The method according to claim 5, characterized in that The SensorHub determines corresponding target eye movement data according to the first image data, including: The DSP in the SensorHub converts the first image data into an infrared IR image; The DSP in the SensorHub determines the target eye movement data based on the IR image and a preconfigured eye movement recognition model.

7. The method according to claim 5 or 6, characterized in that: The first information includes notification push information and a reminder capsule control.

8. The method according to any one of claims 1 to 4, characterized in that: The electronic device further includes a display screen, and before the SensorHub controls the TOF camera sensor to power on, the method further includes: After the display screen displays the fourth interface, the AP sends the first start request to the Sensor Hub.

9. The method according to claim 8, characterized in that After the SensorHub receives the first image data from the TOF camera sensor, the method further includes: The DSP in the SensorHub determines target face data according to the first image data, wherein the target face data includes the position of a rectangular frame of a face area and an array of key face points; The DSP sends the target face data to the AP; when the target face data indicates that the relative position between the user's face and the display screen satisfies the first condition, the AP displays the fourth interface in a horizontal screen in response to the target face data; When the target facial data indicates that the relative position between the user's face and the display screen satisfies the second condition, the AP displays the fourth interface in vertical screen in response to the target facial data, and the first condition and the second condition respectively indicate different positional relationships.

10. The method according to claim 9, characterized in that The first condition includes: the angle between the first direction axis of the user's face and the second direction axis of the electronic device is greater than a preset angle; the second condition includes: the angle between the first direction axis and the second direction axis is not greater than the preset angle.

11. An electronic device, characterized in that: The electronic device includes one or more camera sensors, a memory and one or more processors; the one or more camera sensors include a TOF camera sensor, the one or more processors include an application processor AP and a sensor management component SensorHub, the AP and the SensorHub are independent chip systems; the TOF camera sensor is connected to the SensorHub; The camera sensor is used to collect image data, the memory is used to store computer program code, the SensorHub is used to manage and control the TOF camera sensor, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1-10.

12. A readable storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 10.

Images