Control of brightness of display

Through the time-of-flight sensor and neural network system, the user's head direction is estimated, and the energy-consuming display brightness adjustment is achieved, which solves the problems of high camera energy consumption and confidentiality, and provides efficient brightness control and user's head orientation detection.

CN118942425APending Publication Date: 2024-11-12STMICROELECTRONICS INT NV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410575074.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-05-01
Filing Date
2024-05-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, using a camera to monitor the user's head orientation to automatically adjust the brightness of the display has problems of high energy consumption and confidentiality, and brightness control cannot be achieved without energy consumption.

Method used

The time-of-flight sensor and neural network system are used to measure the user's head distance and photon signal value, calculate the standard deviation and confidence, estimate the user's direction, and control the display brightness and refresh rate based on this to achieve energy-consuming brightness adjustment.

Benefits of technology

Energy-free brightness control is realized, the user's confidentiality is ensured, and the power consumption of the display is reduced, providing user head orientation estimation accuracy similar to that of the camera.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942425A_ABST
    Figure CN118942425A_ABST
Patent Text Reader

Abstract

The invention relates to control of brightness of a display. The present disclosure relates to a system comprising a microcontroller, the microcontroller comprising a neural network; a time-of-flight sensor comprising a plurality of pixels and configured to perform capture of a scene comprising a user, the capture comprising, for each pixel, a measurement of a distance from the user and a measurement of a signal value. The sensor is further configured to calculate a value of a standard deviation associated with the distance value, and a value of a standard deviation associated with the signal value and a confidence value. The sensor is further configured to provide a value to the neural network. The neural network is further configured to generate an estimate of a direction associated with the user based on the value. The system further includes a display, the microcontroller configured to control the display, or another circuit, based on the estimate.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to French patent application No. FR2304677, filed on May 11, 2023, entitled “CONTROL OF THE BRIGHTNESS OF A DISPLAY”, which is incorporated herein by reference to the maximum extent permitted by law. Technical Field

[0003] The present disclosure relates to controlling the brightness of a display by utilizing data obtained by a time-of-flight sensor. Background Art

[0004] Controlling the brightness of displays allows their power consumption to be reduced. Typically, the control is performed by monitoring the orientation of the user's head via a camera (typically via a webcam integrated into the display). The camera is configured to detect when the user is staring in a direction other than the display, whereupon the brightness is reduced, automatically setting the display to, for example, standby. The camera is also configured to detect when the user returns to the display, whereupon the brightness of the display is increased.

[0005] However, even when the display is in reduced operation and / or when the user is no longer in front of the display, using the camera also compromises the user's confidentiality. Further, as a peripheral device with high power consumption cost, saving power by reducing brightness is inconsistent with the power consumption of the camera.

[0006] There is a need for improved methods of automatically adjusting the brightness of a display. In particular, it is desirable to implement the brightness control without consuming energy. It is further desirable to implement the brightness control without compromising the confidentiality of the user. Summary of the invention

[0007] One embodiment provides a system comprising a microcontroller comprising a neural network; a time-of-flight sensor coupled to the microcontroller and configured to perform a first capture of an image scene comprising a user, the sensor comprising a plurality of pixels, the first capture comprising, for each pixel, a measurement of a distance from a user of the system, and a measurement of a signal value corresponding to a number of photons returning to the sensor per unit time, the sensor being further configured to, after the first capture and for each pixel, calculate a value of a standard deviation associated with the distance value, a value of a standard deviation associated with the signal value, and a confidence value, the sensor being further configured to provide the distance, the signal, and the standard deviation value associated with the distance, the standard deviation value associated with the signal, and the confidence value to the neural network associated with each pixel, the neural network being configured to generate an estimate of a direction associated with the user based on the values ​​provided by the sensor, the system further comprising a display coupled to the microcontroller, the microcontroller being configured to control the display or another circuit coupled to the microcontroller based on the estimate of the direction associated with the user.

[0008] According to one embodiment, the microcontroller is further configured to generate a focus value associated with the user, and the microcontroller is configured to control the brightness of the display further based on the focus value.

[0009] According to one embodiment, each pixel of the sensor is further configured to measure a reflectivity value, and the neural network is configured to generate an estimate of the direction further based on the reflectivity value.

[0010] According to one embodiment, the system further comprises a memory storing a software application, and the microcontroller is further configured to control execution of the software application based on the estimate of the direction generated by the neural network.

[0011] According to one embodiment, the system further comprises a backlight unit (BLU), and the microcontroller is configured to turn off the backlight unit based on the estimation of the direction.

[0012] According to one embodiment, the microcontroller is configured to control a refresh rate of the display based on the estimate of the direction.

[0013] According to one embodiment, the sensor is configured to perform a second capture after a first capture, the time interval between the first capture and the second capture being determined by an estimate of the direction generated by the neural network and / or based on a focus value calculated after the first capture.

[0014] According to one embodiment, the sensor is an 8*8 pixel sensor.

[0015] According to one embodiment, the aforementioned system further comprises at least one other display, and the microcontroller is further configured to control the brightness of the at least one other display based on the estimate of the direction associated with the user.

[0016] According to one embodiment, the estimated direction is a direction describing the orientation of the user's head, among north, northeast, northwest, east, west, and south directions, with the north direction indicating that the user is facing the display and the south direction indicating that the user is facing away from the display.

[0017] According to one embodiment, the microcontroller is configured to control the reduction of the brightness of the display when, between at least two consecutive captures, the estimated direction undergoes the following transitions:

[0018] from north to northwest or to northeast; and / or

[0019] from north to south; and / or

[0020] From northwest or northeast to south,

[0021] And wherein the microcontroller is configured to control increasing the brightness of the display when, between at least two consecutive captures, the estimated direction undergoes the following transition:

[0022] from south to north; and / or

[0023] from south to northwest, or from south to northeast; and / or

[0024] From northwest or northeast to north.

[0025] According to one embodiment, the confidence value is a Boolean value indicating whether a measurement performed by a pixel detects a user, and the sensor is configured to not provide the neural network with measurements obtained by pixels that did not detect a user.

[0026] According to one embodiment, the neural network includes: at least one convolutional layer; and at least one dense layer.

[0027] One embodiment provides a method for learning a neural network, comprising:

[0028] capturing a plurality of image scenes of a user including a display by a time-of-flight sensor, each pixel measuring a value of a distance from the user, a signal value, and being configured to calculate a standard deviation associated with the distance value and a standard deviation associated with the signal value;

[0029] removing abnormal images from the captured images by a processor;

[0030] Classifying, by a processor, each of the unremoved images into one of the classes of north, northwest, northeast, east, west, and south;

[0031] The number of images distributed in each class is balanced by the processor;

[0032] selecting, by the processor, an architecture of the neural network; and

[0033] The neural network is trained based on the captured and classified images and based on the values ​​measured for each image pixel, the training comprising finding the parameters of the selected neural network.

[0034] One embodiment provides a method, comprising: capturing, by a time-of-flight sensor, an image scene of a user including a display, the capture comprising measuring, by each sensor pixel, a distance value, a signal value, and a standard deviation value associated with the distance value and a standard deviation value associated with the signal value, the sensor being further configured to generate a confidence value for each pixel; providing, by the sensor, the measurements performed by the pixels to a microcontroller, the microcontroller comprising a neural network configured to estimate a direction describing the orientation of the user's head based on the provided measurements; and controlling, by the microcontroller, the display, or another circuit coupled to the microcontroller, based on the estimate of the orientation of the user's head. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above-mentioned features and advantages and other features and advantages will be described in detail in the following description of specific embodiments given by way of example and not limitation with reference to the accompanying drawings, in which:

[0036] Figure 1 is a block diagram illustrating a system including a display and a time-of-flight sensor according to one embodiment of the present specification;

[0037] Figure 2 illustrates, according to one embodiment of the present specification, processing data obtained by a time-of-flight sensor to estimate the orientation of a user's head of the system;

[0038] Figure 3 The invention illustrates computing data for estimating the orientation of the user's head for each pixel of a time-of-flight sensor;

[0039] Figure 4 illustrates an example application implemented following the estimation of the orientation of the user's head;

[0040] Figure 5 is a flow chart illustrating the steps of a method for processing data obtained by a time-of-flight sensor;

[0041] Fig. 6A An embodiment of a method for estimating attention of a user of a system according to one embodiment of the present specification is illustrated;

[0042] Figure 6B Another embodiment of a method for estimating attention of a user of a system according to one embodiment of the present specification is illustrated; and

[0043] Figure 6C Another embodiment of a method for estimating attention of a user of a system according to one embodiment of the present specification is illustrated. DETAILED DESCRIPTION

[0044] Similar features have been designated by similar reference numerals in the various drawings. In particular, common structural and / or functional features between the various embodiments may have the same reference numerals and may have the same structure, dimensions, and material properties.

[0045] For clarity, only operations and elements useful for understanding the embodiments described herein have been illustrated and described in detail. In particular, the operation of a time-of-flight sensor is known to those skilled in the art and will not be described in detail.

[0046] Unless otherwise indicated, when referring to two elements being connected together, this means a direct connection without any intermediate elements except conductors, and when referring to two elements being coupled together, this means the two elements may be connected or they may be coupled via one or more other elements.

[0047] In the following disclosure, unless otherwise indicated, when referring to absolute position qualifiers such as terms "front", "back", "top", "bottom", "left", "right", etc., or relative position qualifiers such as terms "above", "below", "higher", "lower", etc., or orientation qualifiers such as "horizontal", "vertical", etc., reference is made to the orientation shown in the figure.

[0048] Unless otherwise indicated, the expressions "about," "approximately," "substantially," and "on the order of" mean within 10%, and preferably within 5%.

[0049] Figure 1 is a block diagram illustrating a system 100 including a display (Display) 102 and a time-of-flight sensor (Time-of-Flight Sensor) 104 according to one embodiment of the present description.

[0050] As an example, system 100 illustrates a portable computer, or a desktop computer. Figure 1 A single display is illustrated, but the system 100 may include several displays, for example, two or three displays. In another example, the system 100 is a mobile phone, a tablet, an advertising display, an ATM, and more generally any system including a display and a time-of-flight sensor.

[0051] The system 100 further comprises a microcontroller 106 (MCU) coupled to the time-of-flight sensor 104 via a bus 108. By way of example, the bus 108 is an I2C (Inter-Integrated Circuit) type bus.

[0052] Further, the system 100 includes, for example, a memory 110 (MEM), which includes, for example, a non-volatile memory 112 (NVMEM) and / or a volatile memory 114 (RAM). As an example, the non-volatile memory 112 is a flash memory, and the volatile memory 114 is a random access volatile memory. As an example, the memory 110 is coupled to the microcontroller 106 via a bus 116.

[0053] As an example, system 100 further includes a control circuit 118 (control circuit) coupled to bus 116 and configured to control the brightness of display 102 .

[0054] According to one embodiment, microcontroller 106 includes neural network 120 (neural network) which is configured to estimate the orientation of system 100 and in particular the user's head of display 102 based on data obtained by time-of-flight sensor 104 .

[0055] As an example, after estimating the user head orientation, the microcontroller 106 controls adjusting the brightness of the display 102. As an example, adjusting the brightness of the display 102 is performed via the control circuit 118.

[0056] As an example, the microcontroller 106, and / or the control circuit 118, is configured to reduce the brightness of the display 102 when the estimate of the head orientation shows that the user is looking away from the display 102. When the estimate shows that the user is not looking at the display 102, the microcontroller 106, and / or the control circuit 118, is configured to maintain the brightness of the display 102 or reduce the brightness of the display 102 again. Conversely, when the estimate shows that the user is looking at the display 102 again, or returning to the display 102, the microcontroller 106, and / or the control circuit 118, is configured to increase the brightness of the display 102. Similarly, when the estimate shows that the user stays in front of the display, the microcontroller 106, and / or the control circuit 118, is configured to maintain the brightness.

[0057] Figure 2 Processing data obtained by the time-of-flight sensor 102 to estimate the orientation of the head of the user 200 of the system 100 is illustrated in accordance with one embodiment of the present description.

[0058] According to one embodiment, the time-of-flight sensor 104 includes a plurality of pixels. Figure 2In the example shown in , the time-of-flight sensor 104 comprises 64 pixels arranged in a square with eight pixels on each side.

[0059] The user 200 is, for example, in front of the time-of-flight sensor 104. The user 200 can move his head according to rotational movements around three axes x, y, z: rotation around the z-axis (yaw) corresponds to turning the head to the right or to the left. Rotation around the y-axis (flip) corresponds to rotating the head in a plane parallel to the user 100, i.e., rotating the head towards the shoulders without turning the head to the right or to the left. Finally, rotation around the x-axis (tilt) corresponds to lowering the head towards the ground, or raising the head towards the sky.

[0060] During the capture performed by sensor 104, each pixel of sensor 104 measures different information. As an example, the pixels of sensor 104 are configured to measure the distance between sensor 104 and an object facing sensor 104, etc. The object (such as user 200) does not need to occupy the entire field measured by the sensor. Therefore, when the measured distance is a finite value, or a value less than a threshold, pixel 202 will detect user 200. Other pixels 204 will not detect user 200, and are associated with the background of the captured scene, for example.

[0061] According to one embodiment, after performing the measurement for each pixel, a state value is further generated for each pixel. As an example, the state value is a Boolean value, and when the measured distance indicates that the user 200 is detected by the pixel, the Boolean value has a true value (TRUE), for example. When the value measured by the pixel is abnormal, the state value has a false value (FALSE), for example. As an example, when the user 200 is not detected, the value FALSE is assigned to the pixel.

[0062] exist Figure 2 In the example illustrated in , the state value of pixel 202 is assigned the value TRUE, and the state value of pixel 204 is assigned the value FALSE.

[0063] In another example, the state value is a value belonging to the interval, ie [0, 100]. The state value then represents the confidence index of the pixel.

[0064] According to one embodiment, the measurements performed by the pixels 202 are forwarded to the microprocessor 106 and processed by the neural network 120. The microprocessor 106 is then configured to generate an estimate 206 (user focus) based on the received measurements. The estimate 206 includes an estimate of the orientation of the user's head by the neural network 120. As an example, the estimate of the orientation takes the form of an angle or a direction. The estimate 206, for example, includes an estimate of the user's focus by the microprocessor 106. As an example, the estimate of focus is a Boolean value, taking the value FALSE when the user is not focusing, and taking the value TRUE when the user is focusing. In another example, the estimate of the user's focus is a value belonging to an interval. As an example, the interval considered is the interval [0, 100], the value 0 indicating that the user is not focusing at all, and the value 100 indicating that the user is focusing on the display 102. Given the interval [0, 100] by way of example and not limitation, other user focus intervals or measurements may be considered. The user's focus is estimated, for example, based on the orientation of the user's head estimated by the neural network 120.

[0065] Figure 3 It is illustrated that data for estimating the orientation of the user's head is calculated for each pixel of the time-of-flight sensor 104. As an example, the user data is directly calculated by the sensor 104 based on measurements performed by the pixels.

[0066] The time-of-flight sensor 104 is configured to transmit a pulse of light (generally in the infrared field) to a scene. For each pulse, light reflected by an object located in the scene (here, a user 200 of the system 100) is detected by each pixel. As an example, each pixel includes one or more photosensitive elements configured to detect reflected light. As an example, the photosensitive element is a single photon avalanche diode (SPAD). Generally, in order to obtain a suitable signal-to-noise ratio, each pixel of the time-of-flight sensor 104 is configured to accumulate the signal generated by one or more photosensitive elements during a series of thousands of light pulses emitted by the sensor 104. The distance (distance) between the user 200 and the sensor 104 is then inferred based on the average duration of the pulse sequence, which is measured in millimeters (mm).

[0067] Graph 300 illustrates the accumulation of signals generated by one or more photosensitive elements during a series of thousands of light pulses for a pixel of sensor 104. Axis d represents the distance measured during each flow, while axis "number of photons" represents the number of photons returned per second. The signal value (Signal) corresponds to the average value of the number of photons returned to the pixel per second.

[0068] According to one embodiment, each pixel is further configured to calculate a standard deviation associated with the distance (Σdistance), and a standard deviation associated with the signal (Σsignal).

[0069] According to one embodiment, each pixel is further configured to measure a reflectance value. As an example, the reflectance value is a percentage calculated based on the number of photons sent back by the target and based on the estimated distance to the target. As an example, human skin has a reflectance of about 60% regardless of its skin color. As an example, hair has a reflectance of at least 10% depending on its color.

[0070] As an example, reflectivity values ​​of pixels having a state value of TRUE or above a threshold are passed to the microcontroller 106 .

[0071] According to one embodiment, the sensor 104 is configured to, for each pixel, transmit the value of the distance of the pixel, the value of the signal, the value of the distance standard deviation, and the value of the signal standard deviation, and the state value to the microcontroller 106. As an example, when the state value associated with a pixel is equal to the value FALSE, the microcontroller 106 ignores other values ​​associated with the pixel. The relevant pixel is processed as an invalid pixel by the microcontroller 106, for example. In the example where the state value belongs to an interval, the measurement of the pixel having a value below a threshold as the state value is ignored by the microcontroller 106, for example.

[0072] As an example, distance, signal, standard deviation, and reflectivity measurements for invalid pixels are automatically assigned a default value by the microprocessor 106, such as 0, or a Not a Number (NaN) value.

[0073] Figure 4 An example application implemented after data processing is illustrated.

[0074] As an example, the estimation 206 including the estimation 300 of the orientation of the head of the user 200 (head orientation direction estimation) and the estimation 302 of the attention of the user 200 (user attention TRUE / FALSE interval) is performed by, for example, the microprocessor 106 or (not in Figure 1 Another processor (shown in FIG. 1 ) is used to implement one or more applications 304 (adaptive dimming), 306 (adaptive refresh), and / or 308 (turning on / off BLU and / or T-CON).

[0075] As an example, the estimate 300 of the user's head orientation takes the form of a direction in the directions of north (N), northeast (NE), northwest (NW), east (E), west (W), and south (S). The north direction indicates, for example, that the user is facing the display 102. The east and west directions indicate, for example, that the user is looking 90° to the right and 90° to the left of the display 102, respectively. The south direction indicates, for example, that the user is facing away from the display 102.

[0076] Implementing the application 304 allows adjusting the brightness of the display 102 according to the focus and head orientation of the user's head. As an example, the memory 112 includes instructions that allow implementing the application 304.

[0077] Implementing application 306 allows, for example, slowing down or speeding up the refresh rate of display 102 according to estimation 206. As an example, when it is estimated that the user is not facing display 102, the refresh rate of the display is less than when it is estimated that the user is facing the display.

[0078] The implementation application 308 allows, for example, the system 100 to be turned on or off (not in Figure 1 As an example, when the estimation 206 shows that the user is not facing the display and / or the user is not paying attention, the peripheral circuit is turned off. When the estimation shows that the user returns to the display 102 and / or the user pays attention again, the circuit is turned back on. As an example, the peripheral circuit includes a T-CON (timing control) board and / or a backlight unit BLU.

[0079] Figure 5 is a flow chart illustrating the steps of a method for processing data obtained by a time-of-flight sensor.

[0080] Specifically, combined Figure 5 The described flow chart includes two phases. The training phase (Training Phase) includes the establishment of the neural network 120 and its training. The training phase is performed, for example, before the manufacture of the system 100. The training phase includes, for example, steps 500 to 502. The application phase (Application Phase) includes the implementation of the neural network 120 by the system 100. As an example, the application phase includes steps 503 to 507.

[0081] Step 500 (data capture) of the training phase includes obtaining a plurality of captures of one or more time-of-flight sensors similar to sensor 104. Obtaining a plurality of captures includes capturing a plurality of image scenes by one or more sensors. The captured scenes may include, for example, a user of a display, and the sensors used may be, for example, integrated into the relevant display.

[0082] Each capture includes measuring the distance, the signal, and the standard deviation associated with the distance and the standard deviation associated with the signal by each pixel. As an example, each capture further includes measuring the reflectivity by each pixel.

[0083] In step 501 (data cleaning, labeling, preprocessing), the captures performed in step 500 are, for example, preprocessed. As an example, step 501 includes a cleaning step. Cleaning the captures, for example, includes removing all captures in which no user appears, or several users appear. As an example, the cleaning further includes removing captures that include NaN values ​​for one or more measurements performed. Further, step 501 includes, for example, a labeling step. As an example, the labeling step includes associating each capture that was not removed during cleaning with a direction from the north, northeast, northwest, east, west, and south directions. Further, step 501 includes, for example, a balancing step, in which the number of captures assigned to each direction is balanced. In other words, performing the balancing allows the number of captures associated with each direction to be the same.

[0084] In step 502 (AI training), the architecture of the neural network is searched. As an example, this step is performed by a computer and via an architecture search tool.

[0085] After selecting the neural network, step 502 further includes training the neural network, for example by a computer and via a computational tool or via an architecture search tool. Training the selected neural network allows optimization of parameters contained in the network, such as synaptic weights for dense layers, the number of filters for convolutional layers, etc. The selected and trained neural network is then the neural network 120.

[0086] The neural network 120 is then implemented, for example, in the microprocessor 106 of the system 100 .

[0087] As an example, the neural network 120 includes an input layer configured to receive input data. The input data is, for example, associated with each pixel of the sensor 104, at least 4 measurements: distance, signal, and standard deviation associated with the distance or signal. As an example, a reflectivity measurement is also passed to the neural network 120. In another example, a state value is also passed to the neural network 120. The number of input data is then equal to the number of pixels of the sensor 104 multiplied by the number of measurements performed by each pixel. The values ​​associated with invalid pixels are assigned by the microprocessor 106 as default values, such as null values ​​or non-numeric values. Further, the neural network 120, for example, includes two convolutional layers after the input layer. As an example, the first convolutional layer includes a number of filters equal to the number of measurements passed by each pixel. As an example, the second convolutional layer includes 2 filters. The neural network 120 further includes a flattening layer after the convolutional layer. As an example, during step 502, a so-called dropout layer and / or a Gaussian noise layer is added to reduce overtraining of the neural network 120.

[0088] As an example, the neural network 120 further includes two dense layers. The first dense layer has, for example, a number of neurons equal to the number of output neurons in the second convolutional layer. The second dense layer has, for example, a number of neurons equal to the number of directions that the user's head can have. As an example, the second dense layer includes 6 neurons. The neural network 120 is configured to generate a probability of belonging to each of the directions of north, northeast, northwest, east, west, and south based on the input data. The most likely direction is then an estimate of the user's head orientation.

[0089] The example of the neural network 120 considered is not given in a limiting way and naturally other architectures can be implemented. However, in the case where the number of pixels included by the sensor 104 is reduced, for example when the sensor includes less than one hundred pixels, it is desirable that the neural network 120 has a small size. As an example, the neural network 120 includes less than ten layers. In fact, the small size of the neural network 120 can contribute to energy saving of the system 100, in particular when the display 102 has a low brightness and when the user is not facing the display 102.

[0090] After the neural network 120 is implemented in the microprocessor 106, and during the operation of the system 100, step 503 (capture) of the application phase includes combining Figure 3 The capture of the sensor 104 is described. The measurements performed during the capture are transmitted to the microprocessor 106, for example.

[0091] In step 504 (preprocessing), the microprocessor 106 performs processing of the received measurements. As an example, the processing includes identifying the pixels 202 that detect the user of the display 102. As an example, the pixels that have a state value of TRUE, but are not identified as detecting the user of the display 102, have their state values ​​reallocated to the value FALSE. The measurements performed for these pixels are then replaced by default values. Further, step 504, for example, includes normalizing these measurements. As an example, the value of the distance is normalized to belong to an interval (e.g., the interval [-1, 1]).

[0092] In step 505, the measurements of the pixels are provided as input data to the neural network 120. As an example, these measurements are normalized again when performing step 504. As an example, the value associated with the invalid pixel is a default value. The neural network 120 is then configured to estimate the orientation of the user's head of the display 102, for example, among the north, northeast, northwest, east, west, and south directions based on the provided input data.

[0093] At step 506 (User Focus), a focus value is calculated, for example, by the microprocessor 106, based on the user head orientation. As an example, the sensor 104 is configured to perform a plurality of measurements at moments separated from each other by fixed time intervals or at moments separated from each other by time intervals depending on the estimation performed by the neural network 120. Associated with each moment, the neural network 120 estimates the user head orientation, and this estimation is stored, for example, in a volatile memory. Further, the value of the user focus at a given moment is calculated, for example, based on the estimation of the user head orientation at at least two previous moments.

[0094] In step 507 (APPLY), the estimate of the orientation of the user's 200 head, and the focus value are used, for example, by a microprocessor or by another processor to implement one or more applications, such as, for example, applications 304 , and / or 306 , and / or 308 .

[0095] Fig. 6A , Figure 6B ,and Figure 6C An embodiment of a method for estimating attention of a system user according to one embodiment of the present specification is illustrated.

[0096] As an example, the system includes a volatile memory, such as, for example, a circular buffer, or a first-in, first-out (FIFO) memory. As an example, the volatile memory is included in the microcontroller 106. In another example, the volatile memory is included in the memory 110. For each direction estimate of the neural network 120, the volatile memory is configured to store the direction. Therefore, for each new direction estimate of the neural network 120, the volatile memory is configured to remove the oldest direction and store the new direction. The directions stored by the volatile memory are therefore from consecutive captures of the sensor 104. As an example, the volatile memory is configured to store a number n of directions, for example 10 directions. The size of the volatile memory can naturally vary, and it can store a number higher than 10 consecutive direction estimates, or a number less than 10 consecutive direction estimates.

[0097] exist Fig. 6A , Figure 6B ,and Figure 6C In the example illustrated in FIG. 6 , the volatile memory is represented by a graph 600. Graph 600 includes 10 consecutive direction estimates generated by neural network 120. As an example, the 10 direction estimates are performed from the capture of an image scene 601 performed by sensor 104 at consecutive times T-9 to T. Fig. 6A , Figure 6B ,and Figure 6C In the example illustrated in , the direction estimated at time T is the northwest direction, the direction estimated at time T-1 is the north direction, and so on.

[0098] As an example, the value of interest is a value belonging to an interval, such as the interval [0, 100]. The interval [0, 100] is given only by way of example, and any other interval may be considered, such as, for example, an interval of the form [0, 1], [-1, 1], etc. In general, a confidence interval is characterized by a minimum value and a maximum value.

[0099] exist Fig. 6A In the first embodiment 602 (user focus: small FOA) illustrated in FIG. 1 , the focus value is set to the maximum value when the presence of the user of the display 102 is detected and at least one of the following conditions is met:

[0100] The estimated north direction at time T and time T-1; or

[0101] The north direction is estimated at least six times in the estimation from time T-9 to time T; or

[0102] The distance between the user 200 and the display 102 measured by the sensor 104 is less than a reference distance, the reference distance value being, for example, less than 500 mm. As an example, the reference distance value is equal to 320 mm. As an example, the user presence is detected by applying a detection algorithm, such as, for example, a human and object detection HOD algorithm.

[0103] When none of the above conditions are met, the attention value gradually decreases. As an example, the attention value decreases from a maximum value to a minimum value within a reference time period. As an example, the reference time period is about a few seconds, such as between 2 seconds and 3 seconds.

[0104] exist Figure 6B In the second embodiment 603 (user attention: large FOA) illustrated in FIG. 1 , the attention value is set to a maximum value when the presence of a user of the display 102 is detected (e.g. after applying an algorithm of the HOD type) and at least one of the following conditions is met:

[0105] The estimated north direction at time T and time T-1; or

[0106] The sum of the estimated north, northwest, and northeast directions from time T-9 to time T is at least 8; or

[0107] The distance between the user 200 and the display 102 measured by the sensor 104 is less than a reference distance, the reference distance value being, for example, less than 500 mm. As an example, the reference distance value is equal to 320 mm.

[0108] When none of the above conditions are met, the attention value gradually decreases. As an example, the attention value decreases from a maximum value to a minimum value within a reference time period. As an example, the reference time period is about a few seconds, such as between 2 seconds and 3 seconds.

[0109] exist Figure 6CIn the third embodiment 604 (user focus: angle threshold) illustrated in FIG. 6 , the direction stored in the volatile memory 600 is used to calculate the angle 605 (head angle). As an example, the calculated angle is an angle between -180° and 180°, and corresponds to the average of all directions stored in the volatile memory. As an example, each direction is transformed into an angle, for example, the north direction corresponds to an angle of 0°, the northeast and northwest directions correspond to angles of -45° and 45°, respectively. The east and west directions correspond to angles of -90° and 90°, respectively. The south direction corresponds to an angle of 180°, or corresponds to an angle of -180°.

[0110] Then, the focus value is set to a maximum value when the presence of a user of the display 102 is detected (e.g. after applying an algorithm of the HOD type) and at least one of the following conditions is met:

[0111] The calculated angle belongs to a reference interval, for example a symmetric interval of the form [-θ, θ], where θ is an angle less than 180°. In an example, θ is less than 90°, or less than 45°. In another example, the angle θ is smaller, for example less than 30°, or even less than 20°; or

[0112] The distance between the user and the display 102 measured by the sensor 104 is less than a reference distance, the reference distance value being, for example, less than 500 mm. As an example, the reference distance value is equal to 320 mm.

[0113] When none of the above conditions are met, the attention value gradually decreases. As an example, the attention value decreases from a maximum value to a minimum value within a reference time period. As an example, the reference time period is about a few seconds, such as between 2 seconds and 3 seconds.

[0114] As an example, in combination Fig. 6A , Figure 6B ,and Figure 6C In the described embodiment, the brightness of the display is gradually reduced with the attention value. As an example, as the attention value reaches a minimum value, the display 102 is set to a standby mode.

[0115] One advantage of the described embodiments is that the user head orientation estimation is performed based on measurements performed by a time-of-flight sensor, which consumes much less energy than a camera. Furthermore, the use of a time-of-flight sensor allows the confidentiality of the user to be guaranteed.

[0116] Another advantage of the described embodiments is that, due to the use of time-of-flight sensors, the neural network configured for data processing has a particularly small size, for example less than about ten layers and less than 100,000 parameters, thus allowing not using too many energy resources.

[0117] Another advantage of the described embodiments is that the estimation is performed based on distance measurements or the like, which allows obtaining an estimate with a similar reliability as an estimate obtained based on camera images with a relatively small number of pixels compared to the image resolution of the camera.

[0118] Various embodiments and variations have been described. Those skilled in the art will appreciate that the specific features of these embodiments may be combined and other variations will readily occur to those skilled in the art.

[0119] Finally, the actual implementation of the embodiments and variations described herein is within the capabilities of a person skilled in the art based on the functional description provided above. In particular, with regard to adaptation, the brightness of the display may be changed according to the estimation performed by the neural network. Similarly, the calculation of user attention and its use may be changed. Furthermore, the system may include multiple displays. In such a case, the reference display is the display that the user looks at the most time. The reference display does not need to include a time-of-flight sensor configured to perform capture. A person skilled in the art will be able to calibrate the sensor so that the estimation performed by the neural network corresponds to the orientation of the user's head compared to the reference display. Similarly, when the system includes multiple displays, the user may look at a display other than the reference display without that meaning that he is not paying attention. A person skilled in the art will be able to adjust the brightness control and / or application according to the system configuration, such as, for example, in combination with Figure 3 Implementation of the described application.

Claims

1. A system comprising: microcontrollers, including neural networks; A time-of-flight (ToF) sensor is coupled to the microcontroller and includes a plurality of pixels, the ToF sensor being configured to: performing a first capture of an image scene, the image scene including a user, the first capture including, for each pixel, a measurement of a distance from the user of the system and a measurement of a signal value corresponding to the number of photons returned to the sensor per unit time, After said first capture and for each pixel, calculating a value of a standard deviation associated with said distance value, and a value of a standard deviation associated with said signal value and a confidence value, and providing the distance, the signal, and the standard deviation value associated with the distance and the standard deviation value associated with the signal, and the confidence value to the neural network associated with each pixel, and The neural network is configured to generate an estimate of a direction based on the values ​​provided by the sensor, the direction being associated with the user; as well as A display is coupled to the microcontroller, the microcontroller being configured to control the display, or another circuit coupled to the microcontroller, based on the estimate of the direction associated with the user.

2. The system of claim 1 , wherein each pixel of the sensor is further configured to measure a reflectivity value, and wherein the neural network is configured to generate the estimate of the direction further based on the reflectivity value.

3. The system of claim 1 , further comprising a memory storing a software application, and wherein the microcontroller is further configured to control execution of the software application based on the estimate of the direction generated by the neural network.

4. The system of claim 1, further comprising a backlight unit (BLU), and wherein the microcontroller is further configured to turn off the backlight unit based on the estimate of the direction.

5. The system of claim 1, wherein the microcontroller is configured to control a refresh rate of the display based on the estimate of the direction.

6. The system of claim 1 , wherein the ToF sensor is configured to perform a second capture after the first capture, a time interval between the first capture and the second capture being determined by the estimate of the direction generated by the neural network, and / or based on a focus value calculated after the first capture.

7. The system of claim 1, wherein the microcontroller is further configured to generate an attention value associated with the user, and wherein the microcontroller is configured to control the brightness of the display further based on the attention value. 8 . The system of claim 7 , wherein the ToF sensor is configured to perform a second capture after the first capture, the time interval between the first capture and the second capture being determined based on the value of interest calculated after the first capture.

9. The system of claim 1, further comprising at least one other display, the microcontroller being further configured to control a brightness of the at least one other display based on the estimate of the direction associated with the user.

10. A system according to claim 9, wherein the estimate of the direction is a direction describing the orientation of the user's head among north, northeast, northwest, east, west, and south, the north direction indicating that the user is facing the display and the south direction indicating that the user is facing away from the display.

11. The system of claim 10, wherein the microcontroller is configured to control a reduction in the brightness of the display when, between at least two consecutive captures, the estimate of the direction: - from north to northwest or to northeast; and / or - from north to south; and / or - from northwest or northeast to south, and wherein the microcontroller is configured to control an increase in the brightness of the display when, between at least two consecutive captures, the estimate of the direction: - from south to north; and / or - from south to northwest or from south to northeast; and / or - From northwest or northeast to north.

12. The system of claim 1 , wherein the confidence value is a Boolean value indicating whether the measurement performed by the pixel allows the user to be detected, the sensor being configured to not provide the neural network with the measurement obtained by a pixel that did not detect the user.

13. The system of claim 1, wherein the neural network comprises: At least one convolutional layer; as well as At least one dense layer.

14. A method comprising: capturing a plurality of image scenes by a time-of-flight ToF sensor comprising a plurality of pixels, the plurality of image scenes comprising a user of a display, each pixel measuring a value of a distance from the user, a signal value, and being configured to calculate a standard deviation associated with the distance value, and a standard deviation associated with the signal value; Removing abnormal images from the captured images by a processor; classifying, by the processor, each of the unremoved images into one of the classes of north, northwest, northeast, east, west, and south; balancing, by the processor, the number of images distributed in each class; selecting, by the processor, an architecture of a neural network; as well as The neural network is trained based on the captured and classified images and based on the values ​​measured for each image pixel, the training comprising finding selected parameters of the neural network.

15. The method according to claim 14, further comprising: performing a first capture of an image scene, the image scene including a user, the first capture including, for each pixel, a measurement of a distance to the user and a measurement of a signal value corresponding to a number of photons returned to the sensor per unit time, After said first capture and for each pixel, calculating a value of a standard deviation associated with said distance value, and a value of a standard deviation associated with said signal value and a confidence value, and providing the distance, the signal, and the standard deviation value associated with the distance and the standard deviation value associated with the signal, and the confidence value to the neural network associated with each pixel, and At the neural network, an estimate of a direction associated with the user is generated based on the values ​​provided by the sensors.

16. A method comprising: capturing an image scene by a time-of-flight (ToF) sensor, the image scene including a user of the display, the capturing including measuring, by each sensor pixel, a distance value, a signal value, and a standard deviation value associated with the distance value and a standard deviation value associated with the signal value, the sensor being further configured to generate a confidence value for each pixel; providing, by the sensor, the measurements performed by the pixels to a microcontroller, the microcontroller comprising a neural network configured to estimate a direction based on the provided measurements, the direction describing the orientation of the user's head; as well as The display, or another circuit coupled to the microcontroller, is controlled by the microcontroller based on the estimate of the orientation of the head.

17. The method of claim 16, further comprising generating a focus value associated with the user, and controlling the brightness of the display further based on the focus value.

18. The method of claim 16, further comprising measuring a reflectivity value at each pixel, and generating the estimate of the orientation further based on the reflectivity values. The method of claim 16 , further comprising shutting down a backlight unit based on the estimate of the orientation.

20. A method according to claim 16, wherein the estimate of the orientation is a direction describing the orientation of the user's head among north, northeast, northwest, east, west, and south, the north direction indicating that the user is facing the display and the south direction indicating that the user is facing away from the display.