Screen brightness control
A time-of-flight sensor and neural network system addresses privacy and energy concerns in screen brightness control by estimating user orientation and attention, ensuring efficient and privacy-preserving screen adjustments.
Patent Information
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- STMICROELECTRONICS INT NV
- Filing Date
- 2023-05-11
- Publication Date
- 2026-04-17
AI Technical Summary
Existing screen brightness control methods using cameras compromise user privacy and are energy-intensive, offsetting the energy savings gained from reducing screen brightness.
A system utilizing a time-of-flight sensor and a neural network to estimate user orientation and attention, controlling screen brightness without a camera, by measuring distance, signal, and standard deviations, and adjusting brightness based on user direction and attention.
This approach conserves energy and maintains user privacy by using a time-of-flight sensor, which is less energy-intensive than cameras, while providing reliable screen brightness adjustments based on user presence and orientation.
Smart Images
Figure 00000021_0000 
Figure 00000022_0000 
Figure 00000023_0000
Abstract
Description
Title of the invention: Screen brightness control technical field
[0001] The present description relates to the control of the brightness of a screen by the exploitation of data acquired by a time-of-flight sensor. Previous technique
[0002] Screen brightness control helps reduce energy consumption. Generally, this control is achieved by monitoring the user's head orientation via a camera, usually the webcam integrated into the screen. The camera is configured to detect when the user looks in a direction other than the screen, at which point the screen brightness is reduced, for example, automatically putting the screen into standby mode. The camera is also configured to detect when the user turns their face back towards the screen, at which point the screen brightness is increased.
[0003] However, by operating even when the screen is in a reduced mode and / or when the user is, for example, no longer in front of the screen, the use of the camera compromises the user's privacy. Furthermore, since the camera is a power-intensive device, the energy savings achieved by reducing brightness are offset by the camera's energy consumption.
[0004] There is a need to improve methods for automatically adjusting the brightness of a screen. In particular, it is desirable that the implementation of brightness control not be energy-intensive. It is also desirable that the implementation of brightness control not compromise user privacy. Summary of the invention
[0005] One embodiment provides a system comprising: - a microcontroller including a neural network; - a time-of-flight sensor, connected to the microcontroller, configured to perform an initial capture of an image scene including a user, the sensor comprising a plurality of pixels, the initial capture comprising the measurement, by each pixel, of a distance from the system to the user and a signal value corresponding to the number of photons returning to the sensor per unit of time, the sensor being further configured to, following the initial capture and for each pixel, calculate a standard deviation value associated with the distance value and a standard deviation value associated with the signal value and a confidence value, the sensor being further configured to provide, in association with each pixel, the distance, signal, and standard deviation values associated with the distance and the signal and the confidence value to the neural network, the neural network being configured to generate, on the basis of the values provided by the sensor, an estimate of a direction associated with the user, the system further comprising a screen, connected to the microcontroller, the microcontroller being configured to control the screen, or another circuit connected to the microcontroller, on the basis of the estimate of the direction associated with the user.
[0006] According to one embodiment, the microcontroller is further configured to generate an attention value associated with the user, and wherein the microcontroller is further configured to control the screen brightness based on the attention value.
[0007] According to one embodiment, each pixel of the sensor is further configured to measure a reflectance value and in which the neural network is configured to generate the direction estimate further on the basis of the reflectance values.
[0008] According to one embodiment, the above system further includes a memory comprising an application program and in which the microcontroller is further configured to control the execution of the application program on the basis of the direction estimation generated by the neural network.
[0009] According to one embodiment, the above system further includes a backlight unit (BLU- "Back Light Unit"), and in which the microcontroller is configured to disable the backlight unit, based on direction estimation.
[0010] According to one embodiment, the microcontroller is configured to control a screen refresh rate based on direction estimation.
[0011] According to one embodiment, the sensor is configured to perform, following the first capture, a second capture, the time interval between the first and second captures being determined by the estimation of the direction generated by the neural network and / or on the basis of the attention value calculated following the first capture.
[0012] According to one embodiment, the sensor is an 8x8 pixel sensor.
[0013] According to one embodiment, the above system further comprises at least one another screen, the microcontroller being further configured to control the brightness of at least one other screen based on an estimation of a direction associated with the user.
[0014] According to one embodiment, the estimated direction indicates a direction, describing the orientation of the user's head, among the north, northeast, northwest, east, west and south directions, the north direction indicating that the user is facing the screen and the south direction indicating that the user is facing away from the screen.
[0015] According to one embodiment, the microcontroller is configured to control the Screen brightness decreases when, between at least two successive captures, the estimated direction passes: - from north to northwest or northeast; and / or - from north to south; and / or - from northwest or northeast to south, and in which the microcontroller is configured to control the increase in screen brightness when, between at least two successive captures, the estimated direction passes: - from south to north; and / or - from south to northwest or from south to northeast; and / or - from northwest or northeast to north.
[0016] According to one embodiment, the confidence value is a boolean value indicating whether the measurements taken by the pixel allow the user to be detected; the sensor is configured not to provide the measurements taken by a pixel that does not detect the user to the neural network.
[0017] According to one embodiment, the neural network comprises: - at least one convolution layer; and - at least one dense layer.
[0018] One embodiment provides a method for learning a neural network comprising: - the capture, by a time-of-flight sensor, of a plurality of image scenes including a user of a screen, each pixel measuring a distance value to the user, a signal value and being configured to calculate a standard deviation associated with the distance value and a standard deviation associated with the signal value; - the suppression, by a processor, of aberrant images in the captured images; - the classification, by the processor, of each undeleted image, into a class among the classes north, northwest, northeast, west, east and south; - the balancing, by the processor, of the number of images distributed in each class; - the selection, by the processor, of a neural network architecture; and - training the neural network based on the captured and classified images and on the measured values for each pixel of the images, the training including the search for parameters for the selected neural network.
[0019] One embodiment provides a method comprising: - the capture, by a time-of-flight sensor, of an image scene including a user of a screen, the capture including the measurement, by each pixel of the sensor, of a distance value, a signal value and standard deviation values associated with the distance value and the signal value, the sensor being further configured to generate, for each pixel, a confidence value; - the provision, by the sensor, to a microcontroller of the measurements taken by the pixels, the microcontroller including a neural network configured to estimate, on the basis of the measurements provided, a direction describing the orientation of the user's head; - control, by the microcontroller of the screen or of another circuit connected to the microcontroller, based on the estimation of the orientation of the user's head. Brief description of the drawings
[0020] These features and advantages, as well as others, will be described in detail in the following description of particular embodiments, given by way of non-limiting example, in relation to the accompanying figures, among which:
[0021] [Fig.1] is a block diagram illustrating a system comprising a screen and a time-of-flight sensor, according to an embodiment of the present description;
[0022] [Fig.2] illustrates the processing of data acquired by the time-of-flight sensor in order to estimate the orientation of the head of a user of the system, according to an embodiment of the present description;
[0023] Figure 3 illustrates the calculation, for each pixel of the time-of-flight sensor, of the data used for estimating the orientation of the user's head;
[0024] [Fig.4] illustrates examples of applications implemented following the estimation of the user's head orientation;
[0025] [Fig.5] is a flowchart illustrating steps in a process for processing data acquired by a time-of-flight sensor;
[0026] [Fig. A] illustrates an embodiment of a method for estimating the attention of a user of the system, according to an embodiment of the present description;
[0027] [Fig. B] illustrates another embodiment of a method for estimating the attention of a system user, according to an embodiment of the present description; and
[0028] [Fig. 00C] illustrates another embodiment of a method for estimating the attention of a user of the system, according to an embodiment of the present description. Description of the implementation methods
[0029] The same elements have been designated by the same reference numerals in the different figures. In particular, the structural and / or functional elements common to the different embodiments may have the same reference numerals and may have identical structural, dimensional and material properties.
[0030] For the sake of clarity, only the steps and elements useful for understanding the described embodiments have been shown and are detailed. In particular, the operation of a time-of-flight sensor is known to those skilled in the art and is not not described in detail.
[0031] Unless otherwise specified, when referring to two elements connected together, this means directly connected without intermediate elements other than conductors, and when referring to two elements connected (in English "coupled") together, this means that these two elements can be connected or linked through one or more other elements.
[0032] In the following description, when reference is made to absolute position qualifiers, such as the terms "front", "back", "top", "bottom", "left", "right", etc., or relative position qualifiers, such as the terms "above", "below", "superior", "inferior", etc., or to orientation qualifiers, such as the terms "horizontal", "vertical", etc., reference is made, unless otherwise specified, to the orientation of the figures.
[0033] Unless otherwise specified, the expressions "approximately", "roughly", and "in the order of" mean within 10%, preferably within 5%.
[0034] The [Fig.1] is a block diagram illustrating a system 100 comprising a display 102 and a time-of-flight sensor 104, according to an embodiment of the present description.
[0035] By way of example, system 100 represents a laptop or desktop computer. Although only one screen is illustrated in relation to [Fig. 1], system 100 may include several, for example, two or three screens. In other examples, system 100 is a mobile phone, a tablet, an advertising display, an ATM, and more generally any system comprising a screen and a time-of-flight sensor.
[0036] The system 100 further includes a microcontroller 106 (MCU) connected to the time-of-flight sensor 104 via a bus 108. As an example, bus 108 is an I2C (Inter-Integrated Circuit) type bus.
[0037] The system 100 includes, for example, additional memories 110 (MEM) comprising, for example, a non-volatile memory 112 (NV MEM) and / or a volatile memory 114 (RAM). By way of example, the non-volatile memory 112 is a Flash memory and the memory 114 is a random access memory. By way of example, the memories 110 are connected to the microcontroller 106 via a bus 116.
[0038] By way of example, the system 100 further includes a control circuit 118 (CTRL CIRCUIT), connected to the bus 116, and configured to control the brightness of the screen 102.
[0039] According to one embodiment, the microcontroller 106 includes a neural network 120. The neural network 120 is configured to estimate, from data acquired by the time-of-flight sensor 104, the orientation of the head of a user of the system 100, and in particular of the screen 102.
[0040] For example, following the estimation of the orientation of the user's head, the microcontroller 106 controls the brightness adjustment of the screen 102. For example, the brightness adjustment of the screen 102 is carried out via the control circuit 118.
[0041] By way of example, the microcontroller 106, and / or the control circuit 118, is configured to lower the brightness of the screen 102 when the head orientation estimation shows that the user is looking away from the screen 102. When the estimations show that the user is not looking in the direction of the screen 102, the microcontroller 106, and / or the control circuit 118, is configured to leave the screen 102 at its current brightness, or to lower it again. Conversely, when the estimations show that the user is looking back at the screen 102, the microcontroller 106, and / or the control circuit 118, is configured to increase the screen 102's brightness. Similarly, when the estimations show that the user remains facing the screen, the microcontroller 106, and / or the control circuit 118, is configured to leave the brightness as is.
[0042] Fig. 2 illustrates the processing of data acquired by the time-of-flight sensor 102 in order to estimate the orientation of the head of a user 200 of the system 100, according to an embodiment of the present description.
[0043] According to one embodiment, the time-of-flight sensor 104 comprises a plurality of pixels. In the example illustrated in relation to [Fig. 2], the time-of-flight sensor 104 comprises 64 pixels, arranged in a square comprising 8 pixels per side.
[0044] User 200 is, for example, facing the time-of-flight sensor 104. User 200 can move their head in rotational movements around the three axes x, y, z: a rotation around the z-axis (YAW) corresponds to turning the head to the right or left. A rotation around the y-axis (ROLL) corresponds to pivoting the head in a plane parallel to user 200, that is, pivoting the head towards one shoulder without turning it to the right or left. Finally, a rotation around the x-axis (PITCH) corresponds to lowering the head towards the ground or raising it towards the sky.
[0045] During a capture performed by sensor 104, each pixel of sensor 104 measures different information. For example, the pixels of sensor 104 are configured to measure, among other things, the distance between sensor 104 and a target facing sensor 104. The target, such as user 200, does not necessarily occupy the entire field measured by the sensor. Thus, pixels 202 will detect user 200, for example, when the measured distance is a finite value, or less than a threshold value. Other pixels 204 will not detect user 200 and are, for example, associated with the background of the captured scene.
[0046] According to one embodiment, following the measurements performed by each pixel, a status value is further generated for each pixel. For example, the value of The status is a Boolean value that takes the value true (TRUE) when the measured distance shows that user 200 has been detected by the pixel. The status value takes the value false (FALSE) when the measured values by the pixel are anomalous. For example, when user 200 is not detected, the value FALSE is assigned to the pixel.
[0047] In the example illustrated by [Fig.2], the status values of pixels 202 are assigned the value TRUE and the status values of pixels 204 are assigned the value FALSE.
[0048] In another example, the status value is a value belonging to an interval, for example [0, 100]. The status value then represents a confidence index of the pixel.
[0049] According to one embodiment, the measurements taken by the pixels 202 are transmitted to the microcontroller 106 and processed by the neural network 120. The microcontroller 106 is then configured to generate, based on the received measurements, estimates 206 (USER ATTENTION). The estimates 206 include an estimate, by the neural network 120, of the user's head orientation. For example, the orientation estimate takes the form of an angle or a direction. The estimates 206 also include, for example, an estimate, by the microcontroller 106, of the user's attention. For example, the attention estimate is a Boolean value that takes the value FALSE when the user is not attentive and the value TRUE when the user is attentive. In another example, the user's attention estimate is a value belonging to a range.As an example, the interval considered is [0, 100], with a value of 0 indicating that the user is not paying attention at all, and a value of 100 indicating that the user is focused on the screen. The interval [0, 100] is given as an example and is, of course, not limiting; other intervals or measures of user attention can be considered. For example, user attention is estimated based on the orientation of the user's head, as estimated by the neural network.
[0050] Figure 3 illustrates the calculation, for each pixel of the time-of-flight sensor 104, of the data used to estimate the user's head orientation. As an example, the data used is calculated directly by the sensor 104, based on measurements taken by the pixels.
[0051] The time-of-flight sensor 104 is configured to emit light pulses, generally in the infrared range, towards a scene. For each pulse, the light reflected by objects, here by the user 200 of the system 100, located in the scene is detected by each pixel. By way of example, each pixel comprises one or more light-sensitive elements configured to detect the reflected light. By way of example, the light-sensitive elements are avalanche diodes. single photon (from the English "Single Photon Avalanche Diode" - SPAD). In general, to obtain an acceptable signal-to-noise ratio, each pixel of the time-of-flight sensor 104 is configured to accumulate the signal generated by the photosensitive element(s) during a sequence of several thousand light pulses, emitted by the sensor 104. A distance (DISTANCE), in mm, between the user 200 and the sensor 104 is then deduced from an average time of the pulse sequence.
[0052] Graph 300 illustrates, for one pixel of sensor 104, the accumulation of signals generated by the photosensitive element(s) during a sequence of several thousand light pulses. A d-axis represents the distances measured during each pulse, and a NB PHOTONS axis represents the number of photons returning per second. A signal value (SIGNAL) corresponds to the average number of photons returning to the pixel per second.
[0053] According to one embodiment, each pixel is further configured to calculate a standard deviation associated with the distance (E DISTANCE) and a standard deviation associated with the signal (E SIGNAL).
[0054] By way of example, each pixel is further configured to measure a reflectance value. By way of example, the reflectance value is a percentage calculated based on the number of photons reflected by the target and based on the estimated distance to the target. By way of example, human skin, regardless of its color, has a reflectance of approximately 60%. By way of example, hair has a reflectance of at least 10%, depending on its color.
[0055] By way of example, the reflectance values of pixels having a status value of TRUE, or having a value greater than the threshold value, are transmitted to the microcontroller 106.
[0056] According to one embodiment, the sensor 104 is configured to transmit to the microcontroller 106 the distance, signal, distance standard deviation, and signal standard deviation values of the pixels, as well as the status values for each pixel. For example, when the status value associated with a pixel is FALSE, the microcontroller 106 ignores the other values associated with that pixel. The pixel in question is, for example, treated by the microcontroller 106 as an invalid pixel. In the example where the status value belongs to a range, the measurements of pixels with a status value lower than a threshold value are, for example, ignored by the microcontroller 106.
[0057] By way of example, the distance, signal, standard deviation and reflectance measurements of invalid pixels are automatically assigned, by the microcontroller 106, to default values, for example to 0, or to a value that is not a number (in English "Not A Number" - NaN).
[0058] Figure 4 illustrates examples of applications implemented following data processing.
[0059] By way of example, the estimates 206, comprising an estimate 300 (HEAD ORIENTATION DIRECTIONS ESTIMATIONS) of the orientation of the user's head 200 and an estimate 302 (USER ATTENTION TRUE / FALSE INTERVAL) of the attention of the user 200, are used, for example, by the microcontroller 106 or by another processor (not illustrated in [Fig.1]), for the implementation of one or more applications 304 (ADAPTIVE DIMMING), 306 (ADAPTIVE REFRESH) and / or 308 (TURN-ON / OFF TBLU AND / OR TCON).
[0060] By way of example, the estimation 300 of the user's head orientation takes the form of one of the following directions: north (N), northeast (NE), northwest (NW), east (E), west (W), and south (S). The north direction indicates, for example, that the user is facing the screen 104. The east and west directions indicate, for example, that the user is looking 90° to the right and left of the screen 104, respectively. The south direction indicates, for example, that the user has their back to the screen 104.
[0061] The implementation of application 304 allows the brightness of the screen 102 to be adjusted according to the user's attention and head orientation. For example, memory 112 includes instructions for implementing application 304.
[0062] The implementation of application 306 allows, for example, the refresh rate of screen 102 to be slowed down or accelerated according to the estimates 206. For example, when it is estimated that the user is not facing the screen 102, the screen refresh rate is lower than when it is estimated that the user is facing the screen.
[0063] The implementation of application 308 allows, for example, the activation or deactivation of peripheral circuits (not shown in [Fig. 1]) of the system 100. For example, when the estimates 206 show that the user is not facing the screen and / or is not paying attention, peripheral circuits are deactivated. The circuits are then reactivated when the estimates show that the user returns to facing the screen 102 and / or becomes attentive again. For example, the peripheral circuits include a T-CON (Timing Control Board) and / or a BLU (Back Light Unit).
[0064] Fig. 5 is a flowchart illustrating steps in a process for processing data acquired by a time-of-flight sensor.
[0065] In particular, the flowchart described in relation to [Fig. 5] comprises two phases. A training phase includes the construction of the neural network 120 and its training. The training phase is, for example carried out prior to the manufacture of system 100. The training phase includes, for example, steps 500 to 502. An application phase (APPLICATION PHASE) includes the implementation of the neural network 120 by system 100. As an example, the application phase includes steps 503 to 507.
[0066] Step 500 (DATA CAPTURE) of the training phase comprises the acquisition of a plurality of captures by one or more time-of-flight sensors similar to sensor 104. The acquisition of a plurality of captures includes the capture by one or more sensors of a plurality of image scenes. The captured scenes include, for example, a user of a screen, the sensors used being, for example, integrated into the screen in question.
[0067] Each capture includes the measurement, by each pixel, of the distance, the signal, and the standard deviations associated with the distance and the signal. By way of example, each capture also includes the measurement, by each pixel, of the reflectance.
[0068] In a step 501 (DATA CLEANING, LABELLING, PRE-PROCESSING), the captures performed in step 500 are, for example, preprocessed. By way of example, step 501 includes a cleaning step. Cleaning the captures includes, for example, deleting all captures where no, or more, users appear. By way of example, cleaning further includes deleting captures containing NaN values for one or more of the measurements performed. Step 501 also includes, for example, a labeling step. By way of example, the labeling step includes associating each of the captures, not deleted during cleaning, with a direction among the directions north, northeast, northwest, east, west, and south. Step 501 also includes, for example, a balancing step, in which the number of captures assigned to each direction is balanced.In other words, achieving balance allows for a similar number of captures associated with each direction.
[0069] In a step 502 (AI TRAINING), a neural network architecture is sought. As an example, this step is carried out, for example, by computer and using an architecture search tool.
[0070] Following the selection of a neural network, step 502 further includes training the neural network, for example, by computer and using a computing tool or an architecture search tool. Training the selected neural network allows for the optimization of the network's parameters, such as synaptic weights for dense layers, the number of filters for a convolutional layer, etc. The selected and trained neural network is then neural network 120.
[0071] The neural network 120 is then implemented, for example, in the microcontroller 106 of the system 100.
[0072] By way of example, the neural network 120 includes an input layer configured to receive input data. The input data are, for example, associated with each pixel of the sensor 104, the at least four measurements: distance, signal, and standard deviations associated with the distance and the signal. By way of example, reflectance measurements are also transmitted to the neural network 120. In another example, status values are also transmitted to the neural network 120. The number of input data points is then equal to the number of pixels of the sensor 104 multiplied by the number of measurements taken by each pixel. The values associated with invalid pixels have been assigned default values by the microcontroller 106, such as zero values or non-numerical values. The neural network 120 further includes, for example, two convolutional layers following the input layer.For example, the first convolutional layer includes a number of filters equal to the number of measurements transmitted by each pixel. For example, the second convolutional layer includes 2 filters. The 120 neural network also includes, following the convolutional layers, a flattening layer. For example, during step 502, a dropout layer and / or a Gaussian noise layer are added to reduce overfitting of the 120 neural network.
[0073] By way of example, the neural network 120 further comprises two dense layers. The first dense layer has, for example, a number of neurons equal to the number of output neurons of the second convolutional layer. The second dense layer has, for example, a number of neurons equal to the number of possible directions the user's head could be facing. For example, the second dense layer comprises 6 neurons. The neural network 120 is configured to generate, for each of the directions among north, northeast, northwest, east, west, and south, a probability of belonging, based on the input data. The most probable direction is then the estimate of the user's head direction.
[0074] The given example of neural network 120 is, of course, not limiting, and other architectures can certainly be implemented. However, in cases where the sensor 104 has a small number of pixels, for example, when the sensor has fewer than one hundred pixels, it is desirable for the neural network 120 to be small. For example, the neural network 120 may have fewer than ten layers. Indeed, a small neural network 120 helps to conserve the energy of the system 100, particularly when the screen 102 is dim and the user is not facing the screen 102.
[0075] Following the implementation of the neural network 120 in the microcontroller 106, and during the operation of the system 100, a step 503 (CAPTURE) of the application phase includes a capture by the sensor 104 described in relation to [Fig.3]. The measurements taken during capture are, for example, transmitted to the microcontroller 106.
[0076] In a step 504 (PRE-PROCESSING), the microcontroller 106 processes the received measurements. For example, the processing includes identifying the pixels 202 that detect the user of the screen 102. For example, pixels with a status value of TRUE but not identified as detecting the user of the screen 102 have their status value reassigned to FALSE. The measurements taken for these pixels are then replaced with the default values. Step 504 also includes, for example, the normalization of the measurements. For example, the distance values are normalized to belong to an interval, for example, the interval [-1,1].
[0077] In step 505, pixel measurements are provided as input data to the neural network 120. By way of example, these measurements were renormalized during step 504. By way of example, the values associated with invalid pixels are default values. The neural network 120 is then configured to estimate, based on the input data provided, the orientation of the user's head in relation to the screen 102, for example, among the directions north, northeast, northwest, east, west, or south.
[0078] In a step 506 (USER ATTENTION), the attention value is calculated, for example by the microcontroller 106, based on the user's head orientation. For example, the sensor 104 is configured to perform several measurements at times separated by a regular time interval, or by a time interval dependent on the estimates made by the neural network 120. At each time, the neural network 120 estimates the user's head orientation, and this estimate is, for example, stored in volatile memory. The user's attention value at a given time is further calculated, for example, based on at least two estimates of the user's head orientation for previous times.
[0079] In a step 507 (APPLICATION), the estimation of the user's head orientation 200 and the attention value are used, for example, by the microcontroller 106, or by another processor, for the implementation of one or more applications such as, for example, applications 304 and / or 306 and / or 308.
[0080] Figures 6A, 6B and 6C illustrate embodiments of a method for estimating the attention of a user of the system, according to one embodiment of the present description.
[0081] By way of example, the system includes volatile memory, such as, for example, a circular buffer or a first-in, first-out (FIFO) memory. By way of example, volatile memory is included in the micro Controller 106. In another example, volatile memory is contained within the memories 110. Each time a direction is estimated by the neural network 120, the volatile memory is configured to store that direction. Thus, with each new direction estimate by the neural network 120, the volatile memory is configured to delete the most recent direction and store the new one. The directions stored in volatile memory therefore come from consecutive captures by the sensor 104. As an example, the volatile memory is configured to store a number n of directions, for example, 10 directions. The size of the volatile memory can, of course, vary, and it can store more than 10 consecutive direction estimates, or fewer than 10 consecutive direction estimates.
[0082] In the examples illustrated by Figures 6A, 6B and 6C, volatile memory is represented by an array 600. The array 600 comprises 10 consecutive direction estimates, generated by the neural network 120. As an example, the 10 estimated directions were obtained from captures of an image scene 601, by the sensor 104, taken at consecutive times T-9 to T. In the example illustrated by Figures 6A, 6B and 6C, the estimated direction for time T is northwest, the direction for time T1 is north, etc.
[0083] By way of example, the attention value is a value belonging to an interval, for example the interval [0,100]. The interval [0,100] is given by way of example, and any other interval can be considered, such as, for example, an interval of the form [0,1], [-1,1], etc. In general, the confidence interval is characterized by a minimum value and a maximum value.
[0084] In a first embodiment 602 (USER ATTENTION: SMALL FOA), illustrated by [Fig. 6A], the attention value is set to the maximum value when the presence of the user of the screen 102 is detected and at least one of the following conditions is met: - the north direction was estimated for times T and Tl; or - the north direction was estimated at least six times among the estimates for times T-9 to T; or - The distance between the user 200 and the screen 102, measured by the sensor 104, is less than a reference distance, the value of the reference distance being, for example, less than 500 mm. For example, the value of the reference distance is equal to 320 mm. For example, the presence of the user is detected by applying a detection algorithm, such as, for example, a human-versus-object detection (HOD) algorithm.
[0085] When none of the above conditions are met, the attention value gradually decreases. For example, the attention value decreases from the maximum value to the minimum value in a reference time period. For example, the reference time period is on the order of a few seconds, for example between 2 and 3 seconds.
[0086] In a second embodiment 603 (USER ATTENTION: BIG FOA), illustrated by [Fig. 6B], the attention value is set to the maximum value when the presence of the user of the screen 102 is detected, for example following the application of a HOD-type algorithm, and when at least one of the following conditions is met: - the north direction was estimated for times T and Tl; or - the sum of the north, northwest, and northeast directions among the estimates for times T-9 to T is at least equal to 8; or - The distance between the user and the screen 102, measured by the sensor 104, is less than a reference distance, the value of the reference distance being, for example, less than 500mm. As an example, the value of the reference distance is equal to 320mm.
[0087] When none of the above conditions are met, the attention value gradually decreases. For example, the attention value decreases from the maximum value to the minimum value within a reference time period. For example, the reference time period is on the order of a few seconds, for example, between 2 and 3 seconds.
[0088] In a third embodiment 604 (USER ATTENTION: ANGLE THRESHOLD), illustrated by [Fig. 6C], the directions stored in volatile memory 600 are used to calculate an angle 605 (HEAD ANGLE). For example, the calculated angle is between -180° and 180° and corresponds to an average of all the directions stored in volatile memory. For example, each direction is translated into an angle; for instance, the north direction corresponds to an angle of 0°, the northeast and northwest directions correspond to angles of -45° and 45° respectively. The east and west directions correspond to angles of -90° and 90° respectively. The south direction corresponds, for example, to an angle of 180°, or to an angle of -180°.
[0089] The attention value is then set to the maximum value when the presence of the user of screen 102 is detected, for example following the application of a HOD type algorithm, and when at least one of the following conditions is met: - the calculated angle belongs to a reference interval. The reference interval being, for example, a symmetrical interval of the form [-0, 0] or O is an angle less than 180°. In one example, ® is less than 90°, or 45°. In another example, the angle ® is smaller and is, for example, less than 30° or even 20°; or - the distance between the user and the screen 102, measured by the sensor 104 is in less than a reference distance, the value of the reference distance being for example less than 500m. As an example, the value of the reference distance is equal to 320mm.
[0090] When none of the above conditions are met, the attention value gradually decreases. For example, the attention value decreases from the maximum value to the minimum value within a reference time period. For example, the reference time period is on the order of a few seconds, for example, between 2 and 3 seconds.
[0091] By way of example, in the embodiments described in relation to Figures 6A, 6B and 6C, the screen brightness decreases progressively with the attention level. For example, the screen 102 is put into standby mode when the attention level reaches the minimum value.
[0092] One advantage of the described embodiments is that the estimation of the user's head orientation is based on measurements taken by a time-of-flight sensor, which consumes much less energy than a camera. Furthermore, the use of the time-of-flight sensor ensures user privacy.
[0093] Another advantage of the writing embodiments is that by using a time-of-flight sensor, the neural network configured for data processing is extremely small, for example less than ten layers and less than 100,000 parameters, thus avoiding the use of excessive energy resources.
[0094] Another advantage of the described embodiments is that the estimates are made, among other things, on the basis of a distance measurement, which makes it possible to obtain, for a relatively small number of pixels compared to the resolution of an image from a camera, estimates of reliability similar to those obtained on the basis of images obtained by a camera.
[0095] Various embodiments and variations have been described. A person skilled in the art will understand that certain features of these various embodiments and variations could be combined, and other variations will become apparent to a person skilled in the art.
[0096] Finally, the practical implementation of the described embodiments and variants is within the reach of a person skilled in the art, based on the functional specifications given above. In particular, the screen brightness can be adjusted according to the estimates made by the neural network. Similarly, the calculation of the user's attention and its usage can vary. Furthermore, the system can include several screens. In this case, a reference screen is the screen that the user looks at most of the time. The reference screen does not necessarily include the time-of-flight sensor configured to perform the captures. A person skilled in the art will know how to calibrate the sensor so that the estimates made by the neural network Neurons correspond to the orientation of the user's head relative to the reference screen. Similarly, when the system includes multiple screens, the user can look at a screen other than the reference screen without this indicating a lack of attention. A person skilled in the art will be able to adapt the brightness control and / or the implementation of applications, such as those described in relation to [Fig. 3], to the system configuration.
Claims
Demands
1. System comprising: - a microcontroller (106) including a neural network (120); - a time-of-flight sensor (104), connected to the microcontroller, configured to perform a first capture of an image scene including a user (200), the sensor including a plurality of pixels, the first capture including the measurement, by each pixel, of a distance to the user (200) from the system and a signal value corresponding to a number of photons returning to the sensor per unit of time, the sensor being further configured to, following the first capture and for each pixel, calculate a standard deviation value associated with the distance value and a standard deviation value associated with the signal value and a confidence value, the sensor being further configured to provide, in association with each pixel, the distance values, signal values, standard deviations associated with the distance and the signal, and the confidence value to the neural network,the neural network being configured to generate, based on the values provided by the sensor, an estimate of a direction associated with the user, the system further comprising a screen (102), connected to the microcontroller, the microcontroller being configured to control the screen, or another circuit connected to the microcontroller, based on the estimate of the direction associated with the user.
2. System according to claim 1, wherein the microcontroller (106) is further configured to generate an attention value associated with the user (200), and wherein the microcontroller is further configured to control the brightness of the screen (102) on the basis of the attention value.
3. System according to claim 1 or 2, wherein each pixel of the sensor (104) is further configured to measure a reflectance value and wherein the neural network (120) is configured to generate the direction estimate further on the basis of the reflectance values.
4. System according to any one of claims 1 to 3, further comprising a memory (110) comprising an application program and in which the microcontroller (106) is further configured to control the execution of the application program on the basis of the direction estimation generated by the neural network (120).
5. System according to any one of claims 1 to 4, further comprising a backlight unit (BLU- "Back Light Unit"), and wherein the microcontroller (106) is configured to disable the backlight unit, based on direction estimation.
6. System according to any one of claims 1 to 5, wherein the microcontroller (106) is configured to control a refresh rate of the screen (102) based on direction estimation.
7. System according to any one of claims 1 to 6, wherein the sensor (104) is configured to perform, following the first capture, a second capture, the time interval between the first and second captures being determined by the direction estimation generated by the neural network (120) and / or on the basis of the attention value calculated following the first capture.
8. System according to any one of claims 1 to 7, wherein the sensor (104) is an 8x8 pixel sensor.
9. System according to any one of claims 1 to 8, further comprising at least one other display, the microcontroller further configured to control the brightness of the at least one other display based on an estimation of a direction associated with the user.
10. System according to claim 9, wherein the estimated direction indicates a direction, describing the orientation of the user's head (200), among the directions north, northeast, northwest, east, west and south, the north direction indicating that the user (200) is facing the screen (102) and the south direction indicating that the user is facing away from the screen.
11. System according to claim 10, wherein the microcontroller (106) is configured to control the decrease in screen brightness when, between at least two successive captures, the estimated direction changes: - from north to northwest or northeast; and / or - from north to south; and / or - from northwest or northeast to south, and wherein the microcontroller is configured to control the increase in screen brightness when, between at least two successive captures, the estimated direction changes: - from south to north; and / or - from south to northwest or south to northeast; and / or - from northwest or northeast to north.
12. A system according to any one of claims 1 to 11, in which the confidence value is a boolean value indicating whether the measurements taken by the pixel allow the user to be detected (200), the sensor (104) is configured not to provide measurements taken by a pixel (204) not detecting the user to the neural network (120).
13. System according to any one of claims 1 to 12, wherein the neural network (120) comprises: - at least one convolution layer; and - at least one dense layer.
14. A method for learning a neural network (120) comprising: - capturing, by a time-of-flight sensor, a plurality of image scenes including a user (200) of a screen, each pixel measuring a distance value from the user, a signal value, and being configured to calculate a standard deviation associated with the distance value and a standard deviation associated with the signal value; - removing, by a processor, aberrant images from the captured images; - classifying, by the processor, each non-removed image into one of the classes north, northwest, northeast, west, east, and south; - balancing, by the processor, the number of images distributed in each class; - selecting, by the processor, an architecture of a neural network;and - training the neural network based on the captured and classified images and on the measured values for each pixel of the images, the training including the search for parameters for the selected neural network.
15. A method comprising: - capturing, by a time-of-flight sensor (104), an image scene including a user of a screen (102), the capture comprising the measurement, by each pixel of the sensor, of a distance value, a signal value and standard deviation values associated with the distance value and the signal value, the sensor being further configured to generate, for each pixel, a confidence value; - providing, by the sensor (104), to a microcontroller (106) the measurements made by the pixels, the microcontroller comprising a neural network configured to estimate, on the basis of the measurements provided, a direction describing the orientation of the user's head; - control, by the microcontroller (106) of the screen or of another circuit connected to the microcontroller, based on the estimation of the orientation of the user's head.