Screen brightness control
A time-of-flight sensor and neural network system estimates user orientation to control screen brightness efficiently and privately, addressing energy consumption and privacy concerns in existing camera-based methods.
Patent Information
- Application Number
- FR2023004677
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-05-11
AI Technical Summary
Existing methods for controlling screen brightness, such as using a camera to detect user orientation, consume energy and compromise user privacy.
A system utilizing a time-of-flight sensor and a neural network to estimate user orientation and adjust screen brightness, which includes a microcontroller to control the display based on the estimated direction and attention value, without the need for a camera.
This approach reduces energy consumption and maintains user privacy by using a time-of-flight sensor, which is more energy-efficient and privacy-preserving compared to camera-based systems.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Control of the brightness of a screen Technical field
[0001] The present description relates to the control of the brightness of a screen by the exploitation of data acquired by a time-of-flight sensor. Prior art
[0002] Controlling the brightness of screens makes it possible to reduce their energy consumption. Generally, the control is carried out by monitoring, via a camera, generally via the webcam implemented on the screen, the orientation of the user's head. The camera is configured to detect when the user is looking in a direction other than the screen and the brightness of the screen is then reduced, for example automatically putting the screen on standby. The camera is also configured to detect when the user returns to face the screen and the brightness of the screen is then increased.
[0003] However, by operating even when the screen is in reduced operation and / or when the user is for example no longer in front of his screen, the use of the camera compromises the confidentiality of the user. In addition, the camera being an expensive peripheral in terms of energy consumption, the energy saving achieved by reducing the brightness is counteracted by the energy consumption of the camera.
[0004] There is a need to improve methods for automatically adapting the brightness of a screen. In particular, it is desirable that the implementation of brightness control is not energy-intensive. It is further desirable that the implementation of brightness control does not compromise user privacy. Summary of the invention
[0005] One embodiment provides a system comprising: - a microcontroller comprising a neural network; - a time-of-flight sensor, connected to the microcontroller, configured to perform a first capture of an image scene comprising a user, the sensor comprising a plurality of pixels, the first capture comprising the measurement, by each pixel of a distance to the user of the system and of a signal value corresponding to a number of photons returning to the sensor per unit of time, the sensor being further configured to, following the first capture and for each pixel, calculate a standard deviation value associated with the distance value and a standard deviation value associated with the signal value and a confidence value, the sensor being further configured to provide, in association with each pixel, the distance, signal, standard deviation values associated with the distance and the signal and the confidence value to the neural network, the neural network being configured to generate, based on the values provided by the sensor, an estimate of a direction associated with the user, the system further comprising a screen, connected to the microcontroller, the microcontroller being configured to control the screen, or another circuit connected to the microcontroller, based on the estimate of the direction associated with the user.
[0006] According to one embodiment, the microcontroller is further configured to generate an attention value associated with the user, and wherein the microcontroller is configured to control the brightness of the screen further based on the attention value.
[0007] According to one embodiment, each pixel of the sensor is further configured to measure a reflectance value and wherein the neural network is configured to generate the direction estimate further based on the reflectance values.
[0008] According to one embodiment, the above system further comprises a memory comprising an application program and wherein the microcontroller is further configured to control the execution of the application program based on the direction estimation generated by the neural network.
[0009] According to one embodiment, the above system further comprises a backlight unit (BLU), and wherein the microcontroller is configured to deactivate the backlight unit, based on the estimation of the direction.
[0010] According to one embodiment, the microcontroller is configured to control a refresh rate of the screen based on the estimation of the direction.
[0011] According to one embodiment, the sensor is configured to perform, following the first capture, a second capture, the time interval between the first and second captures being determined by the estimation of the direction generated by the neural network and / or on the basis of the attention value calculated following the first capture.
[0012] According to one embodiment, the sensor is an 8x8 pixel sensor.
[0013] According to one embodiment, the above system further comprises at least one another display, the microcontroller further configured to control the brightness of the at least one other display based on the estimation of a direction associated with the user.
[0014] According to one embodiment, the estimated direction indicates a direction, describing the orientation of the user's head, among the north, northeast, northwest, east, west and south directions, the north direction indicating that the user is facing the screen and the south direction indicating that the user has his back to the screen.
[0015] According to one embodiment, the microcontroller is configured to control the decrease in screen brightness when between at least two successive captures, the estimated direction changes: - from north to northwest or northeast; and / or - from north to south; and / or - from northwest or northeast to south, and wherein the microcontroller is configured to control the increase in screen brightness when, between at least two successive captures, the estimated direction passes: - from south to north; and / or - from south to northwest or from south to northeast; and or - from northwest or northeast to north.
[0016] According to one embodiment, the confidence value is a Boolean value indicating whether the measurements taken by the pixel make it possible to detect the user, the sensor is configured not to provide the measurements taken by a pixel not detecting the user to the neural network.
[0017] According to one embodiment, the neural network comprises: - at least one convolution layer; and - at least one dense layer.
[0018] One embodiment provides a method for training a neural network comprising: - capturing, by a time-of-flight sensor, a plurality of image scenes comprising a user of a screen, each pixel measuring a distance value to the user, a signal value and being configured to calculate a standard deviation associated with the distance value and a standard deviation associated with the signal value; - the deletion, by a processor, in the captured images, of aberrant images; - the classification, by the processor, of each undeleted image, into a class among the classes north, northwest, northeast, west, east and south; - balancing, by the processor, of the number of images distributed in each class; - the selection, by the processor, of an architecture of a neural network; and - training the neural network based on the captured and classified images and on the measured values for each pixel of the images, the training including the search for parameters for the selected neural network.
[0019] One embodiment provides a method comprising: - capturing, by a time-of-flight sensor, an image scene comprising a user of a screen, the capturing comprising the measurement, by each pixel of the sensor, of a distance value, a signal value and standard deviation values associated with the distance value and the signal value, the sensor being further configured to generate, for each pixel, a confidence value; - the provision, by the sensor, to a microcontroller of the measurements made by the pixels, the microcontroller comprising a neural network configured to estimate, on the basis of the measurements provided, a direction describing the orientation of the user's head; - control, by the microcontroller of the screen or another circuit connected to the microcontroller, on the basis of the estimation of the orientation of the user's head. Brief description of the drawings
[0020] These characteristics and advantages, as well as others, will be explained in detail in the following description of particular embodiments given without limitation in relation to the attached figures among which:
[0021] [Fig.l] is a block diagram illustrating a system comprising a display and a time-of-flight sensor, according to one embodiment of the present disclosure;
[0022] [Fig.2] illustrates the processing of data acquired by the time-of-flight sensor in order to estimate the orientation of the head of a user of the system, according to an embodiment of the present description;
[0023] [Fig.3] illustrates the calculation, for each pixel of the time-of-flight sensor, of the data used for estimating the orientation of the user's head;
[0024] [Fig.4] illustrates examples of applications implemented following the estimation of the orientation of the user's head;
[0025] [Fig.5] is a flowchart illustrating steps of a method for processing data acquired by a time-of-flight sensor;
[0026] [Fig.6A] illustrates an embodiment of a method for estimating the attention of a user of the system, according to an embodiment of the present description;
[0027] [Fig.6B] illustrates another embodiment of a method for estimating the attention of a user of the system, according to an embodiment of the present description; and
[0028] [Fig. 6C] illustrates another embodiment of a method for estimating the attention of a user of the system, according to an embodiment of the present description. Description of the embodiments
[0029] The same elements have been designated by the same references in the different figures. In particular, the structural and / or functional elements common to the different embodiments may have the same references and may have identical structural, dimensional and material properties.
[0030] For the sake of clarity, only the steps and elements useful for understanding the described embodiments have been shown and are detailed. In particular, the operation of a time-of-flight sensor is known to those skilled in the art and is not not described in detail.
[0031] Unless otherwise specified, when referring to two elements connected to each other, this means directly connected without intermediate elements other than conductors, and when referring to two elements connected (in English "coupled") to each other, this means that these two elements can be connected or be connected by means of one or more other elements.
[0032] In the following description, when reference is made to absolute position qualifiers, such as the terms "front", "back", "top", "bottom", "left", "right", etc., or relative position qualifiers, such as the terms "above", "below", "upper", "lower", etc., or to orientation qualifiers, such as the terms "horizontal", "vertical", etc., reference is made unless otherwise specified to the orientation of the figures.
[0033] Unless otherwise specified, the expressions "about", "approximately", "substantially", and "of the order of" mean to within 10%, preferably to within 5%.
[0034] [Fig.l] is a block diagram illustrating a system 100 comprising a screen (DISPLAY) 102 and a time-of-flight sensor (TOF SENSOR) 104, according to an embodiment of the present description.
[0035] For example, the system 100 represents a laptop computer, or a desktop computer. Although only one screen is illustrated in relation to [Fig.l], the system 100 may comprise several, for example, two or three screens. In other examples, the system 100 is a mobile phone, a tablet, an advertising screen, an ATM, and more generally any system comprising a screen and a time-of-flight sensor.
[0036] The system 100 further comprises a microcontroller 106 (MCU) connected to the time-of-flight sensor 104 via a bus 108. For example, the bus 108 is an I2C (Inter-Integrated Circuit) type bus.
[0037] The system 100 further comprises, for example, memories 110 (MEM) comprising for example a non-volatile memory 112 (NV MEM) and / or a volatile memory 114 (RAM). For example, the non-volatile memory 112 is a Flash type memory and the memory 114 is a volatile random access memory (from the English “Random Access Memory”). For example, the memories 110 are connected to the microcontroller 106 via a bus 116.
[0038] By way of example, the system 100 further comprises a control circuit 118 (CTRL CIRCUIT), connected to the bus 116, and configured to control the brightness of the screen 102.
[0039] According to one embodiment, the microcontroller 106 comprises a neural network 120 (NEURAL NETWORK). The neural network 120 is configured to estimate, from data acquired by the time-of-flight sensor 104, the orientation of the head of a user of the system 100, and in particular of the screen 102.
[0040] For example, following the estimation of the orientation of the user's head, the microcontroller 106 controls the adjustment of the brightness of the screen 102. For example, the adjustment of the brightness of the screen 102 is carried out via the control circuit 118.
[0041] For example, the microcontroller 106, and / or the control circuit 118, is configured to lower the brightness of the screen 102 when the estimation of the orientation of the head shows that the user is looking away from the screen 102. When the estimations show that the user is not looking in the direction of the screen 102, the microcontroller 106, and / or the control circuit 118, is configured to leave as is, or lower again, the brightness of the screen 102. On the contrary, when the estimations show that the user returns his gaze, or returns to the screen 102, the microcontroller 106, and / or the control circuit 118, is configured to increase the brightness of the screen 102. Similarly, when the estimations show that the user remains facing the screen, the microcontroller 106, and / or the control circuit 118, is set to leave the brightness as is.
[0042] [Fig. 2] illustrates the processing of data acquired by the time-of-flight sensor 102 in order to estimate the orientation of the head of a user 200 of the system 100, according to an embodiment of the present description.
[0043] According to one embodiment, the time-of-flight sensor 104 comprises a plurality of pixels. In the example illustrated in relation to [Fig. 2], the time-of-flight sensor 104 comprises 64 pixels, arranged in a square comprising 8 pixels per side.
[0044] The user 200 is for example facing the time-of-flight sensor 104. The user 200 can move the head according to rotational movements around the 3 axes x, y, z: a rotation around the z axis (YAW) corresponds to turning the head to the right or to the left. A rotation around the y axis (ROLL) corresponds to pivoting the head in a plane parallel to the user 200, that is to say, to pivoting the head towards a shoulder without turning it to the right or the left. Finally, a rotation around the x axis (PITCH) corresponds to lowering the head towards the ground or raising it towards the sky.
[0045] During a capture performed by the sensor 104, each pixel of the sensor 104 measures different information. For example, the pixels of the sensor 104 are configured to measure, among other things, a distance between the sensor 104 and a target facing the sensor 104. The target, such as the user 200, does not necessarily occupy the entire field measured by the sensor. Thus, pixels 202 will detect the user 200, for example when the measured distance is a finite value, or less than a threshold value. Other pixels 204 will not detect the user 200 and are, for example, associated with the background of the captured scene.
[0046] According to one embodiment, following the measurements made by each pixel, a status value is further generated for each pixel. For example, the value of status is a Boolean value, for example, TRUE when the measured distance shows that user 200 was detected by the pixel. The status value is FALSE when the measured values of the pixel are outliers. For example, when user 200 is not detected, the pixel is assigned the value FALSE.
[0047] In the example illustrated by [Fig.2], the status values of pixels 202 are assigned the value TRUE and the status values of pixels 204 are assigned the value FALSE.
[0048] In another example the status value is a value belonging to an interval, for example [0, 100]. The status value then represents a confidence index of the pixel.
[0049] According to one embodiment, the measurements made by the pixels 202 are transmitted to the microcontroller 106 and are processed by the neural network 120. The microcontroller 106 is then configured to generate, on the basis of the measurements received, estimates 206 (USER ATTENTION). The estimates 206 comprise an estimate, by the neural network 120, of the orientation of the user's head. For example, the estimate of the orientation takes the form of an angle or a direction. The estimates 206 comprise, for example, an estimate, by the microcontroller 106, of the user's attention. For example, the estimate of the attention is a Boolean value taking the value FALSE when the user is not attentive and the value TRUE when the user is attentive. In another example, the estimate of the user's attention is a value belonging to an interval.For example, the interval considered is the interval [0, 100], the value 0 indicating that the user is not at all attentive, and the value 100 indicating that the user is focused on the screen 102. The interval [0,100] is given as an example, and is of course not limiting, other intervals or measures of the user's attention can be envisaged. The user's attention is for example estimated on the basis of the orientation of the user's head, estimated by the neural network 120.
[0050] [Fig. 3] illustrates the calculation, for each pixel of the time-of-flight sensor 104, of the data used for estimating the orientation of the user's head. For example, the data used are calculated directly by the sensor 104, based on the measurements made by the pixels.
[0051] The time-of-flight sensor 104 is configured to emit light pulses, generally in the infrared range, towards a scene. For each pulse, the light reflected by objects, here by the user 200 of the system 100, located in the scene is detected by each pixel. For example, each pixel comprises one or more light-sensitive elements configured to detect the reflected light. For example, the light-sensitive elements are avalanche diodes. single photon (from the English "Single Photon Avalanche Diode" - SPAD). Generally speaking, to obtain an acceptable signal-to-noise ratio, each pixel of the time-of-flight sensor 104 is configured to accumulate the signal generated by the photosensitive element(s) during a sequence of several thousand light pulses, emitted by the sensor 104. A distance (DISTANCE), in mm, between the user 200 and the sensor 104 is then deduced from an average time of the pulse sequence.
[0052] A graph 300 illustrates, for a pixel of the sensor 104, the accumulation of the signals generated by the photosensitive element(s) during a sequence of several thousand light pulses. A d axis represents the distances measured during each flow and a NB PHOTONS axis represents the number of photons having returned per second. A signal value (SIGNAL) corresponds to the average number of photons returning to the pixel per second.
[0053] According to one embodiment, each pixel is further configured to calculate a standard deviation associated with the distance (E DISTANCE) and a standard deviation associated with the signal (E SIGNAL).
[0054] For example, each pixel is further configured to measure a reflectance value. For example, the reflectance value is a percentage calculated based on the number of photons returned by the target and based on the estimated target distance. For example, human skin, regardless of its color, has a reflectance of about 60%. For example, hair has a reflectance of at least 10%, depending on its color.
[0055] For example, the reflectance values of pixels having the status value TRUE, or having a value greater than the threshold value, are transmitted to the microcontroller 106.
[0056] According to one embodiment, the sensor 104 is configured to transmit to the microcontroller 106 the values of distance, signal, distance standard deviation and signal standard deviation of the pixels and the status values, for each pixel. For example, when the status value associated with a pixel is equal to the value FALSE, the microcontroller 106 ignores the other values associated with this pixel. The pixel in question is, for example, treated by the microcontroller 106 as being an invalid pixel. In the example where the status value belongs to an interval, the measurements of the pixels having a status value lower than a threshold value are for example ignored by the microcontroller 106.
[0057] For example, the distance, signal, standard deviation and reflectance measurements of the invalid pixels are automatically assigned, by the microcontroller 106, to default values, for example to 0, or to a value not being a number (in English “Not A Number” - NaN).
[0058] [Fig.4] illustrates examples of applications implemented following data processing.
[0059] By way of example, the estimates 206, comprising an estimate 300 (HEAD ORIENTATION DIRECTIONS ESTIMATIONS) of the orientation of the head of the user 200 and an estimate 302 (USER ATTENTION TRUE / FALSE INTERVAL) of the attention of the user 200, are used, for example by the microcontroller 106 or by another processor (not illustrated in [Fig.l]), for the implementation of one or more applications 304 (ADAPTIVE DIMMING), 306 (ADAPTIVE REFRESH) and / or 308 (TURN-ON / OFF TBLU AND / OR TCON).
[0060] By way of example, the estimation 300 of the orientation of the user's head takes, for example, the form of a direction among the directions north (N), northeast (NE), northwest (NW), east (E), west (W) and south (S). The north direction indicates, for example, that the user is facing the screen 104. The east and west directions indicate, for example, respectively, that the user is looking 90° to the right and left of the screen 104. The south direction indicates, for example, that the user has his back to the screen 104.
[0061] The implementation of the application 304 allows the brightness of the screen 102 to be adjusted according to the attention and the orientation of the user's head. For example, the memory 112 includes instructions allowing the implementation of the application 304.
[0062] The implementation of the application 306 makes it possible, for example, to slow down or accelerate the refresh rate of the screen 102 depending on the estimations 206. For example, when it is estimated that the user is not facing the screen 102, the refresh rate of the screen is lower than when it is estimated that the user is facing the screen.
[0063] The implementation of the application 308 allows for example the activation, or the deactivation of peripheral circuits (not illustrated in [Fig.l]) to the system 100. For example, when the estimations 206 show that the user is not facing the screen and / or is not attentive, peripheral circuits are deactivated. The circuits are then reactivated when the estimations show that the user returns to facing the screen 102 and / or becomes attentive again. For example, the peripheral circuits comprise a T-CON card (from the English “Timing Control Board”) and / or a BLU backlight unit (from the English “Back Light Unit”).
[0064] [Fig.5] is a flowchart illustrating steps of a method for processing data acquired by a time-of-flight sensor.
[0065] In particular, the flowchart described in relation to [Fig.5] comprises two phases. A training phase (TRAINING PHASE) comprises the construction of the neural network 120 and its training. The training phase is for example carried out upstream of the manufacture of the system 100. The training phase comprises, for example, steps 500 to 502. An application phase (APPLICATIVE PHASE) comprises the implementation of the neural network 120 by the system 100. By way of example, the application phase comprises steps 503 to 507.
[0066] Step 500 (DATA CAPTURE) of the training phase comprises the acquisition of a plurality of captures by one or more time-of-flight sensors similar to the sensor 104. The acquisition of a plurality of captures comprises the capture by one or more sensors of a plurality of image scenes. The captured scenes comprise for example a user of a screen, the sensors used being for example integrated into the screen in question.
[0067] Each capture includes the measurement, by each pixel, of the distance, of the signal, as well as the standard deviations associated with the distance and the signal. For example, each capture further includes the measurement, by each pixel, of the reflectance.
[0068] In a step 501 (DATA CLEANING, LABELLING, PRE-PROCESSING), the captures made in step 500 are for example pre-processed. For example, step 501 comprises a cleaning step. Cleaning the captures comprises for example the deletion of all the captures where no, or more, users appear. For example, the cleaning further comprises the deletion of the captures comprising NaN values for one or more of the measurements made. Step 501 comprises for example further a labeling step. For example, the labeling step comprises the association of each of the captures, not deleted during the cleaning, with a direction among the north, northeast, northwest, east, west and south directions. Step 501 comprises for example further a balancing step, in which the number of captures assigned to each direction is balanced.In other words, achieving balancing allows for a similar number of captures in association with each direction.
[0069] In a step 502 (AI TRAINING), a neural network architecture is searched for. For example, this step is carried out, for example, by computer and by means of an architecture search tool.
[0070] Following the selection of a neural network, step 502 further comprises training the neural network, for example by computer and via a calculation tool, or the architecture search tool. Training the selected neural network makes it possible to optimize the parameters involved in the network, such as synaptic weights for dense layers, a number of filters for a convolution layer, etc. The selected and trained neural network is then the neural network 120.
[0071] The neural network 120 is then implemented, for example, in the microcontroller 106 of the system 100.
[0072] By way of example, the neural network 120 comprises an input layer configured to receive input data. The input data are, for example, in association with each pixel of the sensor 104, the at least 4 measurements: distance, signal, and standard deviations associated with the distance and the signal. By way of example, the reflectance measurements are also transmitted to the neural network 120. In another example, the status values are also transmitted to the neural network 120. The number of input data is then equal to the number of pixels of the sensor 104 multiplied by the number of measurements made by each pixel. The values associated with the invalid pixels have been assigned, by the microcontroller 106, to default values, such as zero values or values not being a number. The neural network 120 further comprises, for example, following the input layer, two convolution layers.For example, the first convolution layer comprises a number of filters equal to the number of measurements transmitted by each pixel. For example, the second convolution layer comprises 2 filters. The neural network 120 further comprises, following the convolution layers, a flattening layer. For example, during step 502, a so-called dropout layer and / or a Gaussian noise layer are added in order to reduce over-training of the neural network 120.
[0073] By way of example, the neural network 120 further comprises two dense layers. The first dense layer has, for example, a number of neurons equal to the number of output neurons of the second convolution layer. The second dense layer has, for example, a number of neurons equal to the number of directions that the user's head can take. By way of example, the second dense layer comprises 6 neurons. The neural network 120 is configured to generate, for each of the directions among north, northeast, northwest, east, west and south, a probability of belonging, on the basis of the input data. The most probable direction is then the estimation of the direction of the user's head.
[0074] The example given of the neural network 120 is, of course, not limiting and other architectures can, of course, be implemented. Nevertheless, in the case where the sensor 104 comprises a reduced number of pixels, for example when the sensor comprises less than a hundred pixels, it is desirable for the neural network 120 to be small. For example, the neural network 120 comprises less than ten layers. Indeed, a small neural network 120 makes it possible to contribute to the energy saving of the system 100, and particularly when the screen 102 is at low brightness and the user is not facing the screen 102.
[0075] Following the implementation of the neural network 120 in the microcontroller 106, and during the operation of the system 100, a step 503 (CAPTURE) of the application phase comprises a capture by the sensor 104 described in relation to [Fig.3]. The measurements taken during capture are, for example, transmitted to the microcontroller 106.
[0076] In a step 504 (PRE-PROCESSING), the microcontroller 106 performs processing of the received measurements. For example, the processing comprises the identification of the pixels 202 detecting the user of the screen 102. For example, pixels having a status value of TRUE but not identified as detecting the user of the screen 102 have their status value reassigned to the value FALSE. The measurements carried out for these pixels are then replaced by the default values. Step 504 further comprises, for example, the normalization of the measurements. For example, the distance values are normalized so as to belong to an interval, for example to the interval [-1,1].
[0077] In a step 505, the pixel measurements are provided as input data to the neural network 120. For example, these measurements have been renormalized when performing step 504. For example, the values associated with the invalid pixels are default values. The neural network 120 is then configured to, based on the provided input data, estimate the orientation of the head of the user of the screen 102, for example among the directions north, northeast, northwest, east, west or south.
[0078] In a step 506 (USER ATTENTION), the attention value is calculated, for example by the microcontroller 106, on the basis of the orientation of the user's head. For example, the sensor 104 is configured to perform several measurements, at times which are distant from each other by a regular time interval, or by a time interval depending on the estimations made by the neural network 120. In association with each instant, the neural network 120 estimates the orientation of the user's head and this estimation is for example stored in a volatile memory. The attention value of the user at a given instant is, for example, further calculated on the basis of at least two estimations of the orientation of the user's head for previous instants.
[0079] In a step 507 (APPLICATION), the estimation of the orientation of the head of the user 200 and the attention value are for example used, by the microcontroller 106, or by another processor, for the implementation of one or more applications such as for example the applications 304 and / or 306 and / or 308.
[0080] Figures 6A, 6B and 6C illustrate embodiments of a method for estimating the attention of a user of the system, according to an embodiment of the present description.
[0081] For example, the system includes volatile memory, such as, for example, a circular buffer or a first-in, first-out (FIFO) memory. For example, the volatile memory is included in the micro controller 106. In another example, the volatile memory is included in the memories 110. At each estimation of a direction, by the neural network 120, the volatile memory is configured to store this direction. Thus, at each new estimation of direction, by the neural network 120, the volatile memory is configured to delete the least recent direction, and to store the new direction. The directions stored by the volatile memory therefore come from captures, by the sensor 104, consecutive. For example, the volatile memory is configured to store a number n of directions, for example 10 directions. The size of the volatile memory can of course vary, and it can store a number greater than 10 of consecutive direction estimations, or a number less than 10 of consecutive direction estimations.
[0082] In the examples illustrated by Figures 6A, 6B and 6C, the volatile memory is represented by a table 600. The table 600 comprises 10 consecutive direction estimates, generated by the neural network 120. By way of example, the 10 estimated directions were made from captures of an image scene 601, by the sensor 104, made at consecutive times T-9 to T. In the example illustrated by Figures 6A, 6B and 6C, the estimated direction for the instant T is the northwest direction, the direction for the instant T1 is the north direction, etc.
[0083] For example, the attention value is a value belonging to an interval, for example to the interval [0,100]. The interval [0,100] is given as an example, and any other interval can be considered, such as for example an interval of the form [0,1], [-1,1] etc. Generally, the confidence interval is characterized by a minimum value and a maximum value.
[0084] In a first embodiment 602 (USER ATTENTION: SMALL FOA), illustrated by [Fig.6A], the attention value is set to the maximum value when the presence of the user of the screen 102 is detected and at least one of the following conditions is met: - the north direction was estimated for times T and Tl; or - the north direction was estimated at least six times among the estimates for times T-9 to T; or - the distance between the user 200 and the screen 102, measured by the sensor 104, is less than a reference distance, the value of the reference distance being, for example, less than 500 mm. For example, the value of the reference distance is equal to 320 mm. For example, the presence of the user is detected by applying a detection algorithm, such as, for example, a human versus object detection algorithm (in English “Human vs Object Detection” - HOD).
[0085] When none of the above conditions is verified, the attention value decreases progressively. For example, the attention value decreases from the maximum value to the minimum value in a reference period of time. For example, the reference period is of the order of a few seconds, for example between 2 and 3 seconds.
[0086] In a second embodiment 603 (USER ATTENTION: BIG FOA), illustrated by [Fig.6B], the attention value is set to the maximum value when the presence of the user of the screen 102 is detected, for example following the application of a HOD type algorithm, and at least one of the following conditions is met: - the north direction was estimated for times T and Tl; or - the sum of the north, northwest and northeast directions among the estimates for times T-9 to T is at least equal to 8; or - the distance between the user and the screen 102, measured by the sensor 104, is less than a reference distance, the value of the reference distance being, for example, less than 500 mm. For example, the value of the reference distance is equal to 320 mm.
[0087] When none of the above conditions are satisfied, the attention value decreases progressively. For example, the attention value decreases from the maximum value to the minimum value in a reference period of time. For example, the reference period of time is of the order of a few seconds, for example between 2 and 3 seconds.
[0088] In a third embodiment 604 (USER ATTENTION: ANGLE THRESHOLD), illustrated by [Fig.6C], the directions stored in the volatile memory 600 are used to calculate an angle 605 (HEAD ANGLE). For example, the calculated angle is an angle between -180° and 180° and corresponds to an average of all the directions stored in the volatile memory. For example, each direction is translated into an angle, for example the north direction corresponds to an angle of 0°, the northeast and northwest directions correspond respectively to an angle of -45° and 45°. The east and west directions correspond respectively to an angle of -90° and 90°. The south direction corresponds for example to the angle 180°, or to the angle -180°.
[0089] The attention value is then set to the maximum value when the presence of the user of the screen 102 is detected, for example following the application of an HOD type algorithm, and at least one of the following conditions is met: - the calculated angle belongs to a reference interval. The reference interval being for example a symmetrical interval, of the form [ - 0. 0 ] or O is an angle, less than 180°, In one example, ® is less than 90°, or 45°. In another example, the angle ® is smaller and is, for example, less than 30° or even 20°; or - the distance between the user and the screen 102, measured by the sensor 104 is in less than a reference distance, the value of the reference distance being for example less than 500m. For example, the value of the reference distance is equal to 320mm.
[0090] When none of the above conditions are satisfied, the attention value decreases progressively. For example, the attention value decreases from the maximum value to the minimum value in a reference period of time. For example, the reference period of time is of the order of a few seconds, for example between 2 and 3 seconds.
[0091] By way of example, in the embodiments described in relation to FIGS. 6A, 6B and 6C, the brightness of the screen decreases progressively with the attention value. By way of example, the screen 102 is put into standby mode when the attention value reaches the minimum value.
[0092] An advantage of the described embodiments is that the estimation of the orientation of the user's head is carried out on the basis of measurements made by a time-of-flight sensor, which consumes much less energy than a camera. In addition, the use of the time-of-flight sensor makes it possible to guarantee the confidentiality of the user.
[0093] Another advantage of the embodiments of the writings is that by using a time-of-flight sensor, the neural network configured for data processing is extremely small, for example less than ten layers and less than 100,000 parameters, thus making it possible not to use too many energy resources.
[0094] Another advantage of the described embodiments is that the estimations are carried out, inter alia, on the basis of a distance measurement, which makes it possible to obtain, for a relatively small number of pixels compared to the resolution of an image coming from a camera, estimations of reliability similar to those obtained on the basis of images obtained by a camera.
[0095] Various embodiments and variations have been described. Those skilled in the art will understand that certain features of these various embodiments and variations could be combined, and other variations will occur to those skilled in the art.
[0096] Finally, the practical implementation of the embodiments and variants described is within the reach of the person skilled in the art from the functional indications given above. In particular, as regards adaptation, depending on the estimates made by the neural network, the brightness of the screen may vary. Similarly, the calculation of the user's attention and its use may vary. In addition, the system may include several screens. In this case, a reference screen is a screen that the user looks at the majority of the time. The reference screen does not necessarily include the time-of-flight sensor configured to perform the captures. The person skilled in the art will know how to calibrate the sensor so that the estimates made by the neural network neurons correspond to the orientation of the user's head relative to the reference screen. Similarly, when the system includes several screens, the user can look at a screen other than the reference screen without this meaning that he is not paying attention. The person skilled in the art will know how to adapt the brightness control and / or the implementation of applications such as, for example, the applications described in relation to [Fig. 3] to the configuration of the system.
Claims
Claims
1. System comprising: - a microcontroller (106) comprising a neural network (120); - a time-of-flight sensor (104), connected to the microcontroller, configured to perform a first capture of an image scene comprising a user (200), the sensor comprising a plurality of pixels, the first capture comprising the measurement, by each pixel of a distance to the user (200) of the system and of a signal value corresponding to a number of photons returning to the sensor per unit of time, the sensor being further configured to, following the first capture and for each pixel, calculate a standard deviation value associated with the distance value and a standard deviation value associated with the signal value and a confidence value, the sensor being further configured to provide, in association with each pixel, the values of distance, signal, standard deviations associated with the distance and the signal and the confidence value to the neural network,the neural network being configured to generate, based on the values provided by the sensor, an estimate of a direction associated with the user, the system further comprising a screen (102), connected to the microcontroller, the microcontroller being configured to control the screen, or another circuit connected to the microcontroller, based on the estimate of the direction associated with the user.,
2. The system of claim 1, wherein the microcontroller (106) is further configured to generate an attention value associated with the user (200), and wherein the microcontroller is configured to control the brightness of the display (102) further based on the attention value.
3. The system of claim 1 or 2, wherein each pixel of the sensor (104) is further configured to measure a reflectance value and wherein the neural network (120) is configured to generate the direction estimate further based on the reflectance values.
4. The system of any one of claims 1 to 3, further comprising a memory (110) comprising an application program and wherein the microcontroller (106) is further configured to control execution of the application program based on the direction estimate generated by the neural network (120).
5. The system of any one of claims 1 to 4, further comprising a backlight unit (BLU), and wherein the microcontroller (106) is configured to turn off the backlight unit, based on the direction estimation.
6. The system of any one of claims 1 to 5, wherein the microcontroller (106) is configured to control a refresh rate of the screen (102) based on the direction estimation.
7. System according to any one of claims 1 to 6, wherein the sensor (104) is configured to perform, following the first capture, a second capture, the time interval between the first and second captures being determined by the estimation of the direction generated by the neural network (120) and / or on the basis of the attention value calculated following the first capture.
8. The system of any one of claims 1 to 7, wherein the sensor (104) is an 8x8 pixel sensor.
9. The system of any one of claims 1 to 8, further comprising at least one other display, the microcontroller being further configured to control the brightness of the at least one other display based on the estimation of a direction associated with the user.
10. The system of claim 9, wherein the estimated direction indicates a direction, describing the orientation of the head of the user (200), among the north, northeast, northwest, east, west and south directions, the north direction indicating that the user (200) is facing the screen (102) and the south direction indicating that the user is facing away from the screen.
11. System according to claim 10, in which the microcontroller (106) is configured to control the decrease in the brightness of the screen when between at least two successive captures, the estimated direction changes: - from north to northwest or to northeast; and / or - from north to south; and / or - from northwest or northeast to south, and in which the microcontroller is configured to control the increase in the brightness of the screen when between at least two successive captures, the estimated direction changes: - from south to north; and / or - from south to northwest or from south to northeast; and or - from northwest or northeast to north.
12. A system according to any one of claims 1 to 11, wherein the confidence value is a Boolean value indicating whether the measurements made by the pixel make it possible to detect the user (200), the sensor (104) is configured not to provide the measurements taken by a pixel (204) not detecting the user to the neural network (120).
13. System according to any one of claims 1 to 12, wherein the neural network (120) comprises: - at least one convolution layer; and - at least one dense layer.
14. A method for training a neural network (120) comprising: - capturing, by a time-of-flight sensor, a plurality of image scenes comprising a user (200) of a screen, each pixel measuring a distance value to the user, a signal value and being configured to calculate a standard deviation associated with the distance value and a standard deviation associated with the signal value; - deleting, by a processor, in the captured images, aberrant images; - classifying, by the processor, each non-deleted image, into a class among the classes north, northwest, northeast, west, east and south; - balancing, by the processor, the number of images distributed in each class; - selecting, by the processor, an architecture of a neural network;and - training the neural network based on the captured and classified images and based on the measured values for each pixel of the images, the training including searching for parameters for the selected neural network.;
15. A method comprising: - capturing, by a time-of-flight sensor (104), an image scene comprising a user of a screen (102), the capturing comprising measuring, by each pixel of the sensor, a distance value, a signal value and standard deviation values associated with the distance value and the signal value, the sensor being further configured to generate, for each pixel, a confidence value; - providing, by the sensor (104), to a microcontroller (106) the measurements made by the pixels, the microcontroller comprising a neural network configured to estimate, on the basis of the provided measurements, a direction describing the orientation of the user's head; - control, by the microcontroller (106) of the screen or another circuit connected to the microcontroller, on the basis of the estimation of the orientation of the user's head.
Citation Information
Patent Citations
System and method for detecting screen orientation of a device
US11270668B1
Image Display Device
US20100295839A1
Systems and methods for gaze tracking from arbitrary viewpoints
US20190147607A1
Methods and Systems for Energy or Resource Management of a Human-Machine Interface
US20210097960A1
