Terminal device, its control method, and program

Abstract

To provide a terminal apparatus that, when a user puts the terminal apparatus in a bag or a pocket, can accurately make a transition to a power saving mode, a method for controlling the same, and a program.SOLUTION: A digital camera 100 comprises: a luminance distribution acquisition unit 120 that acquires the luminance distribution around the digital camera; a storage area 142 that stores model data generated from time-series information including the luminance distribution acquired by the luminance distribution acquisition unit 120; a determination unit 154 that, based on the time-series information and the model data stored in the storage area 142, determines whether a user has ended operations for the digital camera 100 or still performs the operations; and a control unit 153 that causes the digital camera 100 to transition to a power saving mode according to a result of determination made by the determination unit 154.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a terminal device, a control method and a program therefor, and more particularly to a terminal device having a power saving mode, a control method and a program therefor. [Background technology]

[0002] To extend the operating time of devices such as cameras, it is necessary to reduce power consumption, but users sometimes unintentionally leave the power on when they are not using the device, which results in a problem of reduced battery life.

[0003] In particular, if a terminal device is left turned on while being carried in a bag or pocket, it will consume extra power even though the user is clearly not operating the terminal device, affecting the usage time of the terminal device.

[0004] There is a conventional technology that switches a terminal device to power-saving mode if it has not been operated for a certain period of time. However, if the operation panel of the terminal device touches something in the bag or the proximity sensor of the terminal device reacts, it is determined that an operation has been performed on the terminal device, and the timer count is cleared, causing the terminal device to be unable to enter power-saving mode.

[0005] Therefore, Patent Document 1 proposes a technique for switching a terminal device into a power-saving mode when the illuminance measured by an illuminance sensor provided in the terminal device falls below a predetermined threshold.

[0006] Furthermore, Patent Document 2 presents a technology in which attitude information of a terminal device is stored in a buffer, and the terminal device is shifted to a power-saving mode based on the movement of the terminal device determined based on the stored attitude information. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-147125 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-165243 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the technology of Patent Document 1 switches the terminal device to power-saving mode solely based on whether the illuminance acquired by the illuminance sensor is equal to or lower than a predetermined threshold. However, if only this type of determination is used, the terminal device may switch to power-saving mode at an unintended timing, such as when the ambient light becomes dark while the user is operating the terminal device. Furthermore, if the terminal device is placed in a bag, for example, the user may not always close the bag, so if only this type of determination is used, the terminal device may not switch to power-saving mode at the intended timing.

[0009] On the other hand, there are cases where it is not possible to determine whether the movement of the terminal device detected by the technology of Patent Document 2 is a signal to end the operation or whether the operation is still in progress. Furthermore, Patent Document 2 does not take into account the brightness of the surroundings of the terminal device, so there are cases where it is not possible to switch the terminal device into power-saving mode when it is placed in a bag with the power on.

[0010] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a terminal device that can accurately transition to a power-saving mode when a user places the terminal device in a bag or pocket, and a control method and program therefor. [Means for solving the problem]

[0011] In order to solve the above problem, the terminal device of the present invention is characterized by comprising: a brightness distribution acquisition means for acquiring the brightness distribution around the terminal device; a memory area for storing model data generated from time-series information including the brightness distribution acquired by the brightness distribution acquisition means; a judgment means for judging whether a user has finished operating the terminal device or is in the middle of operating it based on the time-series information and the model data; and a power control means for transitioning the terminal device to a power-saving mode depending on the result of the judgment made by the judgment means. [Effects of the Invention]

[0012] According to the present invention, when a user places a terminal device in a bag or pocket, the terminal device can be accurately switched to a power saving mode. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram illustrating an example of the hardware configuration of a digital camera as a terminal device according to an embodiment of the present invention. [Figure 2] 2 is a block diagram showing a detailed configuration of a system control unit, a system memory, and a memory in FIG. 1. FIG. [Figure 3] FIG. 2 is a conceptual diagram of an estimation process using a trained model executed in the determination unit in FIG. 1. [Figure 4] 2 is a conceptual diagram showing the relationship between input data and teacher data used in the learning process executed by the learning unit in FIG. 1. FIG. [Figure 5] FIG. 10 is a diagram for explaining processing in an estimation phase in a digital camera. [Figure 6A] 10 is a detailed flowchart of processing in an estimation phase and processing in a learning phase in a digital camera. [Figure 6B] This is a continuation of the flowchart of FIG. 6A. [Figure 7A] 10A and 10B are diagrams showing examples of luminance distribution time-series information and motion sensor time-series information when a digital camera is placed in a bag from the Y-axis direction in a scene with bright ambient luminance. [Figure 7B]10A and 10B are diagrams illustrating examples of luminance distribution time-series information and motion sensor time-series information when the digital camera is placed in a bag from the Y-axis direction in a scene with low ambient luminance. [Figure 8] 2 is a diagram showing the directions of the X, Y, and Z axes of the motion sensor in FIG. 1. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0014] The best mode for a terminal device for carrying out the present invention will be described below with reference to the drawings. In the embodiment of the present invention described below, a digital camera 100 is used as an example of the terminal device, but the terminal device is not limited to this. In other words, any terminal device can be used as long as it can determine whether a user has finished operating the terminal device or is in the middle of operating it, and can transition to a power-saving mode based on the determination result. For example, the terminal device may be a smartphone, wireless earphones, a tablet, a laptop, or the like.

[0015] FIG. 1 is a block diagram showing the hardware configuration of a digital camera 100 as a terminal device according to this embodiment.

[0016] The digital camera 100 includes a motion sensor 110, a brightness distribution acquisition unit 120, a display unit 130, a communication unit 131, a system timer 132, an operation unit 133, a memory 140, a system memory 141, a system control unit 150, and a power supply unit 160.

[0017] The motion sensor 110 (motion detection means) is configured with an acceleration sensor, a gyroscope sensor, etc., and detects the movement of the digital camera 100 itself and outputs it as motion information. Fig. 8 shows the directions of the X-axis, Y-axis, and Z-axis of the motion sensor 110.

[0018] The luminance distribution acquisition unit 120 (luminance distribution acquisition means) is composed of an image sensor and the like, and acquires the luminance distribution around the digital camera 100. The luminance distribution acquisition unit 120 forms an optical image of the subject via a lens, aperture, shutter, and the like (not shown), and converts the optical image into an electrical signal. The luminance distribution acquisition unit 120 is also arranged on the same plane as the display unit 130. As a result, when the digital camera 100 is placed in a bag or pocket, the luminance distribution acquisition unit 120 detects, although slight, the light emitted by the backlight illumination of the display unit 130 that is reflected inside the bag or pocket as the luminance distribution.

[0019] The display unit 130 (display means) is composed of a display, backlighting, etc. The display may be an organic EL display, a liquid crystal display, or the like. For example, in the case of a liquid crystal display, it has a transmissive liquid crystal panel that uses a thin-film transistor active matrix drive system. The backlighting is a light source that is fixed to the back of the display and projects light onto the display from behind. For example, the backlighting may be composed of a light source such as an LED, fluorescent tube, or organic EL, as well as a light guide plate, reflector, diffuser, etc. that causes the light from the light source to be emitted across the entire display. This surface emission is performed when the digital camera 100 is in an ON state and has not switched to power saving mode.

[0020] The communication unit 131 is connected wirelessly or via a wired cable to transmit and receive video signals and audio signals. The communication unit 131 can also be connected to a wireless LAN (Local Area Network) or the Internet.

[0021] The system timer 132 measures the time used for various controls and the time of the built-in clock.

[0022] The operation unit 133 is an operation means for inputting various predetermined operation instructions to the system control unit 150. In addition to a power switch, the operation unit 133 is configured by any one or a combination of switches, dials, a touch panel, pointing using line-of-sight detection, a voice recognition device, etc. The state of the power switch (operation means) is switched between ON and OFF in response to user operation. Here, when the power switch is in the ON state, it switches the power of the digital camera 100 ON, and when it is in the OFF state, it switches the power of the digital camera 100 OFF.

[0023] The memory 140 uses a ROM, which is a non-volatile memory that can be electrically erased and stored, and stores constants, programs, etc. for the operation of the system control unit 150. The programs referred to here refer to programs for executing various flowcharts described later in this embodiment. The memory 140 also has the function of storing model data consisting of a trained model 301 (FIG. 3), which will be described later, generated from time-series information on the luminance distribution from the luminance distribution acquisition unit 120 and time-series information on the motion from the motion sensor 110.

[0024] The system memory 141 is configured with RAM, and expands constants, programs, etc. for the operation of the system control unit 150 that are read from the memory 140. The system memory 141 also has the function of storing luminance information from the luminance distribution acquisition unit 120 and time-series information of motion information from the motion sensor 110.

[0025] The system control unit 150 has at least one processor, such as a CPU or GPU, and controls the overall operation of the digital camera 100. Since a GPU can perform efficient calculations by processing a larger amount of data in parallel, it is effective to use a GPU when performing multiple learning rounds using a learning model such as deep learning. Therefore, in this embodiment, a GPU is used in addition to a CPU for processing by a learning unit 155 (FIG. 2), which is part of the system control unit 150 and will be described later. Specifically, when a learning program including a learning model is executed, the CPU and GPU work together to perform calculations to perform learning. The processing of the learning unit 155 may be performed by only the CPU or the GPU. A GPU may also be used for processing by a determination unit 154 (FIG. 2), which is part of the system control unit 150 and will be described later.

[0026] The power supply unit 160 is composed of a battery, a battery detection circuit, a protection circuit, a power supply circuit, etc. Based on commands from the system control unit 150, the power supply unit 160 supplies the desired power supply voltage to each unit of the digital camera 100 for a desired period of time. The power supply unit 160 also has the function of detecting whether a battery is installed, the battery type, and the remaining charge, and notifying the system control unit 150 of the detection results. Furthermore, the power supply unit 160 has the function of protecting the load circuit connected to the power supply circuit by cutting off the power supply when it detects a power supply abnormality such as an overcurrent or overvoltage. The power supply circuit is composed of a DC-DC converter, in particular an LDO regulator.

[0027] Next, the detailed configurations of the system control unit 150, the system memory 141, and the memory 140 will be described with reference to FIG.

[0028] The system control unit 150 includes a data transmission / reception unit 151 , a sensor data acquisition unit 152 , a control unit 153 , a determination unit 154 , and a learning unit 155 .

[0029] The sensor data acquisition unit 152 (generation means) periodically acquires the luminance distribution information from the luminance distribution acquisition unit 120 and the motion information from the motion sensor 110 (hereinafter referred to as "sensor detection information") at a predetermined sampling rate, and generates time-series information of these. The sensor data acquisition unit 152 further transmits the generated time-series information of each sensor detection information to the accumulation area 143 of the system memory 141 via the data transmission / reception unit 151, and controls the storage.

[0030] The data transmission / reception unit 151 is responsible for transmitting and receiving data between the system control unit 150 and blocks of the digital camera 100 other than the brightness distribution acquisition unit 120 and the motion sensor 110. For example, in order to pass the trained model 301 used for estimation to the determination unit 154, the data transmission / reception unit 151 receives the trained model 301 from the storage area 142 of the memory 140 and controls transmission of the trained model 301 to the determination unit 154. The data transmission / reception unit 151 also receives time-series information of each sensor detection information stored in the accumulation area 143 of the system memory 141 and controls transmission of the information to the learning unit 155 and the determination unit 154. Furthermore, the data transmission / reception unit 151 receives a command to transition to a power-saving mode from the determination unit 154 and transmits the command to the power supply unit 160.

[0031] The accumulation area 143 of the system memory 141 is configured by a finite buffer or the like, and stores time-series information of each piece of sensor detection information transmitted from the sensor data acquisition unit 152. Furthermore, the accumulation area 143 transmits the time-series information of each piece of sensor detection information to the determination unit 154 in response to a data transmission request from the determination unit 154.

[0032] The storage area 142 of the memory 140 stores operational constants, programs, and a trained model 301 that control the entire digital camera 100. In response to a request to transmit the trained model 301 from the determination unit 154 of the system control unit 150, the storage area 142 transmits the trained model 301 stored therein via the data transmission / reception unit 151. Note that the storage area 142 may store internal parameters of the trained model 301 instead of the trained model 301. The storage area 142 may also store a list of rule bases equivalent to the trained model 301, and transmit the list when the determination unit 154 performs estimation processing.

[0033] The control unit 153 reads out a program for controlling the digital camera 100 from the memory 140 and loads part of the program into the system memory 141, thereby controlling the entire digital camera 100. Furthermore, when a predetermined condition is satisfied, the control unit 153 issues a command to the learning unit 155 (additional learning means) to perform additional learning of the trained model 301.

[0034] The determination unit 154 (determination means) uses a CPU or GPU to perform estimation processing to determine whether the user has placed the digital camera 100 in a bag (whether operation has been completed or is in the middle of being performed). This estimation processing is performed based on the time-series information of each sensor detection information stored in the accumulation area 143 of the system memory 141 and the trained model 301 stored in the storage area 142 of the memory 140. The determination unit 154 notifies the control unit 153 of the result of this estimation processing. When the control unit 153 (power supply control means) is notified by the determination unit 154 of the result of the estimation processing that the user has placed the digital camera 100 in a bag, the control unit 153 transmits a command to the power supply unit 160 via the data transmission / reception unit 151 to transition to power-saving mode.

[0035] Determination unit 154 also determines whether the time-series information of each sensor detection information stored in accumulation area 143 of system memory 141 satisfies a predetermined condition. If determination unit 154 determines that the predetermined condition is satisfied, control unit 153 determines whether the condition for performing additional learning in learning unit 155 is satisfied. If control unit 153 determines that the condition for performing additional learning is satisfied, it commands learning unit 155 to perform additional learning. Details of this processing flow will be described with reference to FIGS. 6A and 6B.

[0036] When the learning unit 155 receives a command to perform (additional) learning processing from the control unit 153, it receives the time series information of each sensor detection information stored in the storage area 143 of the system memory 141 as input data and executes the (additional) learning processing using the CPU or GPU.

[0037] Furthermore, the learning unit 155 may include an error detection unit and an update unit.

[0038] The error detection unit obtains the error between the training data and the output data output from the output layer of the neural network in response to the input data (learning data) input to the input layer. The error detection unit may use a loss function to calculate the error between the training data and the output data from the neural network.

[0039] The update unit updates the connection weighting coefficients between the nodes of the neural network based on the error obtained by the error detection unit so as to reduce the error. This update unit updates the connection weighting coefficients using, for example, an error backpropagation method. The error backpropagation method is a technique for adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the error.

[0040] It should be noted that the operation of the present invention is possible as long as the trained model 301 used in the above estimation process is stored in the storage area 142 of the memory 140, and the operation of the present invention is possible even without the learning unit 155 that executes the learning process to generate this trained model 301. Furthermore, the additional learning process by the learning unit 155, which will be described below, is necessary to improve the accuracy of the present invention, but is not an essential component of the present invention.

[0041] The power supply unit 160 executes power supply control based on instructions from the control unit 153 .

[0042] Next, the content of estimation performed by the determination unit 154 using the trained model 301 will be described with reference to FIGS. 3 and 4. FIG.

[0043] FIG. 3 is a conceptual diagram of the estimation process using the trained model 301 executed by the determination unit 154.

[0044] The trained model 301 is a neural network trained by a learning process (machine learning) executed by the learning unit 155. Note that the trained model 301 is not limited to a neural network, and may be a nearest neighbor method, a naive Bayes method, a decision tree, a support vector machine, or the like, as long as it is obtained by machine learning. Furthermore, the training method may be deep learning, which uses a neural network to generate features and connection weighting coefficients for training. Furthermore, in this embodiment, since time-series data of each sensor detection information that changes over time is used as input data, it is also preferable to use an RNN (recurrent neural network).

[0045] The luminance distribution time series information 302 is one of the input data to the trained model 301 and is time series information of the luminance distribution information acquired by the luminance distribution acquisition unit 120 of the digital camera 100.

[0046] The motion sensor time-series information 303 is one of the input data to the trained model 301 and is time-series information of the motion information acquired by the motion sensor 110 of the digital camera 100.

[0047] The output data 304 is data that is output when the judgment unit 154 inputs the brightness distribution time series information 302 and the motion sensor time series information 303 into the trained model 301 and executes an estimation process using the trained model 301. The output data 304 includes a judgment result indicating whether the user has placed the digital camera 100 in a bag (or pocket, etc.) and has finished operating it, or whether the user is in the middle of operating it.

[0048] FIG. 4 is a conceptual diagram showing the relationship between input data (learning data) and teacher data used in the learning process executed by learning unit 155.

[0049] As shown in Figure 4, the input data, brightness distribution time series information 302 and motion sensor time series information 303, are, for example, a set of brightness distribution time series information (BD_01) and motion sensor time series information (ACC_01), as shown in row 401 of Figure 4.

[0050] The luminance distribution time-series information (BD_01) and the motion sensor time-series information (ACC_01) are information acquired by the luminance distribution acquisition unit 120 and the motion sensor 110 at the same time, respectively.

[0051] The training data 402 is data indicating whether the user had placed the digital camera 100 in a bag, pocket, etc. and had finished operating it, or was in the middle of operating it, during the period when the input data for row 401 was acquired by the brightness distribution acquisition unit 120 and the motion sensor 110. Specifically, training data such as training data 402, which is True, indicates that the digital camera 100 had finished operating. On the other hand, training data such as False indicates that the digital camera 100 was in the middle of operating it.

[0052] Next, the relationship between the luminance distribution time series information 302 and the motion sensor time series information 303, which are used as input data to the trained model that determines whether the digital camera 100 is in a bag, will be explained using the examples shown in Figures 7A and 7B.

[0053] 7A is a diagram showing an example of luminance distribution time-series information 302 and motion sensor time-series information 303 when digital camera 100 is placed into a bag in the Y-axis direction (FIG. 8) from a scene with bright ambient luminance. Here, times t0 to t3 are times when sensor data acquisition unit 152 acquires each piece of sensor detection information from luminance distribution acquisition unit 120 and motion sensor 110 at a predetermined sampling rate. Also, in the example of FIG. 7A, the user places digital camera 100 into the bag immediately after time t0, and then digital camera 100 remains in the bag from times t1 to t3.

[0054] When the ambient brightness is bright, at time t0 before the digital camera 100 is placed in the bag, the brightness distribution 701 acquired by the brightness distribution acquisition unit 120 is bright. On the other hand, at times t1 to t3 after the digital camera 100 is placed in the bag, the brightness gradually becomes darker from the direction in which the digital camera 100 is inserted into the bag (hereinafter simply referred to as the insertion direction), as shown in brightness distributions 702 to 704 acquired by the brightness distribution acquisition unit 120. After that, the entire surface finally becomes dark, as shown in brightness distribution 704 at time t3.

[0055] On the other hand, from time t0 to t3, in the motion sensor time-series information 303, a noticeable change in acceleration in the Y axis, which is the insertion direction, is detected.

[0056] In this way, when the user places the digital camera 100 in a bag, a characteristic time-series change occurs in the brightness distribution time-series information 302 and the motion sensor time-series information 303. Therefore, by using this characteristic time-series change, it becomes possible to determine whether or not the user has placed the digital camera 100 in a bag.

[0057] On the other hand, Fig. 7B is a diagram showing an example of luminance distribution time-series information 302 and motion sensor time-series information 303 when digital camera 100 is placed into a bag from the Y-axis direction from a scene with low ambient luminance, as in Fig. 7A. Note that in the example of Fig. 7B, as in Fig. 7A, the user places digital camera 100 into the bag immediately after time t0, and thereafter digital camera 100 remains in the bag from time t1 to t3.

[0058] When the ambient brightness is low, at time t0 before the digital camera 100 is placed in the bag, the brightness distribution 801 acquired by the brightness distribution acquisition unit 120 is completely dark. On the other hand, at times t1 to t3 after the digital camera 100 is placed in the bag, the brightness distribution gradually becomes slightly brighter from the insertion direction, as shown in brightness distributions 802 to 804 acquired by the brightness distribution acquisition unit 120. Thereafter, the entire surface finally becomes slightly bright, as shown in brightness distribution 804 at time t3. This is because, when the digital camera 100 is placed in a bag or pocket, the brightness distribution acquisition unit 120, which is disposed on the same surface as the display unit 130, can detect the light from the backlight illumination of the display unit 130 reflected inside the bag or pocket.

[0059] On the other hand, from time t0 to t3, in the motion sensor time-series information 303, a noticeable change in acceleration in the Y axis, which is the insertion direction, is detected in FIG. 7B as well as in FIG. 7A.

[0060] In this way, even when the user puts the digital camera 100 into a bag from a dark scene, a characteristic time-series change occurs in the brightness distribution time-series information 302 and the motion sensor time-series information 303. Therefore, using this characteristic time-series change, it becomes possible to determine whether or not the user has put the digital camera 100 into a bag.

[0061] Next, the processing in the estimation phase in the digital camera 100 will be described with reference to FIG.

[0062] First, in step S501, the sensor data acquisition unit 152 periodically acquires each piece of sensor detection information from the brightness distribution acquisition unit 120 and the motion sensor 110 at a predetermined sampling rate, and stores the information in the accumulation area 143.

[0063] In step S502, the determination unit 154 requests the accumulation area 143 to transmit input data, that is, each piece of sensor time-series information stored in the accumulation area 143 (luminance distribution time-series information 302, motion sensor time-series information 303).

[0064] In step S503, in response to a transmission request from the determination unit 154 in step S502, the storage area 143 transmits each sensor time series information (brightness distribution time series information 302, motion sensor time series information 303) stored in the storage area 143 to the determination unit 154.

[0065] In step S504, the determination unit 154 requests the storage area 142 to transmit the trained model 301.

[0066] In step S505, the storage area 142 transmits the trained model 301 to the determination unit 154 in response to the transmission request from the determination unit 154 in step S504.

[0067] In step S506, the determination unit 154 performs estimation processing based on the trained model 301 received from the storage area 142 and the input data received in step S503. This estimation processing determines whether the user has put the digital camera 100 into their bag (or pocket) (operation has ended) or not (operation is still in progress). Thereafter, the determination unit 154 transmits the determination result of this estimation processing to the control unit 153.

[0068] In step S507, if the judgment unit 154 in step S506 sends a judgment result that the user has put the digital camera 100 in the bag (operation has ended), the control unit 153 issues a command to the power supply unit 160 to transition to power saving mode.

[0069] In step S508, the power supply unit 160 executes power supply control based on the command from the control unit 153 in step S507, and transitions to a power saving mode.

[0070] As a result of the estimation process using the trained model 301 and the input data, when it is determined that the user has put the digital camera 100 into a bag (or pocket) (the operation has ended), the power supply unit 160 can immediately switch to power-saving mode, which has the effect of improving the battery life.

[0071] Next, using the flowcharts of Figures 6A and 6B, we will explain the processing in the estimation phase in digital camera 100 and the processing in the learning phase (additional learning processing) that is executed if the processing in the estimation phase does not result in a transition to power saving mode.

[0072] 6A, first, in step S601, control unit 153 determines whether the state of the power switch of operation unit 133 is ON or OFF. If it is determined to be ON (YES in step S601), the process proceeds to step S602. On the other hand, if the state of the power switch has been switched from ON to OFF in response to a user operation (NO in step S602), the process proceeds to step S621 (FIG. 6B).

[0073] In step S602, the control unit 153 determines whether the motion sensor 110 has detected a change in motion information equal to or greater than a predetermined threshold. If it is determined that a change has been detected (YES in step S602), the control unit 153 proceeds to step S603. On the other hand, if it is determined that a change has not been detected (NO in step S602), the control unit 153 proceeds to step S604.

[0074] In step S603, the control unit 153 increases the sampling rate from the initial value when the sensor data acquisition unit 152 acquires each piece of sensor detection information from the luminance distribution acquisition unit 120 and the motion sensor 110. Note that, although the sampling rates for both the luminance distribution acquisition unit 120 and the motion sensor 110 are increased in this embodiment, the present invention is not limited to this as long as the sampling rate for at least one of them is increased.

[0075] The reason for increasing the sampling rate in step S603 is that when the user puts the digital camera 100 into a bag or pocket, there is a high possibility that a change in motion information (change in acceleration) greater than or equal to a predetermined threshold will occur. This can improve the accuracy of the estimation process by the determination unit 154. On the other hand, if no change in motion information greater than or equal to the predetermined threshold is detected, it is unlikely that the user put the digital camera 100 into a bag or pocket, so the sampling rate is not increased and remains at the initial value. This can reduce the power consumption of the sensor data acquisition unit 152, the brightness distribution acquisition unit 120, and the sensor data acquisition unit 152.

[0076] In step S604, the determination unit 154 acquires, as input data, the luminance distribution time-series information 302 and the motion sensor time-series information 303 stored in the storage area 143 of the system memory 141. Furthermore, the determination unit 154 acquires the trained model 301 stored in the storage area 142 of the memory 140.

[0077] In step S605, the determination unit 154 inputs the input data (luminance distribution time-series information 302 and motion sensor time-series information 303) to the trained model 301 and acquires output data 304. Hereinafter, the input data (luminance distribution time-series information 302 and motion sensor time-series information 303) will be referred to as input data (302, 303). Thereafter, the determination unit 154 notifies the control unit 153 of the acquired output data 304.

[0078] In step S606, control unit 153 determines whether the output data notified by determination unit 154 in step S605 indicates that the user has placed digital camera 100 in the bag (end of operation). If it is determined that digital camera 100 has been placed in the bag (end of operation: second condition is met) (YES in step S606), the process proceeds to step S607. On the other hand, if it is determined that digital camera 100 has not been placed in the bag (operation is in progress) (NO in step S606), the process proceeds to step S615 (FIG. 6B).

[0079] In step S607, control unit 153 determines whether or not an operation has been performed on operation unit 133 based on a signal from operation unit 133. If it is determined that no operation has been performed (YES in step S607), the process proceeds to step S608. On the other hand, if it is determined that an operation has been performed (NO in step S607), the process proceeds to step S610.

[0080] In step S608, control unit 153 uses system timer 132 to determine whether a predetermined time (second time) has elapsed since it was determined in step S606 that the device was placed in the bag. If it is determined that the predetermined time has elapsed (YES in step S608), the process proceeds to step S609. On the other hand, if it is determined that the predetermined time has not elapsed (NO in step S608), the process returns to step S607.

[0081] In step S609, the control unit 153 resets the sampling rate increased in step S603 to the initial value, and issues a power saving mode command to the power supply unit 160. When the power supply unit 160 transitions to the power saving mode in response to this command, this process ends.

[0082] In step S610, the control unit 153 generates training data indicating "operation in progress (False)" for the input data (302, 303) acquired in step S604. Thereafter, the control unit 153 issues a command to the learning unit 155 to update the trained model 301 by additional training using the input data and the generated training data. Based on the command from the control unit 153, the learning unit 155 executes additional training processing to update the trained model held by the learning unit 155, using the input data (302, 303) and the generated training data.

[0083] In step S611, it is determined whether or not the additional learning process by the learning unit 155 has been completed. If it is determined that the additional learning has been completed (YES in step S611), the process proceeds to step S612. On the other hand, if it is determined that the additional learning has not been completed (NO in step S612), the determination in step S611 is repeated.

[0084] In step S612, the learning unit 155 updates the trained model 301 stored in the storage area 142 of the memory 140 with the updated trained model held by the learning unit 155.

[0085] In step S613, the control unit 153 uses the system timer 132 to determine whether a predetermined time has elapsed since the sampling rate was increased in step S603. If it is determined that the predetermined time has elapsed (YES in step S613), the process proceeds to step S614, where the sampling rate is reset to the initial value, and the process returns to step S601. On the other hand, if it is determined that the predetermined time has not elapsed (NO in step S613), the same operations are repeated from step S601.

[0086] 6B, in step S615, the determination unit 154 determines whether the change directions of the input data (302, 303) acquired in step S605 match. If it is determined that the change directions of the input data (302, 303) match (the first condition is satisfied) (YES in step S615), the process proceeds to step S616. On the other hand, if it is determined that they do not match (NO in step S615), the process returns to step S613.

[0087] 7A and 7B, when the user places digital camera 100 in a bag perpendicular to the Y direction, the change in acceleration in the Y direction increases, and the brightness distribution also gradually changes from the Y direction. In other words, if the directions of change in the input data (302, 303) in step S605 are consistent, the determination in step S606 that the digital camera is not in the bag (operation in progress) may be incorrect, and the digital camera may actually have been placed in the bag. For example, when digital camera 100 is placed in the bag, the operation unit 133 may touch something, resulting in a determination in step S606 that operation is in progress. Therefore, in the following steps, if this is highly likely, additional learning is performed to update trained model 301 using the input data and training data indicating operation completion (True).

[0088] In step S616, control unit 153 determines whether or not an operation has been performed on operation unit 133 based on a signal from operation unit 133. If it is determined that no operation has been performed (YES in step S616), the process proceeds to step S617. On the other hand, if it is determined that an operation has been performed (NO in step S616), the process returns to step S613.

[0089] In step S617, the control unit 153 uses the system timer 132 to determine whether a predetermined time (first time) has elapsed since it was determined in step S606 that an operation was in progress. If it is determined that the predetermined time has elapsed (YES in step S617), the process proceeds to step S618. On the other hand, if it is determined that the predetermined time has not elapsed (NO in step S617), the process returns to step S616.

[0090] In step S618, the control unit 153 generates teacher data for the input data (302, 303) indicating that the item has been placed in the bag, i.e., that the operation has ended (True), to the learning unit 155. Thereafter, the control unit 153 issues a command to the learning unit 155 to update the trained model 301 by additional learning using the input data and the generated teacher data. Based on this command from the control unit 153, the learning unit 155 executes additional learning processing to update the trained model held by the learning unit 155, using the input data (302, 303) and the generated teacher data.

[0091] In step S619, it is determined whether or not the additional learning process by the learning unit 155 has been completed. If it is determined that the additional learning has been completed (YES in step S619), the process proceeds to step S620. On the other hand, if it is determined that the additional learning has not been completed (NO in step S619), the determination in step S619 is repeated.

[0092] In step S620, the learning unit 155 updates the trained model 301 stored in the storage area 142 of the memory 140 with the updated trained model held by the learning unit 155, and then returns to step S609.

[0093] In step S621, the control unit 153 increases the sampling rate at which the sensor data acquisition unit 152 acquires the sensor detection information from the brightness distribution acquisition unit 120 and the motion sensor 110 from the initial value.

[0094] In step S622, the determination unit 154 acquires the luminance distribution time-series information 302 and the motion sensor time-series information 303 stored in the storage area 143 of the system memory 141 as input data.

[0095] In step S623, the determination unit 154 determines whether the change directions of the input data (302, 303) acquired in step S622 match. If it is determined that the change directions of the input data (302, 303) match (the third condition is satisfied) (YES in step S623), the process proceeds to step S618. On the other hand, if it is determined that they do not match (NO in step S623), the process proceeds to step S624.

[0096] In step S624, the control unit 153 uses the system timer 132 to determine whether a predetermined time (third time) has elapsed since it was determined in step S601 that the power switch was turned off. If it is determined that the predetermined time has not elapsed (YES in step S624), the process returns to step S622. On the other hand, if it is determined that the predetermined time has elapsed (NO in step S624), the process proceeds to step S609, where the process shifts to normal power saving mode processing without learning.

[0097] As described above, in this embodiment, the luminance distribution time-series information 302 and the motion sensor time-series information 303 are input as input data to the trained model 301, and the digital camera 100 has an estimation phase in which it is determined whether the operation has ended or is still in progress. If it is determined in this estimation phase that the operation has ended, the digital camera 100 transitions to power-saving mode.

[0098] Furthermore, in this embodiment, after the estimation phase, additional learning processing is performed by the learning unit 155. This allows the trained model 301 to adapt to the individual habits of each user, and as the user uses the digital camera 100, the accuracy of judgment in the estimation phase improves, enabling a quick transition to power-saving mode without false detection.

[0099] In this embodiment, the determination unit 154 performs estimation processing using the machine-learned trained model 301 as model data. However, rule-based estimation processing may also be performed using a lookup table (LUT) or the like as model data. In this case, for example, the relationship between input data and output data may be created in advance as an LUT, and the created LUT may be stored in the memory 140. This allows the determination unit 154 to acquire output data by performing pattern matching with reference to the stored LUT. That is, in this case, the determination unit 154 can use the LUT as a program that exhibits the same function as the trained model 301.

[0100] In this embodiment, a program that implements one or more functions may be provided to a computer in a system or device via a network or storage medium, and the program may be read and executed by a system controller of the system or device. The system controller may have one or more processors or circuits, and may include multiple separate system controllers or a network of multiple separate processors or circuits to read and execute the executable instructions.

[0101] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0102] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0103] 110 Motion Sensor 120 Luminance distribution acquisition unit 142 Storage area 143 Accumulation area 153 Control Unit 154 Judgment Department 155 Learning Department 160 Power supply section 301 trained models 302 Luminance distribution time series information 303 Motion Sensor Time Series Information

Claims

1. a brightness distribution acquisition means for acquiring a brightness distribution around the terminal device; a storage area for storing model data generated from time-series information including the luminance distribution acquired by the luminance distribution acquisition means; a determination means for determining whether a user has completed an operation on the terminal device or is in the middle of an operation based on the time-series information and the model data; A terminal device characterized by comprising power control means for shifting the terminal device to a power saving mode in accordance with the result of the determination made by the determination means.

2. The device further includes a motion detection unit that detects motion of the terminal device at the same time that the brightness distribution acquisition unit acquires the brightness distribution and outputs the motion information, 2. The terminal device according to claim 1, wherein the time-series information further includes motion information output by the motion detection means.

3. a generating means for generating the time-series information by periodically acquiring the luminance distribution from the luminance distribution acquiring means and the motion information from the motion detecting means at a predetermined sampling rate; The terminal device according to claim 2, further comprising a control means for controlling the generation means to increase the sampling rate for at least one of the brightness distribution acquisition means and the motion detection means when there is a change in the motion information output by the motion detection means that exceeds a predetermined threshold.

4. The terminal device according to any one of claims 1 to 3, characterized in that the model data stored in the memory area is a trained model trained using the time series information as training data and data indicating whether the user had finished operating the terminal device or was in the middle of operating it during the period when the training data was acquired by the brightness distribution acquisition means as its teacher data.

5. The terminal device according to claim 4, further comprising additional learning means for performing additional learning of the trained model.

6. The terminal device described in claim 5, characterized in that if the time series information satisfies a first condition and no operation is performed on the terminal device before a first time has elapsed, the additional learning means uses the time series information as the learning data and performs additional learning of the learned model using data indicating that the user has finished operating the terminal device as the teaching data.

7. The terminal device described in claim 5 or 6, characterized in that when the time series information satisfies a second condition and operations are performed on the terminal device until a second time has elapsed, the additional learning means uses the time series information as the learning data and performs additional learning of the learned model using data indicating that the user was in the middle of operating the terminal device as the teaching data.

8. further comprising an operation means for switching the power of the terminal device on and off in response to a user operation; The terminal device according to any one of claims 5 to 7, characterized in that when the operation means switches the terminal device from power ON to power OFF in response to the user operation, if the time series information satisfies a third condition before a third time has elapsed, the additional learning means uses the time series information as the learning data and performs additional learning of the learned model using data indicating that the user has finished operating the terminal device as the teaching data.

9. The terminal device further includes a display means for emitting surface light when the terminal device is in a power-on state and has not yet entered the power-saving mode, 9. The terminal device according to claim 1, wherein the luminance distribution acquisition means is disposed on the same plane as the display means.

10. A method for controlling a terminal device having an image sensor, a memory, and a processor, comprising: a brightness distribution acquisition step of acquiring a brightness distribution around the terminal device by the image sensor; a storage step of storing, in the memory, model data generated from time-series information including the luminance distribution acquired in the luminance distribution acquisition step; a determining step of determining, by the processor, whether the user has finished operating the terminal device or is in the middle of operating the terminal device based on the time-series information and the model data; a power control step of causing the processor to transition the terminal device to a power saving mode depending on the result of the determination made in the determination step.

11. A program for causing a computer to function as each of the means of the terminal device according to any one of claims 1 to 9.

Images