Information processing method, information processing apparatus, and computer program

Abstract

To provide a technique for accurately determining a person who operates a device, in an environment in which people who operate the device vary every day.SOLUTION: An information processing apparatus 11 acquires an image captured by a camera 12. The information processing apparatus 11 detects a hand appearing in the acquired image. The information processing apparatus 11 determines that the detected hand is a hand of a non-operating person when feature information of the detected hand meets predetermined conditions for non-operating persons who are not allowed to operate a smart mirror 10. The information processing apparatus 11 suppresses gesture operation on the smart mirror 10 from being recognized based on motion of the detected hand when the detected hand is determined to be a hand of a non-operating person.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to data processing technology, and more particularly to an information processing method, an information processing device, and a computer program. [Background technology]

[0002] A technology has been proposed in which a user operates a terminal using a non-contact operation such as a gesture operation, and the display contents on the terminal are changed in accordance with the operation. For example, Patent Document 1 describes a technology in which, when there are multiple operators, the priority of each of the multiple operators is determined based on the appearance characteristics of each operator, thereby accurately detecting a gesture operation with a high priority. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-88721 A Summary of the Invention [Problem to be solved by the invention]

[0004] As shown in Patent Document 1, in order to determine the priority of each of multiple operators based on the external characteristics of each operator, it is necessary to store the external characteristics of each operator. However, in an environment where operators change daily (such as a store), it is not easy to store the external characteristics of each operator.

[0005] The present disclosure has been made based on the inventor's recognition of the above problem, and one objective is to provide a technology for accurately determining the operator of an equipment in an environment where the operator of the equipment changes from day to day. [Means for solving the problem]

[0006] In order to solve the above problems, an information processing method of one aspect of the present disclosure is executed by a computer to acquire an image captured by a camera, detect a hand appearing in the acquired image, and if characteristic information of the detected hand satisfies predetermined conditions related to a non-operator who is not permitted to operate a specified device, determine that the detected hand is the hand of a non-operator, and if it is determined that the detected hand is the hand of a non-operator, suppress recognition of a gesture operation on the device based on the movement of the detected hand.

[0007] Another aspect of the present disclosure is an information processing device including: an image acquisition unit that acquires an image captured by a camera; a detection unit that detects a hand captured in the image acquired by the image acquisition unit; a determination unit that determines that the detected hand is the hand of a non-operator when feature information of the hand detected by the detection unit satisfies a predetermined condition related to a non-operator who is not permitted to operate a predetermined device; and an operation recognition unit that suppresses recognition of a gesture operation on the device based on a movement of the detected hand when it is determined that the detected hand is the hand of the non-operator.

[0008] In addition, any combination of the above components, and any conversion between the expressions of the present disclosure, such as a system, a computer program, a recording medium having a computer program recorded thereon, a smart mirror, etc., are also valid aspects of the present disclosure. Effect of the Invention

[0009] According to the technology of the present disclosure, the operator of the device can be accurately determined in an environment where the operator changes from day to day. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of use of the smart mirror according to the first embodiment. [Diagram 2] 1 is a block diagram showing functional blocks of an information processing apparatus according to a first embodiment; [Diagram 3] 4 is a flowchart showing the operation of the information processing device of the first embodiment. [Figure 4] 4 is a flowchart showing details of the hairdresser determination process in S13 of FIG. 3. [Diagram 5] FIG. 13 is a diagram illustrating an example of deriving the speed of a hand movement. [Figure 6] FIG. 11 is a block diagram showing functional blocks of an information processing apparatus according to a second embodiment. [Figure 7] 4 is a diagram showing an example of biometric information stored in a biometric information storage unit; FIG. [Figure 8] 10 is a flowchart showing the operation of an information processing device according to a second embodiment. [Figure 9] 13 is a flowchart showing the operation of an information processing device according to a third embodiment. [Figure 10] 13 is a flowchart showing the operation of an information processing device according to a fourth embodiment. [Figure 11] FIG. 13 is a block diagram showing functional blocks of an information processing apparatus according to a fifth embodiment. [Figure 12] FIG. 11 is a diagram showing an example of scene information. [Figure 13] 13 is a flowchart showing the operation of an information processing device according to the fifth embodiment. [Figure 14] 14 is a flowchart showing the details of the scene determination process in S61 of FIG. 13. [Figure 15] FIG. 11 is a diagram illustrating an example of condition information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The subject of the device or method in this disclosure includes a computer. The computer executes a program to realize the function of the subject of the device or method in this disclosure. The computer includes a processor that operates according to a program as a main hardware configuration. The type of processor is not important as long as it can realize the function by executing the program. The processor is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or an LSI (Large Scale Integration). Although it is called an IC or an LSI here, the name changes depending on the degree of integration, and it may be called a system LSI, a VLSI (Very Large Scale Integration), or an ULSI (Ultra Large Scale Integration). A Field Programmable Gate Array (FPGA), which is programmed after the LSI is manufactured, or a reconfigurable logic device that can reconfigure the connection relationship inside the LSI or set up the circuit partition inside the LSI, can also be used for the same purpose. Multiple electronic circuits may be integrated into one chip or provided on multiple chips. The multiple chips may be integrated in one device, or may be provided in multiple devices. The program may be recorded in a non-transitory recording medium such as a computer-readable Read Only Memory (ROM), an optical disk, or a hard disk drive, or may be recorded in a temporary storage medium such as a computer-readable Random Access Memory (RAM). The program may be stored in the recording medium in advance, or may be supplied to the recording medium or storage medium via a wide area communication network including the Internet.

[0012] First Embodiment An overview of the first embodiment will be described. Conventionally, a technology has been proposed in which a user operates a terminal using non-contact operations such as gesture operations, and the display contents of a display terminal are switched in accordance with the operation. However, as shown in Patent Document 1, in order to determine the priority of each of a plurality of operators based on the appearance characteristics of each operator, it is necessary to store the appearance characteristics of each operator, which is inconvenient in an environment (such as a store) where operators change daily.

[0013] The present inventor has developed a smart mirror to be installed in a beauty salon. This smart mirror detects gesture operations input by a customer during a hair treatment and displays various information according to the gesture operations. The smart mirror installed in a store such as a beauty salon has a feature that the operators are an unspecified number of people and change daily. In addition, from the viewpoint of privacy protection, it is not desirable to store the appearance characteristics of the customer, such as face information. Therefore, it is difficult to determine the priority of the operator based on the appearance characteristics of the operator as described in Patent Document 1.

[0014] Hereinafter, a person who is not permitted to operate a specific device (e.g., a smart mirror) will be referred to as a "non-operator," and a person who is permitted to operate the device will be referred to as an "authorized operator." The information processing device of the first embodiment determines whether a person who inputs an operation to a device (hereinafter referred to as an "operator") is a non-operator or an authorized operator, based on the characteristics of non-operators registered in advance. This allows the operator of the device to be accurately determined in an environment where the operator of the device changes daily.

[0015] The first embodiment will be described in detail. FIG. 1 shows an example of use of the smart mirror 10 of the first embodiment. FIG. 1 shows a customer sitting in a seat in front of the smart mirror 10 and a hairdresser treating the customer's hair. The treatment includes cutting, styling, coloring, and perming the hair. A camera 12 and a display unit 14 are provided on the mirror surface of the smart mirror 10. The camera 12 photographs the hairdresser and the customer facing the smart mirror 10. As a variant, the camera 12 may be externally attached to the frame of the smart mirror 10. The display unit 14 displays various electronic contents in response to gesture operations by an operator (the hairdresser or the customer in FIG. 1) on the smart mirror 10.

[0016] 1, the smart mirror 10 includes an information processing device that executes various information processes including a gesture operation recognition process. The information processing device can also be said to be a control device that controls the display content of the display unit 14.

[0017] 2 is a block diagram showing the functional blocks of the information processing device 11 of the first embodiment. Each block shown in the block diagram of the present disclosure can be realized in terms of hardware by elements and mechanical devices such as a computer CPU and memory, and in terms of software by a computer program, etc., but here, the functional blocks realized by their cooperation are depicted. Those skilled in the art will understand that these functional blocks can be realized in various forms by combining hardware and software.

[0018] The information processing device 11 includes a camera 12, a display unit 14, a control unit 16, a storage unit 18, and a communication unit 20. In the first embodiment, the camera 12 and the display unit 14 are included in the information processing device 11. However, as a modified example, the camera 12 and the display unit 14 may be provided outside the information processing device 11.

[0019] The camera 12 photographs a person or object in front of the smart mirror 10, in other words, photographs a person or object reflected on the mirror surface of the smart mirror 10. The camera 12 may be a far-infrared camera that detects far-infrared rays emitted by a person or object in front of the smart mirror 10. This is particularly useful when there is no lighting on the person or object in front of the smart mirror 10. The display unit 14 may include a liquid crystal display or an organic EL display.

[0020] The control unit 16 executes various data processing operations and controls the operation of the information processing device 11. The control unit 16 may be realized by a processor (such as a CPU) included in the information processing device 11. The storage unit 18 stores data referenced or updated by the control unit 16. The communication unit 20 communicates with an external device (not shown) according to a predetermined communication protocol. The control unit 16 communicates with the external device via the communication unit 20.

[0021] The storage unit 18 includes a position storage unit 22 and a hand storage unit 24. The position storage unit 22 stores information indicating the characteristics and position (e.g., coordinate information) of a hand detected from an image captured by the camera 12 (hereinafter also referred to as a "captured image"). The position storage unit 22 stores the characteristic information and position information of each hand in association with the ID of each hand. The position storage unit 22 also stores the time-series transition of the position of each of one or more hands appearing in the captured image.

[0022] The hand storage unit 24 stores feature information related to the external features of one or more hands detected from the captured image. The feature information of the hand may include, for example, information related to the shape and color of the hand.

[0023] The control unit 16 includes an image acquisition unit 30, a detection unit 32, a hairdresser determination unit 34, a hand registration unit 36, a gesture recognition unit 38, and a display control unit 40. The functions of these multiple functional blocks may be implemented in a computer program, and this computer program may be installed in the storage of the information processing device 11. The processor of the information processing device 11 may fulfill the functions of the multiple functional blocks by reading this computer program into a main memory and executing it.

[0024] The image acquisition unit 30 acquires data of an image captured by the camera 12 (hereinafter also referred to as a "captured image") from the camera 12. As shown in Fig. 1, the captured image may show one or both of a hairdresser and a customer receiving a treatment from the hairdresser.

[0025] The detection unit 32 detects the hand appearing in the photographed image acquired by the image acquisition unit 30. The detection unit 32 detects the feature and position of the hand appearing in the photographed image by using a known object detection technique (which can also be called an object recognition technique). For example, the detection unit 32 may input the data of the photographed image to a function approximator (in other words, a mathematical model) of a convolutional neural network, and acquire feature information and position information of the hand appearing in the photographed image as the output of the function approximator. Alternatively, the detection unit 32 may acquire feature information and position information of the hand appearing in the photographed image by performing template matching on the photographed image based on a template image of the hand stored in the storage unit 18. For example, feature information of a template image with a high degree of match may be applied as the feature of the hand appearing in the photographed image.

[0026] The detection unit 32 associates the feature information and position information of one or more hands detected from the photographed image with different IDs and stores them in the position storage unit 22. The detection unit 32 also tracks the position of each hand appearing in the photographed image using a known technique. The detection unit 32 stores information indicating the time-series transition of the positions of one or more hands appearing in the photographed image in the position storage unit 22.

[0027] If the feature information of the hand detected by the detection unit 32 satisfies a predetermined condition related to a non-operator (hereinafter also referred to as a "non-operator condition"), the hairdresser determination unit 34 determines that the detected hand is the hand of a non-operator. In addition, if the feature information of the hand detected by the detection unit 32 does not satisfy the non-operator condition, the hairdresser determination unit 34 determines that the detected hand is the hand of an authorized operator.

[0028] The permitted operator in the first embodiment is a customer of the beauty salon. The non-operator in the first embodiment is a hairdresser who performs a treatment on the customer's hair. The device to be operated in the first embodiment is the smart mirror 10, which can also be called the information processing device 11 or the display unit 14.

[0029] In the first embodiment, the hairdresser determination unit 34 determines whether or not the owner of the hand in the captured image is a non-operator (i.e., a hairdresser) based on the movement of the hand. The hairdresser determination unit 34 includes a speed calculation unit 42. The speed calculation unit 42 derives the speed of the hand movement detected by the detection unit 32 based on the time-series transition of the hand position stored in the position storage unit 22. If the speed of the hand movement detected by the detection unit 32 is faster than a predetermined threshold, the hairdresser determination unit 34 determines that the detected hand is the hand of a non-operator (i.e., a hairdresser). An appropriate value for the speed threshold may be determined based on the developer's knowledge or an experiment using the smart mirror 10.

[0030] The hand registration unit 36 ​​associates the feature information of the hand determined by the hairdresser determination unit 34 as that of a non-operator (i.e., a hairdresser) with the non-operator in the hand storage unit 24. The hand registration unit 36 ​​also associates the feature information of the hand determined by the hairdresser determination unit 34 as that of an authorized operator in the hand storage unit 24.

[0031] When the hand detected by the detection unit 32 is determined to be the hand of an authorized operator, the gesture recognition unit 38 recognizes a gesture operation on the smart mirror 10 (information processing device 11) based on the movement of the detected hand. When the hand detected by the detection unit 32 is determined to be the hand of a non-operator, the gesture recognition unit 38 suppresses recognition of a gesture operation on the smart mirror 10 (information processing device 11) based on the movement of the detected hand.

[0032] The gesture recognition unit 38 recognizes a gesture operation input by the operator (operation authorized person) to the smart mirror 10 based on the time-series transition of the hand position in the captured image stored in the position storage unit 22. The gesture operation is an operation by the body movement of the operator (so-called gestures), and in the first embodiment, an operation by the hand movement. The gesture operation includes, for example, a swipe operation in which a finger is slid in any direction, a flick operation in which a finger is quickly moved, and a pinch operation in which two fingers are narrowed or widened.

[0033] The display control unit 40 controls the display of electronic content on the display unit 14. The electronic content may include a web page provided by an external web server, or information indicating the execution result of an application installed in the information processing device 11. For example, the display control unit 40 may obtain electronic content to be displayed on the display unit 14 from an external device (e.g., a web server, etc.) via the communication unit 20 in response to a gesture operation by an operator (authorized operator) recognized by the gesture recognition unit 38. The display control unit 40 may transmit the electronic content obtained from the external device to the display unit 14 to display it.

[0034] The operation of the information processing device 11 of the first embodiment having the above configuration will be described. 3 is a flowchart showing the operation of the information processing device 11 of the first embodiment. The camera 12 of the information processing device 11 outputs captured image data at a predetermined cycle. The image acquisition unit 30 of the information processing device 11 acquires the captured image data output from the camera 12 (S10). The detection unit 32 of the information processing device 11 detects one or more hands appearing in the captured image. The detection unit 32 acquires feature information and position information of each hand appearing in the captured image and stores the information in the position storage unit 22 (S11). The position storage unit 22 accumulates time-series data on the position of each hand appearing in the captured image.

[0035] The subsequent processes are executed for each hand detected from the captured image. The hairdresser determination unit 34 of the information processing device 11 determines whether or not the feature information of the hand appearing in the captured image acquired by the detection unit 32 has been registered in the hand storage unit 24. If the feature information of the hand appearing in the captured image has not been registered in the hand storage unit 24 (No in S12), the hairdresser determination unit 34 executes a hairdresser determination process (S13).

[0036] If the hand shown in the captured image is determined not to be the hand of a non-operator (here, the hairdresser) in the hairdresser determination process of S13 (No in S14), the hand registration unit 36 ​​of the information processing device 11 registers the feature information of the hand in the hand storage unit 24 in association with the authorized operator (here, the customer) (S15). The gesture recognition unit 38 of the information processing device 11 recognizes a gesture operation by the hand based on the time-series transition of the position of the hand stored in the position storage unit 22 (S16). The display control unit 40 of the information processing device 11 controls and updates the display content of the display unit 14 in accordance with the gesture operation recognized by the gesture recognition unit 38 (S17).

[0037] If the hand shown in the captured image is determined to be the hand of a non-operator (here, the hairdresser) in the hairdresser determination process of S13 (Yes in S14), the hand registration unit 36 ​​associates the feature information of the hand with the non-operator (here, the hairdresser) and registers it in the hand storage unit 24 (S18). In this case, the gesture recognition unit 38 suppresses the process of recognizing a gesture operation by the hand.

[0038] If the feature information of the hand shown in the captured image has already been registered in the hand storage unit 24 as feature information of the hand of a non-operator (No in S19), the hairdresser determination process is skipped and the process proceeds to the gesture recognition process in S16. If the feature information of the hand shown in the captured image has already been registered in the hand storage unit 24 as feature information of the hand of a non-operator (Yes in S19), the hairdresser determination process and the gesture recognition process are skipped and the process in this figure ends.

[0039] Fig. 4 is a flowchart showing details of the hairdresser determination process in S13 in Fig. 3. The hairdresser determination unit 34 selects one hand to be determined from one or more hands detected from the captured image. The hairdresser determination unit 34 (speed calculation unit 42) derives the speed of the movement of the hand based on the time-series transition of the position of the selected hand stored in the hand storage unit 24 (S20).

[0040] FIG. 5 shows an example of deriving the speed of hand movement. In the first embodiment, the area captured in the image 100 captured by the camera 12 is divided into a plurality of small areas (hereinafter referred to as "frames"). In FIG. 5, 16 frames included in the captured image 100 are shown, and the movement of the hand in a predetermined time (here, 1 second) is indicated by an arrow. The hairdresser determination unit 34 derives the speed of the hand movement based on how many frames the hand has moved in a predetermined time. In the example of FIG. 5, since the hand moves 5 frames (5, 6, 10, 14, and 15) per second, the speed of the hand movement may be derived as 5 frames / second. As a modified example, the hairdresser determination unit 34 (speed calculation unit 42) may derive the speed of the hand movement based on the transition of the hand position (coordinate information) in three-dimensional space.

[0041] Returning to Fig. 4, if the speed of the hand movement is faster than a predetermined threshold (Yes in S21), the hairdresser determination unit 34 determines that the hand detected from the captured image is the hand of a non-operator (i.e., a hairdresser) (S22). On the other hand, if the speed of the hand movement is equal to or lower than the threshold (No in S21), the hairdresser determination unit 34 determines that the hand detected from the captured image is not the hand of a non-operator. In other words, the hairdresser determination unit 34 determines that the hand detected from the captured image is the hand of an authorized operator (i.e., a customer) (S23).

[0042] According to the information processing device 11 of the first embodiment, in an environment where the operator of the smart mirror 10 changes daily, it is possible to determine whether the operator who inputs a gesture operation to the smart mirror 10 is an authorized operator who is permitted to operate, or a non-operator who is not permitted to operate. Particularly in a beauty salon, the hands of a hairdresser as a non-operator usually move faster than the hands of a customer as an authorized operator, so that a non-operator can be accurately identified. In addition, it is possible to prevent the hand movement of a person who does not intend to operate (for example, a hairdresser during a treatment) from being recognized as a gesture operation to the smart mirror 10, which causes the smart mirror 10 to perform an unintended operation.

[0043] The present disclosure has been described above based on the first embodiment. The first embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process of the first embodiment, and that such modifications are also within the scope of the present disclosure.

[0044] <Second embodiment> The second embodiment of the present disclosure will be described focusing on the differences from the first embodiment, and the description of the common points will be omitted as appropriate. The features of the second embodiment can be arbitrarily combined with the features of other embodiments or the features of the modified examples. The components of the second embodiment that are the same as or correspond to the components of the first embodiment will be described with the same reference numerals as appropriate.

[0045] The information processing device 11 built into the smart mirror 10 of the second embodiment recognizes a non-operator (hairdresser) of the smart mirror 10, similar to the information processing device 11 of the first embodiment. However, the information processing device 11 of the second embodiment identifies a non-operator (hairdresser) using biometric authentication technology.

[0046] The biometric information in the second embodiment is biometric information based on the appearance of the hand, and includes, for example, at least one of vein patterns, fingerprints, nail shapes, finger shapes, and palm lines. Hereinafter, the biometric information of the operator captured in the captured image is referred to as "first biometric information." The first biometric information can also be said to be biometric information detected from the image of the operator's hand included in the captured image. Moreover, the biometric information of a non-operator registered and stored in advance in the information processing device 11 is referred to as "second biometric information."

[0047] Fig. 6 is a block diagram showing functional blocks of an information processing device 11 according to the second embodiment. The information processing device 11 according to the second embodiment further includes a biometric information storage unit 26. The biometric information storage unit 26 stores biometric information of one or more non-operators (hairdressers). Fig. 7 shows an example of the biometric information stored in the biometric information storage unit 26. The biometric information stored in the biometric information storage unit 26 includes second biometric information for each hairdresser ID and for each angle information.

[0048] The hairdresser ID is an ID unique to each hairdresser. The angle information is information indicating the posture of the hand from which the corresponding second biometric information was detected. The angle information is information indicating the angle of the hand from a predetermined reference posture, and may include, for example, values ​​of three-dimensional angles (i.e., roll angle, pitch angle, yaw angle) from the predetermined reference posture. The second biometric information is biometric information recognized from the hand in the posture indicated by the angle information.

[0049] Returning to Figure 6, the hairdresser determination unit 34 of the information processing device 11 of the second embodiment determines that the detected hand is the hand of a non-operator if the biometric information related to the hand detected from the captured image matches the biometric information of a non-operator stored in the biometric information storage unit 26.

[0050] Specifically, the hairdresser determination unit 34 of the second embodiment includes an angle recognition unit 44 and a biometric information recognition unit 46. The angle recognition unit 44 recognizes angle information (e.g., a combination of pitch angle, roll angle, and yaw angle) related to the hand captured in the captured image using a known method.

[0051] The biometric information recognition unit 46 recognizes first biometric information (such as vein patterns as described above) related to the hand shown in the captured image. The biometric information recognition unit 46 may estimate the value of the first biometric information related to the hand shown in the captured image by analyzing the captured image using a known method. The biometric information recognition unit 46 stores the value of the first biometric information in the biometric information storage unit 26 in association with the hairdresser ID and the angle information recognized by the angle recognition unit 44.

[0052] Furthermore, when the biometric information recognition unit 46 detects that a hand has stopped moving based on a plurality of captured images, it detects the first biometric information related to the hand. For example, the biometric information recognition unit 46 may determine whether or not a hand has stopped moving based on a transition in the position of a hand detected by the detection unit 32 based on a plurality of captured images stored in the position storage unit 22. The biometric information recognition unit 46 may detect the first biometric information related to the hand on the condition that the hand has stopped moving.

[0053] The operation of the information processing device 11 of the second embodiment configured as above will be described. The operation of the information processing device 11 of the second embodiment is the same as the operation of the information processing device 11 of the first embodiment shown in Fig. 3, except for the hairdresser determination process. The hairdresser determination process of the second embodiment will be described in detail below.

[0054] As a preliminary preparation, the biometric information storage unit 26 of the information processing device 11 stores biometric information (second biometric information) of each hairdresser extracted from images of the hands of each hairdresser taken at various angles in association with the hairdresser ID and angle information.

[0055] FIG. 8 is a flowchart showing the operation of the information processing device 11 of the second embodiment. FIG. 8 corresponds to FIG. 4 and shows details of the hairdresser determination process of S13 in FIG. 3. The hairdresser determination unit 34 selects one hand to be determined from one or more hands detected from the captured image (S30). The hairdresser determination unit 34 determines whether the hand to be determined is stationary or not. The hairdresser determination unit 34 may determine that a hand whose movement speed is equal to or less than a predetermined threshold is stationary by deriving the speed of the hand movement as in the first embodiment. In addition, the hairdresser determination unit 34 may determine that a hand is stationary when it detects that the direction of hand movement has been reversed between captured images.

[0056] If it is determined that the hand to be judged is not stationary (No in S31), the process returns to S11 in Fig. 3. Alternatively, the hand to be judged may be switched to another hand.

[0057] If it is determined that the hand of the subject is stationary (Yes in S31), the hairdresser determination unit 34 acquires angle information and first biometric information of the hand of the subject from the captured image (S32).The hairdresser determination unit 34 determines whether or not there is a combination of angle information and second biometric information stored in the biometric information storage unit 26 that matches the combination of angle information and first biometric information acquired in S32 (S33).

[0058] If a match is found (Yes in S34), the hairdresser determination unit 34 determines that the hand being determined is the hand of a non-operator (i.e., a hairdresser) (S35). If a match is not found (No in S34), the hairdresser determination unit 34 determines that the hand being determined is not the hand of a non-operator, in other words, that the hand being determined is the hand of an authorized operator (i.e., a customer) (S36). In the second embodiment, by using the angle information, the search range for the second biometric information can be narrowed and the search efficiency can be improved.

[0059] As a modified example, the biometric information stored in the biometric information storage unit 26 may include the hairdresser ID and the second biometric information, but may not include the angle information. In this case, the hairdresser determination unit 34 may search for the second biometric information that matches the first biometric information acquired in S32 from among all the second biometric information stored in the biometric information storage unit 26.

[0060] The information processing device 11 of the second embodiment also achieves the same effects as the information processing device 11 of the first embodiment. Moreover, the information processing device 11 of the second embodiment can accurately determine whether or not the photographed hand is the hand of a non-operator. Moreover, by detecting biometric information (first biometric information) when the hand is stationary, it becomes easier to obtain biometric information (first biometric information) with less error regarding the photographed hand.

[0061] The present disclosure has been described above based on the second embodiment. The second embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process of the second embodiment, and that such modifications are also within the scope of the present disclosure.

[0062] <Third embodiment> The third embodiment of the present disclosure will be described focusing on the differences from the first embodiment, and the description of the common points will be omitted as appropriate. The features of the third embodiment can be arbitrarily combined with the features of the other embodiments or the features of the modified examples. The components of the third embodiment that are the same as or correspond to the components of the first embodiment will be described with the same reference numerals as appropriate.

[0063] The information processing device 11 of the third embodiment recognizes a non-operator (hairdresser) of the smart mirror 10, similarly to the information processing device 11 of the first embodiment. However, in the third embodiment, it is determined whether or not a hand shown in a captured image is the hand of a non-operator (a hairdresser in the third embodiment) based on whether or not the hand shown in the captured image is holding a predetermined tool. The predetermined tool is a tool whose characteristic information is registered in advance in the information processing device 11, and in the third embodiment, it is a tool used by a hairdresser for treatment (hereinafter also referred to as "hairdresser's tools"). The hairdresser's tools may include scissors, combs, hair clips, rods, hair dryers, etc.

[0064] The functional blocks of the information processing device 11 of the third embodiment are similar to the functional blocks of the information processing device 11 of the first embodiment shown in Fig. 2. However, the hairdresser determination unit does not include a speed calculation unit .

[0065] The detection unit 32 of the information processing device 11 further detects the positions and types of tools that appear in the captured image and are used by a non-operator (beautician's tools used by a hairdresser in the third embodiment) using a known object detection technique, and stores the positions and types in the position storage unit 22. For example, the detection unit 32 may detect the positions and types of the hairdresser's tools appearing in the captured image by inputting the data of the captured image into a function approximator (in other words, a mathematical model) of a convolutional neural network. Alternatively, the detection unit 32 may acquire position information of the hairdresser's tools appearing in the captured image by performing template matching based on template images of the hairdresser's tools stored in the storage unit 18.

[0066] The hairdresser determination unit 34 of the information processing device 11 determines whether or not the hairdresser's tools are present in the vicinity of the hand detected from the captured image, based on the distance between the hand position stored in the position storage unit 22 and the position of the hairdresser's tools. The range of the vicinity may be set to an appropriate value based on the developer's knowledge or an experiment using the smart mirror 10. Alternatively, the hairdresser determination unit 34 may determine that the hairdresser's tools are present in the vicinity of the hand detected from the captured image, when the outer edge of the hand area and the outer edge of the tool area are in contact with each other on the captured image.

[0067] FIG. 9 is a flowchart showing the operation of the information processing device 11 of the third embodiment. FIG. 9 corresponds to FIG. 4 and shows details of the hairdresser determination process of S13 in FIG. 3. The detection unit 32 detects the position and type of hairdresser tools shown in the captured image and records them in the position storage unit 22 (S40). The hairdresser determination unit 34 selects one hand to be determined from one or more hands detected from the captured image (S41). If hairdresser tools are present near the hand to be determined (Yes in S42), the hairdresser determination unit 34 determines that the hand to be determined is the hand of a non-operator (i.e., a hairdresser) (S43). If hairdresser tools are not present near the hand to be determined (No in S42), the hairdresser determination unit 34 determines that the hand to be determined is not the hand of a non-operator (S44).

[0068] The information processing device 11 of the third embodiment also achieves the same effects as the information processing device 11 of the first embodiment. Moreover, according to the information processing device 11 of the third embodiment, it is possible to accurately determine whether or not a photographed hand is the hand of a non-operator, depending on the characteristic of the non-operator (e.g., a hairdresser) who uses a tool.

[0069] The present disclosure has been described above based on the third embodiment. The third embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process of the third embodiment, and that such modifications are also within the scope of the present disclosure.

[0070] A modified example of the third embodiment will be described. The hairdresser determination unit 34 of the information processing device 11 may derive the number of movements of the hairdresser's tools present in the vicinity of the hand detected from the captured image (for example, the number of times the same movement is repeated within a predetermined period) using a known image analysis method. The hairdresser determination unit 34 may determine that the hand detected from the captured image is the hand of a non-operator (hairdresser) under the additional condition that the number of movements of the hairdresser's tools is greater than a predetermined threshold (for example, once per second). In other words, the hairdresser determination unit 34 may determine that the hand detected from the captured image is the hand of a non-operator (hairdresser) when a hairdresser's tool is present in the vicinity of the hand detected from the captured image and the number of movements of the hairdresser's tool is greater than a predetermined threshold. An appropriate value for the threshold for the number of movements may be determined based on the developer's knowledge or an experiment using the smart mirror 10.

[0071] The number of times of motion of the hairdresser's tool may be the interval at which the same motion is repeated, and if the interval is shorter than a predetermined threshold, the hand detected from the captured image may be determined to be the hand of a non-operator (hairdresser). The number of times of motion of the hairdresser's tool may be the number of times the same motion is repeated within a predetermined time, and if the number of times is greater than a predetermined threshold, the hand detected from the captured image may be determined to be the hand of a non-operator (hairdresser). For example, if the number of times the same motion is repeated in one minute is 6 or more or 7 or more, the hand detected from the captured image may be determined to be the hand of a hairdresser. The number of times of motion of the hairdresser's tool may be the speed of motion when the same motion is repeated, and if the speed of motion is faster than a predetermined threshold, the hand detected from the captured image may be determined to be the hand of a non-operator (hairdresser).

[0072] According to this modified example, it is possible to more accurately determine whether the photographed hand is that of a non-operator, depending on the typical manner in which a tool is used by a non-operator (for example, the way a hairdresser moves scissors).

[0073] <Fourth embodiment> The fourth embodiment of the present disclosure will be described focusing on the differences from the first embodiment, and the description of the common points will be omitted as appropriate. The features of the fourth embodiment can be arbitrarily combined with the features of the other embodiments or the features of the modified examples. The components of the fourth embodiment that are the same as or correspond to the components of the first embodiment will be described with the same reference numerals as appropriate.

[0074] The information processing device 11 built into the smart mirror 10 of the fourth embodiment recognizes a non-operator (hairdresser) of the smart mirror 10, similar to the information processing device 11 of the first embodiment. In the fourth embodiment, the non-operator (hairdresser) wears gloves to protect his / her hands. In the fourth embodiment, it is determined whether or not the hand shown in the captured image is the hand of a non-operator depending on whether or not a predetermined condition related to the hand shown in the captured image wearing gloves is satisfied. In other words, in the fourth embodiment, it is determined that the hand shown in the captured image is the hand of a non-operator when a predetermined condition that is satisfied when the hand shown in the captured image wears gloves is satisfied. Specifically, in the fourth embodiment, it is determined whether or not a color condition related to the hand detected from the captured image is satisfied.

[0075] The functional blocks of the information processing device 11 of the fourth embodiment are similar to the functional blocks of the information processing device 11 of the first embodiment shown in Fig. 2. However, the hairdresser determination unit does not include a speed calculation unit .

[0076] The detection unit 32 of the information processing device 11 detects the position of the hand shown in the captured image, and also detects the color of the hand using a known color recognition technique. The detection unit 32 stores the position information and feature information (including information on the color of the hand) of the hand shown in the captured image in the position memory unit 22. The hairdresser determination unit 34 of the information processing device 11 determines that the hand detected in the captured image is the hand of a non-operator when the color of the hand detected in the captured image matches a color predetermined as the color of gloves used by a non-operator (hairdresser).

[0077] Fig. 10 is a flowchart showing the operation of the information processing device 11 of the fourth embodiment. In Fig. 10, the hairdresser determination unit 34 selects one hand to be determined from one or more hands detected from a captured image (S50). The hairdresser determination unit 34 obtains information about the color of the hand to be determined stored in the position storage unit 22, and determines whether the color of the hand to be determined matches a predetermined glove color (e.g., green).

[0078] If the color of the hand to be judged matches the color of the gloves (Yes in S51), the hairdresser judgment unit 34 judges that the hand to be judged is the hand of a non-operator (i.e., a hairdresser) (S52). If the color of the hand to be judged does not match the color of the gloves (No in S51), the hairdresser judgment unit 34 judges that the hand to be judged is not the hand of a non-operator (S53).

[0079] The information processing device 11 of the fourth embodiment also achieves the same effects as the information processing device 11 of the first embodiment. Moreover, according to the information processing device 11 of the fourth embodiment, it is possible to accurately determine whether or not a photographed hand is the hand of a non-operator, depending on the characteristic of the non-operator (e.g., a hairdresser) that he or she wears gloves.

[0080] The present disclosure has been described above based on the fourth embodiment. The fourth embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process of the fourth embodiment, and that such modifications are also within the scope of the present disclosure.

[0081] A modified example of the fourth embodiment will be described. The predetermined condition regarding whether the hand shown in the captured image is wearing a glove may include at least one of the following: nails are hidden, fingerprints are hidden, veins are hidden, and the hand has a vinyl texture. The detection unit 32 of the information processing device 11 may detect at least one of the following for the hand shown in the captured image: nails are hidden, fingerprints are hidden, veins are hidden, and the hand has a vinyl texture.

[0082] The hairdresser determination unit 34 of the information processing device 11 may determine that the hand to be determined is the hand of a non-operator if the above conditions are met. Moreover, the hairdresser determination unit 34 may determine that the hand to be determined is not the hand of a non-operator if the above conditions are not met. Even in this modified example, it is possible to accurately determine whether the photographed hand is the hand of a non-operator according to the characteristic of the non-operator (e.g., a hairdresser) that he or she wears gloves.

[0083] A modified example applicable to any of the first to fourth embodiments will be described. In the first to fourth embodiments, the hairdresser is a non-operator who is not permitted to operate the smart mirror 10. As a modified example, an operating member (herein called a "hairdresser operation button") for instructing the hairdresser to operate the smart mirror 10 may be provided on the smart mirror 10 or a remote control device (not shown).

[0084] When the hairdresser operation button is pressed, that is, when a signal instructing the hairdresser to operate the smart mirror 10 is input, the gesture recognition unit 38 of the information processing device 11 may exceptionally accept and recognize the gesture operation regardless of whether the person who inputs the gesture operation is a hairdresser or a customer. For example, in the flowchart of FIG. 3, even if the detected hand has been registered as the hand of a non-operator (Yes in S19) and even if it is determined to be the hand of a hairdresser by the hairdresser determination process (Yes in S14), the gesture recognition process of S16 may be executed. Alternatively, the hairdresser determination process of S13 may be skipped and the gesture recognition process of S16 may be executed.

[0085] According to this modified example, it is possible to prevent the smart mirror 10 from recognizing operations that the hairdresser does not intend during treatment, while enabling the hairdresser to operate the smart mirror 10 if he or she wishes to do so.

[0086] Although not mentioned in the first to fourth embodiments, the hairdresser determination unit 34 may add points to each of the multiple hands detected by the detection unit 32 according to the tracking time. In other words, the longer the tracking time, the higher the score may be given to the hand. The hairdresser determination unit 34 may determine that a hand with a score equal to or greater than a predetermined threshold is the hand of the hairdresser (i.e., a non-operator), and may determine that a hand with a score less than the threshold is the hand of a customer (i.e., an authorized operator). This is because, during a hair treatment, the hairdresser's hands are often in front of the camera 12, while the customer's hands are often hidden from the camera 12 by a cutting cape, cutting cloth, magazine, or the like.

[0087] <Fifth embodiment> The fifth embodiment of the present disclosure will be described focusing on the differences from the first embodiment, and the description of the common points will be omitted as appropriate. The features of the fifth embodiment can be arbitrarily combined with the features of the other embodiments or the features of the modified examples. The components of the fifth embodiment that are the same as or correspond to the components of the first embodiment will be described with the same reference numerals as appropriate.

[0088] As in the first embodiment, the fifth embodiment also proposes a technology for accurately determining the operator of a device in an environment where the operator of the device changes daily. In a beauty salon, it is the customer who operates the smart mirror 10 while the hairdresser performs a treatment on the customer's hair. Therefore, during the treatment, it is desired to give priority to accepting the gesture operation of the customer. However, during times other than the treatment (for example, during counseling before the treatment or after the treatment), the hairdresser may also operate the smart mirror 10, so it is necessary to adjust the priority.

[0089] In the fifth embodiment, a technique is proposed for determining a scene in a captured image and appropriately switching between a non-operator and an authorized operator. Specifically, the information processing device 11 of the fifth embodiment stores a scene in which an authorized operator should be switched, and appropriately switches between the authorized operator by determining a current scene.

[0090] 11 is a block diagram showing functional blocks of an information processing device 11 according to a fifth embodiment. The information processing device 11 according to the fifth embodiment includes the functional blocks included in the information processing device 11 according to the first embodiment, and further includes a scene storage unit 28, a scene determination unit 50, and a switching unit 52.

[0091] The scene storage unit 28 stores information (scene information) relating to a number of predetermined scenes. FIG. 12 shows an example of scene information. The scene information includes a scene ID, a scene name, conditions, and whether or not the hairdresser can operate the smart mirror 10. Scene 0 is a scene where there is no one in front of the smart mirror 10, and the hairdresser is not allowed to operate the smart mirror 10. Scene 1 is a scene of counseling before a treatment or a scene after the treatment is completed (such as a scene where the results of the treatment are confirmed with the customer), and the hairdresser is allowed to operate the smart mirror 10. Scene 2 is a scene where the hairdresser is performing a treatment, and the hairdresser is not allowed to operate the smart mirror 10.

[0092] The "conditions" of the scene information are data used to identify the scene. P1 indicates that a person is captured in the captured image, and !P1 indicates that a person is not captured in the captured image. P2 indicates that a hand holding a tool is captured in the captured image, and !P2 indicates that a hand holding a tool is not captured in the captured image. Therefore, when a person is not captured in the captured image (!P1), it becomes scene 0. Also, when a person is captured in the captured image and a hand holding a tool is not captured in the captured image (P1&&!P2), it becomes scene 1. Also, when a person is captured in the captured image and a hand holding a tool is captured in the captured image (P1&&P2), it becomes scene 2. Note that the tools in condition P2 are hairdresser's tools.

[0093] 11, the detection unit 32, as an object recognition unit, recognizes one or more objects appearing in the captured image acquired by the image acquisition unit 30. The detection unit 32 executes a process of recognizing whether or not a person appears in the captured image, and a process of recognizing whether or not a hand holding a predetermined tool (hairdresser's tool) appears in the captured image.

[0094] The detection unit 32 may detect the position of a person, a hand, and a hairdresser's tools in the captured image using a known object detection technique. The detection unit 32 may determine that a hand holding a hairdresser's tool is captured in the captured image when the position of the hand and the position of the hairdresser's tool are within a predetermined vicinity range. The vicinity range may be determined to be an appropriate range based on the developer's knowledge or an experiment using the smart mirror 10. For example, the detection unit 32 may determine that a hand holding a hairdresser's tool is captured in the captured image when the outer edge of the hand area and the outer edge of the hairdresser's tool area are in contact with each other in the captured image. As described above, a convolutional neural network or template matching may be used to detect objects from the captured image.

[0095] The scene determination unit 50 refers to the scene storage unit 28 and identifies a scene corresponding to one or more objects recognized by the detection unit 32 from among a plurality of predetermined scenes. In other words, the scene determination unit 50 identifies a scene according to an object recognition situation by the detection unit 32. The switching unit 52 switches at least one of the users who can operate a specific device (smart mirror 10) and the operations that can be input to the device, based on the scene identified by the scene determination unit 50.

[0096] When no person is captured in the captured image, the scene determination unit 50 determines that the scene is an empty seat (scene 0). When the scene is determined to be an empty seat (scene 0), the switching unit 52 restricts the operations that can be input to the smart mirror 10. For example, in the empty seat scene (scene 0), the switching unit 52 controls to suppress updates to the display content based on operations by a non-operator (e.g., a hairdresser).

[0097] If a person is captured in the captured image and a hand holding a predetermined tool (e.g., a hairdresser's tool) is not captured in the captured image, the scene determination unit 50 determines that the scene is after counseling or treatment (scene 1). If the scene is determined to be after counseling or treatment (scene 1), the switching unit 52 includes the hairdresser in the users who can operate the smart mirror. In the fifth embodiment, the switching unit 52 allows both the customer and the hairdresser to operate the smart mirror 10 in the scene after counseling or treatment (scene 1).

[0098] When a person is captured in the captured image and a hand holding a predetermined tool (e.g., a hairdresser's tool) is captured in the captured image, the scene determination unit 50 determines that the captured image is a scene in which a treatment is in progress (scene 2). When the captured image is determined to be a scene in which a treatment is in progress (scene 2), the switching unit 52 excludes hairdressers from users who can operate the smart mirror 10.

[0099] As a hairdresser determination process, the hairdresser determination unit 34 determines whether or not the hand shown in the captured image is the hand of an authorized operator (e.g., a customer) who is authorized to operate the smart mirror 10, based on whether or not the feature information of the hand shown in the captured image satisfies a predetermined condition related to a non-operator (e.g., a hairdresser) who is not authorized to operate the smart mirror 10. The hairdresser determination process described in any one of the first to fourth embodiments may be applied to this hairdresser determination process.

[0100] 13 is a flowchart showing the operation of the information processing device 11 of the fifth embodiment. The camera 12 of the information processing device 11 outputs captured image data at a predetermined cycle. The image acquisition unit 30 of the information processing device 11 acquires the captured image data output from the camera 12 (S60). The information processing device 11 executes a scene determination process based on the acquired captured image (S61).

[0101] Fig. 14 is a flowchart showing details of the scene determination process of S61 in Fig. 13. The detection unit 32 of the information processing device 11 recognizes whether or not a person appears in the captured image acquired in S60, and also recognizes whether or not a hand holding a hairdresser's tool appears in the captured image (S80).

[0102] The scene determination unit 50 of the information processing device 11 derives condition information that matches the object recognition result in S80 (S81). FIG. 15 shows an example of the condition information. Based on the object recognition result by the detection unit 32, the scene determination unit 50 derives condition information indicating a combination of whether a person is captured in the captured image (P1) or not (!P1) and whether a hand holding a tool is captured in the captured image (P2) or not (!P2). Returning to FIG. 14, the scene determination unit 50 refers to the scene information in the scene storage unit 28 to identify a scene that matches the condition information (S82). In the fifth embodiment, the scene is identified as one of scene 0 "empty seat", scene 1 "after counseling or treatment", and scene 2 "during treatment" shown in FIG. 12.

[0103] 13, if the scene is determined to be scene 0 in the scene determination process of S61 (Yes in S62), the process returns to S60 and a new captured image is acquired. If the scene is determined to be scene 1 or scene 2 in the scene determination process of S61 (No in S62), the detection unit 32 acquires feature information and position information of each hand appearing in the captured image and stores them in the position storage unit 22 (S63).

[0104] If it is determined to be scene 2 in the scene determination process of S61 (No in S64), the hairdresser determination unit 34 executes a hairdresser determination process corresponding to S13 in the first to fourth embodiments (S65). The hairdresser determination unit 34 may apply the techniques described in the first to fourth embodiments or their modified examples to determine whether or not a hand shown in a captured image is a hairdresser's hand.

[0105] If the hairdresser determination unit 34 determines that the hand shown in the captured image is not a hairdresser's hand (No in S66), the hand registration unit 36 ​​associates the feature information of the hand with the authorized operator (customer) and registers it in the hand storage unit 24 (S67). The gesture recognition unit 38 of the information processing device 11 recognizes a gesture operation by the hand based on the time-series transition of the position of the hand stored in the position storage unit 22 (S68). The display control unit 40 of the information processing device 11 controls the display content of the display unit 14 in accordance with the gesture operation recognized by the gesture recognition unit 38 (S69).

[0106] If the hairdresser determination unit 34 determines that the hand in the captured image is the hand of a hairdresser (Yes in S66), the hand registration unit 36 ​​associates the feature information of the hand with the non-operator (hairdresser) and registers it in the hand storage unit 24 (S70). In this case, the gesture recognition unit 38 suppresses the process of recognizing a gesture operation by the hand.

[0107] If it is determined in the scene determination process of S61 that it is scene 1 (Yes in S64), the processes of S65 to S67 are skipped and the process proceeds to the gesture recognition process of S68. In the information processing device 11 of the fifth embodiment, in scene 1 “after counseling or treatment”, the information processing device 11 does not distinguish between a hairdresser and a customer and accepts either gesture operation.

[0108] Although not shown in Fig. 13, as described in Fig. 3 (S12, S19) of the first embodiment, after N in S64 and before the hairdresser determination process in S65, it may be determined whether the hand to be determined has already been registered as the hand of an authorized operator or whether it has already been registered as the hand of a non-operator. If it has already been registered as the hand of an authorized operator, the hairdresser determination process may be skipped and the process may proceed to gesture recognition process. On the other hand, if it has already been registered as the hand of a non-operator, the process related to the hand to be determined may be terminated.

[0109] According to the information processing device 11 of the fifth embodiment, the user who can operate the smart mirror 10 or the operations that can be input to the smart mirror 10 can be appropriately switched based on which of multiple scenes related to hair treatments in a beauty salon the scene captured by the camera 12 is.

[0110] The present disclosure has been described above based on the fifth embodiment. The fifth embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process of the fifth embodiment, and that such modifications are also within the scope of the present disclosure.

[0111] Sixth embodiment The sixth embodiment of the present disclosure will be described focusing on the differences from the fifth embodiment, and the description of the common points will be omitted as appropriate. The features of the sixth embodiment can be arbitrarily combined with the features of the other embodiments or the features of the modified examples. The components of the sixth embodiment that are the same as or correspond to the components of the fifth embodiment will be described with the same reference numerals as appropriate.

[0112] As described above, the information processing device 11 of the fifth embodiment does not distinguish between the hairdresser and the customer in scene 1 "After counseling or treatment is completed," and accepts gesture operations from either the hairdresser or the customer. On the other hand, in scene 1, it is considered that the hairdresser may want to display various information on the display unit 14 while talking with the customer. Therefore, the information processing device 11 of the sixth embodiment gives operation priority to the hairdresser in scene 1, in other words, treats the hairdresser's hands as the hands of a priority operator.

[0113] The functional blocks of the information processing device 11 of the sixth embodiment are similar to the functional blocks of the information processing device 11 of the fifth embodiment shown in Fig. 11. When the scene determination unit 50 determines that it is scene 1 "after counseling or treatment", the gesture recognition unit 38 of the information processing device 11 gives priority to handling operations inputted to the smart mirror 10 by the hairdresser over operations inputted to the smart mirror 10 by a customer (in other words, a person other than the hairdresser).

[0114] The operation of the information processing device 11 of the sixth embodiment is the same as the operation of the information processing device 11 of the fifth embodiment shown in Fig. 13. However, if the scene determination unit 50 determines that it is scene 1 "After counseling or treatment", in S68, the gesture recognition unit 38 recognizes the gesture operation by the hairdresser's hand with priority over the gesture operation by the customer's hand. For example, in S68, when a gesture operation by the customer's hand and a gesture operation by the hairdresser's hand are input simultaneously, the gesture recognition unit 38 may ignore the gesture operation by the customer's hand and accept only the gesture operation by the hairdresser's hand.

[0115] The information processing device 11 of the sixth embodiment has the same effects as the information processing device 11 of the fifth embodiment. Moreover, the information processing device 11 of the sixth embodiment can more appropriately set the operating subject of the smart mirror 10 in a scene during counseling before a treatment or a scene after the treatment is completed.

[0116] The present disclosure has been described above based on the sixth embodiment. The sixth embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component or each treatment process of the sixth embodiment, and that such modifications are also within the scope of the present disclosure.

[0117] Modifications of the first to sixth embodiments will be described. The hairdresser determination unit 34 of the information processing device 11 may determine whether the hand to be determined is a hairdresser's hand, using the additional condition that the hand to be determined appears in a captured image for a predetermined time or more in addition to the hairdresser determination conditions described in the first to fourth embodiments. In other words, the hairdresser determination unit 34 may determine that the hand to be determined is a hairdresser's hand when the hairdresser determination conditions described in the first to fourth embodiments are met and the additional condition that the hand to be determined appears in a predetermined number of temporally consecutive captured images is met.

[0118] In addition, the hairdresser determination unit 34 may determine that the hand to be determined is the hand of an assistant when the hairdresser determination conditions described in the first to fourth embodiments are satisfied but the hand to be determined is not shown in the captured image for a predetermined time or more. In addition, the assistant's hand may not be given operation authority. For example, when the scene determination unit 50 determines that the scene is scene 1 "After counseling or treatment", the gesture recognition unit 38 may recognize the gesture operation by the hand determined to be the hairdresser's hand, but may ignore the gesture operation by the hand determined to be the assistant's hand. The time threshold value in the above determination may be set to an appropriate value based on the developer's knowledge or experiments using the smart mirror 10.

[0119] Another modified example of the first to sixth embodiments will be described. In each embodiment, the device to be operated is the smart mirror 10, but the device to be operated is not limited to the smart mirror 10. For example, even when multiple people are in front of a tablet terminal equipped with a camera and a display unit, the technology for recognizing a non-operator described in the first to fourth embodiments can be applied, and the technology for determining a scene and switching operators described in the fifth and sixth embodiments can also be applied.

[0120] Any combination of the above-mentioned embodiments and modifications is also useful as an embodiment of the present disclosure. A new embodiment resulting from the combination has the effects of each of the combined embodiments and modifications. In addition, it will be understood by those skilled in the art that the functions to be performed by each component described in the claims can be realized by each component shown in the embodiments and modifications alone or in cooperation with each other.

[0121] <Additional Notes> The above description of the embodiment and modifications discloses the following techniques. [Technology 1-1] Acquire an image captured by a camera, Detect hands in the captured images, If the feature information of the detected hand satisfies a predetermined condition related to a non-operator who is not permitted to operate a predetermined device, the detected hand is determined to be the hand of the non-operator; when the detected hand is determined to be the hand of the non-operator, suppressing recognition of a gesture operation on the device based on the movement of the detected hand. An information processing method implemented by a computer. According to this information processing method, it is possible to determine whether a person who inputs a gesture operation to a device is a non-operator who is not permitted to operate the device, and it is possible to accurately determine the operator of the device in an environment where the operator of the device changes from day to day. Also, it is possible to prevent the hand movement of a person who does not intend to operate the device from being recognized as a gesture operation to the device, and to prevent the person from causing the device to perform an unintended operation. [Technology 1-2] If the feature information of the detected hand does not satisfy the condition, it is determined that the detected hand is a hand of an authorized user who is authorized to operate the device; when it is determined that the detected hand is the hand of the permitted operation person, recognizing a gesture operation based on the movement of the detected hand; The information processing method described in Technology 1-1. This information processing method can accurately determine the operator of the device in an environment where the operator changes daily. Also, it can restrict the operation of the device by non-operators (e.g., hairdressers at a beauty salon), while permitting operation of the device by authorized operators (e.g., customers at the beauty salon). [Technology 1-3] The process of determining includes determining that the detected hand is the hand of the non-operator when a speed of the detected hand movement is faster than a predetermined threshold. The information processing method according to Technology 1-1 or 1-2. According to this information processing method, it is possible to accurately determine whether or not a photographed hand is that of a non-operator, depending on the characteristic of the non-operator (for example, a hairdresser at a beauty salon) that the hand moves quickly. [Technology 1-4] The process of determining includes determining that the detected hand is the hand of the non-operator when the biometric information on the detected hand matches the biometric information stored in a storage unit. The information processing method according to Technology 1-1 or 1-2. According to this information processing method, it is possible to accurately determine whether or not a photographed hand is the hand of a non-operator. [Technology 1-5] When it is detected that the hand is stationary based on the plurality of images, the computer further detects biometric information related to the hand. The information processing method described in Technology 1-4. According to this information processing method, it becomes easier to obtain biometric information with fewer errors regarding the photographed hand, and it becomes possible to more accurately determine whether the photographed hand is the hand of a non-operator. [Technology 1-6] The detecting process further detects a tool used by the non-operator that appears in the acquired image; The process of determining includes determining that the detected hand is the hand of the non-operator when the tool is present in the vicinity of the detected hand. The information processing method according to Technology 1-1 or 1-2. According to this information processing method, it is possible to accurately determine whether or not the photographed hand is the hand of a non-operator, depending on the characteristic of the non-operator (eg, a hairdresser) who uses a tool. [Technology 1-7] The process of determining determines that the detected hand is the hand of the non-operator on the additional condition that the number of movements of an object present in the vicinity of the detected hand is greater than a predetermined threshold. The information processing method described in Technology 1-6. According to this information processing method, it is possible to more accurately determine whether or not a photographed hand is the hand of a non-operator, depending on the manner in which the non-operator uses a tool (for example, a hairdresser using scissors). [Technology 1-8] The process of determining includes determining that the detected hand is the hand of the non-operator when a color of the detected hand matches a color of a glove used by the non-operator. The information processing method according to Technology 1-1 or 1-2. According to this information processing method, it is possible to accurately determine whether or not a photographed hand is the hand of a non-operator, depending on the characteristic of the non-operator (eg, a hairdresser at a beauty salon) that he or she wears gloves. [Technology 1-9] an image acquisition unit that acquires an image captured by a camera; a detection unit that detects a hand appearing in an image acquired by the image acquisition unit; a determination unit that determines that the detected hand is a hand of a non-operator when feature information of the hand detected by the detection unit satisfies a predetermined condition related to a non-operator who is not permitted to operate a predetermined device; and an operation recognition unit that, when it is determined that the detected hand is the hand of the non-operator, suppresses recognition of a gesture operation on the device based on the movement of the detected hand; An information processing device comprising: According to this information processing device, it is possible to determine whether a person who inputs a gesture operation to a device is a non-operator who is not permitted to operate the device, and to accurately determine the operator of the device in an environment where the operator of the device changes daily. Also, it is possible to prevent the hand movement of a person who does not intend to operate the device from being recognized as a gesture operation to the device, thereby preventing the person from causing the device to perform an unintended operation. [Technology 1-10] Acquire an image captured by a camera, Detect hands in the captured images, If the feature information of the detected hand satisfies a predetermined condition related to a non-operator who is not permitted to operate a predetermined device, the detected hand is determined to be the hand of the non-operator; when the detected hand is determined to be the hand of the non-operator, suppressing recognition of a gesture operation on the device based on the movement of the detected hand. A computer program that causes a computer to carry out a task. According to this computer program, it is possible to determine whether a person who inputs a gesture operation to a device is a non-operator who is not permitted to operate the device, and it is possible to accurately determine the operator of the device in an environment where the operator of the device changes from day to day. In addition, it is possible to prevent the hand movement of a person who does not intend to operate the device from being recognized as a gesture operation to the device, thereby preventing the person from causing the device to perform an unintended operation.

[0122] [Technology 2-1] Acquire an image captured by a camera, Recognizing one or more objects in the captured image; Identifying a scene from among a plurality of predetermined scenes that corresponds to the recognized object or objects; switching at least one of a user who can operate a predetermined device and an operation that can be input to the device based on the identified scene; A control method in which a computer executes the process. According to this control method, it is possible to appropriately switch at least one of the users who can operate the device and the operations that can be input to the device based on the scene captured by the camera. [Technology 2-2] The recognition process includes a process of recognizing whether or not a person is captured in the image, and a process of recognizing whether or not a hand holding a predetermined tool is captured in the image. The control method described in Technology 2-1. According to this control method, it becomes easier to more appropriately specify the scene, and it becomes easier to more appropriately switch at least one of the users who can operate the device and the operations that can be input to the device. [Technology 2-3] The device is a smart mirror. The control method according to Technique 2-1 or 2-2. According to this control method, it is possible to appropriately switch at least one of the users who can operate the smart mirror and the operations that can be input to the smart mirror based on the scene captured by the camera. [Technology 2-4] The identifying process identifies the scene as an empty seat scene when the recognizing process recognizes that no person is captured in the image, The switching process limits operations that can be input to the smart mirror when the identifying process identifies the scene as an empty seat. The control method described in Technology 2-3. According to this control method, if the seat near the smart mirror is vacant, the operations that can be input to the smart mirror can be appropriately restricted. [Technology 2-5] The identifying process identifies the image as a scene after counseling or treatment has ended when the recognizing process recognizes that the image contains a person and that the image does not contain a hand holding a predetermined tool, When the identifying process identifies the scene as a scene after the counseling or treatment, the switching process includes a hairdresser as a user who can operate the smart mirror. The control method according to any one of the preceding claims. According to this control method, in the case of a scene after counseling or treatment, users who are allowed to operate the device can be appropriately set. [Technology 2-6] When the scene is identified as a scene after the counseling or treatment is completed by the identification process, the computer further executes the following: giving priority to the operation input by the hairdresser to the smart mirror over the operation input by the customer receiving the treatment to the smart mirror. The control method described in Technology 2-5. According to this control method, in the case of a scene after counseling or treatment, the subject of operation of the device can be appropriately set. [Technology 2-7] The identifying process identifies the scene as being during treatment when the recognition process recognizes that the image contains a person and a hand holding a predetermined tool, When the identifying process identifies the scene as the scene during the treatment, the switching process excludes a hairdresser from users who can operate the smart mirror. A control method according to any one of techniques 2-3 to 2-6. According to this control method, when a treatment scene is occurring, users who are permitted to operate the device can be appropriately set. [Technology 2-8] The computer further executes determining whether or not the hand shown in the image is a hand of a person who is permitted to operate the device, based on whether or not feature information of the hand shown in the image satisfies a predetermined condition related to a non-operator who is not permitted to operate the device. A control method according to any one of techniques 2-1 to 2-7. According to this control method, it is possible to appropriately determine whether or not the hand shown in the image is the hand of a person authorized to operate the device. [Technology 2-9] an image acquisition unit that acquires an image captured by a camera; an object recognition unit that recognizes one or more objects appearing in the image acquired by the image acquisition unit; a scene determination unit that identifies a scene corresponding to one or more objects recognized by the object recognition unit from among a plurality of predetermined scenes; a switching unit that switches at least one of a user that can operate a predetermined device and an operation that can be input to the device based on the scene identified by the scene determination unit; A control device comprising: [Technology 2-10] Acquire an image captured by a camera, Recognizing one or more objects in the captured image; Identifying a scene from among a plurality of predetermined scenes that corresponds to the recognized object or objects; switching at least one of a user who can operate a predetermined device and an operation that can be input to the device based on the identified scene; A computer program that causes a computer to carry out a task. [Explanation of symbols]

[0123] 10 smart mirror, 11 information processing device, 12 camera, 22 position memory unit, 24 hand memory unit, 30 image acquisition unit, 32 detection unit, 34 hairdresser determination unit, 38 gesture recognition unit, 50 scene determination unit, 52 switching unit.

Claims

1. Acquire an image captured by a camera, Detect hands in the captured images, If the feature information of the detected hand satisfies a predetermined condition related to a non-operator who is not permitted to operate a predetermined device, the detected hand is determined to be the hand of the non-operator; when the detected hand is determined to be the hand of the non-operator, suppressing recognition of a gesture operation on the device based on the movement of the detected hand. An information processing method implemented by a computer.

2. If the feature information of the detected hand does not satisfy the condition, it is determined that the detected hand is a hand of an authorized user who is authorized to operate the device; when it is determined that the detected hand is the hand of the permitted operation person, recognizing a gesture operation based on the movement of the detected hand; The information processing method according to claim 1 .

3. The process of determining includes determining that the detected hand is the hand of the non-operator when a speed of the detected hand movement is faster than a predetermined threshold.

3. The information processing method according to claim 1.

4. The process of determining includes determining that the detected hand is the hand of the non-operator when the biometric information on the detected hand matches the biometric information stored in a storage unit.

3. The information processing method according to claim 1.

5. When it is detected that the hand is stationary based on the plurality of images, the computer further detects biometric information related to the hand. The information processing method according to claim 4.

6. The detecting process further detects a tool used by the non-operator that appears in the acquired image; The process of determining includes determining that the detected hand is the hand of the non-operator when the tool is present in the vicinity of the detected hand.

3. The information processing method according to claim 1.

7. The process of determining determines that the detected hand is the hand of the non-operator on the further condition that the number of movements of an object present in the vicinity of the detected hand is greater than a predetermined threshold. The information processing method according to claim 6.

8. The process of determining includes determining that the detected hand is the hand of the non-operator when a color of the detected hand matches a color of a glove used by the non-operator.

3. The information processing method according to claim 1.

9. an image acquisition unit that acquires an image captured by a camera; a detection unit that detects a hand appearing in an image acquired by the image acquisition unit; a determination unit that determines that the detected hand is a hand of a non-operator when feature information of the hand detected by the detection unit satisfies a predetermined condition related to a non-operator who is not permitted to operate a predetermined device; and an operation recognition unit that, when it is determined that the detected hand is the hand of the non-operator, suppresses recognition of a gesture operation on the device based on the movement of the detected hand; An information processing device comprising:

10. Acquire an image captured by a camera, Detect hands in the captured images, If the feature information of the detected hand satisfies a predetermined condition related to a non-operator who is not permitted to operate a predetermined device, the detected hand is determined to be the hand of the non-operator; when the detected hand is determined to be the hand of the non-operator, suppressing recognition of a gesture operation on the device based on the movement of the detected hand. A computer program that causes a computer to carry out a task.

Images