Information processing device and information processing program

Abstract

To improve the accuracy of identifying an object person on the basis of photographing data.SOLUTION: An information processing device includes a detection part and an identification part. The detection part detects a prescribed gesture on the basis of photographing data. The identification part identifies one or more candidates satisfying a condition among one or more persons whose prescribed gesture is detected by the detection part on the basis of the photographing data. The identification part identifies an object person on the basis of the one or more candidates.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing device and an information processing program. [Background technology]

[0002] In restaurants and other establishments, self-ordering systems have been introduced in which customers themselves input their orders into a terminal. One way of inputting orders is touchless, without the customer having to touch the terminal.

[0003] As a touchless input method, a method is being considered in which orders are input into a terminal based on the recognition of customer hand gestures using a camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-521576 Summary of the Invention [Problem to be solved by the invention]

[0005] In the embodiment using hand gestures, the self-ordering system needs to identify a single subject who is trying to input an order and recognize the order by the single subject through hand gestures. The self-ordering system needs to improve the accuracy of identifying the subject to avoid inputting an erroneous order due to recognition of hand gestures of a person other than the subject.

[0006] The problem to be solved by the embodiments of the present invention is to provide a technology for improving the accuracy of identifying a target person based on photographic data. [Means for solving the problem]

[0007] In one embodiment, the information processing device includes a detection unit and an identification unit. The detection unit detects a predetermined gesture based on the captured image data. The identification unit identifies, based on the captured image data, one or more candidates who satisfy a condition from among one or more people whose predetermined gesture has been detected by the detection unit. The identification unit identifies a target person based on the one or more candidates. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an external view illustrating a terminal according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a terminal according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating a first region associated with a terminal according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of measurement by a terminal according to the embodiment. [Figure 5] FIG. 5 is a flowchart illustrating an information processing procedure by the terminal according to the embodiment. [Figure 6] FIG. 6 is a flowchart illustrating a procedure of a candidate specification process by a terminal according to the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating the procedure of the area identification process by the terminal according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of a procedure of a target person identification process by the terminal according to the embodiment. [Figure 9] FIG. 9 is a flowchart illustrating another example of the procedure of the target person identification process by the terminal according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment will be described with reference to the drawings.

[0010] (Configuration example) FIG. 1 is an external view illustrating a terminal 1. As shown in FIG. Terminal 1 is an electronic device that allows customers to input orders touchlessly using gestures without touching the terminal 1. Terminal 1 is an example of an information processing device. For example, terminal 1 is installed in a store such as a restaurant in association with a table 2. A store may have multiple terminals and multiple tables, but for the sake of simplicity, FIG. 1 shows one terminal 1 and one table 2 associated with terminal 1.

[0011] Terminal 1 has a camera whose shooting range is a range including a first area associated with terminal 1. The first area is a three-dimensional area in which all customers seated around table 2 and eating and drinking are generally assumed to be present. For example, the first area is an area including table 2 and the seats placed around table 2. The range of the first area can be set as appropriate. The first area is an example of a predetermined area.

[0012] Because the imaging range of terminal 1 includes the first area, the imaging data acquired by terminal 1 includes all customers present in the first area. The imaging range of terminal 1 includes not only the first area but also part of the second area. The second area is an area different from the first area. Therefore, the imaging data acquired by terminal 1 may include one or more people present in the second area. For example, the imaging data acquired by terminal 1 may include one or more customers seated around a table that is different from table 2 and is included in the second area. For example, the imaging data acquired by terminal 1 may include one or more customers or store clerks passing through an aisle near table 2 that is included in the second area.

[0013] All people present in the first area are people who may input an order into the terminal 1 by a gesture. When starting to input an order, a customer performs a first gesture to be recognized as a person who will input an order into the terminal 1. The first gesture may be, but is not limited to, a gesture of raising a hand. The first gesture can be set as appropriate. The first gesture is an example of a predetermined gesture.

[0014] The customer recognized as a single target person by terminal 1 performs various gestures to input an order to terminal 1 after the first gesture. The various gestures may include a gesture to input food and beverages to be ordered. The various gestures may include a gesture to move a cursor to select food and beverages to be ordered from a plurality of food and beverages displayed on terminal 1. The various gestures may include a gesture to input a confirmation of the food and beverages to be ordered. The various gestures may include a gesture to input an order quantity of food and beverages to be ordered. The various gestures may include a gesture to move a cursor to select an order quantity of food and beverages to be ordered from a plurality of numbers displayed on terminal 1. The various gestures may include a gesture to input a confirmation of the order quantity of food and beverages to be ordered.

[0015] FIG. 2 is a block diagram illustrating the terminal 1. As shown in FIG. The terminal 1 has a processor 10, a main memory 11, an auxiliary storage device 12, a communication interface 13, an input device 14, a display device 15, a microphone 16, a speaker 17, and a camera 18. The components constituting the terminal 1 are connected to each other so that signals can be input and output. In Fig. 2, the interface is indicated as "I / F".

[0016] The processor 10 corresponds to the central part of the computer of the terminal 1. For example, the processor 10 is a CPU (Central Processing Unit), but is not limited to this. The processor 10 may be composed of various circuits. The processor 10 loads a program stored in the main memory 11 or the auxiliary storage device 12 into the main memory 11. The program is a program that causes the processor 10 of the terminal 1 to realize each unit described below. The processor 10 performs various operations by executing the program loaded into the main memory 11.

[0017] The main memory 11 corresponds to the main storage portion of the computer of the terminal 1. The main memory 11 includes a non-volatile memory area and a volatile memory area. The main memory 11 stores an operating system or a program in the non-volatile memory area. The main memory 11 uses the volatile memory area as a work area where data is rewritten by the processor 10 as appropriate. For example, the main memory 11 includes a ROM (Read Only Memory) as a non-volatile memory area. For example, the main memory 11 includes a RAM (Random Access Memory) as a volatile memory area.

[0018] The auxiliary storage device 12 corresponds to an auxiliary storage portion of the computer of the terminal 1. For example, the auxiliary storage device 12 is, but is not limited to, an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disc Drive), or an SSD (Solid State Drive). The auxiliary storage device 12 stores the above-mentioned programs, data used by the processor 10 in performing various processes, and data generated by the processes of the processor 10.

[0019] The auxiliary storage device 12 includes a reference value storage area 121 that stores a reference value DF for area identification. The reference value storage area 121 is an example of a storage unit. The reference value DF for area identification is a value that serves as a reference for area identification to identify whether an area in which a person exists is a first area or a second area by comparison with the measurement value F.

[0020] The reference value DF for region identification is a value based on one or more distances between features included in the face of a reference person measured in frames of a moving image constituting the shooting data. The reference person is a person with an average face that serves as a reference. The reference person may be a real person or a virtual person. For example, the one or more distances between features included in the face of the reference person are at least one of the width of the inner canthus of the reference person's eye and the width of the outer canthus of the eye, but are not limited to this. The width of the inner canthus of the eye is the linear distance connecting the inner and outer canthus of the eye. The width of the outer canthus of the eye is the linear distance connecting the outer and inner canthus of the eye. The one or more distances between features included in the face of the reference person may also be one or more distances between features other than the left and right eyes of the reference person. Here, the one or more distances between features included in the face of the reference person are the width of the inner canthus of the reference person and the width of the outer canthus of the eye. The value based on the one or more distances between features included in the face of the reference person is the sum of the width of the inner canthus of the eye and the width of the outer canthus of the eye of the reference person. The reference value DF for identifying the region may be the average value of the sum of the widths of the inner and outer corners of the eyes of a plurality of reference persons.

[0021] The measurement value F is a value based on one or more distances between parts included in the face of a person measured in frames of a moving image constituting the photographic data. For example, one or more distances between parts included in the face of a person may be the width of the inner corner of the eye of the person, f in and lateral canthus width f out The one or more distances between the features included in the person's face may be one or more distances between features other than the left and right eyes of the person. Here, the one or more distances between the features included in the person's face may be at least one of the following: in and lateral canthus width f out The measurement value F is the width of the inner corner of the eye of a person f in and lateral canthus width f out The value is the sum of the above.

[0022] The measurement value F satisfying the region identification criterion corresponds to the region where the person exists being the first region, and the measurement value F not satisfying the region identification criterion corresponds to the region where the person exists being the second region.

[0023] The reference value DF for region identification includes a lower limit value DF1 associated with the position of the first region farthest from the camera 18. The lower limit value DF1 is a value based on one or more distances between parts included in the face of a reference person assumed to be located at the position of the first region farthest from the camera 18. Here, the lower limit value DF1 is determined by the medial canthus width f of the reference person. in1 and lateral canthus width f out1 The value is the sum of the above.

[0024] The size of the subject within the frames of the moving images constituting the shooting data increases as the subject approaches camera 18. When the measurement value F is equal to or greater than the lower limit value DF1, the person is said to be present in the first region. When the measurement value F is equal to or greater than the lower limit value DF1, this is an example of the measurement value F satisfying the region identification criterion. When the measurement value F is less than the lower limit value DF1, the person is said to be present in the second region. When the measurement value F is less than the lower limit value DF1, this is an example of the measurement value F not satisfying the region identification criterion.

[0025] It should be noted that the camera 18 may be installed away from the table 2 without being adjacent to the first region. The reference value DF for region identification may include an upper limit value DF2 associated with the position of the first region closest to the camera 18. The upper limit value DF2 is a value based on one or more distances between parts included in the face of a reference person assumed to be present at the position of the first region closest to the camera 18. Here, the upper limit value DF2 is determined based on the medial canthus width f of the reference person. in2 and lateral canthus width f out2 The value is the sum of the above.

[0026] When the measurement value F is equal to or greater than the lower limit DF1 and equal to or less than the upper limit DF2, the person is said to be present in the first area. When the measurement value F is equal to or greater than the lower limit DF1 and equal to or less than the upper limit DF2, this is an example of the measurement value F satisfying the area identification criterion. When the measurement value F is greater than the upper limit DF2, the person is said to be present in the second area. When the measurement value F is greater than the upper limit DF2, this is an example of the measurement value F not satisfying the area identification criterion.

[0027] The communication interface 13 includes various interfaces that connect the terminal 1 to other devices via a network in accordance with a predetermined communication protocol so that the terminal 1 can communicate with other devices.

[0028] The input device 14 is a device that allows data or instructions to be input by touch operation to the terminal 1. For example, the input device 14 is a keyboard or a touch panel, but is not limited to these.

[0029] The display device 15 is a device that can display various images under the control of the terminal 1. For example, the display device 15 is a liquid crystal display or the like, but is not limited to this.

[0030] The microphone 16 is a device that can input sounds from the surrounding environment of the terminal 1 .

[0031] The speaker 17 is a device that can output sound under the control of the terminal 1 .

[0032] Camera 18 is a device capable of acquiring photographic data of a photographing range. For example, camera 18 acquires photographic data of a moving image. Camera 18 is an example of a photographing unit.

[0033] The hardware configuration of the terminal 1 is not limited to the above configuration. The terminal 1 allows the omission or modification of the above components and the addition of new components as appropriate. For example, the display device 15 may be a device independent of the terminal 1. The camera 18 may be a device independent of the terminal 1.

[0034] Each unit realized by the above-mentioned processor 10 will be described below. The processor 10 realizes a first detection unit 100, a second detection unit 101, a measurement unit 102, and an identification unit 103. Each unit realized by the processor 10 can also be referred to as each function. Each unit realized by the processor 10 can also be referred to as being realized by a control unit including the processor 10 and main memory 11.

[0035] The first detection unit 100 detects a first gesture based on the imaging data. The first detection unit 100 may detect the first gesture based on the imaging data using a known image processing technique. For example, the first detection unit 100 detects the hands of each person from frames of a moving image constituting the imaging data. The frames of the moving image may be a predetermined number of consecutive frames. The predetermined number of frames may be, but is not limited to, 15 frames, for example. The predetermined number of frames can be set as appropriate. The first detection unit 100 may detect the hands of each person for every predetermined number of frames.

[0036] The first detection unit 100 detects multiple key points (coordinates) from each person's hand. The first detection unit 100 acquires key point information including multiple key points for each person's hand. The first detection unit 100 classifies each person's hand gesture into a first gesture or a second gesture based on the key point information of each person's hand. The second gesture is a gesture different from the first gesture.

[0037] The first detection unit 100 may classify each person's hand gesture using a gesture classification model based on a deep neural network. The gesture classification model may be a trained model that classifies a person's hand gesture into either a first gesture or a second gesture based on keypoint information. The gesture classification model may be stored in the auxiliary storage device 12. The first detection unit 100 detects the first gesture based on the classification of each person's hand gesture. The first detection unit 100 is an example of a detection unit.

[0038] The second detection unit 101 detects the posture of each person based on the shooting data. The second detection unit 101 may detect the posture of each person based on the shooting data using a known image processing technique. For example, the second detection unit 101 detects the body of each person from frames of a moving image constituting the shooting data. The second detection unit 101 detects multiple key points from the body of each person. The second detection unit 101 acquires key point information including multiple key points for the body of each person. The second detection unit 101 classifies the posture of each person into a first posture or a second posture based on the key point information of the body of each person. The first posture is, but is not limited to, a sitting posture. The first posture can be set as appropriate. The first posture is an example of a predetermined posture. The second posture is a posture different from the first posture.

[0039] The second detection unit 101 may classify the posture of each person using a posture classification model based on a deep neural network. The posture classification model may be a trained model that classifies the posture of a person into either a first posture or a second posture based on keypoint information. The posture classification model may be stored in the auxiliary storage device 12. The second detection unit 101 detects the posture of each person based on the classification of the posture of each person. The second detection unit 101 is an example of a detection unit.

[0040] The measurement unit 102 measures one or more distances between parts included in the face of each person based on the imaging data. The measurement unit 102 acquires a measurement value F based on the measurement of one or more distances between parts included in the face of each person. The measurement unit 102 may measure one or more distances between parts included in the face of each person based on the imaging data using a known image processing technique, and acquire the measurement value F.

[0041] For example, the measurement unit 102 detects the face of each person from frames of a moving image constituting the shooting data. The measurement unit 102 detects multiple key points included in the face of each detected person. The measurement unit 102 acquires key point information including multiple key points for the face of each person. The measurement unit 102 measures one or more distances between parts included in the face of each detected person based on the key point information.

[0042] The identification unit 103 identifies one or more first persons based on the shooting data. The first persons are persons for whom a first gesture has been detected by the first detection unit 100. For example, the identification unit 103 identifies one or more first persons from frames of a moving image constituting the shooting data based on the detection of the first gesture by the first detection unit 100.

[0043] The identifying unit 103 identifies one or more candidates who satisfy the candidate identifying conditions from among one or more first persons based on the photographed data. The candidate identifying conditions are an example of a condition. The candidate identification conditions include being present in a first area. Satisfying the candidate identification conditions includes the first person's measurement value F satisfying the area identification criteria. Not satisfying the candidate identification conditions includes the first person's measurement value F not satisfying the area identification criteria. In this example, the identification unit 103 identifies one or more second persons from one or more first persons based on the photographic data. The second persons are first persons who are present in the first area.

[0044] For example, the identification unit 103 identifies one or more second persons from among the one or more first persons based on the measurement value F of each of the one or more first persons that is based on one or more distance measurements by the measurement unit 102. In this example, the identification unit 103 identifies an area in which each of the one or more first persons exists based on the measurement value F of each of the one or more first persons acquired by the measurement unit 102. The identification unit 103 compares the measurement value F of each of the one or more first persons acquired by the measurement unit 102 with a reference value DF for area identification. The identification unit 103 identifies an area in which each of the one or more first persons exists based on the comparison of the measurement value F of each of the one or more first persons with the reference value DF for area identification.

[0045] If the measurement value F satisfies the region identification criterion, the identification unit 103 identifies the region where the first person associated with the measurement value F that satisfies the region identification criterion as the first region. If the measurement value F does not satisfy the region identification criterion, the identification unit 103 identifies the region where the first person associated with the measurement value F that does not satisfy the region identification criterion as the second region.

[0046] An example will be described in which the region identification reference value DF includes a lower limit value DF1. The identification unit 103 identifies, as a first region, a region in which a first person associated with a measurement value F equal to or greater than the lower limit value DF1 exists. The identification unit 103 identifies, as a second region, a region in which a first person associated with a measurement value F less than the lower limit value DF1 exists.

[0047] An example will be described in which the region identification reference value DF includes a lower limit value DF1 and an upper limit value DF2. The identification unit 103 identifies, as a first region, an area in which a first person associated with a measurement value F that is equal to or greater than the lower limit value DF1 and equal to or less than the upper limit value DF2 exists. The identification unit 103 identifies, as a second region, an area in which a first person associated with a measurement value F that is less than the lower limit value DF1 or greater than the upper limit value DF2 exists.

[0048] The identification unit 103 identifies one or more second persons from among the one or more first persons based on the identification of the areas where each of the one or more first persons exists. The identification unit 103 identifies one or more candidates based on the one or more second persons. The identification unit 103 identifies the one or more second persons as one or more candidates who satisfy the candidate identification conditions.

[0049] The candidate identification conditions may include being present in the first area and having a first posture. Satisfying the candidate identification conditions includes the first person's measurement value F satisfying the area identification criterion and the first person's posture being a first posture. Not satisfying the candidate identification conditions includes the first person's measurement value F not satisfying the area identification criterion. Not satisfying the candidate identification conditions includes the first person's measurement value F satisfying the area identification criterion, but the first person's posture being a second posture. In this example, the identification unit 103 identifies one or more third persons from one or more first persons based on the photographic data. The third person is a first person who is present in the first area and has a first posture. The third person is also a second person who has the first posture.

[0050] For example, as described above, the identification unit 103 identifies one or more second persons from among one or more first persons based on the photographic data. The identification unit 103 identifies one or more third persons from among the one or more second persons based on the postures of the one or more second persons detected by the second detection unit 101.

[0051] The identification unit 103 identifies one or more candidates based on one or more third persons. Identifying one or more candidates based on one or more third persons is an example of identifying one or more candidates based on one or more second persons. The identification unit 103 identifies the one or more third persons as one or more candidates who satisfy the candidate identification conditions.

[0052] The identification unit 103 identifies one target person based on one or more candidates. When the identification unit 103 identifies one candidate who satisfies the candidate identification conditions, it identifies the one candidate as one target person. When the identification unit 103 identifies multiple candidates who satisfy the candidate identification conditions, it identifies one candidate who satisfies the target identification conditions from among the multiple candidates as one target person based on the photographed data. The target identification conditions are conditions for identifying one target person from among the multiple candidates who satisfy the candidate identification conditions.

[0053] In one example, the target person identification condition includes the earliest detection timing of the first gesture by the first detection unit 100. In this example, when multiple candidates are identified, the identification unit 103 identifies, as the target, one candidate from among the multiple candidates whose first gesture was detected earliest by the first detection unit 100. For example, the identification unit 103 compares the chronological order of frames indicating the start of the first gesture by each of the multiple candidates among a predetermined number of frames. The identification unit 103 identifies, as the target, the candidate associated with the first gesture indicated by the earliest chronological frame. The identification unit 103 identifies, as the target, the candidate who satisfies the target person identification condition.

[0054] In another example, the condition for identifying a subject includes being closest in distance from a reference position. The reference position is, but is not limited to, the position of camera 18. The reference position may be any position within the first region. The reference position is an example of a predetermined position. In this example, when multiple candidates are identified, the identification unit 103 identifies one candidate from the multiple candidates who is closest in distance from the reference position as one subject. For example, the identification unit 103 compares the measurement values ​​F of the multiple candidates acquired by the measurement unit 102. The identification unit 103 identifies one candidate who satisfies the condition for identifying a subject as one subject based on the comparison of the measurement values ​​F of the multiple candidates. The identification unit 103 identifies one candidate who satisfies the condition for identifying a subject as one subject.

[0055] An example will be described in which the reference position is the position of camera 18. The closer a person is to the reference position, the larger the measurement value F. The identification unit 103 identifies the candidate with the largest measurement value F among multiple candidates as the candidate who satisfies the target person identification conditions.

[0056] An example where the reference position is an arbitrary position will be described. The identification unit 103 identifies one candidate among multiple candidates whose measurement value F is closest to the reference value DF3 at the arbitrary position as one candidate who satisfies the conditions for identifying a target person. The reference value DF3 is a distance measured in a frame of a moving image, and is a value based on one or more distances between parts included in the face of a reference person assumed to be present at an arbitrary position. Here, the reference value DF3 is calculated by multiplying the width f of the inner corner of the eye of the reference person by the distance F. in3 and lateral canthus width f out3 The reference value DF3 may be stored in the auxiliary storage device 12.

[0057] The first area associated with terminal 1 will now be described. FIG. 3 is a diagram illustrating a first area 3 associated with a terminal 1. As shown in FIG. FIG. 3 is a plan view of the terminal 1 and the table 2 as seen from above in the vertical direction. The first area 3 is an area including the table 2 and the seats placed around the table 2 . The terminal 1 acquires video data with at least the first area 3 as the shooting range.

[0058] FIG. 4 is a diagram showing an example of measurement by the terminal 1. FIG. 4 shows the width f of the inner canthus of a person measured by the measurement unit 102. in and lateral canthus width f out The measurement unit 102 measures the width of the inner corner of the eye of a person f in and lateral canthus width f out The measured value F is obtained by adding up the above.

[0059] (operation) Next, an example of the operation of the terminal 1 configured as above will be described.

[0060] FIG. 5 is a flowchart illustrating an example of the procedure of information processing by the terminal 1. The processing procedures described below are merely examples, and each process may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate depending on the embodiment.

[0061] The first detection unit 100 detects a first gesture based on the captured image data (ACT 1). In ACT 1, the first detection unit 100 may detect the first gesture of one person, or may detect the first gestures of each of multiple people.

[0062] The identification unit 103 identifies, based on the photographed data, one or more first persons whose first gestures have been detected by the first detection unit 100 (ACT2).

[0063] The identifying unit 103 identifies one or more candidates who satisfy the candidate identifying conditions from among one or more first persons based on the photographed data (ACT 3). An example of the processing in ACT 3 will be described later.

[0064] The identifying unit 103 identifies one target person based on one or more candidates (ACT 4). An example of the processing in ACT 4 will be described later.

[0065] FIG. 6 is a flowchart illustrating the procedure of the candidate specification process by the terminal 1 in ACT3. FIG. 6 shows an example in which the candidate identifying conditions include being in a first position in addition to being in a first area.

[0066] The processing procedures described below are merely examples, and each process may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate depending on the embodiment.

[0067] The identification unit 103 identifies an area where each of the one or more first persons exists based on the measurement value F of each of the one or more first persons based on one or more distance measurements by the measurement unit 102 (ACT31). An example of the processing in ACT31 will be described later.

[0068] Based on the identification of the area where each of the one or more first persons exists, the identification unit 103 determines, for each first person, whether the area where the first person exists is the first area (ACT32).

[0069] If the area where the first person exists is the first area (ACT32, YES), the process transitions from ACT32 to ACT33. If the area where the first person exists is the second area (ACT32, NO), the process transitions from ACT32 to ACT34. The identification unit 103 does not identify the first person existing in the second area as the second person. In other words, the identification unit 103 does not identify the first person existing in the second area as a candidate. The identification unit 103 identifies the first person existing in the first area as the second person (ACT33).

[0070] The identification unit 103 determines whether or not the processing of ACT32 to ACT33 has been performed for all of the one or more first persons (ACT34). If the identification unit 103 has performed the processing for all of the one or more first persons (ACT34, YES), the processing transitions from ACT34 to ACT 35. If the identification unit 103 has not performed the processing for all of the one or more first persons (ACT34, NO), the processing transitions from ACT34 to ACT32.

[0071] The identification unit 103 executes the processes of ACT32 to ACT33 for all of the one or more first persons. As a result, the identification unit 103 identifies one or more second persons who exist in the first area from among the one or more first persons based on the shooting data. In a typical example, the identification unit 103 identifies one or more second persons from among the one or more first persons based on the measurement value F of each of the one or more first persons. The identification unit 103 identifies one or more second persons from among the one or more first persons based on the identification of the area where each of the one or more first persons exists based on the measurement value F.

[0072] The second detection unit 101 detects the posture of each of the one or more second persons based on the photographic data (ACT35).

[0073] The identification unit 103 determines, based on the posture of each of the detected one or more second persons, whether or not the posture of each second person is the first posture (ACT 36).

[0074] If the posture of the second person is the first posture (ACT36, YES), the process transitions from ACT36 to ACT37. If the posture of the second person is the second posture (ACT36, NO), the process transitions from ACT36 to ACT38. The identification unit 103 does not identify the second person in the second posture as the third person. In other words, the identification unit 103 does not identify the second person in the second posture as a candidate. The identification unit 103 identifies the second person in the first posture as the third person (ACT37).

[0075] The identification unit 103 determines whether or not the processing of ACT36 to ACT37 has been performed for all of the one or more second persons (ACT38). If the identification unit 103 has performed the processing for all of the one or more second persons (ACT38, YES), the processing proceeds from ACT38 to ACT 39. If the identification unit 103 has not performed the processing for all of the one or more second persons (ACT38, NO), the processing proceeds from ACT38 to ACT36.

[0076] The identification unit 103 executes the processes of ACT 36 to ACT 37 for all of the one or more second persons. As a result, the identification unit 103 identifies one or more third persons from among the one or more second persons based on the postures of each of the detected one or more second persons.

[0077] The identification unit 103 identifies one or more candidates based on the one or more third persons (ACT 39). In ACT 39, for example, the identification unit 103 identifies the one or more third persons as one or more candidates who satisfy the candidate identification conditions.

[0078] The identification unit 103 performs the candidate identification process illustrated in FIG. 6 to identify one or more candidates who satisfy the candidate identification conditions from among one or more first persons based on the photographed data.

[0079] The candidate identification conditions include being in the first area, but do not necessarily include having the first posture. In this example, the processes of ACT 35 to ACT 38 may be omitted. In ACT 39, the identification unit 103 identifies one or more candidates based on one or more second persons. For example, the identification unit 103 identifies one or more second persons as one or more candidates who satisfy the candidate identification conditions.

[0080] FIG. 7 is a flowchart illustrating the procedure of the area identification process by the terminal 1 in ACT31. The processing procedures described below are merely examples, and each process may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate depending on the embodiment.

[0081] The measurement unit 102 measures one or more distances between parts included in the faces of one or more first persons based on the photographic data (ACT 311).

[0082] The measurement unit 102 acquires a measurement value F of each of the one or more first persons based on one or more distance measurements (ACT 312).

[0083] The identifying unit 103 acquires the area identifying reference value DF from the auxiliary storage device 12 (ACT 313).

[0084] The identification unit 103 compares the measurement value F of each first person with the reference value DF for area identification (ACT 314).

[0085] The identification unit 103 determines whether the measurement value F of the first person satisfies the criteria (ACT315). If the measurement value F of the first person satisfies the criteria for area identification (ACT315, YES), the process proceeds from ACT315 to ACT316. If the measurement value F of the first person does not satisfy the criteria for area identification (ACT315, NO), the process proceeds from ACT315 to ACT317.

[0086] The identification unit 103 identifies an area where the first person associated with the measurement value F that satisfies the area identification criterion as a first area (ACT 316). The identification unit 103 identifies an area where the first person associated with the measurement value F that does not satisfy the area identification criterion as a second area (ACT 317).

[0087] The identification unit 103 determines whether or not the processing from ACT314 to ACT317 has been performed for all of the one or more first persons (ACT318). If the identification unit 103 has performed the processing for all of the one or more first persons (ACT318, YES), the processing ends. If the identification unit 103 has not performed the processing for all of the one or more first persons (ACT318, NO), the processing transitions from ACT318 to ACT314.

[0088] The identification unit 103 executes the processes of ACT314 to ACT317 for all of the one or more first persons. As a result, the identification unit 103 identifies the area in which each of the one or more first persons exists, based on the measurement value F of each of the one or more first persons. In a typical example, the identification unit 103 compares the measurement value F of each of the one or more first persons with the area identification reference value DF. The identification unit 103 identifies the area in which each of the one or more first persons exists, based on the comparison of the measurement value F of each of the one or more first persons with the area identification reference value DF.

[0089] FIG. 8 is a flowchart showing an example of the procedure of the target person identification process by the terminal 1 in ACT4. The processing procedures described below are merely examples, and each process may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate depending on the embodiment.

[0090] The identification unit 103 determines whether or not one candidate who satisfies the candidate identification conditions has been identified (ACT41). If the identification unit 103 has identified one candidate who satisfies the candidate identification conditions (ACT41, YES), the process transitions from ACT41 to ACT43. If the identification unit 103 has identified multiple candidates who satisfy the candidate identification conditions (ACT41, NO), the process transitions from ACT41 to ACT42.

[0091] The identifying unit 103 identifies one candidate from among a plurality of candidates who satisfy the candidate identifying conditions, the candidate whose first gesture was detected earliest by the first detecting unit 100 (ACT42).

[0092] The identification unit 103 identifies one target person (ACT43). In ACT43, for example, when the identification unit 103 identifies one candidate who satisfies the candidate identification conditions, the identification unit 103 identifies the one candidate as one target person. When the identification unit 103 identifies multiple candidates who satisfy the candidate identification conditions, the identification unit 103 identifies the one candidate whose first gesture was detected earliest as one target person.

[0093] FIG. 9 is a flowchart showing another example of the procedure of the target person identification process by the terminal 1 in ACT4. The processing procedures described below are merely examples, and each process may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate depending on the embodiment.

[0094] The identification unit 103 determines whether or not one candidate who satisfies the candidate identification conditions has been identified (ACT44). If the identification unit 103 has identified one candidate who satisfies the candidate identification conditions (ACT44, YES), the process transitions from ACT44 to ACT46. If the identification unit 103 has identified multiple candidates who satisfy the candidate identification conditions (ACT44, NO), the process transitions from ACT44 to ACT45.

[0095] The identifying unit 103 identifies one candidate who is closest to the reference position from among the plurality of candidates who satisfy the candidate identifying conditions (ACT45).

[0096] The identification unit 103 identifies one target person (ACT46). In ACT46, for example, when the identification unit 103 identifies one candidate who satisfies the candidate identification conditions, the identification unit 103 identifies the one candidate as one target person. When the identification unit 103 identifies multiple candidates who satisfy the candidate identification conditions, the identification unit 103 identifies the one candidate who is closest in distance from the reference position as one target person.

[0097] (effect) The terminal 1 according to the embodiment identifies one or more candidates who satisfy the candidate identification condition from among one or more people from whom the first gesture was detected, based on the photographed data. The terminal 1 identifies a target person based on the one or more candidates. For example, a person making the first gesture is likely to be a customer about to input an order into terminal 1. Terminal 1 can identify the target person by narrowing down the candidates to those making the first gesture. Therefore, terminal 1 can improve the accuracy of identifying the target person based on the captured data from among the customers at table 2 associated with terminal 1.

[0098] The terminal 1 according to the embodiment identifies one or more people present in the first area from one or more people who have detected a first gesture based on the captured image data. The terminal 1 identifies one or more candidates based on the one or more people present in the first area. For example, a person making a first gesture and present in the first area is likely not a store clerk but a customer about to enter an order into terminal 1. A person making a first gesture but not present in the first area is likely to be a store clerk or other person. Terminal 1 can specify a target person by narrowing down candidates to people making the first gesture and present in the first area. Terminal 1 can exclude from the candidates people making the first gesture of a store clerk or other person but not present in the first area. Therefore, terminal 1 can improve the accuracy of specifying candidates.

[0099] The terminal 1 according to the embodiment identifies one or more people who are in a first pose from one or more people who are present in a first area. The terminal 1 identifies one or more candidates based on the one or more people who are in the first pose. For example, the posture of a customer at table 2 associated with terminal 1 is different from the posture of a waiter who is serving food, etc. A person who makes a first gesture and assumes a first posture in a first area is likely to be a customer who is about to input an order into terminal 1. A person who makes a first gesture but does not assume the first posture in the first area is likely to be a waiter, etc. Terminal 1 can narrow down the candidates to people who make the first gesture and assume the first posture in the first area. Terminal 1 can exclude people who make the first gesture of a waiter, etc., but do not assume the first posture in the first area from the candidates. As a result, terminal 1 can improve the accuracy of identifying candidates.

[0100] The first posture according to the embodiment is a seated posture. For example, the posture of a customer at table 2 associated with terminal 1 is likely to be a seated posture. A waiter who is temporarily present near table 2 to serve food, etc., is likely to be standing. Terminal 1 can narrow down the candidates to people who make a first gesture and are seated in a first area. Terminal 1 can exclude from the candidates people who make a first gesture like a waiter but are not in a first posture in the first area. As a result, terminal 1 can improve the accuracy of identifying candidates.

[0101] The terminal 1 according to the embodiment measures one or more distances between parts included in the faces of one or more people for whom a first gesture has been detected, based on the captured image data. The terminal 1 identifies one or more people who are present in the first area from among the one or more people for whom a first gesture has been detected, based on the measurement values ​​of each of the one or more people for whom a first gesture has been detected based on the one or more distance measurements. The one or more distances between the features included in the face vary depending on the distance between the camera and the person. By measuring the one or more distances between the features included in the face, terminal 1 can improve the accuracy of determining whether a person making a first gesture is present in the first area.

[0102] The measurement value according to the embodiment is the sum of the width of the inner canthus and the width of the outer canthus. By using a plurality of distances such as the width of the inner corner of the eye and the width of the outer corner of the eye, the terminal 1 can improve the accuracy of determining whether or not a person making the first gesture is present in the first area.

[0103] When a plurality of candidates are identified, the terminal 1 according to the embodiment identifies the candidate who has the earliest detection timing of the first gesture among the plurality of candidates as the target candidate. For example, when there are multiple candidates, it is an appropriate condition to select the candidate who initiated the first gesture as the target. This allows the terminal 1 to identify, under appropriate conditions, a candidate who is suitable as the target for inputting an order to the terminal 1 when there are multiple candidates.

[0104] The terminal 1 according to the embodiment identifies, as a target candidate, the candidate who is closest to the reference position among a plurality of candidates. This allows the terminal 1 to identify a suitable candidate for the person entering an order to the terminal 1 in accordance with fair rules when there are multiple candidates.

[0105] (Variation) In the above embodiment, an example in which the terminal 1 identifies one target person has been described, but this is not limiting. The terminal 1 may also be configured to identify multiple predetermined targets. In this example, the target identification conditions are conditions for identifying multiple predetermined targets from multiple candidates who satisfy the candidate identification conditions. In one example, the target identification conditions include identifying the multiple predetermined targets in order of earliest detection of the first gesture by the first detection unit 100. In this example, the identification unit 103 identifies the multiple predetermined candidates as the multiple predetermined targets, starting with the candidate who detected the first gesture earliest. In another example, the target identification conditions include identifying the multiple predetermined candidates in order of closest distance from the reference position. In this example, the identification unit 103 identifies the multiple predetermined candidates as the multiple predetermined targets, starting with the candidate who is closest distance from the reference position.

[0106] In the above embodiment, an example has been described in which the terminal 1 used in a store such as a restaurant recognizes a target person who inputs an order to the terminal 1, but the present invention is not limited to this. The above embodiment can be applied to various other places where a target person needs to be recognized, other than a store. For example, the terminal 1 may be applied in a conference. In this example, the terminal 1 may recognize a person who wishes to speak or ask a question as a target person.

[0107] The information processing device may be realized by one device such as the terminal 1, or may be realized by a plurality of devices with distributed functions.

[0108] The program may be transferred in a state where it is stored in a device, or in a state where it is not stored in a device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The form of the recording medium is not important as long as it is a medium that can store the program and is computer-readable, such as a CD-ROM or a memory card.

[0109] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0110] 1...terminal, 2...table, 3...first area, 10...processor, 11...main memory, 12...auxiliary storage device, 13...communication interface, 14...input device, 15...display device, 16...microphone, 17...speaker, 18...camera, 100...first detection unit, 101...second detection unit, 102...measurement unit, 103...identification unit, 121...reference value storage area.

Claims

1. a detection unit that detects a predetermined gesture for recognizing the person as a person who is to input an order based on the photographed data; an identification unit that identifies one or more candidates who satisfy a condition from among one or more people for whom the predetermined gesture has been detected by the detection unit based on the photographed data, and identifies the target person based on the one or more candidates; means for receiving an order input by the subject's gesture after the subject makes the predetermined gesture; An information processing device comprising:

2. the condition includes being in a predetermined area; the identification unit identifies one or more people present in the predetermined area from among one or more people for whom the predetermined gesture has been detected based on the photographic data, and identifies the one or more candidates based on the one or more people present in the predetermined area. The information processing device according to claim 1 .

3. the condition includes being in a predetermined posture; the identification unit identifies one or more people taking the predetermined posture from among one or more people present in the predetermined area, and identifies the one or more candidates based on the one or more people taking the predetermined posture; The information processing device according to claim 2 .

4. a measuring unit that measures, based on the photographing data, one or more distances between parts included in the faces of the one or more people for whom the predetermined gesture has been detected; the identification unit identifies one or more people present in the predetermined area from the one or more people for whom the predetermined gesture has been detected, based on respective measurement values ​​of the one or more people for whom the predetermined gesture has been detected based on the measurement of the one or more distances by the measurement unit. The information processing device according to claim 2 .

5. When the identification unit identifies one candidate, the identification unit identifies the one candidate as the target person; When a plurality of candidates are identified, the identification unit identifies, as the target, the candidate who has the earliest timing at which the detection unit detects the predetermined gesture among the plurality of candidates, or identifies, as the target, the candidate who is closest to a predetermined position among the plurality of candidates. The information processing device according to claim 1 .

6. On the computer, A function to detect a predetermined gesture based on the photographed data to recognize the person as the person entering the order; a function of identifying one or more candidates who satisfy a condition from one or more people who have detected the predetermined gesture based on the photographed data, and identifying the target person based on the one or more candidates; a function of accepting an order input by the subject's gesture after the subject's predetermined gesture; An information processing program for executing the above.

Images