Mobile information terminal and information processing method

Abstract

With conventional devices for displaying meeting-partner information on a mobile information terminal held by a user, when the user meets a meeting-partner in person, it is not possible to quickly acquire information relating to the meeting-partner at a timing immediately before a meeting, and said information were acquired after the start of the meeting. Thus, there had been awkward situations where the user and the meeting-partner could not be on the same topic at the start of the meeting. A mobile information terminal according to a representative embodiment of the present invention determines whether or not a person in the vicinity of a user is a potential meeting-partner by performing behavior analysis on said person, and, when it is determined that the person is a potential meeting-partner, acquires information relating to the person, and presents same to the user. Consequently, the user is allowed to previously be aware of the information relating to the meeting-partner before the start of a meeting or at the time at which the person is recognized as a meeting-partner.

Description

Technical Field

[0001] The present invention relates to a portable information terminal and an information processing method.

Background Art

[0002] There are cases where, regarding a person with whom one has had a direct face-to-face conversation in the past (hereinafter also referred to as the "person met in person"), when it is several years until the next meeting with that person, or when a person who has met several times in the past but not frequently suddenly visits one day.

[0003] And when directly facing the person met in the past again, one may completely forget the information (such as the name) of the other person. When the conversation starts, the other person (the person met in person) can say their own name, but this side cannot say (cannot remember) the name of the other person, and an embarrassing situation may occur.

[0004] For the purpose of avoiding the occurrence of the above situation as much as possible, there is also a method of recording information of many friends and related persons in the user's notebook etc. (regardless of the medium such as paper or electronic). On the other hand, even for users who have implemented such a method, for example, when there are a very large number of people met in the past (in other words, information of the people recorded in the notebook etc.), the same problem as above may occur.

[0005] More specifically, the more information there is about the people (past people met in person) recorded in the notebook etc., the more the recorded information of those people gradually (for example, starting from the older ones) is lost from the user's memory (conscious awareness) in the brain. Therefore, when a person met in the past suddenly visits one day, the user cannot quickly retrieve the keyword of the person met in person from the brain, and an embarrassing situation may occur where the user cannot say (cannot remember) the name of the other person when the conversation starts.

[0006] Recently, it has become common for users to carry around a mobile information terminal that has electronic information with a face photo attached, and check the information of the person they are meeting with beforehand, refresh their memory, and prepare for the meeting. A mobile information terminal that stores such electronic information can be an effective tool when the person is known in advance. However, in a similar situation to the above, that is, when a person whose electronic information such as a face photo is stored in the mobile information terminal suddenly visits the user one day and they meet, the same problem as the above can occur.

[0007] From another perspective, users of such tools must frequently check the information recorded in the tool and constantly refresh their memory in order to be able to instantly provide names and other information to unexpected visitors. However, this type of work becomes more cumbersome and time-consuming as the amount of recorded information increases. In addition, even if there are not actually that many unexpected visitors, it is likely that many users find the task of recalling the information of a large number of people in preparation for the near future inefficient or resist it.

[0008] In general, the portable information terminals described above are not considered to be effective tools because users would not be able to respond to unexpected visitors.

[0009] In recent years, advances have been made in facial recognition technology and portable information terminals equipped with small cameras have become widespread. By using these technologies, it is becoming possible to identify a person (an interviewee) and obtain information about the person.

[0010] For example, Patent Document 1 describes a technique related to a method in which a user meets a person face-to-face and acquires information about the person. [Prior art documents] [Patent documents]

[0011]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0012] The method described in Patent Document 1 is performed in the following procedure. (1) The user takes a picture of the interview partner with a camera provided in an HMD (Head Mounted Display), which is a portable information terminal. (2) By performing face image authentication processing on the captured image, the face image is identified and the interview partner is specified. (3) Information about the specified interview partner is acquired (obtained). (4) The information about the acquired interview partner is notified to the user.

[0013] However, in the technique described in Patent Document 1, the user needs to face the interview partner, and after such a face-to-face meeting, information about the interview partner is acquired, so there is a time lag until the information is acquired. In other words, in the technique described in Patent Document 1, the user needs to determine that "this person is (presumably) my interview partner". As a result, the user starts the interview in a state where there is no or insufficient information about the partner, and inconveniences such as the conversation not going well at the beginning of the face-to-face meeting occur. In particular, when the interview partner recognizes information about the user (name, occupation, etc.) before the user, in addition to the above-mentioned time lag, extra lost time occurs until the user recognizes that "this person is (presumably) my interview partner".

[0014] An object of the present invention is to provide a portable information terminal and an information processing method capable of more quickly providing information about an interview partner to a user.

Means for Solving the Problems

[0015] Among the inventions disclosed in the present application, the outline of representative ones will be briefly described as follows.

[0016] According to a representative embodiment of the present invention, a mobile information terminal determines whether a person around the user can be an interviewee by analyzing the behavior of the person, and if it is determined that the person can be an interviewee, the mobile information terminal acquires the supplementary information regarding the person. As a result, when the user recognizes the person as an interviewee, the user knows the supplementary information regarding the interviewee.

Effect of the Invention

[0017] Among the inventions disclosed in the present application, the effects obtained by representative ones will be briefly described as follows.

[0018] That is, according to a representative embodiment of the present invention, the mobile information terminal automatically determines whether a person around the user can be an interviewee, and if it is determined that the person can be an interviewee, the supplementary information regarding the person is acquired. Therefore, the information of the interview partner can be provided to the user more quickly.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22A

Figure 22B

Figure 23

Embodiments for Carrying Out the Invention

[0020] Hereinafter, specific examples of embodiments to which the present invention is applied will be described in detail with reference to the drawings. Each of the embodiments described below is an example for realizing the present invention and does not limit the technical scope of the present invention. In the embodiments, members having the same function are denoted by the same reference numerals, and repeated descriptions thereof are omitted unless particularly necessary.

[0021] <<Embodiment 1>> First, with reference to FIGS. 1 to 3, Embodiment 1 according to the present invention will be described. Here, FIG. 1 is a schematic diagram for explaining the outline of Embodiment 1. FIG. 2 is an external view showing an example of the HMD in Embodiment 1. Further, FIG. 3 is a system configuration diagram showing an example of the internal configuration of the HMD in Embodiment 1.

[0022] In FIG. 1, a scene in which a single person 15 is in front of a user 10 wearing the glasses-shaped transmissive HMD1 is schematically shown. For the sake of convenience of explanation, in FIG. 1, the transmissive HMD1 is shown separated from the user 10, but in reality, it is assumed that the user 10 is looking forward while wearing the HMD1 on the front surface of the head (glasses-wearing position). Further, hereinafter, the person 15 may be referred to as the "interviewer 15".

[0023] The transmissive HMD1 of the present embodiment includes a semi-transparent (transmissive) display screen 75 (display unit) at the lens position of the glasses. The user can visually recognize the real space through such a transmissive display screen 75. Further, an AR object (interviewer information) of augmented reality can be displayed on the display screen 75. Therefore, the wearer of the HMD1 (user 10 in this example) can simultaneously visually recognize both the AR object (interviewer information) of augmented reality displayed on the display screen 75 and the situation of the real space.

[0024] In Fig. 1, there is no interview partner 15 in the line-of-sight direction 19 of user 10, indicating a state where user 10 does not recognize the presence of interview partner 15. More specifically, since user 10 is wearing HMD1, the field of view of user 10 in the real space other than the line-of-sight direction 19 is slightly narrower than the field of view when viewing with the naked eye. On the other hand, interview partner 15 recognizes the presence of user 10 earlier than user 10 recognizes the presence of interview partner 15, and is in a state of raising the right hand.

[0025] In the present embodiment, for example, in the scenario shown in Fig. 1, after being activated, HMD1 quickly acquires the surrounding information of HMD1 using the surrounding information acquisition device. In one specific example, the "surrounding information" is any one of video, distance measurement information, sound, or a combination thereof.

[0026] In relation to the blocks shown in Fig. 3, the surrounding information acquisition device corresponds to a sensor unit 5 including an imaging unit 71 that acquires the video around HMD1 (and thus user 10 who carries HMD1), a distance measurement sensor 55 that acquires distance measurement data (the distance between the user and the object), a human sensor 56 that more senses the presence and approach of people, etc., and an audio input unit 81 such as a microphone that picks up (acquires) the sound around the user. HMD1 recognizes the person 15 who is (or has appeared) around the user by analyzing the surrounding information acquired through the above-mentioned surrounding information acquisition device.

[0027] Note that in Fig. 1 according to Embodiment 1, only the imaging unit 71 is used as the surrounding information acquisition device, and an example is shown in which the imaging unit 71 acquires the video around the user as the surrounding information. In contrast, in Embodiment 2 described later, the imaging unit 71 and the audio input unit 81 are used as the surrounding information acquisition device. Also, in Embodiment 3 described later, only the audio input unit 81 is used as the surrounding information acquisition device.

[0028] HMD1 determines whether the person 15 can be an interview partner by the action analysis processing unit 74, and if it is determined that the person 15 can be an interview partner, the incidental information regarding the person 15 is acquired. The acquired incidental information is presented to the user through the information presentation unit.

[0029] The presentation of this additional information is made by either or both of an image and a sound. When the information is presented by an image, the information presentation unit corresponds to the display screen 75 controlled by the display unit 72. On the other hand, when the information is presented by a sound, the information presentation unit corresponds to the sound output unit 82.

[0030] FIG. 1 shows an example in which additional information about the person 15 is displayed on the display screen 75. That is, in FIG. 1, as the additional information about the person 15, the name 18 (in this example, "Taro Yamada") is displayed on the display screen 75.

[0031] In this specification, the terms "video" and "image" are assumed to include both moving images and still images.

[0032] As shown in FIG. 1, the HMD1 is configured to be connectable to the network server 32 on the network network 33. More specifically, the HMD1 is connected to the access point 31 by the communication processing unit 6 described later in FIG. 2, and communicates with the network server 32 connected to the network network 33 via the access point 31. The network server 32 shown in FIG. 1 includes various servers such as a processing server that performs various arithmetic processes and a data server that stores various data. Therefore, the HMD1 can utilize various external resources by communicating with the above servers as necessary.

[0033] FIG. 2 is an external view showing an example of the HMD1 used in the present embodiment. The display screen 75 is configured at the left and right lens positions of the glasses, and the right camera 711 is arranged at the end of the right lens position of the glasses, and the left camera 712 is arranged at the end of the left lens position of the glasses.

[0034] Although the microphone is not shown, it is arranged near the right camera 711 and near the left camera 712. Also, the right speaker 821 and the left speaker 822 are arranged at the part corresponding to the temple of the glasses.

[0035] In addition, the electronic components such as the circuits of the HMD1 are separately stored in the right housing 111 and the left housing 112.

[0036] Hereinafter, the problems in the present disclosure, that is, the specific methods for more quickly providing the information of the interview partner to the user will be described in more detail with reference to the drawings.

[0037] [Example of HMD system configuration] The main body of the HMD1 used in the present invention is composed of various blocks described below.

[0038] FIG. 3 is a system configuration diagram showing an example of the internal configuration of the above-described HMD1. As shown in FIG. 3, the HMD1 includes a main control unit 2, a system bus 3, a storage unit 4, a sensor unit 5, a communication processing unit 6, a video processing unit 7, an audio processing unit 8, an operation input unit 9, and the like.

[0039] The main control unit 2 is a microprocessor unit that controls the entire HMD1 according to a predetermined operation program. The system bus 3 is a data communication path for transmitting and receiving various commands, data, etc. between the main control unit 2 and each component block in the HMD1.

[0040] The storage unit 4 is composed of a program unit 41 that stores programs for controlling the operation of the HMD1, a data storage unit 42 that stores various data such as operation setting values, detection values from sensors, objects including contents, and library information downloaded from libraries, and a rewritable program function unit 43 such as a work area used in various program operations.

[0041] In addition, the storage unit 4 can store operation programs downloaded from the network, various data created by the operation programs, etc. Also, it can store contents such as videos, still images, and audio downloaded from the network. Further, it can store data such as videos and still images captured using the camera's shooting function. Moreover, the storage unit 4 can store necessary information (setting values such as thresholds and image data, etc.) in advance.

[0042] In addition, the storage unit 4 needs to retain the stored information even when no external power is supplied to the HMD1. Therefore, for the storage unit 4, devices such as semiconductor element memories such as flash ROMs and SSDs (Solid State Drives), and magnetic disk drives such as HDDs (Hard Disc Drives) are used. Note that each operation program stored in the storage unit 4 can be updated and functionally expanded by download processing from each server device on the network.

[0043] The sensor unit 5 is a group of various sensors (in other words, a "sensor device") for detecting the state of the HMD1. The sensor unit 5 is composed of a GPS (Global Positioning System) receiver 51, a geomagnetic sensor 52, an acceleration sensor 53, a gyro sensor 54, a distance measurement sensor 55, a human presence sensor 56, etc.

[0044] Through the various sensors described above, the sensor unit 5 can detect the position, inclination, direction, movement, etc. of the HMD1, and can also measure the distance to an object (interview partner, various other things). For this reason, the sensor unit 5 constitutes a part of a surrounding information acquisition device that acquires surrounding information including information on the interview partner.

[0045] Among the above, the distance measurement sensor 55 is, for example, of the optical ToF (Time of Flight) type, and measures the distance to objects (persons and accessories of persons (such as glasses, hats, canes, flags, clothes, masks, etc.), and buildings, roads, etc.) around the HMD 1 and the user 10 (hereinafter, may simply be referred to as the surroundings for simplicity).

[0046] For simplicity, hereinafter, "the surroundings of the HMD 1 and the user 10" may simply be referred to as "the surroundings".

[0047] Also, the human presence sensor 56 is, for example, of the infrared type, and can selectively detect a person (human) among the various objects existing in the surroundings.

[0048] In addition, the GPS (Global Positioning System) receiver 51 can obtain the location of the HMD 1, in other words, the location where the surrounding information is obtained, by using satellite communication to obtain the current location information. Also, in order to obtain the current location information, other systems, for example, other systems in the GNSS (Global Navigation Satelite System) may be used.

[0049] Note that the sensor unit 5 may further include other sensors, for example, detection or measurement devices such as an illuminance sensor and an altitude sensor, and these sensors can also be components of the surrounding information acquisition device.

[0050] The communication processing unit 6 is a communication device including a LAN (Local Area Network) communication unit 61, a telephone network communication unit 62, etc. Among these, the LAN communication unit 61 is connected to a network network 33 such as the Internet (refer to FIG. 1 as appropriate) via an access point 31 or the like, and transmits and receives data to and from each network server 32 on the network network 33. The connection between the LAN communication unit 61 and the access point 31 or the like is performed by wireless communication such as Wi-Fi (registered trademark).

[0051] Note that the main control unit 2 can cause an external server (network server 32) to perform at least a part of the characteristic processing performed by the HMD 1 via the communication processing unit 6 (communication device).

[0052] The telephone network communication unit 62 performs telephone communication (call) and data transmission / reception by wireless communication with a base station or the like of a mobile telephone communication network. The communication with the base station or the like may be performed by an LTE (Long Term Evolution) method, a 5G method (a fifth-generation mobile communication system aiming at high speed, large capacity, low latency, and multiple simultaneous connections), or other communication methods.

[0053] The LAN communication unit 61 and the telephone network communication unit 62 each include an encoding circuit, a decoding circuit, an antenna, and the like. Further, the communication processing unit 6 may further include other communication units such as an infrared communication unit.

[0054] The video processing unit 7 includes an imaging unit 71, a display unit 72, a face information processing unit 73, and a behavior analysis processing unit 74.

[0055] The imaging unit 71 is a camera that inputs image data (video) of the surroundings or an object by converting light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. In the present embodiment, the imaging unit 71 includes a right camera 711, a left camera 712, and the like.

[0056] Such an imaging unit 71 (right camera 711, left camera 712, etc.) constitutes a part (imaging device or image acquisition device) of a surrounding information acquisition device that acquires surrounding information including information on the interview partner.

[0057] The display unit 72 is, for example, a display device (liquid crystal display device) of a transmissive type display using a semi-transparent liquid crystal. The display unit 72 constitutes a display screen 75 (refer to FIGS. 22A, 22B, and 23 as appropriate), and provides incidental information about the interview partner or the like to the user 10 of the HMD 1.

[0058] The face information processing unit 73 is a processing unit that extracts face information from the video of the interviewee captured by the imaging unit 71. Details of the processing mainly executed by the face information processing unit 73 will be described later.

[0059] The behavior analysis processing unit 74 is a processing unit that performs behavior analysis on a person from the video of the person captured by the imaging unit 71 and the distance to the person measured by the distance measurement sensor 55. Details of the processing mainly executed by the behavior analysis processing unit 74 will be described later.

[0060] In one specific example, the face information processing unit 73 and the behavior analysis processing unit 74 are each composed of separate processors. As another example, these processing units 73 and 74 may be composed of the same processor.

[0061] The voice processing unit 8 is composed of a voice input unit 81 and a voice output unit 82.

[0062] The voice input unit 81 is a microphone (hereinafter may be abbreviated as a mic) that converts sounds in the real space, the user's voice, etc. into voice data and inputs them. In the present embodiment, mics are respectively arranged near the right camera 711 and the left camera 712.

[0063] The voice input unit 81 constitutes a part (a sound collection device or a voice acquisition device) of a surrounding information acquisition device that acquires surrounding information including information on the interview partner.

[0064] The voice output unit 82 is a speaker that outputs necessary voice information and the like to the user. In the present embodiment, the voice output unit 82 arranges a right speaker 821 and a left speaker 822 at positions close to the user's ears. Although not shown, the voice output unit 82 may be provided with wired or wireless terminals for connecting external voice output devices such as earphones and headphones. According to the HMD1 having such a configuration, the voice output method or path can be appropriately used according to the application and the like.

[0065] The operation input unit 9 is a hardware device equipped with key switches and the like for inputting operation instructions and the like to the HMD 1, and outputs an operation input signal according to the operation content (input instruction) of the user to the main control unit 2.

[0066] In the present disclosure, the operation input unit 9 and the main control unit 2 function as a setting unit or a setting processing device for setting characteristics functions (for example, surrounding information acquisition, behavior analysis processing, information presentation, etc.) in the HMD 1. As other components of the setting unit or the setting processing device, the above-described display unit 72 may be included.

[0067] Note that in the hardware configuration example of the HMD 1 shown in FIG. 3, components and the like that have low relevance to the configuration for solving the above-described problems are also included. Therefore, even a configuration that does not include components and the like with low relevance to problem solving does not impair the specific effects in the present embodiment. Further, a configuration not shown, such as an electronic money settlement function, may be further added.

[0068] [Functional Blocks of the Present Embodiment] FIG. 4 is a functional block diagram showing an example of the functional block configuration in the HMD 1 of the present embodiment. The control function 21 is a function for controlling the entire HMD 1, and as shown in FIG. 4, it is mainly composed of the main control unit 2, the program unit 41 and the program function unit 43 of the storage unit 4.

[0069] Among the above, the communication processing function 22 is a function for performing communication processing for connecting to the network network 33 by the LAN communication unit 61 of the communication processing unit 6 or the telephone network communication unit 62 of the communication processing unit 6 (also refer to FIGS. 1 and 3 as appropriate).

[0070] The photographing data acquisition function 23 is a function for photographing an interviewee by the imaging unit 71 (right camera 711, left camera 712) of the video processing unit 7 and acquiring photographing data.

[0071] The face information processing function 24 is a function that analyzes face information from the video of the interviewee obtained by the shooting data acquisition function 23 by the face information processing unit 73 to identify the interviewee. Details of the face information processing will be described later.

[0072] The face information storage function 25 is a function that stores the face information for identifying the interviewee obtained by the face information processing function 24 in the data storage unit 42 of the storage unit 4.

[0073] The interviewee information storage function 26 is a function that stores the additional information regarding the interviewee in the data storage unit 42 of the storage unit 4.

[0074] The interviewee information output function 27 is a function that reads out the additional information regarding the interviewee stored in the interviewee information storage function 26 and displays it on the display unit 72 of the video processing unit 7.

[0075] The behavior analysis processing function 30 is a function that analyzes the behavior of a person by the behavior analysis processing unit 74 from the video of the person obtained by the shooting data acquisition function 23 and the distance between the person and the person obtained by the distance measurement data acquisition function 1000, and determines whether the person becomes an interview candidate. Details of the interview candidate determination processing will be described later.

[0076] [Processing procedure of the present embodiment] FIG. 5 is a flowchart showing the procedure of a new interviewee process (step S400) for acquiring information of a new interviewee in the present embodiment. Hereinafter, the processing procedure shown in FIG. 5 will be described with reference to the functional block diagram of FIG. 4.

[0077] In executing the new interviewee process (step S400), it is premised that the consent of the new interviewee has been obtained in advance from the viewpoint of personal information protection.

[0078] The novel interviewer processing shown in FIG. 5 (step S400) is executed, for example, by the following procedure. That is, after the start processing (step S401) such as software startup or memory reset, first, a novel interviewer is photographed (step S402). This corresponds to preprocessing for acquiring face information from the video of the novel interviewer.

[0079] Specifically, in step S402, under the control of the main control unit 2, the imaging unit 71 of the video processing unit 7 operates, and thus the background or subject in front of the user of the HMD1 is photographed. Hereinafter, the case where there is a novel interviewer in the subject in front of the user will be described on the premise.

[0080] Next, face information detection processing (step S420), which is a defined processing (subroutine), is performed. This face information detection processing (step S420) is processing for acquiring the face information of the novel interviewer. Specifically, in step S420, under the control of the main control unit 2, the face information processing unit 73 of the video unit processing 7 analyzes the image of the subject photographed in step S402, thereby acquiring the face information of the novel interviewer. By this processing, face information for identifying the novel interviewer is obtained.

[0081] Here, a more detailed processing procedure of step S420 (face information detection processing) will be described with reference to FIG. 6. FIG. 6 is a flowchart showing the processing procedure of the subroutine of face information detection processing (step S420).

[0082] Hereinafter, the processing procedure shown in FIG. 6 will be described with reference to the block diagram of FIG. 3 and the functional block diagram of FIG. 4. In order to perform the face information processing function 24, the face information processing unit 73 reads out and sequentially executes the program of the face recognition method stored in the program unit 41 of the storage unit 4.

[0083] Specifically, after the start processing (step S421) such as software startup or memory reset, the face information processing unit 73 first performs a process of detecting the face contour of the novel interviewer in the photographed frame by the face contour detection program (step S422).

[0084] In the subsequent step S423, the face information processing unit 73 determines whether the face contour of the new interviewee has been detected by the face contour detection process (step S422).

[0085] Here, when the face information processing unit 73 determines that the face contour of the new interviewee has not been detected (step S423: NO), it proceeds to the face detection error setting process (step S428) for setting a face detection error.

[0086] On the other hand, when the face information processing unit 73 determines that the face contour of the new interviewee has been detected (step S423: YES), it proceeds to the face element detection process (step S424).

[0087] In the face element detection process (step S424), the face information processing unit 73 performs a process of detecting face elements such as eyes, nose, mouth, etc. inside the face contour by the face element detection unit program.

[0088] In the subsequent step S425, the face information processing unit 73 determines whether the face elements of the new interviewee have been detected by the face contour detection process (step S424).

[0089] Here, when the face information processing unit 73 determines that the face elements of the new interviewee have not been detected by the face contour detection process (step S424) (step S425: NO), it proceeds to the face detection error setting process (step S428) for setting a face detection error.

[0090] On the other hand, when the face information processing unit 73 determines that the face elements of the new interviewee have been detected by the face contour detection process (step S424), it proceeds to the next face feature amount detection process (step S426).

[0091] In the face information feature amount detection process (step S426), the face information processing unit 73 performs a process of detecting face feature amounts such as the size, position of each element, and the positional relationship between elements by the face feature amount detection program.

[0092] In the subsequent step S427, the face information processing unit 73 determines whether the face feature amount of the new interviewee has been detected by the face information feature amount detection process (step S426).

[0093] Here, when the face information processing unit 73 determines that the face feature amount of the new interviewee has not been detected by the face information feature amount detection process (step S426) (step S427: NO), it proceeds to the face detection error setting process (step S428) for setting a face detection error.

[0094] On the other hand, when the face information processing unit 73 determines that the face elements of the new interviewee have been detected by the face information feature amount detection process (step S426) (step S427: YES), it ends the face information detection process (step S420) (step S429).

[0095] Also, in the face detection error setting process (step S428), the face information processing unit 73 specifies at which stage the face detection error occurred and ends the face information detection process (step S420) (step S429).

[0096] As another example, the above-described face information detection process (step S420) may be configured to be performed by the network server 32. In this case, the main control unit 2 of the HMD1 controls the communication processing unit 6 to transmit the video of the new interviewee captured by the video processing unit 7 (imaging unit 71) to the network server 32 that performs the face information detection process via the network network 33. Subsequently, the main control unit 2 of the HMD1 receives only the detection result of the face information executed by the network server 32 from the network server 32 via the network network 33.

[0097] Here, returning to the description of the processing procedure of the flowchart in FIG. 5, in step S403 following the above-described face information detection process (step S420), the face information processing unit 73 determines whether the face information of the new interviewee has been acquired by the face information detection process (step S420).

[0098] Here, when the face information processing unit 73 determines that the face information of the new interviewee cannot be obtained by the process of the face information detection process (step S420) (step S403: NO), it determines that there is no face information to be saved, and proceeds to the new interviewee information processing (step S405).

[0099] On the other hand, when the face information processing unit 73 determines that the face information of the new interviewee has been obtained by the process of the face information detection process (step S420) (step S403: YES), it proceeds to the face information saving process (step S404).

[0100] In the face information saving process (step S404), the face information processing unit 73 saves the face feature amount of the interviewee related to the face recognition of the interviewee in the data storage unit 42 of the storage unit 4 by performing the face information saving function 25 (see FIG. 4). Next, the face information processing unit 73 proceeds to the new interviewee information acquisition process (step S405).

[0101] The new interviewee information acquisition process (step S405) is a process of acquiring the additional information of the new interviewee. In this step S405, the face information processing unit 73 performs a process of acquiring the additional information about the new interviewee, such as the name and age of the new interviewee.

[0102] Next, the face information processing unit 73 determines whether the additional information about the new interviewee has been obtained by the new interviewee information acquisition process (step S405) (step S406).

[0103] Here, when the face information processing unit 73 determines that the additional information about the new interviewee cannot be obtained (step S406: NO), it ends the new interviewee processing (step S400) shown in FIG. 5 (step S408).

[0104] On the other hand, when the face information processing unit 73 determines that the additional information about the new interviewee has been acquired (step S406: YES), it proceeds to step S407. In step S407, the face information processing unit 73 saves the acquired additional information about the new interviewee in the data storage unit 42 of the storage unit 4 by executing the interviewee information storage function 26 (see FIG. 4), and then ends the new interviewee process (step S400) shown in FIG. 5 (step S408).

[0105] In addition, when the additional information about the new interviewee is stored in the network server 32, the HMD1 can also acquire the additional information about the new interviewee from the network server 32 via the network network 33 under the control of the main control unit 2. Also in this case, after the face information processing unit 73 saves the additional information about the new interviewee acquired from the network server 32 in the data storage unit 42 of the storage unit 4, it ends the new interviewee process (step S400) (step S408).

[0106] FIG. 7 is a table showing an example of the interviewee information stored in the data storage unit 42 of the storage unit 4.

[0107] The interviewee table (T840) shown in FIG. 7 has a configuration in which a person column 860 indicating the type of person (such as an interviewee) is associated with an item column 850 indicating information (items) about the user and each interviewee (interviewee 1, interviewee 2,... interviewee n).

[0108] Among the above, the item column 850 is composed of two items: face information 851 and additional information 852 about the interviewee.

[0109] On one hand, in the person column 860, in addition to the interviewers (862 to 864), the user 861 is registered. Note that the presence of the user 861 in the person column 860 that originally indicates the type of interviewer is, figuratively speaking, something like a profile in a mobile phone. Also, the face information 851 of the user (861) can be obtained, for example, by photographing using a mirror (in this case, an image with left and right reversed is obtained), or by so-called "selfie", etc.

[0110] The interviewer table (T840) as shown in FIG. 7 can also be stored in the network server 32 via the network network 33. In particular, regarding the face feature amount, by utilizing the network server 32 specialized in the process of identifying the interviewer from the face feature amount, the speed of the process of identifying the interviewer can be increased.

[0111] In the present embodiment, by using the processes described above with reference to FIGS. 5 and 6 and the table having the data structure as shown in FIG. 7, the face information of the new interviewer and the additional information regarding the new interviewer can be obtained and stored in the HMD1.

[0112] 〔Interviewer Identification / Information Acquisition Process〕 Next, the process of identifying the interviewer and acquiring the additional information regarding the interviewer will be described. FIG. 8 is a flowchart showing the procedure of the process (interviewer identification / information acquisition process) of determining whether a person can be an interviewer and, if so, acquiring the additional information regarding the interviewer in advance, which is the essence of the present embodiment.

[0113] The processing procedure of FIG. 8 will be described with reference to the block diagram of FIG. 3 and the functional block diagram of FIG. 4. In the following description, the main body of each process shown in FIG. 8 will be described as the main control unit 2, but a part or all of the process may be performed by the behavior analysis processing unit 74.

[0114] When the HMD1 is activated, the main control unit 2 promptly starts the process (step S431) to perform processes such as interviewer identification and information acquisition.

[0115] When the interviewer identification and information acquisition process starts (step S431), the main control unit 2 first performs surrounding shooting processing (step S432). This is a process of shooting the environment (scenery or view) around the HMD1, that is, around the user 10, by the shooting data acquisition function 23 to acquire shooting data.

[0116] Here, the shooting data to be acquired may be a video or a still image. When the shooting data is a video, it can be expected that the accuracy of action analysis will be higher compared to the case of a still image. On the other hand, when the shooting data is a still image, it can be expected that the power consumption of the HMD1 will be suppressed compared to the case of a video. Note that from the viewpoint of ensuring the accuracy of action analysis to a certain level or more, when the shooting data is a still image, it is good to capture images, that is, acquire still images, at a predetermined period.

[0117] To execute the above processing, the main control unit 2 controls the video processing unit 7 to start imaging by the imaging unit 71. At this time, the video processing unit 7 captures an image by the imaging unit 71 (camera), analyzes the captured image by the face information processing unit 73 and the action analysis processing unit 74, and outputs the analysis result to the main control unit 2.

[0118] In step S433 after receiving the analysis result from the video processing unit 7, the main control unit 2 determines whether a person exists in the acquired image (hereinafter referred to as "shooting data").

[0119] Here, when the main control unit 2 determines that no person exists in the shooting data (step S433, NO), it proceeds to the end instruction determination process in step S434.

[0120] On the other hand, when the main control unit 2 determines that a person exists in the shooting data (step S433, YES), it proceeds to the interview candidate determination process in step S900.

[0121] In step S900 (interview candidate determination process), the main control unit 2 determines whether a person around the user can be an interviewer. This determination is made based on the action analysis result of the person in the captured data by the action analysis processing unit 74. When the attention of that person is directed towards the user, it is determined that the person can be an interviewer. For this determination, additional shooting of a video or a still image may be performed.

[0122] Here, the conditions for determining that a person's attention is directed towards the user include, for example, the following actions (person's behavior). (Condition 1) The direction of the person's line of sight is towards the user. (Condition 2) The person is performing a greeting action such as raising a hand to the user. (Condition 3) The person is approaching the user. (Condition 4) The person is calling the user's name or uttering their own name (or the name of the company or organization to which the person belongs, etc.).

[0123] Among the above, since Condition 4 is an action based on the person's voice (sound), it is considered that it is not always easy to obtain (extract) from an image. More specifically, for example, by analyzing the lip movement of a person in a moving image, the content (spoken sound) uttered by the person can be estimated. On the other hand, in the current social situation, the number of cases where people wear masks for the purpose of preventing infection against various diseases (such as the novel coronavirus) is increasing. In this case, it is considered that it becomes difficult to analyze the lip movement of a person.

[0124] Taking the above circumstances into consideration, in Embodiment 1, mainly Conditions 1 to 3 are considered, and Condition 4 will be described in Embodiments 2 and 3.

[0125] The above conditions 1 to 3 are actions based on the movements of a person's body and can generally be defined as "actions indicating interest in the user". Therefore, when the action of the person included in the shooting data (surrounding information) indicates interest in the user 10, the main control unit 2 (action analysis processing device) determines that the person can be an interviewer (step S900, YES).

[0126] Hereinafter, a person (person) who can be an interviewer will be appropriately referred to as an "interview candidate" for explanation.

[0127] Note that the above conditions 1 to 3 only show a part of the examples of "actions indicating interest in the user", and in actual operation, various other conditions (modes of the person's actions) can be added.

[0128] Furthermore, as an exceptional process (judgment criterion), the main control unit 2 may determine that the person is an interview candidate (step S900, YES) regardless of the above conditions when, for example, based on the detection result of the human sensor 56 described above, the distance between the user and the person approaches a predetermined fixed distance. This is because, for example, when the user 10 is wearing a mask, etc., it is possible that the other person only notices the identity of the user 10 when coming close to the user 10.

[0129] Also, as another exceptional process (judgment criterion), the main control unit 2 may perform a process of not considering the person as an interview candidate, that is, determining NO in step S900, even when the person's attention is directed to the user (for example, even when all of conditions 1 to 3 are satisfied).

[0130] This is considered, for example, in the case of clerks in stores, receptionists in facilities such as receptionists in company receptions, security guards, etc., who are only showing interest in the user as a professional response. More specifically, it is usually the case that no additional information is registered for these people, and performing the process of acquiring additional information for these people may prevent the acquisition of additional information for truly necessary people.

[0131] Also, from the perspective of prioritizing the acquisition of supplementary information for truly necessary persons, in a specific place such as one's own home where one only meets known persons, the process shown in FIG. 8 may not be performed (the function is automatically stopped). In this case, the main control unit 2 may determine whether it is a "specific place" based on the reception information of the GPS reception unit 51 (see FIG. 3).

[0132] Furthermore, from the perspective of prioritizing the acquisition of supplementary information for truly necessary persons, for persons who frequently meet, such as family members, they may be excluded from the processing targets in steps S420 and S450 (so-called "exclusion target persons" setting). Also, for persons who have undergone the processing in steps S420 and S450 once, the processing in steps S420 and S450 may not be performed for a certain period (so-called "display stop period" setting).

[0133] The above-described various exception processing settings (so-called exclusion settings) can be made, for example, by the user operating the operation input unit 9. By performing the above-described various exclusion settings or exclusion processing, unnecessary information presentation can be suppressed, contributing to quickly obtaining the supplementary information for truly necessary persons, thereby improving convenience.

[0134] If the main control unit 2 determines in the interview candidate determination process of step S900 that the persons around are not interview candidates, it proceeds to the end instruction determination process of step S434.

[0135] In step S434, the main control unit 2 monitors, for example, the input signal from the operation input unit 9 to determine whether the user 10 or the like has instructed the end of the processing of this embodiment.

[0136] Here, when the main control unit 2 determines that the end of the processing has been instructed (step S434: YES), it ends the routine shown in FIG. 8 (interview candidate determination / information acquisition processing) (step S436).

[0137] On the other hand, when the main control unit 2 determines that the processing has not yet been instructed to end (step S434: NO), it returns to the surrounding imaging process (step S432) that captures the surroundings of the HMD 1 in order to continue the routine shown in FIG. 8.

[0138] Thus, when the main control unit 2 determines in the interviewee candidate determination process of step S900 that the person in the surroundings is an interviewee candidate (step S900: YES), it performs a face information detection process (step S420), which is a defined process (subroutine).

[0139] Note that even before acquiring the information of the interviewee candidate, the main control unit 2 may perform a display 1100 indicating that the mobile information terminal has recognized the interviewee candidate (in the illustrated example, a message display saying "There is an interviewee candidate"), as shown in FIG. 22A. At that time, the main control unit 2 may superimpose a mark 1101 indicating the person to inform the user of the position of the interviewee candidate. Regarding the mark 1101, in the illustrated example, a figure surrounding the person 15 as the interviewee candidate is displayed, but other figures such as an "arrow" indicating the person 15 may also be used.

[0140] Also, since the details of the face information detection process (step S420) have been described in the flowchart of FIG. 6, the description here is omitted.

[0141] After the end of the face information detection process (step S420), the main control unit 2 performs an interviewee information process (step S450), which is a defined process (subroutine). This interviewee information process (step S450) is a process of identifying the interviewee and acquiring additional information regarding the corresponding interviewee.

[0142] The main control unit 2 waits for the end of the interviewee information process (step S450) and then ends the interviewee identification / information acquisition process of the present embodiment (step S436).

[0143] Here, the more specific content of the interviewee information processing (the processing of the subroutine in step S450) will be described. FIG. 9 is a flowchart showing the processing procedure of the interviewee information processing (step S450) which is a subroutine. Regarding the processing procedure shown in FIG. 9, reference will be made to the hardware block diagram of FIG. 3 and the functional block diagram of FIG. 4 as appropriate for explanation.

[0144] When the processing of step S450 (interviewee information processing) starts (step S451), the main control unit 2 first determines whether the face information detected in the face information detection process (step S420) is the face information of a known interviewee (step S452).

[0145] In this example, the main control unit 2 compares the face information (face feature amount) detected in the face information detection process (step S420) with the face information (face feature amount) stored by the face information storage function 25, and if they are very similar (for example, when the degree of coincidence of the outer shape (outline) of the face is within a preset threshold), it discriminates as a known interviewee. In recent years, since the number of people wearing masks has increased for preventing infectious diseases and the like, for mask wearers, the main control unit 2 determines whether the degree of coincidence of the outer shape (outline) of the face other than the mask part among the outer shapes (outlines) of the face is within the above threshold.

[0146] When the main control unit 2 determines NO in step S452, that is, when the detected face information (face feature amount) and the stored face information (face feature amount) do not match, or when in the face information detection process (step S420), face information sufficient to identify a person cannot be detected, it determines that the person is not a known interviewee and proceeds to step S400.

[0147] On the other hand, when the main control unit 2 determines YES in step S452, that is, when the detected face information (face feature amount) and the stored face information (face feature amount) match, it proceeds to step S453. In step S453, the main control unit 2 acquires the additional information regarding the known interviewee stored by the interviewee information storage function 26 and proceeds to step S454.

[0148] In step S454, the main control unit 2 determines whether it is necessary to correct the additional information regarding the known interviewee.

[0149] Here, when the main control unit 2 determines that it is not necessary to correct the additional information regarding the known interviewee (step S454: NO), it proceeds to the interviewee information output process (step S457).

[0150] On the other hand, when the main control unit 2 determines that it is necessary to correct the additional information regarding the known interviewee (step S454: YES), it proceeds to the interviewee correction information storage process (step S455) for storing the corrected interviewee information.

[0151] In the interviewee correction information storage process (step S455), the main control unit 2 corrects the interviewee information stored by the interviewee information storage function 26 and stores the corrected interviewee information. After the completion of the interviewee correction information storage process (step S455), the main control unit 2 proceeds to the interviewee information output process (step S457).

[0152] On the contrary, in the determination process of step S452, when it is determined that the interviewee is not a known interviewee (step S452: NO), since there is no information regarding the interviewee, it is necessary to newly obtain the information regarding the interviewee. Therefore, the main control unit 2 performs the new interviewee process (step S400) of the present embodiment. Note that the details of the new interviewee process (step S400) have been described in the flowchart of FIG. 5, so the description here is omitted.

[0153] Next, the main control unit 2 determines whether the interviewee information has been obtained in the new interviewee process (step S400) (step S456).

[0154] Here, when the main control unit 2 determines that new interviewee information has been obtained (step S456: YES), it proceeds to the interviewee information output process (step S457). In the new interviewee process (step S400), if additional information regarding the interviewee has been acquired, since the new interviewee information has already been saved, it is possible to proceed to the interviewee information output process (step S457).

[0155] In the interviewee information output process (step S457), the main control unit 2 outputs the additional information regarding the interviewee to the outside through the interviewee information output function 27. In the present embodiment, the main control unit 2 displays and outputs the information 1102 of the interviewee on the display unit 72 of the video processing unit 7 (see FIG. 22B).

[0156] After the completion of the interviewee information output process (step S457), the main control unit 2 ends the interviewee information process (step S450) (step S458). Also, when the main control unit 2 determines in the determination process of step S456 that no interviewee information has been obtained, it also ends the interviewee information process (step S450) (step S458).

[0157] In the present embodiment, the face information of the new interviewee and the additional information regarding the new interviewee are acquired and saved in advance. Then, before the user recognizes, the main control unit 2 determines whether a person in the vicinity can be the interviewee through the behavior analysis of that person. If it is determined that the person can be the interviewee, the main control unit 2 presents the character information such as the name of that person as additional information to the user 10 by displaying it on the display screen 75 (see FIG. 22B, etc.).

[0158] As another example, the additional information to be saved and presented may be graphic information such as an illustration, or may be voice information using the output from the right speaker 821 or the left speaker 822.

[0159] As described above, the HMD1 of Embodiment 1 (portable information terminal) includes a surrounding information acquisition device (sensor unit 5, imaging unit 71, voice input unit 81) that acquires the surrounding information of the terminal and the user 10, and analyzes the actions of the people included in the acquired surrounding information to determine whether there is an interview candidate (a person who intends to interview the user) for the user 10. It is provided with an action analysis processing device (main control unit 2, action analysis processing unit 74) and an information presentation device (display unit 72) that presents the user 10 with additional information corresponding to the person when it is determined that there is an interview candidate.

[0160] According to such an HMD1, it is possible to provide the user 10 with the additional information of the interview partner more quickly, and when the user 10 recognizes the person as an interviewer, the user 10 knows the additional information about the interviewer.

[0161] Therefore, according to the HMD1 of the present disclosure, it is possible to effectively prevent the problem of time lag that has been a problem in conventional devices, that is, starting an interview in a state where there is no or insufficient information about the other party, and causing inconveniences such as the conversation not going well at the beginning of the face-to-face meeting.

[0162] <<Embodiment 2>> Hereinafter, Embodiment 2 of the present disclosure will be described. The basic hardware configuration and software configuration of Embodiment 2 are the same as those of Embodiment 1 described above. Hereinafter, the differences between this embodiment (Embodiment 2) and the above-described Embodiment 1 will be mainly described, and the common parts will be omitted as much as possible to avoid duplication.

[0163] In the above-described Embodiment 1, the interview candidate was identified using the face information of the interviewer. In contrast, in this embodiment, the voice information of the person is taken into account to identify the interview candidate. Hereinafter, this embodiment will be described.

[0164] [System configuration example of Embodiment 2] FIG. 10 is a system configuration diagram showing an example of the internal configuration of the present HMD1 used in the present embodiment. The system configuration diagram shown in FIG. 10 is substantially the same as the system configuration diagram of FIG. 3, and a voice information processing unit 83 is added to the system configuration diagram of FIG. 3. Here, the configuration of the voice information processing unit 83 will be described exclusively.

[0165] The voice information processing unit 83 performs a function of extracting voice information from the voice of the interviewee input from the voice input unit 81. In one specific example, the voice information processing unit 83 uses a hardware processor separate from the main control unit 2 and performs the above function under the control of the main control unit 2. The details of the processing executed by the voice information processing unit 83 will be described later.

[0166] [Functional Blocks of the Present Embodiment] FIG. 11 is a functional block diagram showing an example of the functional block configuration of the HMD1 of the present embodiment.

[0167] The functional block diagram shown in FIG. 11 is substantially the same as the functional block diagram shown in FIG. 4 already described, and a voice information processing function 28 and a voice information storage function 29 are added to the functional block diagram of FIG. 4. The following describes the added voice information processing function 28 and voice information storage function 29.

[0168] The voice information processing function 28 is a function of analyzing voice information by the voice information processing unit 83 from the voice of the interviewee input from the voice input unit 81 and discriminating the interviewee, and is one of the functions performed by the voice information processing unit 83 described above in FIG. 10.

[0169] The voice information storage function 29 is a function of storing the voice information for discriminating the interviewee obtained by the voice information processing function 28 in the data storage unit 42 of the storage unit 4.

[0170] [Processing Procedure of Embodiment 2] FIG. 12 is a flowchart showing the procedure of a new interviewee process (step S460) for obtaining information taking into account the voice information of a new interviewee in the present embodiment. Hereinafter, the processing procedure of FIG. 12 will be described with reference to the functional block diagram of FIG. 11.

[0171] In executing this new interviewee process (step S460), from the viewpoint of personal information protection, it is desirable to obtain the prior consent of the new interviewee. However, such matters of consent do not mean technical limitations.

[0172] The flowchart showing the procedure of the new interviewee process (step S460) in FIG. 12 is almost the same as the flowchart showing the procedure of the new interviewee process (step S400) in FIG. 5. The differences are that a voice information detection process (step S470) which is a defined subroutine, a determination process (step S462) for determining the detection result of the voice information detection process (step S470), and a storage process (step S463) for storing the voice information obtained in the voice information detection process (step S470) are added. Here, only the processes added in FIG. 12 will be described.

[0173] When the new interviewee process (step S460) in the present embodiment is started (step S461), processes equivalent to those in FIG. 5 (steps S402 to S404) are executed, and the process regarding the face information is completed.

[0174] Here, the process of step S470 (voice information detection process) which is a subroutine will be described. FIG. 13 is a flowchart showing the processing procedure of the voice information detection process (step S470) which is a subroutine. Hereinafter, the processing procedure of FIG. 13 will be described with reference to the functional block diagram of FIG. 11.

[0175] The voice information processing unit 83 reads out the program of the voice recognition method stored in the program unit 41 of the storage unit 4 under the control of the main control unit 2 in order to perform the function of the voice information processing function 28 (step S471), and sequentially executes the processes from step S472 below.

[0176] When the process of step S470 (voice information detection process) starts, first, the voice information processing unit 83 determines whether a sound has been detected (step S472).

[0177] Here, when the voice information processing unit 83 determines that no sound has been detected (step S472: YES), it proceeds to the voice detection error setting process (step S477). On the other hand, when the voice information processing unit 83 determines that a sound has been detected (step S472: NO), it proceeds to step S473 (sound source separation process).

[0178] In step S473 (sound source separation process), the voice information processing unit 83 confirms the direction of sound generation and identifies (separates) the position of the sound source. In the present embodiment, the position of the mouth where the new interviewee vocalizes is the sound source position.

[0179] In the subsequent step S474, the voice information processing unit 83 determines whether the sound for which the sound source has been identified (separated) is human voice. Whether it is human voice can be determined (identified) from, for example, the frequency band of the sound, the characteristics of the waveform, etc. Since such technology is well-known, detailed description is omitted.

[0180] Here, when the voice information processing unit 83 determines that it is not human voice (step S474, NO), it proceeds to the voice detection error setting process (step S477). On the other hand, when the voice information processing unit 83 determines that it is human voice (step S474, YES), it proceeds to the voice feature amount detection process (step S475).

[0181] In the voice feature amount detection process (step S475), the voice information processing unit 83 extracts elements due to an individual (way of speaking, habit, intonation, etc.) as voice feature amounts. Note that as long as it is a method capable of identifying an individual's characteristics, other methods (for example, identifying a specific person when a specific rare language is extracted, etc.) may be used.

[0182] In the subsequent step S476, the voice information processing unit 83 determines whether it has been able to detect an individual's characteristics (in this example, voice feature amounts) from the processing result of the voice feature amount detection process (step S475).

[0183] Here, when the voice information processing unit 83 determines that it has not been able to detect the voice feature amounts (step S476, NO), it proceeds to the voice detection error setting process (step S477).

[0184] On the other hand, when the voice information processing unit 83 determines that it has been able to detect the voice feature amounts (step S476, YES), it ends the present voice information detection process (step S470) (step S478).

[0185] In the voice detection error setting process (step S477), the voice information processing unit 83 causes the display unit 72 to display so as to clarify at which stage the voice detection error has occurred. After that, it ends the voice information detection process (step S470) (step S478).

[0186] As another example of the present voice information detection process (step S470), the HMD 1 transmits the acquired voice of the new interviewer to the network server 32 that performs the voice information detection process via the network network 33, and the network server 32 can perform the voice information detection process. At that time, under the control of the main control unit 2 of the HMD 1, the above-described communication processing unit 6 receives only the detection result of the voice information from the network server 32 via the network network 33.

[0187] Also, the main control unit 2 of the HMD 1 can cause the face information detection process and the voice information detection process to be performed by different network servers 32 through the communication processing unit 6.

[0188] Here, returning to the processing procedure of FIG. 12, the description of the processing performed by the main control unit 2 or the voice information processing unit 83 is continued. The main control unit 2 (or the voice information processing unit, hereinafter the same applies to the processing subject up to S464) determines whether voice information of a new interviewee has been acquired by the voice information detection process (step S470) after the voice information detection process (step S462).

[0189] If the main control unit 2 determines in the determination process of step S462 that the voice information of the new interviewee could not be acquired by the process of the voice information detection process (step S470), since there is no voice information to be saved, the process proceeds to the new interviewee information acquisition process (step S405).

[0190] If the main control unit 2 determines in the determination process of step S462 that the voice information of the new interviewee has been acquired by the process of the voice information detection process (step S470), the process proceeds to the voice information storage process (step S463). In the voice information storage process (step S463), the main control unit 2 stores the voice feature amount of the interviewee related to the voice identification of the interviewee in the data storage unit 42 of the storage unit 4 by the voice information storage function 29. Next, the main control unit 2 proceeds to the new interviewee information acquisition process (step S405).

[0191] Regarding the processes after the new interviewee information acquisition process (step S405), the main control unit 2 performs processes (steps S406, S407) equivalent to the flowchart of FIG. 5, and ends the new interviewee process (step S460) of the present embodiment (step S464).

[0192] FIG. 14 is a table (T870) showing an example of the interviewee information stored in the present embodiment. The interviewee table (T870) shown in FIG. 14 is composed of the type 860 of the interviewee (person) and the information items 850 regarding each interviewee.

[0193] The information item 850 regarding each interviewee is composed of three elements: face information 851, voice information 853, and supplementary information 852 regarding the interviewee. In addition to the interviewees (persons) 862 - 864, there is a user 861 in the type 860 of interviewee.

[0194] The existence of the user 861 in the interviewee type has the significance of a profile in a mobile phone and the significance of separating the user's voice information from the voice of a new interviewee during a conversation with the new interviewee.

[0195] This interviewee table (T870) can be transmitted from the HMD1 to the network server 32 via the network network 33 (see Fig. 1) and stored in the storage medium of the network server 32. In particular, regarding the face feature amount and voice information, by utilizing the network server 32 specialized in the process of identifying the interviewee from the face feature amount and voice feature amount, the speed of the process of identifying the interviewee by the HMD1 can be increased.

[0196] As described above, in this embodiment, the information of a new interviewee taking voice information into account can be obtained and stored by the processes of Figs. 12 to 13 and the table of Fig. 14.

[0197] <Interviewee Identification and Information Acquisition Process> Next, the process of identifying the interviewee and acquiring the supplementary information regarding the interviewee will be described. Fig. 15 is a flowchart showing the procedure of the process (interviewee identification and information acquisition process) of determining whether a person can be an interviewee taking voice information into account, which is the essence of this embodiment, and acquiring the supplementary information regarding the interviewee in advance if the person can be an interviewee. The procedure of the process in Fig. 15 will be described with reference to the functional block diagram of Fig. 11.

[0198] The flowchart of Embodiment 2 shown in Fig. 15 is almost the same as the flowchart of Embodiment 1 described above in Fig. 8, but the content of the interviewee information process is different. In this regard, for the purpose of distinction, the interviewee information process of Embodiment 2 is indicated by a step number (step S490) different from that of Embodiment 1 (step S450).

[0199] In addition, in the second embodiment, it is different from the first embodiment in that a voice information detection process (step S470), which is a defined subroutine, is added. Since the details of the voice information detection process (step S470) are described in the flowchart of FIG. 13, the description will not be repeated here.

[0200] FIG. 16 is a flowchart (subroutine) showing the details of the interviewer information processing (step S490) in the second embodiment. The details of the processing procedure shown in FIG. 16 will be described with reference to the hardware block diagram of FIG. 10 and the functional block diagram of FIG. 11.

[0201] The flowchart of the second embodiment shown in FIG. 16 is almost the same as the flowchart of the first embodiment described above with reference to FIG. 9, except for the content of the process for determining (distinguishing) whether the interviewer is a known person or not. In this regard, for the purpose of distinction, the determination (discrimination) process of the second embodiment is indicated by a step number (step S492) different from that of the first embodiment (step S452).

[0202] In addition, in the second embodiment, the processing for a new interviewer is different from that of the first embodiment. For the purpose of distinction, the processing for a new interviewer is indicated by a step number (step S460) different from the step number (step S400) in the first embodiment.

[0203] When the process of step S490 (interviewer information processing) is started (step S491), first, based on the face information detected in the face information detection process (step S420), it is determined whether the interviewer is a known person (step S492).

[0204] More specifically, the face information processing unit 73 compares the face information (face feature amount) detected in the face information detection process (step S420 in FIG. 6) with the face information (face-voice feature amount) stored by the face information storage function 25. If they match within a preset threshold value, it is determined that the interviewer is a known person (step S492, YES).

[0205] In another example, when voice can be acquired, the main control unit 2 compares the voice information (voice feature amount) detected in the voice information detection process (step S470) with the voice information (voice feature amount) stored by the voice information storage function 29. If they match within a preset threshold value, it is determined that the person is a known interviewee (step S492, YES).

[0206] Here, it is possible to determine that the person is a known interviewee only based on the match of face information or voice information, or it is also possible to determine that the person is a known interviewee when both face information and voice information match. If neither face information nor voice information matches, or if neither face information nor voice information sufficient to identify the person is detected, it is determined that the person is not a known interviewee (step S492, NO), and the process proceeds to step S460.

[0207] Note that the details of step S460 (new interviewee process) have been described in the flowchart of FIG. 12, so the description here is omitted.

[0208] Thus, when it is determined that the person is a known interviewee (step S492, YES), the processes of steps S453 to S457 described in FIG. 9 are executed, and the interviewee information process (step S490) of the present embodiment is terminated (step S493).

[0209] As described above, according to the configuration of the present embodiment, it is possible to identify an interviewee candidate taking voice information into account, and it is possible to improve the accuracy of information provision by the interviewee.

[0210] <<Embodiment 3>> Hereinafter, Embodiment 3 of the present invention will be described. The basic hardware configuration and software configuration of Embodiment 3 are the same as those of the foregoing embodiments. Hereinafter, the differences between the present embodiment (Embodiment 3) and the foregoing embodiments will be mainly described, and the common parts will be omitted as much as possible to avoid duplication.

[0211] In the foregoing embodiments, it was described on the premise that the user is wearing a glasses-shaped HMD, and for the interviewer to be recognized, it was described on the premise that the interviewer is present in front of the user. In this embodiment, while assuming that the user is wearing a glasses-shaped HMD, a case where it is difficult for the user to easily recognize the interview (candidate) person, such as when the interviewer is present behind, will be considered. Hereinafter, this embodiment will be described.

[0212] 〔Outline of operations, etc.〕 FIG. 17 is a schematic diagram for explaining the background of this embodiment. As can be seen by comparing with FIG. 1 described above, in FIG. 17, it shows a state where there is no interviewer in the line-of-sight direction 19 (refer to the dotted arrow in the figure) of the user 10 wearing the glasses-shaped HMD 1 and within the visual field of the user 10. Also, in FIG. 17, the interview partner 16 approaches from behind (the back) of the user 10, and the interview partner 16 recognizes the presence of the user 10 earlier than the user 10 recognizes the presence of the interview partner 16, and is in a state of emitting a voice 14 of "Oh!" indicated by a speech bubble.

[0213] In this embodiment, in a scenario as shown in FIG. 17, the HMD 1 is configured to start up in response to the voice 14 of "Oh!", and immediately after startup, perform a process of analyzing the surrounding video and audio, and recognize the person 16.

[0214] Here, as a process of analyzing the surrounding video and audio, the HMD 1 determines whether the person 16 can be an interviewer (whether the person is an interview candidate), and if the person can be an interviewer, acquires the incidental information regarding the person 16 and displays the acquired incidental information on the display screen 75. In the example shown in FIG. 17, the HMD 1 displays the name (Jiro Yamada) 17 as the incidental information regarding the person 16 on the display screen 75.

[0215] In addition, as shown in FIG. 17, the HMD 1 is connected to a network 33 to which a network server 32 is connected via an access point 31. Here, the network server 32 includes a network server that performs various arithmetic processes, a network server that stores various data, etc., and the HMD 1 can utilize them as necessary.

[0216] In one specific example, the main control unit 2 can control the communication processing unit 6 (communication device) to cause an external server (network server 32) to perform the processing of the behavior analysis processing unit 74 (behavior analysis processing device) of the HMD 1.

[0217] With such a configuration, the resources of the entire HMD 1 can be used efficiently, so the processing speed is improved, which in turn leads to quickly presenting necessary information to the user.

[0218] 〔Processing for obtaining additional information〕 Next, the process of identifying the interviewee and obtaining additional information about the interviewee will be described.

[0219] FIG. 18 is a flowchart showing the procedure of step S500 of the process (interviewee identification / information acquisition process) of determining whether a person can be an interviewee based on voice information and, if so, acquiring in advance additional information about the interviewee, which is the essence of the present embodiment. For the description of the processing procedure in FIG. 18, reference will be made to the functional block diagram in FIG. 11.

[0220] The flowchart showing the processing procedure in FIG. 18 is almost the same as the flowchart showing the procedure in FIG. 15, and a defined subroutine, voice information alone processing (step S510), is added.

[0221] When the interviewer identification and information acquisition process (step S500) starts (step S501), the same process as the flowchart in FIG. 15 is performed. However, when the HMD1 determines from the image information that there is no person around (step S433, NO), and when it determines from the image information that there is no interview candidate around (step S900, NO), it executes a subroutine, the voice information only process (step S510).

[0222] Here, the process of step S510 (voice information only process), which is a subroutine, will be described. FIG. 19 is a flowchart showing the processing procedure of the voice information only process (step S510), which is a subroutine. For the explanation of the processing procedure in FIG. 19, reference will be made to the functional block diagram in FIG. 11 for the description.

[0223] The voice only process (step S510) of Embodiment 3 shown in FIG. 19 is almost the same as the voice information detection process (step S470) shown in FIG. 13 (Embodiment 2). The differences are that in Embodiment 3, compared with FIG. 13, there is no determination process in step S476, and the interview candidate determination (step S901) and the interviewer information process (step S490), which is a subroutine, are added.

[0224] After the start (step S511) of the voice only process (step S510), the HMD1 performs the same processes as those described above in FIG. 13 from step S472 to step S474. Also, the voice detection error setting process in step S477 is the same as described above.

[0225] In step S901 following step S474, the HMD1 determines whether the main of the voice is the voice of an interview candidate. In a specific example, in step S901, the HMD1 determines whether the content of the voice may be a call to the user.

[0226] Here, as a case where the content of the voice may be a call to the user, for example, (1) When the user's name is included, (2) When it is a voice calling to a person (for example, in the case of a call such as "Ah, you over there", "Could it be you"), are cited.

[0227] Therefore, in the case of the above (1) or (2), the HMD1 determines that the owner of the voice is the voice of the interview candidate (step S901, YES). In this case, the HMD1 shifts to the voice feature detection process (step S475) described above with reference to FIG. 13.

[0228] On the other hand, if it is determined in the interview candidate determination in step S901 that the owner of the voice is not that of the interview candidate (step S901, NO), the process proceeds to the end process of this routine (step S512).

[0229] After the voice feature detection process (step S475), the HMD1 executes the interviewer information process (step S490) described above in the subroutine of FIG. 16, and then ends the voice-only process (step S510) (step S512).

[0230] According to the HMD1 of Embodiment 3 that performs the above-described voice-only process (step S510), even when the interview candidate cannot be determined from the image information, the interview candidate can be judged only from the voice information, and the additional information regarding the interview candidate can be acquired. Therefore, for example, even when the image of a person cannot be completely acquired in a crowded place or when the above-described imaging unit 71 malfunctions, the information regarding the interview candidate can be acquired.

[0231] <<Embodiment 4>> The following describes Embodiment 4 of the present invention. The basic hardware configuration and software configuration of Embodiment 4 are the same as those of the foregoing Embodiments 1 to 3. Hereinafter, the differences between this embodiment (Embodiment 4) and the foregoing Embodiments 1 to 3 will be mainly described, and the common parts will be omitted as much as possible to avoid duplication.

[0232] In the foregoing Embodiments 1 to 3, it was assumed that the user was wearing a glasses-shaped HMD. In contrast, in Embodiment 4, the case where an HMD other than a glasses shape is worn will be considered. Hereinafter, this embodiment will be described.

[0233] FIG. 20 is an external view showing an example of the HMD used in Embodiment 4. The HMD 100 shown in FIG. 20 has a goggle-shaped housing (hereinafter, also simply referred to as a "goggle") and an outer shape, and includes an HMD wearing belt 180. Then, as shown in FIG. 20, the user 101 wears the HMD 100 on the head of the user 101 by hanging the HMD wearing belt 180 on the back of the head.

[0234] The HMD 100 has a display screen (a display for displaying an image) 175 disposed on the front surface of the goggle, and a left camera 172 and a right camera 171 are respectively disposed near the left end and the right end of the front surface of the goggle.

[0235] Also, in the HMD 100, left and right speakers are respectively disposed at positions corresponding to the ears of the user 101. In FIG. 20, the left speaker 182 is shown, and the right speaker is not shown because it is behind the user 101.

[0236] In the example shown in FIG. 20, a left side camera 173, which is a camera different from the above-described left camera 172, is disposed near the left speaker 182. Although not shown, on the HMD wearing belt 180, a rear camera, which is a camera different from the above-described cameras (171, 172, 173), is disposed at a position corresponding to the back of the head of the user 101. Further, although not shown, a right side camera, which is a camera different from the above-described right camera 171, is also disposed near the above-described right speaker.

[0237] As described above, in this embodiment, by adding or expanding the camera, the range for photographing the surroundings of the user 101 can be expanded. In particular, by installing a rear camera on the rear side of the user 101, even in the positional relationship as described in FIG. 17, the person 16 on the rear side can be identified by face information without the user 101 turning around.

[0238] In Embodiment 4, it is characterized in that the camera is expanded and the range for photographing the surroundings of the user 101 is expanded. In other words, in Embodiment 4, as the surrounding information acquisition device, it includes a plurality of cameras that acquire images, and each camera is arranged to acquire an image in a range wider than the field of view of the user 101.

[0239] As described above, according to the configuration in which the device as the surrounding information acquisition device is arranged so as to be able to acquire the surrounding information in a range that the field of view of the user 101 cannot reach, the probability of capturing the interview candidate who is in a position where the user 101 is not aware is improved, and the convenience is also improved.

[0240] Also, although an example of using a camera as the device of the surrounding information acquisition device is given here, as another example, for example, a configuration in which a plurality of distance measurement sensors 55 and human presence sensors 56 (see FIG. 3) are arranged may be used.

[0241] <<Embodiment 5>> Hereinafter, Embodiment 5 of the present invention will be described. The basic hardware configuration and software configuration of Embodiment 5 are the same as those of the foregoing embodiments. Hereinafter, the differences between this embodiment (Embodiment 5) and the foregoing embodiments will be mainly described, and the common parts will be omitted as much as possible to avoid duplication. In the foregoing Embodiments 1 to 4, the case where there is one interview candidate is assumed. In contrast, in Embodiment 5, the case where there are a plurality of interview candidates will be considered. Hereinafter, this embodiment will be described.

[0242] Figure 21 is a flowchart showing the procedure of a process (interviewer identification / information acquisition process) for determining whether a person can be an interviewer and, if so, acquiring additional information about the interviewer in advance, which is the essence of this embodiment.

[0243] The interviewer identification / information acquisition process shown in Figure 21 is almost the same as the interviewer identification / information acquisition process described above with reference to Figure 8. The difference is that a discrimination process (step S522) for determining whether there are multiple potential interviewers and a priority determination process (step S523) are added.

[0244] When the interviewer identification / information acquisition process of this embodiment starts (step S521), the HMD1 performs step S432 (surrounding photography process), step S433 (determination process of the presence or absence of a person), and step S900 (determination process of whether it is an interview candidate) described with reference to Figure 8.

[0245] If the HMD1 determines that it is an interview candidate (step S900, YES), it proceeds to step S522. In step S522, the HMD1 determines whether the number of detected persons who are determined to be interview candidates is one or more than one.

[0246] In step S522, if the detected person is one (step S522, NO), the HMD1 performs a face information detection process (step S420) and an interview information process (step S450) in the same manner as in other embodiments and then ends this routine.

[0247] On the other hand, in step S522, if the number of detected persons who are determined to be interview candidates (hereinafter sometimes simply referred to as "interview candidates") is more than one, the HMD1 proceeds to the priority determination process (step S523).

[0248] Here, the significance of the priority determination process will be explained. If there is enough margin in the processing speed of the processor in the HMD1 and resources such as RAM, it is considered that even if there are multiple interview candidates, all of them can be handled.

[0249] However, in reality, there are often insufficient hardware resources. In particular, when the HMD1 is performing its original function (for example, when playing a video of a predetermined content), if an attempt is made to acquire information on all the interview candidates, the processing time will become long. Such an extension of the processing time may cause at least one of the interviewees to face the user and an interview (conversation, etc.) to start, which may lead to the above-mentioned problems (such as an increase in the psychological burden on the user who cannot say the name of the other party, etc.).

[0250] In view of the above problems, the inventors of the present invention have conceived that when there are a plurality of interview candidates, it is effective to narrow down or order the persons to be the target of information acquisition, and have provided a configuration for determining the priority.

[0251] Specifically, in the priority determination process (step S523), the HMD1 specifies, as the priority person, the person who is most likely to be the interviewer or the person considered to be the most important among the plurality of persons who can be the interviewer.

[0252] More specifically, in step S523, the HMD1, for example, (A) The line-of-sight direction of the person is the direction of the user. (B) The person is performing a greeting act such as raising a hand to the user. (C) The person is approaching the user. (D) The distance to the user is short. For each of the above actions, a predetermined weighting is performed to determine the priority, and the person with the highest priority is specified as the priority person.

[0253] When the priority determination process (step S523) is executed, the candidate persons are narrowed down to one person, resulting in the same state as in other embodiments. Therefore, the HMD1 then sequentially performs the face information detection process (step S420) and the interview information process (step S450) in the same manner as in other embodiments, and then ends this routine.

[0254] As an example of the setting example of the weighting, in step S523, the HMD1 identifies, among the above (A) to (D), (D), that is, the person with the closest distance as the priority person. By such processing, among a plurality of interview candidates, the person with the highest possibility of starting the interview (dialogue) earliest is identified as the priority person, so that the user can quickly know the information of the person (see also step S450).

[0255] In another example of the setting example of the weighting, in step S523, the HMD1 identifies, among the above (A) to (D), (B), that is, the person who is performing a greeting act such as raising a hand to the user as the priority person. This is considering that in a case where there are a plurality of interviewers, when the interviewers are a superior and a subordinate, it is usually considered that the superior (the person with a higher position) performs a greeting act, and the superior does not necessarily stand at the forefront. Rather, there may be a case where the subordinate takes the lead.

[0256] The above-described setting of the weighting may be set arbitrarily in advance by the user through operations of the operation input unit 9 or the like.

[0257] Also, in the example shown in FIG. 21, the configuration is such that the information of only one person is acquired. As another example, when there are a plurality of interview candidates, the processes of step S420 and step S450 may be sequentially performed in the order of decreasing priority specified through the priority determination process (step S523). By performing such processing, while effectively utilizing the hardware resources of the HMD1, the information of all interview candidates can be presented to the user in the order of decreasing priority (in other words, importance).

[0258] As yet another example, when there are multiple interview candidates, the processes of steps S420 and S450 may be performed in the order of decreasing priority specified through the priority determination process (step S523), and only for a predetermined number N (N is an integer of 1 or more) of candidates. Such a process is effective, for example, when the number of interview candidates is large, and while effectively utilizing the hardware resources of the HMD1, information on a certain number of interview candidates can be presented to the user in the order of decreasing priority (in other words, importance).

[0259] Furthermore, as a modification example of the process of assigning priorities, as shown in FIGS. 22A, 22B, and 23, a process may be performed in which simplified information is displayed (FIG. 22A) when the interview candidate is far away, and detailed information is displayed when the interview candidate approaches (FIG. 22B).

[0260] Referring to FIG. 23, the HMD1 displays simplified information 1103a, 1103c (only the name in this example) for the persons 15a, 15c who are far away, and detailed information 1104b (the name and various other information in this example) for the person 15b who is close by.

[0261] By providing a configuration in which the detail level of the information to be displayed (that is, presented or notified to the user) is changed according to the distance of the interview candidate (target person), the user's attention can be focused on the information of the person for whom more information is desired, improving convenience.

[0262] When performing the process of such a modification example, after performing the same processes as in other embodiments for all interview candidates (target persons), the interviewer identification / information acquisition process of this embodiment may be terminated (step S524).

[0263] As described above, with the configuration of Embodiment 5, even when there are multiple candidates who can be interviewers, a prompt response can be made.

[0264] As described in detail above, the mobile information terminal (HMD1, 100) of the present disclosure includes a surrounding information acquisition device (sensor unit 5, imaging unit 71, voice input unit 81) that acquires the surrounding information of the terminal and the user 10, and analyzes the actions of the people included in the acquired surrounding information, thereby determining whether there is an interview candidate (a person who intends to interview the user) for the user 10. An action analysis processing device (main control unit 2, action analysis processing unit 74), and an information presentation device (display unit 72) that presents the user 10 with additional information corresponding to the person when it is determined that there is an interview candidate.

[0265] According to the mobile information terminal (HMD1, 100) having the above configuration, it is possible to more quickly provide the user 10 with the additional information of the interview partner. When the user 10 recognizes the person as an interviewer, the user 10 knows the additional information about the interviewer.

[0266] Further, the above-described mobile information terminal (HMD1, 100) has a configuration in which any one or more of the video captured by the imaging unit 71, the distance information measured by the distance measuring sensor 55 for the object including the person, and the voice picked up by the voice input unit 81 are acquired as surrounding information.

[0267] According to such a configuration, the surrounding information acquisition device can acquire surrounding information in consideration of the advantages of various information and the resources of the HMD1 (100) (such as the execution state of the original function), and thus can realize more quickly providing the user 10 with the additional information of the interview partner.

[0268] Further, the above-described mobile information terminal (HMD1, 100) has a configuration in which the action analysis processing unit 74 does not determine an interview candidate depending on the place where the surrounding information is acquired.

[0269] According to such a configuration, unnecessary information presentation can be suppressed, which contributes to quickly acquiring the additional information for the truly necessary person, thereby improving convenience.

[0270] Also, in the above-described mobile information terminal (HMD1, 100), when there are a plurality of persons determined to be interview candidates, the behavior analysis processing unit 74 assigns a priority according to the analysis result of the behavior to each interview candidate, and determines the number or order of the interview candidates related to the supplementary information presented by the information presentation device (display unit 72) according to the assigned priority.

[0271] Furthermore, in the above-described mobile information terminal (HMD1, 100), when it is determined by the behavior analysis processing device that the interview candidate exists, the information presentation device (display unit 72) presents information indicating that the interview candidate exists and the supplementary information corresponding to the interview candidate in stages.

[0272] According to such a configuration, it is possible to concentrate the user's attention on the information of the person who desires more information, and the convenience is enhanced.

[0273] As described above, the examples of the embodiments of the present invention have been described using Embodiments 1 to 5. However, the configuration for realizing the technology of the present invention is not limited to the above-described embodiments, and various modifications can be considered. For example, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. All of these belong to the scope of the present invention. Also, the numerical values, messages, etc. appearing in the text and drawings are merely examples, and using different ones will not impair the effects of the present invention.

[0274] The functions and the like of the present invention described above may be realized in hardware by designing some or all of them, for example, in an integrated circuit. Alternatively, they may be realized in software by a microprocessor unit or the like interpreting and executing a program for realizing each function and the like. It is also possible to use a combination of hardware and software. The software may be stored in advance in the program section 41 or the like of the HMD1 at the time of product shipment. After product shipment, it may be acquired from various server devices or the like on the Internet. It may also be acquired from the software provided on a memory card, an optical disk, or the like.

[0275] In addition, the control lines and information lines shown in the figures indicate those considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In reality, it may be considered that almost all components are interconnected.

Explanation of Reference Numerals

[0276] 1... HMD (portable information terminal), 2... main control unit, 3... system bus, 4... storage unit, 5... sensor unit, 6... communication processing unit (communication device), 7... video processing unit, 8... audio processing unit, 9... operation input unit, 10... user, 15... person (interview candidate), 16... person (interview partner), 32... network server, 42... data storage unit, 71... imaging unit (surrounding information acquisition device), 72... display unit (information presentation device), 73... face information processing unit, 74... behavior analysis processing unit (behavior analysis processing device), 75... display screen, 81... audio input unit (surrounding information acquisition device), 82... audio output unit, 83... audio information processing unit, 100... HMD (portable information terminal), 171... right camera, 172... left camera, 173... left side camera, 175... display screen, 711... right camera, 712... left camera.

Claims

1. In a portable information terminal carried by a user, a surrounding information acquisition device that acquires surrounding information, an action analysis processing device that analyzes the actions of people included in the acquired surrounding information and determines an interview candidate who is about to interview the user, and an information presentation device that presents the user with additional information corresponding to the interview candidate, configured such that, depending on the type of person included in the surrounding information, the action analysis processing device does not determine the interview candidate, a portable information terminal.

2. In a portable information terminal carried by a user, a surrounding information acquisition device that acquires surrounding information, an action analysis processing device that analyzes the actions of people included in the acquired surrounding information and determines an interview candidate who is about to interview the user, and an information presentation device that presents the user with additional information corresponding to the interview candidate, wherein when there are a plurality of people determined to be the interview candidates by the action analysis processing device, the action analysis processing device assigns a priority order according to the analysis result of the actions to each of the interview candidates, and determines the number or order of the interview candidates related to the additional information presented by the information presentation device according to the assigned priority order, a portable information terminal.

3. The portable information terminal according to claim 2, further configured such that, depending on the type of person included in the surrounding information, the action analysis processing device does not determine the interview candidate, a portable information terminal.

4. The portable information terminal according to claim 1 or 3, wherein the types of people for whom the determination is not made include one or more of facility responders and security guards, a portable information terminal.

5. The portable information terminal according to claim 1 or 2, wherein the surrounding information is any one or more of an imaged video, distance information in which the distance to an object including the person is measured, and recorded sound, a portable information terminal.

6. The portable information terminal according to claim 1 or 2, wherein the action analysis processing device determines that a person is an interview candidate when the action of the person included in the surrounding information is an action indicating interest in the user, a portable information terminal.

7. The portable information terminal according to claim 6, wherein the action indicating interest in the user is one or more of an action of turning the line of sight, a greeting action, an approaching action, and a calling action, a portable information terminal.

8. In the mobile information terminal according to claim 1 or 2, furthermore, depending on the location where the surrounding information is acquired, it is configured not to perform the determination of the interview candidate by the action analysis processing device. Mobile information terminal.

9. In the mobile information terminal according to claim 1, when the action analysis processing device determines that there is an interview candidate, the information presentation device presents information indicating the existence of the interview candidate and the supplementary information corresponding to the interview candidate in stages. Mobile information terminal.

10. In the mobile information terminal according to claim 1, when there are a plurality of persons determined to be interview candidates by the action analysis processing device, the information presentation device presents the supplementary information corresponding to the interview candidate to the user so that the information presented becomes more detailed as the distance from the user becomes closer. Mobile information terminal.

11. In the mobile information terminal according to claim 1 or 2, the surrounding information acquisition device includes a plurality of cameras that acquire images, and the cameras are arranged to acquire the images in a range wider than the field of view of the user. Mobile information terminal.

12. In the mobile information terminal according to claim 1 or 2, it is configured to cause an external server to perform a part of the processing performed by the action analysis processing device via a communication device. Mobile information terminal.

13. In the mobile information terminal according to claim 1 or 2, the mobile information terminal is an HMD (Head Mounted Display). Mobile information terminal.

Images