Accessibility determination device, accessibility determination method, and recording medium

Abstract

An accessibility determination device includes: a photographing section that acquires a photographed image of an environment around a user; a face organ detection section that analyzes the photographed image to detect a face organ of another person; a processing section that determines at least one of an orientation, a moving direction, and an expression of the face of the other person based on a detection result of the face organ detection section, and decides the accessibility based on the determined result; a target person detection section that detects whether the other person is a predetermined target person based on the detection result of the face organ detection section; and an informing section that informs the user in a case where the accessibility decided by the processing section satisfies a predetermined criterion and the target person detection section detects that the other person is the predetermined target person. The processing section changes the accessibility or the predetermined criterion according to a distance of the photographing section from the other person.

Description

Technical Field

[0001] This invention relates to an accessibility determination device that determines accessibility, an index representing the ease with which communication with others is possible. Background Technology

[0002] In social life, which involves communication with others, it is important to determine whether someone is in a state where they can be approached for conversation. Devices are known to determine whether a person being photographed, or someone else, is interested in the device or the device itself. For example, Patent Document 1 discloses a product sales device that can estimate whether a person intending to use the device is someone whose position is shifted relative to the device body.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2014-174859 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, even if someone moves closer to you, you cannot expect smooth communication if their face is not facing you, or if they are angry. Simply knowing someone's position is insufficient to accurately determine whether you can communicate effectively with them.

[0008] The purpose of this disclosure is to provide an accessibility determination device that enables users to smoothly communicate with acquaintances or other individuals.

[0009] Methods for solving problems

[0010] One approachability determination device disclosed herein comprises: a capturing unit that captures multiple images of a user's surrounding environment at time intervals; a facial organ detection unit that analyzes the captured images to detect facial organs of others different from the user; a processing unit that, based on the detection results of the facial organ detection unit, determines at least one of the orientation, direction of movement, and expression of the other person's face, and determines an approachability index representing the ease with which the user can communicate with the other person based on the determined results; an object detection unit that, based on the detection results of the facial organ detection unit, detects whether the other person is a pre-determined object; and a notification unit that, when the approachability determined by the processing unit meets a pre-determined criterion and the object detection unit detects that the other person is the pre-determined object, notifies the user, wherein the processing unit changes the approachability or the pre-determined criterion based on the distance between the capturing unit and the other person.

[0011] Another aspect of the accessibility determination device disclosed herein includes: a camera unit that captures images of the user's surrounding environment; a facial organ detection unit that analyzes the captured images to detect facial organs of another person different from the user; a processing unit that, based on the detection results of the facial organ detection unit, determines the orientation and expression of the other person's face, and based on the determined results, determines an accessibility index representing the ease with which the user can communicate with the other person; an object detection unit that, based on the detection results of the facial organ detection unit, detects whether the other person is a predetermined object; and a notification unit that, when the accessibility determined by the processing unit meets a predetermined benchmark and the object detection unit detects that the other person is the predetermined object, notifies the user, wherein the processing unit changes the accessibility or the predetermined benchmark based on the distance between the camera unit and the other person.

[0012] Invention Effects

[0013] The accessibility determination device disclosed herein enables users to communicate smoothly with acquaintances or other individuals. Attached Figure Description

[0014] Figure 1 This is a schematic diagram illustrating an application example of the accessibility determination device according to an embodiment of the present disclosure.

[0015] Figure 2 This is a block diagram illustrating a structural example of the accessibility determination device according to this embodiment.

[0016] Figure 3 This is a flowchart illustrating an example of the operation of the accessibility determination device in this embodiment.

[0017] Figure 4 It means Figure 3 The flowchart shows the steps involved in determining accessibility.

[0018] Figure 5 It means Figure 3 The flowchart shows the steps involved in determining the threshold Mth.

[0019] Figure 6 This is a flowchart illustrating a variation of the operation of the accessibility determination device in this embodiment. Detailed Implementation

[0020] Hereinafter, embodiments of the guiding device of this disclosure will be described with reference to the accompanying drawings. Furthermore, in the following embodiments, the same reference numerals are used to denote the same constituent elements.

[0021] 1. Application Examples

[0022] Figure 1 This is a schematic diagram illustrating an application example of the accessibility determination device 100 of this embodiment. The accessibility determination device 100 uses the camera 2 to photograph the surrounding environment of the user 90, and if there is an acquaintance or other person 91 in the vicinity who can be approached for conversation, it uses the sound output device 72 to inform the user 90. The camera 2 is an example of the "photographing unit" of this disclosure.

[0023] The accessibility determination device 100 uses an accessibility metric to determine whether an acquaintance or other person 91 is someone who can be approached for conversation. Here, accessibility refers to the ease with which the user 90 can communicate with another person. Accessibility is determined based on facial information captured by the camera 2.

[0024] For example, such as Figure 1 As shown, when the face of the person 91 is facing the user 90 and is smiling, the approachability is high, meaning the person 91 is someone the user 90 can approach and talk to. In this case, the approachability determination device 100 informs the user 90 via the sound output device 72. The information provided includes, for example, the location of the person 91, the name of the person 91, and the level of approachability. Through this information, the user 90 can recognize that there is a person 91 nearby who is in a position to talk to. Therefore, the user 90 can initiate a conversation with the person 91 and successfully establish communication with them.

[0025] In particular, if user 90 is visually impaired, even if they encounter an acquaintance on the street, they may not be able to visually recognize the acquaintance's presence. Even if user 90 wishes to converse with the acquaintance, there is a possibility that they will simply walk past without engaging in conversation. In this case, by informing user 90 through the accessibility determination device 100 that an acquaintance in a highly accessible state is present nearby, user 90 can successfully communicate with the acquaintance. Thus, the accessibility determination device 100 can be used as a device to assist visually impaired users, i.e., user 90, in communication.

[0026] User 90 can also be a robot capable of autonomous movement. In this case, components of the accessibility determination device 100, such as camera 2 and sound output device 72, can also be assembled into the robot. In this case, the robot can, for example, identify whether the person 91 is in a conversational state based on images captured by camera 2 while it is moving autonomously. Thus, in a society where humans and robots coexist, the robot can smoothly communicate with humans.

[0027] 2. Structural Example

[0028] Figure 2 This is a block diagram illustrating a structural example of the accessibility determination device 100 according to this embodiment. The accessibility determination device 100 includes a camera 2, a control unit 4, a storage unit 5, a communication interface (I / F) 6, and a notification unit 7.

[0029] Camera 2 is a photographing device that captures images of the user 90's surroundings. For example, camera 2 captures images of the user 90's surroundings at a predetermined frame rate, sequentially generating image data. Camera 2 forms images using a solid-state imaging element such as CMOS (Complementary MOS) or CCD (Charge Coupled Device). Camera 2 is, for example, a wearable camera that the user 90 can wear. Camera 2 can also be mounted on the user 90's belongings, such as backpacks or suitcases.

[0030] The notification unit 7 is, for example, an output device that outputs information according to the control of the control unit 4. The notification unit 7 informs the user 90. The notification unit 7 includes, for example, a display device 71, a sound output device 72, and a vibrator 73. The display device 71 includes, for example, a liquid crystal display, an organic EL display, and a projector. The sound output device 72 includes, for example, audio devices such as speakers, headphones, and headsets.

[0031] The control unit 4 is an information processing device that includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc., and controls the various components of the accessibility determination device 100 based on information processing. The control unit 4 includes, for example, an image acquisition unit 41, a face detection unit 42, a facial feature detection unit 43, a target detection unit 44, an accessibility determination unit 45, and an output control unit 46 as components. Each component of the control unit 4 can also perform its own processing by executing necessary programs through the control unit 4. Such programs can also be stored in the storage unit 5. When the control unit 4 executes necessary programs, the target program stored in the storage unit 5 is expanded in RAM. The control unit 4 interprets and executes the expanded program in RAM through the CPU, controlling each component. Examples of the operation of each component will be described later.

[0032] Storage unit 5 is a medium that stores recorded program information in a manner that allows computers and other devices or machines to read it, using electrical, magnetic, optical, mechanical, or chemical means. Storage unit 5 may be, for example, an auxiliary storage device such as a hard disk drive or a solid-state drive. Storage unit 5 may store, for example, a facial database 51 and programs executed by control unit 4. Storage unit 5 may also include a main storage device such as RAM.

[0033] The communication interface 6 includes an interface circuit for communication connection between the accessibility determination device 100 and external devices. The communication interface 6 may communicate according to standards such as IEEE 802.3, IEEE 802.11, Wi-Fi (registered trademark), LTE, 3G, 4G, 5G, etc. Alternatively, the communication interface 6 may be an interface circuit that communicates according to standards such as USB (Universal Serial Bus), HDMI (registered trademark), IEEE 1394, Bluetooth (registered trademark).

[0034] In this embodiment, an example of implementing the functions of the control unit 4 using a CPU will be described. However, some or all of the above functions can also be implemented by one or more dedicated processors. Furthermore, the components of the control unit 4 can be appropriately omitted, substituted, or added depending on the embodiment. The control unit 4 can also be composed of various semiconductor integrated circuits such as CPUs, MPUs, GPUs, microcomputers, DSPs, FPGAs, and ASICs.

[0035] 3. Example of an action

[0036] 3-1. Overall Process

[0037] The accessibility determination device 100 detects whether an acquaintance or other person is captured in the image taken by the camera 2. If so, it determines the accessibility of that person. If the determined accessibility is greater than a predetermined threshold, the accessibility determination device 100 informs the user 90. Thus, the user 90 can know whether an acquaintance or other person in the vicinity is someone they can talk to. Therefore, by using the accessibility determination device 100, the user 90 can smoothly communicate with acquaintances or people of interest in daily life. (The following refers to...) Figures 3-5 An example of the operation of the accessibility determination device 100 will be explained.

[0038] Figure 3 This is a flowchart illustrating an example of the operation of the accessibility determination device 100. Figure 3 The process shown is executed by control unit 4 at a predetermined cycle, for example.

[0039] (Step S1)

[0040] First, the image acquisition unit 41 acquires the captured image by the camera 2 (S1). For example, the camera 2 takes pictures at a fixed frame rate. In step S1, the image acquisition unit 41 may also acquire multiple captured images. Hereinafter, an example of processing where the image acquisition unit 41 acquires one captured image and then proceeds to the next step S2 will be described. However, this disclosure is not limited to this. For example, in step S1, the image acquisition unit 41 may acquire a moving image composed of multiple frames, or it may acquire multiple still images.

[0041] (Step S2)

[0042] Next, the face detection unit 42 analyzes the captured image acquired by the image acquisition unit 41 and detects the person's face (S2). Here, detecting the person's face includes detecting the area within the captured image that is estimated to be the area where the person's face was captured.

[0043] (Step S3)

[0044] Next, the facial organ detection unit 43 detects facial organs (hereinafter referred to as "facial organs") within the face detected by the facial detection unit 42 (S3). Here, facial organs are a collection of tissues, including tissues with specific functions. For example, facial organs include the eyes, nose, mouth, and ears. Facial organs may also include skin. The entire face may also be included in facial organs. For example, the facial organ detection unit 43 detects the position of the facial organs.

[0045] (Steps S4 and S5)

[0046] Next, the target detection unit 44, based on the detection results of the facial organ detection unit 43 and the facial database 51 storing information related to the target's face, detects whether the face detected by the face detection unit 42 is the target's face (S4). Step S4 performs identity recognition processing to determine whether the face detected by the face detection unit 42 is identical or similar to the target's face. The target is the other party with whom the user 90 wants to communicate or can communicate, such as an acquaintance or celebrity of the user 90. The facial-related information referenced by the target detection unit 44 is not limited to the facial database 51 stored in the storage unit 5. For example, the target detection unit 44 may also reference a facial database stored on an external server connected to a network via the communication interface 6. If the detected face is the target's face (if "yes" in step S5), the process proceeds to step S6; if the detected face is not the target's face (if "no" in step S5), the control unit 4 terminates. Figure 3 The process is shown below.

[0047] In the detection of the face in step S2, the detection of facial organs in step S3, and the detection of the subject in step S4, known image analysis methods such as template matching can be used, or a recognizer constructed through machine learning such as neural networks and self-organizing maps can be used.

[0048] (Steps S6 and S7)

[0049] In step S6, the accessibility determination unit 45 determines the accessibility M (S6). The accessibility determination unit 45 is an example of the "processing unit" of this disclosure. Accessibility M is an indicator of how easily user 90 can communicate with others. Next, the accessibility determination unit 45 determines a threshold Mth regarding the accessibility M (S7). The threshold Mth is an example of the "pre-determined benchmark" of this disclosure. Details of steps S6 and S7 will be described later.

[0050] (Steps S8 and S9)

[0051] Next, the output control unit 46 determines whether the accessibility M is greater than the threshold Mth (S8). If it is determined in step S8 that the accessibility M is below the threshold Mth (if it is "No" in step S8), the control unit 4 does not provide any notification and the process ends. Figure 3The process is as shown. If it is determined that the accessibility M is greater than the threshold Mth (if "yes" is indicated in step S8), the output control unit 46 controls the notification unit 7 to notify the user 90 (S9). For example, in notification step S9, a display device 71, such as a monitor, displays the name and accessibility M of the acquaintance detected in step S4. This information can also be communicated to the user 90 via sound output device 72. Alternatively, in step S9, vibration by vibrator 73 can convey to the user 90 the presence of an acquaintance and that the acquaintance's accessibility M is greater than the threshold Mth.

[0052] 3-2. Determining Accessibility M

[0053] Figure 4 It means Figure 3 The flowchart shown is for the process of determining the accessibility M step S6.

[0054] (Step S61)

[0055] First, the accessibility determination section 45 is based on... Figure 3 The facial orientation is determined by the information of the facial organs detected in step S3 (S61). The facial orientation can be determined, for example, by... Figure 3 The detection results of step S3 are used to detect feature quantities representing the position of each facial organ, and these feature quantities are compared with a three-dimensional facial shape model to determine the orientation of the face. The orientation of the face can be represented by a vector, for example. The orientation of the face can also be represented by an angle of 0° when the direction towards camera 2 is set. In the method of determining the orientation of the face, the techniques illustrated in International Patent Publication No. 2006 / 051607, Japanese Patent No. 4093273, etc., can also be applied.

[0056] (Step S62)

[0057] Next, the accessibility determination unit 45 determines the first evaluation index F related to the orientation of the face. P (S62). Here, the first evaluation index F P This represents the evaluation index used to assess accessibility (M) based on facial orientation. The first evaluation index is F. P This can also be described as an indicator related to the orientation of the face among the components constituting accessibility M. When the subject is facing the direction of camera 2, the accessibility determination unit 45 will use the first evaluation index F. P Set it to a larger value. For example, the first evaluation index F P It is the maximum value when the face is turned at an angle of 0°, decreases as the angle increases, and is the minimum value at 180°. First evaluation index F PAlternatively, it can be represented by multiplying the inner product of the optical axis of camera 2 and the orientation of the subject's face by -1. First evaluation index F P For example, it is standardized to a value between 0 and 1.

[0058] (Step S63)

[0059] In addition, the accessibility determination unit 45 is based on... Figure 3 The facial information detected in step S2 or the facial organ information detected in step S3 is used to determine the direction of facial movement (S63). For example, the accessibility determination unit 45 compares the position of the face or the position of facial organs in multiple images captured at different times to determine the direction of facial movement. The direction of facial movement is, for example, a 3D position change vector of the subject's face.

[0060] (Step S64)

[0061] Next, the accessibility determination unit 45 determines the second evaluation index F related to the direction of facial movement. M (S64). Here, the second evaluation index F M This represents the evaluation index used to assess accessibility M based on the direction of facial movement. The second evaluation index is F. M This can also be described as an indicator related to the direction of facial movement among the components constituting accessibility M. When the direction of facial movement is towards camera 2, the accessibility determination unit 45 will assign the second evaluation index F. M Set to a larger value. Second evaluation indicator F M For example, the value can be represented by multiplying the inner product of the orientation of the optical axis of camera 2 and the vector of the change in position of the subject's face by -1. Second evaluation index F M For example, it is standardized to a value between 0 and 1.

[0062] (Step S65)

[0063] In addition, the accessibility determination unit 45 is based on... Figure 3 The facial features detected in step S3 are used to estimate facial expressions (S65). For example, the accessibility determination unit 45 detects the distance between feature points of the face such as eyes, eyebrows, and mouth, or the edges of the facial surface, based on the information of the facial features, as information representing facial expressions.

[0064] Facial expression estimation can also utilize known methods. For example, the accessibility determination unit 45 extracts features related to the relative position and shape of facial organs based on their location information. These features include, for example, Haar-like features, distances between feature points, and Fourier descriptors. The extracted features are then input into an expression discriminator, which outputs a score (expression component value). The expression score includes factors such as the degree of smile (smile intensity) and the degree of happiness (happiness intensity). The expression discriminator can be constructed, for example, by learning from multiple sample images of faces using machine learning techniques such as neural networks and self-organizing maps.

[0065] Facial expression estimation involves identifying the types of facial expressions, that is, determining the types of facial expressions being identified by using words that express emotions. Here, facial expression identification can be based on identifying words expressing a single emotion or on identifying combinations of words expressing emotions. In the case of combinations of words expressing emotions, the words expressing each emotion can also be weighted. For example, based on Paul Ekman's expression analysis, facial expressions are classified into seven types: "serious," "joyful," "angry," "disgusted," "surprised," "fearful," and "sad." As the result of expression estimation, the output is a score that quantifies the degree of each of the seven expressions (also called expression similarity or expression score) in a way that sums to 1. The score for each expression is also called the expression component value.

[0066] The aforementioned expression discriminator may not be a single unit; it may consist of seven discriminators, each responsible for one of the seven expressions.

[0067] In methods for estimating facial expressions, techniques exemplified in Japanese Patent Application Publication No. 2019-111092, Japanese Patent Application Publication No. 2016-149063, and Japanese Patent Application Publication No. 2014-206903 may also be applied.

[0068] (Step S66)

[0069] After step S65, the accessibility determination unit 45 determines the third evaluation index F related to facial expression. A (S66). Here, the third evaluation index F A This represents the evaluation index used to assess approachability (M) based on facial expressions. The third evaluation index is F. A This can also be described as an indicator related to facial expressions among the components that constitute approachability M. When facial expressions are positive from a communicative perspective, the approachability determination section 45 will use the third evaluation indicator, F. ASet to a larger value. Facial expressions that are considered positive from a communication perspective include situations where the facial expression is classified as "joyful" based on the expression estimation result of step S65. Alternatively, facial expressions that are considered positive from a communication perspective include situations where the expression estimation result of step S65 shows a smile or happiness level above a predetermined threshold.

[0070] (Step S67)

[0071] Next, the accessibility decision-making unit 45 determines accessibility based on evaluation indicators F1 to F3. P F M F A Accessibility M is determined by the first to third evaluation indicators F. P F M F A The sum. In the first to third evaluation indicators F... P F M F A If the accessibility M is standardized by making it a value of 0 to 1, then the first to third evaluation indicators F can also be standardized in order to standardize the accessibility M. P F M F A The sum of these factors multiplied by 1 / 3 is taken as the accessibility M. Accessibility M can also be the first through third evaluation indicators F. P F M F A .

[0072] The above, such as Figure 4 The example shown illustrates the sequential execution of steps S61 to S67. However, the accessibility M determination step S6 in this embodiment is not limited to this. Specifically, step S62 needs to be executed after step S61, step S64 needs to be executed after step S63, and step S66 needs to be executed after step S65, but steps S61, S63, and S65 can also be executed in any order. Alternatively, steps S61, S63, and S65 can be processed in parallel.

[0073] In addition, as mentioned above, Figure 4 As shown, the accessibility determination unit 45 determines the first to third evaluation indicators F. P F M F A All examples have been illustrated. However, this disclosure is not limited thereto. The accessibility determination section 45 only needs to determine the first to third evaluation indicators F. P F M F A At least one of them is sufficient. For example, when only the first evaluation index F is determined... P In the case that it can be omitted Figure 4 Steps S63 to S66. In this case, in step S67, only the first evaluation index F is considered. P To determine accessibility M.

[0074] Alternatively, the accessibility determination unit 45 can also determine the first to third evaluation indicators F. P F M F A The accessibility M is determined based on two evaluation indicators. In this case, for example, the first evaluation indicator F is determined. P and the third evaluation index F A Accessibility M is determined based on these two factors.

[0075] 3-3. Determination of the accessibility threshold Mth

[0076] Figure 5 It means Figure 3 The flowchart shows the process of determining the threshold Mth in step S7.

[0077] First, the accessibility determination section 45 is based on... Figure 3 The facial information detected in step S2 or the facial organ information detected in step S3 is used to detect the distance d between camera 2 and the subject (S71). Camera 2 is a wearable camera worn by user 90 or a camera mounted on the goods of user 90, so the distance d between camera 2 and the subject can be said to be the same distance as the distance between user 90 and the subject.

[0078] Next, the accessibility determination unit 45 determines whether the distance d detected in step S71 is a long distance, a medium distance, or a short distance (S72, S73). For example, the accessibility determination unit 45 determines that the distance d is longer than 5m, that it is longer than 3m but less than 5m, that it is a medium distance, and that it is less than 3m.

[0079] If the distance d is determined to be a long distance (in the case of "yes" in step S72), the accessibility determination unit 45 sets the threshold Mth to a high level value M1 (S74). The high level value M1 is set to 0.8, for example, when the accessibility M is normalized.

[0080] If the distance d is determined to be a medium distance (in the case of "yes" in step S73), the accessibility determination unit 45 sets the threshold Mth to a medium level value M2 that is lower than the high level value M1 (S75). For example, when the accessibility M is normalized, the medium level value M2 is set to 0.5.

[0081] If the distance d is determined to be a short distance (in the case of "No" in step S73), the accessibility determination unit 45 sets the threshold Mth to a low level value M3, which is lower than the medium level value M2 (S76). For example, when the accessibility M is normalized, the low level value M3 is set to 0.2.

[0082] Thus, the longer the distance d, the more stringent the conditions (step S8) for executing the notification step S9 are made by the control unit 4.

[0083] Control unit 4 can make the conditions for notification more stringent as the distance d increases; the means of stringency are not limited to... Figure 3 as well as Figure 5 The illustrated method. Below, we will explain an example of a method where the longer the distance d, the stricter the conditions for notification. For example, if control unit 4 determines that the distance d is a long distance, it will only notify the first to third evaluation indicators F. P F M F A If all values ​​are 0.8 or higher, proceed to notification step S9. If the control unit 4 determines that the distance d is a medium distance, it will check the first to third evaluation indicators F. P F M F A If two or more of the following are 0.8 or higher, the notification step S9 is executed. If the control unit 4 determines that the distance d is a short distance, and if the first to third evaluation indicators F... P F M F A If any one of them is 0.8 or higher, then proceed to the notification step S9.

[0084] Furthermore, the following will explain other examples of methods where a longer distance d makes the conditions for notification more stringent. The accessibility determination unit 45 can also change the accessibility M based on the distance d between the camera 2 and the other person. For example, the accessibility determination unit 45 can also adjust the first to third evaluation indicators F... P F M F A The sum of the distances divided by the distance d is used as the accessibility M. Therefore, the longer the distance d, the smaller the accessibility M. Even if the threshold Mth is a fixed value, the longer the distance d, the more difficult it is to perform the notification step S9.

[0085] 4. Function / Effect

[0086] As described above, the accessibility determination device 100 of this embodiment includes a camera 2 (an example of an imaging unit), a facial organ detection unit 43, an accessibility determination unit 45 (an example of a processing unit), a target detection unit 44, and a notification unit 7. The camera 2 captures multiple images of the user 90's surrounding environment at time intervals (S1). The facial organ detection unit 43 analyzes the captured images to detect facial organs of others different from the user 90 (S3). Based on the detection results of the facial organ detection unit 43, the accessibility determination unit 45 determines at least one of the other person's facial orientation, direction of movement, and expression. Furthermore, based on the determined results, the accessibility determination unit 45 determines an accessibility M, an index representing the ease with which the user 90 can communicate with others (S6). Based on the detection results of the facial organ detection unit 43, the target detection unit 44 detects whether the other person is a pre-determined target (S4). If the accessibility M determined by the accessibility determination unit 45 meets the predetermined criteria, and the target detection unit 44 detects that another person is the predetermined target, the notification unit 7 notifies the user 90 (S9). The accessibility determination unit 45 changes the accessibility M or the predetermined criteria based on the distance between the camera 2 and the other person (S7).

[0087] According to this structure, user 90 can identify familiar people or other individuals in the surrounding environment whose accessibility M meets a predetermined criterion by being informed. User 90 can recognize these identified individuals as people they can approach and engage in conversation with. Therefore, user 90 can smoothly communicate with familiar people or other individuals in daily life.

[0088] Camera 2 may also capture multiple images by taking pictures of the user 90's surrounding environment instead of taking pictures at time intervals. The accessibility determination unit 45 may also determine the orientation and expression of another person's face based on the detection results of the facial organ detection unit 43, and determine the accessibility M based on the determined results.

[0089] Thus, even by determining the structure of two parameters—the orientation of another person's face and their facial expression—accessibility M can be determined. Therefore, similarly, user 90 is able to communicate smoothly with acquaintances and other individuals in daily life.

[0090] The accessibility determination unit 45 can also determine the orientation, expression, and direction of facial movement of another person's face based on the detection results of the facial organ detection unit 43, and determine the accessibility M based on the determined results.

[0091] In this way, by determining the structure of three parameters—the orientation of another person's face, their expression, and the direction of their facial movement—accessibility M can be determined with high precision. Therefore, similarly, user 90 can smoothly communicate with acquaintances and other individuals in daily life.

[0092] In this embodiment, if the target detection unit 44 detects that another person is a pre-determined target, the accessibility determination unit 45 may determine the accessibility M for that person. On the other hand, if the target detection unit 44 does not detect that another person is a pre-determined target, the accessibility determination unit 45 may not determine the accessibility M for that person.

[0093] Therefore, when the other party is not a pre-determined target such as an acquaintance, the process of determining the accessibility M for that other party can be omitted. Thus, the processing load of the accessibility determination device 100 can be reduced.

[0094] 5. Variations

[0095] The embodiments of this disclosure have been described in detail above, but the above description is merely illustrative in all respects. Various modifications and variations can be made without departing from the scope of this disclosure. For example, the following changes can be made. Furthermore, the same reference numerals are used below for the same constituent elements as in the above embodiments, and descriptions of points that are the same as in the above embodiments are appropriately omitted. The following variations can be appropriately combined.

[0096] Figure 6 This is a flowchart illustrating a modified example of the operation of the accessibility determination device 100. Figure 6 In the flowchart, steps S6 to S8 are executed after step S3. Figure 3 In the flowchart, step S8, which determines whether the accessibility M is greater than the threshold Mth, is a process performed only if the face detected in step S2 is the face of the target (if it is "yes" in step S5). In contrast, in Figure 6 In the flowchart, step S8, which determines whether the accessibility M is greater than the threshold Mth, is always executed. Only if the result in step S8 is "yes" is step S4, which checks whether the face detected in step S2 is the face of the target.

[0097] Thus, in a modified example, the target detection unit 44 detects whether another person is a predetermined target if the accessibility M determined by the accessibility determination unit 45 meets the predetermined criteria; otherwise, it does not detect whether another person is a predetermined target.

[0098] According to this structure, the accessibility determination device 100 only performs identity recognition processing on faces detected from the captured image that meet a predetermined criterion for accessibility M, and identifies the face of the person being approached. If accessibility M does not meet the predetermined criterion, this identity recognition processing can be omitted. Therefore, the processing load of the accessibility determination device 100 can be reduced.

[0099] (Postscript)

[0100] The various methods of this disclosure will be described below. The reference numerals used below are merely illustrative.

[0101] The accessibility determination device 100 of the first aspect of this disclosure includes:

[0102] The camera unit 2 captures multiple images of the user 90’s surrounding environment at time intervals.

[0103] The facial organ detection unit 43 analyzes the captured image to detect facial organs of others who are different from the user 90;

[0104] The processing unit 45, based on the detection results of the facial organ detection unit 43, determines at least one of the orientation, movement direction, and expression of the other person's face, and determines the accessibility index, which represents the ease with which the user 90 can communicate with the other person, based on the determined results.

[0105] The target detection unit 44, based on the detection results of the facial organ detection unit 43, detects whether the other person is a pre-determined target; and

[0106] The notification unit 7 notifies the user 90 when the accessibility determined by the processing unit 45 meets a predetermined criterion and the target detection unit 44 detects that the other person is the predetermined target.

[0107] The processing unit 45 changes the accessibility or the predetermined benchmark based on the distance between the shooting unit 2 and the other person.

[0108] The accessibility determination device 100 of the second aspect of this disclosure includes:

[0109] Camera unit 2 captures images of the user 90's surrounding environment.

[0110] The facial organ detection unit 43 analyzes the captured image to detect facial organs of others who are different from the user 90;

[0111] The processing unit 45 determines the orientation and expression of the other person's face based on the detection results of the facial organ detection unit 43, and determines the accessibility index, which represents the ease with which the user 90 can communicate with the other person, based on the determined results.

[0112] The target detection unit 44, based on the detection results of the facial organ detection unit 43, detects whether the other person is a pre-determined target; and

[0113] The notification unit informs the user 90 when the accessibility determined by the processing unit 45 meets a predetermined criterion and the target detection unit 44 detects that the other person is the predetermined target.

[0114] The processing unit 45 changes the accessibility or the predetermined benchmark based on the distance between the shooting unit 2 and the other person.

[0115] In the third aspect of this disclosure, the accessibility determination device 100 of the second aspect is provided, wherein,

[0116] The camera unit captures multiple images of the user's surrounding environment at time intervals.

[0117] Based on the detection results of the facial organ detection unit 43, the processing unit further determines the direction of movement of the other person's face, and further determines the accessibility based on the determined direction of movement of the other person's face.

[0118] In the fourth aspect of this disclosure, an accessibility determination device 100 is provided, which is an embodiment of any one of the first to third aspects, wherein,

[0119] If the target detection unit 44 detects that the other person is the pre-determined target, the processing unit 45 determines the accessibility for that other person.

[0120] If the target detection unit 44 does not detect that the other person is the predetermined target, the processing unit 45 does not determine the accessibility for the other person.

[0121] In the fifth aspect of this disclosure, an accessibility determination device 100 is provided, which is any one of the first to third aspects, wherein,

[0122] If the accessibility determined by the processing unit 45 meets the predetermined criteria, the target detection unit 44 detects whether the other person is the predetermined target.

[0123] If the accessibility determined by the processing unit 45 does not meet the predetermined criteria, the detection is not performed to determine whether the other person is the predetermined target.

[0124] The accessibility determination method of the sixth method disclosed herein includes:

[0125] Step S1: The control unit acquires multiple captured images of the user 90’s surrounding environment, which are captured at time intervals.

[0126] In the facial organ detection step S3, the control unit analyzes the captured image to detect facial organs of others who are different from the user.

[0127] In the accessibility determination step S6, the control unit determines at least one of the orientation, movement direction, and expression of the other person's face based on the detection results of the facial organ detection step, and determines the accessibility index, which represents the ease with which the user 90 can communicate with the other person, based on the determined results.

[0128] In the target detection step S4, the control unit, based on the detection results of the facial organ detection step, detects whether the other person is a pre-determined target; and

[0129] Step S9: If the accessibility determined in the accessibility determination step meets the predetermined criteria and the other person is detected as the predetermined target in the target detection step, the user 90 is informed.

[0130] In the seventh aspect of this disclosure, a program is provided for causing a computer to execute the accessibility determination method of the sixth aspect of this disclosure.

[0131] Label Explanation

[0132] 2: Camera; 4: Control unit; 5: Storage unit; 6: Communication interface; 7: Notification unit; 41: Image acquisition unit; 42: Face detection unit; 43: Facial organ detection unit; 44: Subject detection unit; 45: Accessibility determination unit; 46: Output control unit; 51: Face database; 71: Display device; 72: Sound output device; 73: Vibrator; 90: User; 100: Accessibility determination device.

Claims

1. An accessibility determination device, comprising: The camera unit takes multiple images of the user's surroundings at time intervals. A facial organ detection unit analyzes the captured image to detect facial organs of others who are different from the user; The processing unit, based on the detection results of the facial organ detection unit, determines at least one of the orientation, movement direction, and expression of the other person's face, and based on the determined results, determines an index representing the ease of access for the user to communicate with the other person, that is, the accessibility of the other person to the state in which the user can approach and talk to them. The target detection unit detects whether the other person is a pre-determined target based on the detection results of the facial organ detection unit; as well as The notification unit informs the user when the accessibility determined by the processing unit meets a predetermined criterion and the target detection unit detects that the other person is the predetermined target. The processing unit changes the accessibility or the predetermined benchmark based on the distance between the shooting unit and the other person.

2. The accessibility determination device according to claim 1, wherein, If the target detection unit detects that the other person is the pre-determined target, the processing unit determines the accessibility for that other person. If the target detection unit does not detect that the other person is the predetermined target, the processing unit does not determine the accessibility for that other person.

3. The accessibility determination device according to claim 1, wherein, If the accessibility determined by the processing unit meets the predetermined criteria, the target detection unit detects whether the other person is the predetermined target. If the accessibility determined by the processing unit does not meet the predetermined criteria, the target detection unit does not detect whether the other person is the predetermined target.

4. A method for determining accessibility, comprising the following steps: The control unit acquires multiple images of the user's surrounding environment captured at time intervals. In the facial organ detection step, the control unit analyzes the captured image to detect facial organs of others who are different from the user; In the accessibility determination step, the control unit determines at least one of the orientation, movement direction, and expression of the other person's face based on the detection results of the facial organ detection step. Based on the determined results, it determines an index representing the accessibility of the user to communicate with the other person, that is, whether the other person is in a state that the user can approach and talk to. In the target detection step, the control unit, based on the detection results of the facial organ detection step, detects whether the other person is a pre-determined target; and If the accessibility determined in the accessibility determination step meets the pre-determined criteria and the other person is detected as the pre-determined target in the target detection step, the user is informed.

5. A recording medium having a program recorded thereon, the program being used to cause a computer to execute the accessibility determination method of claim 4.

Images