COMMUNICATION SUPPORT DEVICE, COMMUNICATION SUPPORT METHOD, COMMUNICATION SUPPORT PROGRAM, AND SERVER DEVICE
The communication support device enhances communication efficiency for visually impaired users by prioritizing category-based image analysis to quickly identify acquaintances, addressing the issue of delayed detection and notification.
Patent Information
- Application Number
- JP2020131580
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-08-03
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-08-03
AI Technical Summary
Existing communication devices for visually impaired individuals take too long to detect and identify acquaintances in images, leading to delayed communication due to inefficient database searches.
A communication support device that prioritizes category-based image analysis using location information to quickly identify acquaintances by associating target persons with categories and searching the database in order of priority, reducing response time.
The device significantly reduces the time required to detect and notify the user about acquaintances, enabling smooth and timely communication.
Smart Images

Figure 0007581691000001 
Figure 0007581691000002 
Figure 0007581691000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a communication support device, a communication support method, a communication support program, and a server device. [Background technology]
[0002] Patent Document 1 discloses a device and method for assisting the movement of visually impaired people. This device acquires images using a visual sensor in an obstacle detection module, detects people, obstacles, etc. in the images through image analysis, and provides audio feedback. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] China Patent Publication No. 108836769 Summary of the Invention [Problem to be solved by the invention]
[0004] When a target person such as a user's acquaintance is captured in an image, the user can smoothly communicate with the target person by receiving an identification result of the target person using a technology such as image analysis. However, it may take time for the device to access a database and detect a person or object captured in the image, increasing the response time and preventing the user from smoothly communicating with the target person.
[0005] An object of the present disclosure is to provide a communication support means that reduces the time required to detect a target person. [Means for solving the problem]
[0006] A communication support device according to an embodiment of the present disclosure includes: A location acquisition unit that acquires location information indicating a user's location; An imaging unit that captures an image of a surrounding environment of the user and acquires a captured image; A storage unit that stores a subject database in which an image of a subject and a category indicating a characteristic of the subject are associated with the subject; a category ranking setting unit that sets priorities of the categories according to the location information; a subject detection unit that detects subjects belonging to the categories in the captured image in the order of priority set by the category ranking setting unit; and a notification unit that notifies the user of information regarding the subject detected by the subject detection unit.
[0007] A server device according to another aspect of the present disclosure includes: A location acquisition unit that acquires location information indicating a user's location; a subject database in which an image of a subject and a category indicating a characteristic of the subject are associated with the subject; a category ranking setting unit that sets priorities of the categories according to the location information; The communication support device includes a transmission unit that transmits information in the target database and information indicating the priority order set by the category rank setting unit to a communication support device that detects the target based on this information and notifies the user.
[0008] A communication support method according to still another aspect of the present disclosure includes: A location acquisition unit acquires location information indicating a location of a user; An imaging unit captures an image of a surrounding environment of the user to obtain an image; storing a subject database in a storage unit in which an image of a subject and a category indicating a characteristic of the subject are associated with the subject; a category ranking setting unit setting a priority order for the categories according to the location information; a subject detection unit detecting, in the captured image, subjects belonging to the category in the order of priority set by the category ranking setting unit; The method includes a step in which a notification unit notifies the user of information regarding the subject detected by the subject detection unit. Effect of the Invention
[0009] The present invention can reduce the time required to detect a target person in a captured image when supporting a user to communicate with a target person based on a captured image. [Brief description of the drawings]
[0010] [Figure 1] 1 is a schematic diagram illustrating an application example of a communication support device according to an embodiment of the present disclosure. [Diagram 2] 1 is a block diagram showing a configuration example of a communication support device according to a first embodiment of the present disclosure. [Diagram 3] 3 is a table showing an example of a target person database stored in a storage unit of the communication support device shown in FIG. 2. [Figure 4A] 3 is a table showing an example of category ranking information stored in a storage unit of the communication support device shown in FIG. 2. [Figure 4B] 10 is a table showing another example of category ranking information stored in the storage unit of the communication support device shown in FIG. 2. [Figure 4C] 10 is a table showing still another example of category ranking information stored in the storage unit of the communication support device shown in FIG. 2. [Diagram 5] 5 is a flowchart showing an operation example of the communication support device according to the first embodiment of the present disclosure. [Figure 6] 6 is a flowchart showing an example of the target person detection process shown in FIG. 5. [Figure 7] FIG. 11 is a schematic diagram for explaining an example of a spatial distance. [Figure 8A] 6 is a flowchart illustrating a detailed flow of the long-distance flow illustrated in FIG. 5 . [Figure 8B]6 is a flowchart illustrating a detailed flow of the long-distance flow illustrated in FIG. 5 . [Figure 9] 8C is a flowchart illustrating a detailed flow of the facial expression identification processing step shown in FIG. 8B. [Figure 10] 8C is a flowchart illustrating a detailed flow of the action identification processing step shown in FIG. 8B. [Figure 11] 8C is a flowchart illustrating a detailed flow of the AC post-processing step shown in FIG. 8B. [Figure 12] 1 is a schematic diagram showing an example of an operation including a long-distance flow of the communication support device according to the first embodiment of the present disclosure. FIG. [Figure 13] 6 is a flowchart illustrating a detailed flow of the middle distance flow shown in FIG. 5 . [Figure 14] 1 is a schematic diagram showing an example of an operation including a mid-distance flow of the communication support device according to the first embodiment of the present disclosure. FIG. [Figure 15] 6 is a flowchart illustrating a detailed flow of the short-distance flow shown in FIG. 5 . [Figure 16] 1 is a schematic diagram showing an example of an operation including a short-distance flow of the communication support device according to the first embodiment of the present disclosure. FIG. [Figure 17] FIG. 4 is a diagram illustrating a configuration example of a vibration unit. [Figure 18] FIG. 11 is a block diagram showing a configuration example of a communication support system according to a second embodiment of the present disclosure. [Figure 19] 13 is a flowchart showing a modified example of the category rank setting process. [Figure 20] 13A to 13C are schematic diagrams for explaining an example of the operation of a communication robot according to a modified example of an embodiment of the present disclosure. [Figure 21] 13 is a table illustrating an example of a subject database according to a modified example of an embodiment of the present disclosure. [Figure 22] 11 is a table illustrating an example of category ranking information stored in a storage unit. [Figure 23]11A and 11B are schematic diagrams illustrating modified examples of output modes of the communication support device according to the embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, embodiments of a communication support device according to the present disclosure will be described with reference to the accompanying drawings. In the following embodiments, similar components are denoted by the same reference numerals.
[0012] 1. Application Examples FIG. 1 is a schematic diagram illustrating an application example of a communication support device 100 according to an embodiment of the present disclosure. The communication support device 100 provides communication support to, for example, a visually impaired user 90. Even if a target person 80 such as an acquaintance approaches, the visually impaired person has difficulty visually perceiving the figure, face, and the like of the target person 80. Therefore, it is difficult for the visually impaired person to recognize the name, facial expression, and actions of the target person 80. Therefore, even if the target person 80 approaches, the user 90 may not be able to take an action such as greeting the target person 80.
[0013] Therefore, the communication support device 100, for example, captures an image of the surroundings using the camera 3, analyzes the captured image to detect the target person 80, and outputs identification information such as the name of the target person 80, as well as information such as the position, distance, facial expression, and movement, to notify the user 90. The communication support device 100 can perform personal recognition while the user 90 is performing a movement (e.g., while walking) and / or while the user 90 is stopped (while not performing a movement).
[0014] The communication support device 100 is mounted on luggage such as a backpack 70 of the user 90. Alternatively, the communication support device 100 itself may be wearable. Information regarding the position of the target person 80 and the distance between the target person 80 and the user 90 can be notified to the user 90 by vibration using the vibration unit 2. On the other hand, it is difficult to notify the target person 80's identification information such as his / her name, facial expression, and movement by vibration, and therefore, for example, the information is notified to the user 90 by voice using the voice output unit 1 such as a speaker.
[0015] In order to detect the target person 80 as described above, for example, the communication support device 100 searches for a target person database in which targets such as acquaintances are registered, for a match or similar to the person appearing in the captured image. At this time, if the target person database is searched randomly, it may take a long time to detect the target person 80. This increases the response time of the communication support device 100, and there is a possibility that the user 90 may pass by the target person 80 while the target person 80 is detected and the voice output unit 1 is reading out information about the target person 80. This makes it difficult for the user 90 to communicate smoothly with the target person 80.
[0016] Therefore, the communication support device 100 according to the present disclosure, for example, associates information about the target person 80 with a category indicating the nature of the target person 80 and records the information in the target person database. For example, as shown in FIG. 1, the categories include classifications such as family, workplace, friends, neighbors, relatives, and others. The communication support device 100 according to the present disclosure acquires location information indicating the location of the user 90 from, for example, the GPS receiver 7, and sets priorities to the categories according to the location information. For example, when the user 90 is near the user 90's home, the priorities are assigned to each category as shown in the category ranking information 50a in FIG. 1. When the user 90 is near the user 90's workplace, the priorities are assigned to each category as shown in the category ranking information 50b in FIG. 1.
[0017] The communication support device 100 detects subjects belonging to categories in the captured image in order of priority. For example, when the user 90 is near the user 90's home, the communication support device 100 first searches only for subjects belonging to the "family" category and detects those who match or are similar to the person in the captured image (see the category order information 50a in FIG. 1). Next, the communication support device 100 searches only for subjects belonging to the "neighbor" category and detects those who match or are similar to the person in the captured image. The same applies to categories with lower priorities.
[0018] In this way, the communication support device 100 detects subjects belonging to categories in the captured image in the order of the set priority. For example, the communication support device 100 searches for subjects in the subject database that match or are similar to the subjects in the captured image, starting from those with high category ranks. This reduces the time required to detect the subject 80. If the subject 80 can be detected quickly, the notification unit 10 can also notify the user 90 of information quickly. Therefore, the user 90 can quickly obtain information about the subjects 80 in the vicinity, and can communicate with the subject 80 smoothly or naturally.
[0019] 2. First embodiment 2-1. Configuration example 2 is a block diagram showing a configuration example of the communication support device 100 according to the first embodiment of the present disclosure. The communication support device 100 includes a GPS (Global Positioning System) receiver 7, a camera 3, a control unit 4, a storage unit 5, a communication interface (I / F) 6, and a notification unit 10.
[0020] The GPS receiver 7 is a positioning device that measures the position of the communication support device 100. The GPS receiver 7 is an example of a "positioning unit" in the present disclosure. The positioning unit includes, for example, a satellite positioning system such as a GNSS (Global Navigation Satellite System) receiver. The GNSS includes, for example, a GPS. The GPS receiver 7 may be mounted in an information processing device such as a mobile phone or smartphone carried by the user 90 as long as it can measure the position of the user 90.
[0021] The camera 3 is an imaging device that captures an image of the environment around the user 90 and forms a captured image. The camera 3 is an example of an "imaging unit" in the present disclosure. The camera 3 captures an image of the user 90's surroundings at a predetermined frame rate, for example, and sequentially generates image data. The camera 3 forms a captured image using a solid-state imaging element such as a complementary MOS (CMOS) or a charge coupled device (CCD). The camera 3 is, for example, a wearable camera that can be worn by the user 90. For example, the camera 3 is a glasses-type camera mounted on glasses, and captures an image in the direction of the user 90's line of sight. The camera 3 may be mounted on the user 90's luggage, such as a backpack or a suitcase.
[0022] The control unit 4 is an information processing device that includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and controls each component of the communication support device 100 according to information processing. The control unit 4 includes, as components, for example, a position acquisition unit 48, a category ranking setting unit 49, an image acquisition unit 41, a target person detection unit 42, a target person tracking unit 43, a target person position detection unit 44, a distance measurement unit 45, a facial expression identification unit 46, and a motion identification unit 47. Each component of the control unit 4 may execute a process that is assigned to the control unit 4 by the control unit 4 executing a necessary program. Such a program may be stored in the storage unit 5. When the control unit 4 executes a necessary program, the target program stored in the storage unit 5 is expanded in the RAM. The control unit 4 interprets and executes the program expanded in the RAM by the CPU, and controls each component. An example of the operation of each component will be described later.
[0023] The communication interface 6 includes an interface circuit for enabling a communication connection between the communication support device 100 and an external device. The communication interface 6 performs communication according to standards such as IEEE802.3, IEEE802.11, Wi-Fi (registered trademark), LTE, 3G, 4G, and 5G. The communication interface 6 may be an interface circuit that performs communication according to standards such as USB (Universal Serial Bus), HDMI (registered trademark), IEEE1394, and Bluetooth (registered trademark).
[0024] The notification unit 10 is an output device for transmitting information to the user 90. The notification unit 10 includes, for example, an audio output unit 1 and a vibration unit 2. The audio output unit 1 is an output device that outputs audio according to the control of, for example, the control unit 4. The audio output unit 1 includes, for example, audio devices such as a speaker, earphones, and headphones. The vibration unit 2 is, for example, a vibrator that generates vibration according to the control of the control unit 4.
[0025] In this embodiment, an example will be described in which each function of the control unit 4 is realized by a CPU. However, some or all of the above functions may be realized by one or more dedicated processors. In addition, regarding the components of the control unit 4, functions may be omitted, replaced, or added as appropriate depending on the embodiment. The control unit 4 may be composed of various semiconductor integrated circuits such as a CPU, MPU, GPU, microcomputer, DSP, FPGA, and ASIC.
[0026] The storage unit 5 is a medium that accumulates information such as a program by electrical, magnetic, optical, mechanical, or chemical action so that a computer or other device, machine, etc. can read the recorded information such as the program. The storage unit 5 is, for example, an auxiliary storage device such as a hard disk drive or a solid state drive. The storage unit 5 stores, for example, category ranking information 50, a subject database 51, a facial expression database 52, a motion database 53, and a program executed by the control unit 4. The storage unit 5 may include a main storage device such as a RAM. Storing these data in the storage unit 5 is merely an example, and these data may be stored in, for example, an external server device with which the communication support device 100 can communicate via the communication interface 6.
[0027] 3 is a table showing an example of the target database 51 stored in the storage unit 5 of the communication support device 100 shown in FIG. 2. The target database 51 stores information about the target, such as the target's identification number (ID), an image including a face, name, category, relationship coefficient, address, telephone number, email address, and date of birth. The target is a person with whom the user 90 wishes to communicate or who can communicate, and includes, for example, an acquaintance of the user 90, a famous person, and the like. Not only people, but also things that can be the subject of communication, such as robots, may be registered as targets.
[0028] In FIG. 3, a category is, for example, a division indicating the nature of the subject. For example, the category is determined based on the address of the subject, the name or address of the place of work, etc. Such a category can be said to be a division indicating the nature of the subject's living area. Alternatively, for example, a category is a division indicating the nature of the subject based on the relationship between the subject and the user 90. Such a category includes, for example, divisions such as family, workplace, friends, neighbors, relatives, etc. The workplace category may include divisions related to the workplace, such as superior, subordinate, colleague, etc. For example, a category is specified by the user 90 when the user 90 registers the subject in the subject database 51.
[0029] For example, each category may be associated with a particular location. For example, the family category, neighbors category, etc. are associated with locations around the home. The friends category, relatives category may also be associated with locations around the home. Similarly, for example, the work category is associated with a work address. Which categories are associated with which locations may be preset by the user 90 and recorded in, for example, the subject database 51.
[0030] Also, in FIG. 3, the relationship coefficient is an index indicating, for example, the closeness of the relationship between the user 90 and the subject. The relationship coefficient is input in advance by the user 90, for example. The relationship coefficient may be determined based on the number of meetings between the user 90 and the subject, blood ties such as kinship, and the like. For example, the closer the relationship between the user 90 and the subject, the higher the relationship coefficient has. The relationship coefficient is expressed by a numerical value of, for example, 0 to 1.
[0031] 4A to 4C are tables showing an example of the category ranking information 50 stored in the storage unit 5 shown in FIG. 2. In the category ranking information 50, a ranking is set for each category included in the target person database 51 shown in FIG. 3. In the category ranking information 50c shown in FIG. 4A, rankings of 1st to 6th are set for the categories of friends, relatives, workplace, neighbors, family, and others. These rankings may be specified by the user 90 as initial values. The category ranking information 50 is updated from the initial value shown in FIG. 4A to category ranking information 50d shown in FIG. 4B in response to a change in the situation such as the location of the communication support device 100 or the user 90. In the category ranking information 50, a ranking according to the location of the communication support device 100 or the user 90 may be stored in advance as shown in the category ranking information 50e in FIG. 4C.
[0032] 2-2. Example of operation 2-2-1. Overall flow The communication support device 100 according to the present embodiment supports a user 90 in smooth communication with a target person such as an acquaintance by informing the user 90 of the name, location, distance, etc. of the target person. For example, the communication support device 100 supports a visually impaired user 90 in smooth communication with a target person such as an acquaintance. An example of the operation of the communication support device 100 will be described below with reference to FIG. 5.
[0033] 5 is a flowchart showing an example of the operation of the communication support device 100 according to this embodiment. The process shown in FIG.
[0034] (Step S91) First, the position acquisition unit 48 acquires, from the GPS receiver 7, position information indicating the position of the communication support device 100 or the user 90 measured by the GPS receiver 7 (S91).
[0035] (Step S92) Next, the category rank setting unit 49 sets a category rank in the category rank information 50 based on the position information acquired by the position acquisition unit 48 in step S91 (S92). For example, if the position acquired by the position acquisition unit 48 in step S91 is within the workplace area, the category rank setting unit 49 updates the category rank information 50 in the storage unit 5 from the initial value shown in Fig. 4A to the current value shown in Fig. 4B. Here, the "workplace area" refers to, for example, the area within the workplace of the user 90, or the area around the workplace including a point a predetermined distance away from the workplace of the user 90.
[0036] Alternatively, if the position acquired by the position acquisition unit 48 in step S91 is within the workplace area, the category rank setting unit 49 may update the category rank information 50 in the storage unit 5 from the rank in the initial value column shown in Fig. 4C to the value in the workplace area column. Also, if the position acquired by the position acquisition unit 48 in step S91 is within the home area, the category rank setting unit 49 may update the category rank information 50 in the storage unit 5 to the value in the home area column shown in Fig. 4C. Here, the "home area" refers to, for example, the area within the premises of the user 90's home, or the area around the home of the user 90 including a point a predetermined distance away from the premises of the user 90's home.
[0037] If the position acquired by the position acquisition unit 48 in step S91 is not included in the predetermined positions, for example, the category rank setting unit 49 may set the category rank information 50 in the storage unit 5 to an initial value.
[0038] (Step S1) Next, the image acquisition unit 41 acquires captured images captured by the camera 3 (S1). For example, the camera 3 captures images of the surrounding environment of the user 90 in time series to generate multiple captured image data. In this manner, the camera 3 may capture images at a constant frame rate. The camera 3 may also capture video. In step S1, the image acquisition unit 41 may acquire multiple captured images. The image acquisition unit 41 may acquire a video consisting of multiple frames, or may acquire multiple still images.
[0039] (Step S2) Next, the subject detection unit 42 analyzes the captured image acquired by the image acquisition unit 41 to detect a person (S2). Here, detecting a person includes detecting an area in the captured image in which a person is presumed to be included. If a person is not detected in step S2 (No in step S2), the control unit 4 ends the flow shown in FIG. 5.
[0040] (Step S93) If a person is detected in step S2 (Yes in step S2), the subject detection unit 42 detects whether the person detected in step S2 is a subject such as an acquaintance, based on the subject database 51 in which information about the face of the subject is stored (S93). What is executed in step S3 is an identity identification process to determine whether the face of the person detected in step S2 matches or resembles the face of the subject.
[0041] For example, in step S93, the subject detection unit 42 refers to the category rank information 50 and the subject database 51, and detects, in the captured image, subjects belonging to a category having the priority order stored in the category rank information 50. For example, in step S93, the subject detection unit 42 searches for people detected in step S2 in order of category rank, from a subject belonging to a category having the highest category rank to a subject belonging to a category having the lowest category rank. The subject detection unit 42 detects, in the captured image, a subject belonging to a category having the highest priority order, and if no subject is detected, detects a subject belonging to a category having a priority order lower than the highest priority order.
[0042] Fig. 6 is a flowchart showing an example of the subject detection process shown in Fig. 5. First, the subject detection unit 42 sets an integer i to 1, which is an initial value (S931).
[0043] Next, the subject detection unit 42 searches for the person detected in step S2 of Fig. 5 among subjects belonging to the i-th category in the subject database 51 (S932). For example, when the integer i is 1, the subject detection unit 42 identifies whether the face of the person detected in step S2 matches or resembles the face of a subject (D and E in Fig. 3) belonging to the category having the first category rank (friends in Fig. 4A).
[0044] If it is determined in step S2 that the person detected matches or is similar to one of the subjects belonging to the i-th category (Yes in step S933), the subject detection unit 42 sets the subject as the detection result (S934). Then, the process proceeds to step S3 in Fig. 5. In this case, since there is no need to search the subject database 51 for subjects other than those already searched, for example, subjects belonging to categories with priorities of i+1 or lower, the communication support device 100 can finish the search process early.
[0045] If it is determined in step S2 that the person detected does not match or resemble the subject belonging to the i-th category (No in step S933), the subject detection unit 42 determines whether or not it has finished searching all subjects in the subject database 51 (S935).
[0046] If all subjects in the subject database 51 have not yet been searched (No in step S935), the subject detection section 42 increments the integer i (S937) and returns to step S932.
[0047] When it is determined that all subjects in the subject database 51 have been searched (Yes in step S935), the subject detection unit 42 determines that no subjects have been detected (S936). In this case, data corresponding to the person detected in step S2 is not stored in the subject database 51. Then, the process proceeds to step S3 in FIG. 5.
[0048] (Step S3) Returning to FIG. 5, if a target person is detected in step S93 (Yes in step S3), the process proceeds to step S4; otherwise (No in step S3), the control unit 4 ends the flow shown in FIG.
[0049] (Step S4) When the target person is detected, the vibration unit 2 notifies the user 90 that the target person has been detected by vibration (S4). Specifically, the vibration unit 2 notifies the user 90 that the target person has been detected by vibrating with the first vibration pattern. In this way, the vibration unit 2 feeds back to the user 90 that the target person has been detected in order to call the attention of the user 90. Details of the vibration pattern will be described later.
[0050] In step S4, when the subject detected in step S3 enters an observation distance, a perception distance, or an exchange distance described below, feedback may be performed by vibrating with a first vibration pattern. In this sense, a predetermined distance such as an observation distance, a perception distance, or an exchange distance is called a feedback distance for performing feedback. That is, when the subject tracking unit 43 is tracking a subject, the vibration unit 2 may vibrate with a first vibration pattern when the distance between the subject detected by the distance measuring unit 45 and the camera 3 is the observation distance, the perception distance, or the exchange distance.
[0051] (Step S5) Next, the distance measurement unit 45 detects the spatial distance between the camera 3 and the target person based on, for example, the information detected in step S2 (S5). The spatial distance is an example of the "distance" in the present disclosure. Since the camera 3 is a camera such as a wearable camera worn by the user 90 or a camera mounted on the luggage of the user 90, the spatial distance between the camera 3 and the target person can be said to be approximately the same as the spatial distance between the user 90 and the target person.
[0052] Spatial distance generally means the distance between two points and may vary depending on the path between these points. The spatial distance between the user 90 and the subject can be roughly divided into three categories, for example, observation distance, perceptual distance, and interaction distance. Corresponding to these spatial distances, the space around the user 90 can be roughly divided into an observation space, a perceptual space, and an interaction space. FIG. 7 is a schematic diagram for explaining an example of spatial distance.
[0053] The observation distance is, for example, a distance at which a person carefully observes an object or person to obtain information. For example, when a person is within the observation distance, the user 90 can observe the other person and identify whether the other person is an acquaintance and who the acquaintance is. The observation distance represents, for example, a distance that is longer than a second predetermined threshold. The observation distance corresponds to a case where the spatial distance between individuals is, for example, greater than 3.6 m.
[0054] The perceptual distance is, for example, a distance at which the actions and / or emotions of another person are interpreted or recognized through senses (e.g., the five senses, particularly sight and hearing). For example, when another person is within the perceptual distance, the user 90 can observe the facial expressions, actions, etc. of the other person and recognize emotions such as joy, anger, and sadness. In addition, for example, when another person is within the perceptual distance, the user 90 can recognize actions such as whether the other person is looking at the user 90, whether the other person is working on a phone, tablet, etc., and whether the other person is waving at the user 90. The perceptual distance represents, for example, a distance that is longer than a first threshold value and equal to or less than a second threshold value that is determined in advance. The perceptual distance corresponds to a case where the spatial distance between individuals is, for example, greater than 1.2 m and equal to or less than 3.6 m.
[0055] The social distance is, for example, a distance at which interactions with other people, such as conversation, usually take place. The social distance represents, for example, a distance equal to or less than a first threshold. The social distance corresponds to a spatial distance between individuals of, for example, 1.2 m or less.
[0056] Here, spatial distances that can be distinguished into three categories, namely, observation distance, perception distance, and interaction distance, are exemplified. However, the concept of spatial distance is not limited to these. For example, spatial distances may be broadly categorized into, for example, public space, social space, and personal space based on Proxemics (Hall, ET, The hidden dimension, New York: Doubleday, 1966). In addition, spatial distances may differ depending on the intimacy between people, the culture to which the people belong, the gender of the people, and the like.
[0057] (Steps S6 to S11) Returning to FIG. 5, if the measurement result in step S5 shows that the spatial distance between the camera 3 and the subject is the observation distance (Yes in step S6), the control unit 4 proceeds to step S7 and executes the long-distance flow. If it is a perception space (Yes in step S8), the control unit 4 proceeds to step S9 and executes the medium-distance flow. If it is an interaction space (Yes in step S10), the control unit 4 proceeds to step S11 and executes the close-distance flow. In this way, communication support device 100 executes different processes depending on the spatial distance between the camera 3 and the subject at the time the subject was discovered in step S3.
[0058] 2-2-2.Long-distance flow 2-2-2-1. Overall flow 8A and 8B are flow charts illustrating detailed flow of the long distance flow (S7) shown in FIG. 5. FIG. 8A and FIG. 8B are connected by a connector A and a connector B. The flows shown in FIG. 8A and FIG. 8B may be executed in real time, that is, every time the image acquisition unit 41 acquires an image. Alternatively, the flows shown in FIG. 8A and FIG. 8B may be executed for a plurality of frames of captured images acquired by the image acquisition unit 41 over a period of several seconds to several tens of seconds.
[0059] (Step S701) First, the audio output unit 1 notifies the user 90 of the name of the target person detected in step S3 by voice (S701). This allows the user 90 to know the name of the target person near him / her and to identify who is near him / her. The name of the target person is an example of identification information for identifying the target person in this disclosure.
[0060] (Step S702) Next, the vibration unit 2 notifies the user 90 of the spatial distance to the subject and the position of the subject by vibration (S702). Specifically, the vibration unit 2 vibrates with the second vibration pattern to notify the user 90 that the subject is within the observation distance and the position of the subject. Details of the vibration pattern will be described later. The spatial distance is measured by the distance measurement unit 45 in step S5 of FIG. 5. The position of the subject is detected, for example, by the subject position detection unit 44. The position of the subject is specified, for example, based on the optical axis of the camera 3, depending on whether the subject is in the right direction, the front direction, or the left direction. Furthermore, the position of the subject may be specified, for example, based on the optical axis of the camera 3, depending on whether the subject is in the upper direction or the lower direction. In other words, the position of the subject may be represented by an index that specifies whether the subject is in the up, down, left, right, or front direction as viewed from the user 90.
[0061] (Step S703) Next, the subject tracking unit 43 tracks the subject detected in step S3 in Fig. 5 (S703). Specifically, the subject tracking unit 43 tracks the subject in the current frame captured after the reference frame based on the position in the image of the subject detected or tracked in the reference frame. The subject can be tracked, for example, by storing the subject in the reference frame as a template in the storage unit 5 and searching the current frame using the template by applying a method such as known template matching.
[0062] For example, the subject tracking unit 43 calculates a matching score indicating the likelihood that the subject detected or tracked in the reference frame and the subject tracked in the current frame are the same subject. The matching score is, for example, a value in the range of 0 to 1, and the larger the value, the higher the likelihood that the subject detected in the reference frame and the subject detected in the current frame are the same subject. If the matching score is equal to or greater than a predetermined threshold, the subject tracking unit 43 determines that the subject detected in the reference frame and the subject detected in the current frame are the same subject, and assumes that tracking of the subject has been successful.
[0063] (Step S704) The subject tracking unit 43 judges whether or not the subject detected or tracked in the tracking process is captured in the captured image (S704). For example, the subject tracking unit 43 judges whether or not the subject is captured in the current frame. If it is judged that the subject is captured in the captured image (Yes in step S704), the process proceeds to step S705, and if it is judged that the subject is not captured (No in step S704), the process proceeds to step S720.
[0064] (Step S705) The distance measurement unit 45 judges whether the subject has approached to the perceptual distance (S705). Specifically, the distance measurement unit 45 judges whether the subject who was at the observation distance in the frame used in step S5 of FIG. 5 is at the perceptual distance in the frame captured at a later time. For example, when the distance between the camera 3 and the subject becomes 3.6 m, the distance measurement unit 45 judges that the subject has approached to the perceptual distance. When the subject has approached to the perceptual distance (Yes in step S705), the process proceeds to step S706 in FIG. 8B, and when the subject has not approached to the perceptual distance (No in step S705), the process returns to step S703. When the subject has not approached to the perceptual distance, the control unit 4 may end the processes of FIG. 8A and FIG. 5.
[0065] (Step S706) In step S706 shown in Fig. 8B, the facial expression identification unit 46 identifies the facial expression of the target person based on the captured image (S706). Details of the facial expression identification process step S706 will be described later.
[0066] (Step S707) Next, the voice output unit 1 notifies the user 90 by voice of the facial expression information related to the facial expression identified by the facial expression identification unit 46 in step S706 (S707). Here, the facial expression information may include not only information representing the person's facial expression itself, but also information representing whether the person is wearing an attachment or a shield such as a mask, an eye patch, glasses, or sunglasses on the face. For example, the voice output unit 1 notifies the user 90 of facial expression information such as "smiling", "angry", "unclear expression", and "wearing a mask". This allows the user 90 to know the facial expressions of the target person close to him / her, and facilitates smooth communication according to the facial expression.
[0067] (Step S708) Next, the vibration unit 2 notifies the user 90 of the spatial distance to the target person and the target person's position by vibration (S708). Specifically, the vibration unit 2 vibrates with the third vibration pattern to notify the user 90 that the target person is within the perceptual distance and the target person's position. Details of the vibration pattern will be described later.
[0068] (Step S709) Next, the motion identification unit 47 identifies the motion of the subject based on the captured image (S709). Details of the motion identification process step S709 will be described later.
[0069] (Step S710) Next, the audio output unit 1 notifies the user 90 by voice of the motion information related to the motion identified by the motion identification unit 47 in step S709 (S710). For example, the audio output unit 1 notifies the user 90 of motion information such as "the target is looking at us", "the target is waving", "the target is talking on the phone", "the target is coughing", and "the target's motion is unknown". This allows the user 90 to know the target's motion and facilitate smooth communication according to the motion.
[0070] (Step S711) Next, the distance measurement unit 45 judges whether the subject has approached to the exchange distance (S711). Specifically, the distance measurement unit 45 judges whether the subject who was at the perception distance in the frame used in step S705 of FIG. 8A is at the exchange distance in the frame captured at a later time. For example, the distance measurement unit 45 judges that the subject has approached to the exchange distance when the distance between the camera 3 and the subject becomes 1.2 m. If the subject has approached to the exchange distance (Yes in step S711), the process proceeds to step S712, and if the subject has not approached to the exchange distance (No in step S711), the process returns to step S709. If the subject has not approached to the exchange distance even after a long time has passed, the control unit 4 may end the processes of FIG. 8B and FIG. 5.
[0071] (Step S712) Next, the vibration unit 2 notifies the user 90 of the spatial distance to the target person and the target person's position by vibration (S712). Specifically, the vibration unit 2 vibrates with the fourth vibration pattern to notify the user 90 that the target person is within the communication distance and the target person's position. Details of the vibration pattern will be described later. This allows the user 90 to know that the target person is within the communication distance and to start communication with the target person.
[0072] (Step S713) Following step S712, for example, after the user 90 finishes communication with the target person, the control unit 4 executes post-interaction processing (S713). For example, the control unit 4 controls the vibration unit 2 to notify the user 90 by vibration that the target person has left the user 90. This allows the user 90 to know that the target person has left, and allows the user 90 to resume the action, such as moving to a destination, that was being performed before the communication with the target person.
[0073] (Step S720) Returning to FIG. 8A, if it is determined in step S704 that the subject is not shown in the captured image, the control unit 4 determines whether the period during which the subject is not shown in the captured image is longer than a predetermined period. The predetermined period is, for example, one second to several minutes, for example, four seconds. Instead of the period, the number of frames may be predetermined. For example, the control unit 4 may determine whether the subject is shown in a predetermined number of consecutive frame images. If the period during which the subject is not shown in the captured image is longer than the predetermined period (Yes in step S720), the process proceeds to step S721, and if it is equal to or shorter than the predetermined period (No in step S720), the process returns to step S703.
[0074] (Step S721) If the period during which the target person is not captured in the captured image is longer than a predetermined period (Yes in step S720), the vibration unit 2 vibrates to notify the user 90 that the target person can no longer be tracked (S721). Specifically, the vibration unit 2 vibrates with the fifth vibration pattern to notify the user 90 that the target person can no longer be tracked. Details of the vibration pattern will be described later. This allows the user 90 to know that the target person can no longer be tracked or that the target person has left the vicinity of the user 90. After step S721, the components such as the control unit 4 and the notification unit 10 may resume a normal operation such as navigation to a destination that was performed before the detection of the target person for the user 90.
[0075] 2-2-2-2. Facial expression identification processing FIG. 9 is a flowchart illustrating a detailed flow of the facial expression identification process step S706 shown in FIG. 8B.
[0076] (Step S7061) First, the facial expression identification unit 46 analyzes the captured image and detects the face of the subject (S7061). Here, detecting the face includes detecting an area in the captured image that is estimated to include a human face.
[0077] (Step S7062) If the face of the target person is detected (Yes in step S7062), the process proceeds to step S7063. If the face of the target person is not detected (No in step S7062), the process proceeds to step S7067.
[0078] (Step S7063) When the subject's face is detected (Yes in step S7062), the facial expression identification unit 46 detects whether the subject is wearing a mask, an eye patch, glasses, sunglasses, or other such covering (S7063). Since the subject's face may not be detected because the subject is wearing a covering, an covering detection process is adopted in this embodiment. For example, the technology disclosed in JP 2018-151919 A is applied as a method for detecting whether the subject is wearing a covering.
[0079] When it is detected in step S7063 that the subject is wearing a mask (Yes in step S7064), the facial expression identification unit 46 identifies the facial expression information related to the subject's facial expression as a face wearing a mask (S7068). The facial expression identification unit 46 may identify what the mask is. For example, in step S7068, the facial expression identification unit 46 may identify the facial expression information related to the subject's facial expression as a face wearing a mask.
[0080] If it is not detected in step S7063 that the subject is wearing a shield (No in step S7064), the facial expression identification unit 46 identifies the facial expression of the subject (S7065). For example, the facial expression identification unit 46 compares the facial expression of the subject in the captured image with the facial expression database 52 that stores information about human facial expressions, and identifies the facial expression of the subject.
[0081] A known method may be used to identify the facial expression. For example, the facial expression identification unit 46 detects facial organs (hereinafter referred to as "facial organs") in the face of the subject. The facial organs are a collection of tissues and include those having a specific function. For example, the facial organs include the eyes, nose, mouth, and ears. The facial organs may include the skin. The entire face may be included in the facial organs. The facial expression identification unit 46 identifies the facial expression based on the information of the detected facial organs. For example, the facial expression identification unit 46 detects the distance between facial feature points such as the eyes, eyebrows, and mouth, or the edges of the face surface from the information of the facial organs as information representing the facial expression.
[0082] For example, the facial expression identification unit 46 extracts features related to the relative positions and shapes of facial organs based on the position information of the facial organs. The features include, for example, Haar-like features, feature point distances, and Fourier descriptors. Next, the extracted features may be input to a facial expression classifier that classifies facial expressions, and a facial expression score (facial expression component value) may be output. The facial expression score may include, for example, a smile level indicating the degree of smile, a happiness level indicating the degree of happiness, and the like. The facial expression classifier is constructed by learning a large number of sample facial images by machine learning such as a neural network and a self-organizing map.
[0083] Facial expression identification includes distinguishing the type of facial expression, that is, identifying the type of facial expression of the target face by a word indicating emotion. Here, facial expression identification may be identification by a single word indicating emotion, or identification by a combination of words indicating emotion. When combining words indicating emotions, each word indicating emotion may be weighted. For example, facial expressions are classified into seven types, namely, "neutral," "happiness," "anger," "disgust," "surprise," "fear," and "sadness," based on Paul Ekman's facial expression analysis. As a result of facial expression identification, a score is output in which the degree of each of the seven types of expressions (also called "expression-likeness" or "expression degree") is quantified so that the sum of the scores is 1. The score of each facial expression is also called the facial expression component value.
[0084] The facial expression classifier does not have to be one, and may be seven classifiers each responsible for one of the seven types of facial expressions.
[0085] The technique for estimating facial expressions may be the techniques exemplified in JP 2019-111092 A, JP 2016-149063 A, JP 2014-206903 A, etc.
[0086] (Step S7066) The facial expression identification unit 46 identifies the facial expression of the target person as the one identified in step S7065 (S7066). For example, the facial expression identification unit 46 identifies the facial expression of the target person as a "joy" facial expression.
[0087] (Step S7067) If the subject's face is not detected in step S7061 (No in step S7062), the facial expression identification unit 46 identifies the subject's facial expression as "unknown" (S7067). The facial expression identification unit 46 may identify the subject's facial expression as "unknown because the face was not detected." Here, "if the subject's face was not detected" includes a case where the subject's face was not detected in one frame of captured image. Also, "if the subject's face was not detected" may include a case where face detection processing was attempted on multiple frames of captured image, but the subject's face was not detected in any of the frames.
[0088] The facial expression information identified in the facial expression identification process step S706 in FIG. 9 is notified to the user 90 in step S707 shown in FIG. 8B, as described above. For example, the voice output unit 1 notifies the user 90 of facial expression information such as "smiling," "angry," "unknown expression," and "wearing a mask." This allows the user 90 to know the facial expression of a target person close to the user, and allows smooth communication according to the facial expression. For example, when the user 90 is notified of facial expression information indicating that the target person is angry, the user 90 can choose not to talk to the target person.
[0089] Also, for example, in a situation where an infectious disease is prevalent, when the user 90 is notified of facial expression information indicating that the target person is wearing a mask, the user 90 can select to perform an action such as moving away from the target person to ensure social distance from the target person (social distancing). Alternatively, when the user 90 is notified of facial expression information indicating that the target person is wearing a mask, the user 90 can ask the target person about his / her physical condition. When the target person is sick or not feeling well, the user 90 can select to perform an action to ensure social distance from the target person. In this way, for example, in a situation where an infectious disease is prevalent, the user 90 can reduce the risk of infection from others such as the target person based on the information notified by the communication support device 100. In this way, communicating with others while adjusting the distance from others according to the situation is also included in smooth communication.
[0090] 2-2-2-3. Action identification process FIG. 10 is a flowchart illustrating a detailed flow of the action identification process step S709 shown in FIG. 8B.
[0091] (Step S7091) First, the motion identification unit 47 detects whether the subject is looking at the user 90 (S7091). For example, the motion identification unit 47 detects whether the subject's face is facing the camera 3. Since the camera 3 is a camera such as a wearable camera worn by the user 90 or a camera mounted on the user's luggage, when the subject's face is facing the camera 3, it can be regarded as the subject looking at the user 90. Alternatively, the motion identification unit 47 may detect the iris or pupil of the subject and detect the subject's line of sight based on the position of the detected iris or pupil. When the difference between the direction of the subject's line of sight and the optical axis direction of the camera 3 is small, it can be regarded as the subject looking at the user 90. When it is detected that the subject is looking at the user 90 (Yes in S7091), the process proceeds to step S7094.
[0092] (Steps S7092, S7093) When it is not detected that the subject is looking at the user 90 (No in S7091), the motion identification unit 47 identifies the motion of the subject (S7092). For example, the motion identification unit 47 compares the motion database 53 in which information on human motion is stored with information on the motion of the subject, such as the posture, in the captured image, and identifies the motion of the subject. The motion of the subject includes, for example, the subject being on the phone, reading a book, waving, running, walking, coughing, and the like. In the above example, it is determined whether the subject is looking at the user 90 in step S7091, but the present embodiment is not limited to this. For example, the determination of whether the subject is looking at the user 90 may be included in the motion identification in step S7092. When the motion of the subject is identified (Yes in step S7093), the process proceeds to step S7094, and when the motion of the subject is not identified (No in step S7093), the process proceeds to step S7095.
[0093] (Step S7094) When the motion of the target person is identified (Yes in step S7093), the motion identifying unit 47 identifies the motion of the target person as the one identified in step S7092 (step S7094). Also, when it is detected in step S7091 that the target person is looking at the user 90 (Yes in S7091), in step S7094, the motion of the target person is identified as looking at the user 90. The identified motion is notified to the user 90 in step S710 of FIG. 8B, as described above.
[0094] (Step S7095) If the subject's motion is not identified in step S7093, motion identification section 47 determines whether or not to attempt motion identification in other frame images captured at a time later than the time the frame image used in step S7092 was captured (S7095). Information regarding the period during which frame images captured are to be subjected to motion identification or information regarding how many frame images are to be subjected to motion identification may be determined in advance.
[0095] (Step S7096) In step S7095, when it is determined that further motion identification is not to be attempted (No in step S7095), the motion identification unit 47 identifies the motion of the subject as “unknown” (S7096). The identified motion is notified to the user 90 in step S710 of FIG. 8B, as described above.
[0096] (Step S7097) The motion identification section 47 acquires another frame image captured at a time later than the time at which the frame image used in step S7092 was captured (S7097).
[0097] (Step S7098) Steps S7098 to S7100 are the same as steps S704, S720, and S721 in Fig. 8A. In step S7098, the motion identification unit 47 judges whether or not the target person is captured in the frame image acquired in step S7097 (S7098). If it is judged that the target person is captured in the frame image (Yes in step S7098), the process returns to step S7091, and if it is judged that the target person is not captured (No in step S7098), the process proceeds to step S7099.
[0098] (Step S7099) The control unit 4 determines whether the period during which the subject is not shown in the captured image is longer than a predetermined period (S7099). If the period during which the subject is not shown in the captured image is longer than the predetermined period (Yes in step S7099), the process proceeds to step S7100, and if the period is equal to or shorter than the predetermined period (No in step S7099), the process returns to step S7091.
[0099] (Step S7100) If the period during which the target person is not captured in the captured image is longer than a predetermined period (Yes in step S7099), the vibration unit 2 vibrates to notify the user 90 that the target person can no longer be tracked (S7100). Specifically, the vibration unit 2 vibrates with the fifth vibration pattern to notify the user 90 that the target person can no longer be tracked. This allows the user 90 to know that the target person can no longer be tracked or that the target person has left the vicinity of the user 90. After the notification in step S7100, the control unit 4 ends the series of processes shown in FIG. 5. The flow in FIG. 5 may be started repeatedly. The flow in FIG. 5 is started, for example, at predetermined intervals.
[0100] 2-2-2-4. AC Post-processing FIG. 11 is a flowchart illustrating a detailed flow of the AC post-processing step S713 shown in FIG. 8B.
[0101] (Step S7131) First, the target person tracking unit 43 tracks the target person (S7131). In the tracking step S7131, the same process as the tracking step S703 shown in FIG. 8A may be executed.
[0102] (Step S7132) Next, the subject tracking unit 43 judges whether or not the subject detected or tracked in the tracking process is captured in the captured image (S7132). For example, the subject tracking unit 43 judges whether or not the subject is captured in the current frame. If it is judged that the subject is captured in the captured image (No in step S7132), the process proceeds to step S7133, and if it is judged that the subject is not captured (Yes in step S7132), the process proceeds to step S7135.
[0103] (Step S7133) The control unit 4 determines whether the period during which the subject is not shown in the captured image is longer than a predetermined period (S7133). If the period during which the subject is not shown in the captured image is longer than the predetermined period (Yes in step S7133), the process proceeds to step S7134, and if the period is equal to or shorter than the predetermined period (No in step S7133), the process returns to step S7131.
[0104] (Step S7134) If the period during which the target person is not captured in the captured image is longer than a predetermined period (Yes in step S7133), the vibration unit 2 vibrates to notify the user 90 that the target person can no longer be tracked (S7134). Specifically, the vibration unit 2 vibrates with the fifth vibration pattern to notify the user 90 that the target person can no longer be tracked. This allows the user 90 to know that the target person can no longer be tracked or that the target person has left the vicinity of the user 90. This allows the user 90 to concentrate on an action such as moving to a destination. Since the user 90 can concentrate on his or her own action, the safety of the user 90 is also ensured. Step S7134 may be the same step as step S721 shown in FIG. 8A.
[0105] (Step S7135) In step S7132, if it is determined that the subject is captured in the captured image (Yes in step S7132), the distance measurement unit 45 detects whether the subject is approaching (S7135). Specifically, the distance measurement unit 45 compares the distance between the camera 3 and the subject in two frame images captured at different times, and detects whether the subject is approaching.
[0106] If it is detected that the target is approaching (Yes in step S7135), there is a possibility that the user 90 will communicate with the target again. Therefore, the control unit 4 may once end the series of processes shown in Fig. 5 and start the process of Fig. 5 again. The flow of Fig. 5 is started, for example, at predetermined intervals. If it is not detected that the target is approaching (No in step S7135), the control unit 4 returns to step S7131 and continues tracking.
[0107] 2-2-2-5. Example of long distance flow 12 is a schematic diagram showing an example of the operation of the communication support device 100 when the spatial distance to the detected target person 80 is the observation distance (Yes in step S6 of FIG. 5). When the communication support device 100 detects the target person 80 in the observation space, it vibrates the vibration unit 2 in a first pattern to call the attention of the user 90. Next, the audio output unit 1 notifies the user 90 of the name of the target person 80 by voice (S701), and the vibration unit 2 notifies the spatial distance to the target person 80 and the position of the target person 80 by vibrating in a second vibration pattern (S702).
[0108] Thereafter, when the target person 80 enters the perception space, the audio output unit 1 notifies the user 90 of the facial expression information of the target person 80 by audio (S707), and the vibration unit 2 notifies the user 90 of the spatial distance from the target person and the position of the target person by vibrating with the third vibration pattern (S708). Furthermore, the audio output unit 1 notifies the user 90 of the facial expression information of the target person 80 by audio (S710).
[0109] Thereafter, when the target person 80 enters the interaction space, the vibration unit 2 notifies the user 90 of the spatial distance from the target person and the target person's position by vibrating with the fourth vibration pattern (S712). This allows the user 90 to know that the target person is within the interaction distance, and the user 90 can start communication with the target person. After the user 90 finishes communication with the target person, for example, the vibration unit 2 notifies the user 90 that the target person has left the user 90 by vibrating with the fifth vibration pattern (S7134). Thereafter, the components such as the control unit 4 and the notification unit 10 may resume the normal operation, such as navigation to the destination, that was performed before the detection of the target person.
[0110] 2-2-3.Mid-distance flow Fig. 13 is a flow chart illustrating a detailed flow of the medium-distance flow (S9) shown in Fig. 5. In the medium-distance flow, steps that are the same as or similar to steps explained in the long-distance flow described above are given the same reference numerals. As such steps, the medium-distance flow includes steps S701, S706 to S710, S712, and S713. Duplicate explanations of such steps may be omitted.
[0111] In the medium distance flow, first, the voice output unit 1 notifies the user 90 of the name of the target person detected in step S3 by voice (S701). Next, the vibration unit 2 notifies the user 90 of the spatial distance to the target person and the position of the target person by vibration (S708). Specifically, the vibration unit 2 vibrates with the third vibration pattern to notify the user 90 that the target person is within the perceptual distance and the position of the target person.
[0112] Next, the facial expression identification unit 46 identifies the facial expression of the target person based on the captured image (S706). Next, the audio output unit 1 notifies the user 90 of facial expression information related to the facial expression identified by the facial expression identification unit 46 in step S706 by voice (S707). The facial expression identification process step S706 may be executed before step S701 of notifying the target person's name.
[0113] Next, the distance measurement unit 45 judges whether the target person has approached to within the communication distance (S94). Step S94 may be the same as step S711 shown in Fig. 8B. If it is judged that the target person has approached to within the communication distance (Yes in step S94), the process proceeds to step S712, and if it is judged that the target person has not approached to within the communication distance (No in step S94), the process proceeds to step S709.
[0114] When it is determined that the target person has approached within the AC distance (Yes in step S94), the vibration unit 2 notifies the user 90 of the spatial distance to the target person and the target person's position by vibration (S712). Specifically, the vibration unit 2 vibrates with the fourth vibration pattern to notify the user 90 that the target person is within the AC distance and of the target person's position.
[0115] Next, for example, after the user 90 finishes communication with the target person, the control unit 4 executes post-interaction processing (S713).
[0116] If it is determined in step S94 that the target person has not approached within the communication distance (No in step S94), the motion identifying unit 47 identifies the motion of the target person based on the captured image (S709). Next, the audio output unit 1 notifies the user 90 of the motion information related to the motion identified by the motion identifying unit 47 in step S709 by audio (S710).
[0117] FIG. 14 is a schematic diagram showing an example of the operation of the communication support device 100 when the spatial distance to the detected target person 80 is the perceived distance (when Yes is selected in step S8 of FIG. 5). Compared with the schematic diagram of the observation distance shown in FIG. 12, in FIG. 14, in particular, step S710 of notifying the target person's action by voice is omitted. When the target person approaches to the exchange distance, if step S710 of notifying the target person's action by voice is executed, for example, the target person may pass the other person during the notification, and smooth communication may not be possible. Therefore, in the medium-distance flow, step S710 is omitted depending on the distance to the target person. However, as described above, in the medium-distance flow, the control unit 4 additionally executes the action identification processing step S709 when, for example, the target person does not approach to the exchange distance and remains at the perceived distance. Thereby, even if the communication support device 100 detects the target person for the first time at the perceived distance, it also provides the user 90 with action information depending on the situation. By receiving sufficient information according to the situation, the user 90 can smoothly communicate with the target person.
[0118] 2-2-4. Close-distance flow Fig. 15 is a flow chart illustrating a detailed flow of the short-distance flow (S11) shown in Fig. 5. In the short-distance flow, steps that are the same as or similar to steps described in the long-distance flow above are given the same reference numerals. As such steps, the short-distance flow includes steps S701, S712, and S713. Duplicate explanations of such steps may be omitted.
[0119] In the short distance flow, first, the audio output unit 1 notifies the user 90 of the name of the target person detected in step S3 by voice (S701). Next, the vibration unit 2 notifies the user 90 of the spatial distance to the target person and the position of the target person by vibration (S712). Specifically, the vibration unit 2 notifies the user 90 that the target person is within communication distance and the position of the target person by vibrating with the fourth vibration pattern. Next, for example, after the user 90 finishes communication with the target person, the control unit 4 executes post-communication processing (S713).
[0120] FIG. 16 is a schematic diagram showing an example of the operation of the communication support device 100 when the spatial distance to the detected target person 80 is the interaction distance (when the answer is Yes in step S10 in FIG. 5). Compared with the schematic diagram of the case of the perception distance shown in FIG. 14, in FIG. 16, in particular, step S707 of notifying the target person's facial expression by voice is further omitted. When the target person is found at a short distance such as the interaction distance, the user 90 can immediately start communication with the target person. In such a case, there is no time to notify the user 90 of a lot of information such as the target person's facial expression and actions by voice or vibration, and if such information is notified to the user 90, it is usually not possible to perform smooth communication. Therefore, in the short-distance flow, the communication support device 100 aims to facilitate smooth communication between the user 90 and the target person by notifying the user 90 of only the minimum information such as the target person's name, distance, and position.
[0121] 2-3.Vibration pattern 2-3-1. Structure of the vibration part The vibration unit 2 and the vibration pattern of the vibration unit 2 will be described with reference to FIG. 17. FIG. 17 is a diagram showing a configuration example of the vibration unit 2. The vibration unit 2 is attached to, for example, a shoulder belt of a backpack 70. The vibration unit 2 includes two or more vibration elements that vibrate according to a control signal from the control unit 4. In the example of FIG. 17, the vibration unit 2 includes first to sixth vibration elements 21 to 26 that are regularly arranged. The first to third vibration elements 21 to 23 are attached to a left shoulder belt 71, and the fourth to sixth vibration elements 24 to 26 are attached to a right shoulder belt 72. The first to third vibration elements 21 to 23 are aligned in the direction in which the left shoulder belt 71 extends. The fourth to sixth vibration elements 24 to 26 are aligned in the direction in which the right shoulder belt 72 extends. The first to third vibration elements 21 to 23 are an example of the "left vibration element" of the present disclosure. The fourth to sixth vibration elements 24 to 26 are aligned in the extension direction of the right shoulder belt 72. The first to third vibration elements 21 to 23 are an example of the "right vibration element" of the present disclosure.
[0122] 2-3-2. First vibration pattern In the above example, the first vibration pattern is a vibration pattern for notifying the user 90 that a target person has been detected, as described in step S4 of Fig. 5. The first vibration pattern has a function of calling the attention of the user 90.
[0123] In the first vibration pattern, the control unit 4 vibrates the first to sixth vibration elements 21 to 26 of the vibration unit 2 in sequence. That is, in the first vibration pattern, the control unit 4 controls the vibrations in the order of the first vibration element 21, the second vibration element 22, the third vibration element 23, the fourth vibration element 24, the fifth vibration element 25, and the sixth vibration element 26. When the backpack 70 is seen from above while being carried by the user 90, in the first vibration pattern, the first to sixth vibration elements 21 to 26 vibrate in sequence counterclockwise. The user 90 can know that the target person is nearby by feeling the counterclockwise vibration. This counterclockwise vibration may be executed once or multiple times.
[0124] 2-3-3. Second vibration pattern In the above example, the second vibration pattern is a vibration pattern for notifying the user 90 that the subject is within the observation distance and the position of the subject, as described in step S702 of FIG. 8A.
[0125] For example, each of the vibration elements 21-26 of the vibration unit 2 can vibrate at three vibration intensities: weak, medium, and strong. In the second vibration pattern, each of the vibration elements 21-26 of the vibration unit 2 notifies the user 90 that the subject is within the observation distance, for example, by weak vibration. Alternatively, the control unit 4 may notify the user 90 of the position of the subject by changing the vibration elements such as the frequency and waveform of the vibration by each of the vibration elements 21-26.
[0126] Regarding the position of the subject, for example, when the subject is on the left, the vibration unit 2 vibrates only the first to third vibration elements 21 to 23 attached to the left shoulder belt 71 to notify the user 90 that the subject is on the left. Also, for example, when the subject is on the right, the vibration unit 2 vibrates only the fourth to sixth vibration elements 24 to 26 attached to the right shoulder belt 72 to notify the user 90 that the subject is on the right. Also, for example, when the subject is in the front direction, the vibration unit 2 vibrates all of the first to sixth vibration elements 21 to 26.
[0127] In this manner, the vibration section 2 can notify the user 90 of both the spatial distance to the target person and the target person's position almost simultaneously by controlling the vibration strength and the vibration position.
[0128] 2-3-4.Third vibration pattern In the above example, the third vibration pattern is a vibration pattern for notifying the user 90 that the target person is within the perception distance and the position of the target person, as described in step S708 of FIG. 8B and FIG. 13. In the third vibration pattern, each of the vibration elements 21-26 of the vibration unit 2 notifies the user 90 that the target person is within the perception distance, for example, by vibrating with a medium intensity. The third vibration pattern is similar to the above-mentioned second vibration pattern except for the vibration intensity, so a duplicated description will be omitted.
[0129] 2-3-5. Fourth vibration pattern In the above example, the fourth vibration pattern is a vibration pattern for notifying the user 90 that the target person is within the AC distance and the target person's position, as described in step S712 of FIG. 8B and FIG. 13. In the fourth vibration pattern, each of the vibration elements 21 to 26 of the vibration unit 2 notifies the user 90 that the target person is within the AC distance, for example, by vibrating with a strong intensity. The fourth vibration pattern is similar to the above-mentioned second and third vibration patterns except for the vibration intensity, so a duplicated description will be omitted.
[0130] 2-3-6. Fifth vibration pattern In the above example, the fifth vibration pattern is a vibration pattern for notifying the user 90 that the target person being tracked can no longer be tracked, as described in step S721 of Fig. 8A and step S7134 of Fig. 11. The fifth vibration pattern allows the user 90 to know that the target person can no longer be tracked or that the target person has left the vicinity of the user 90.
[0131] In the fifth vibration pattern, the control unit 4 vibrates the first to sixth vibration elements 21 to 26 of the vibration unit 2 in sequence in a different order from the first vibration pattern. For example, in the fifth vibration pattern, the control unit 4 controls the vibrations in the order of the sixth vibration element 26, the fifth vibration element 25, the fourth vibration element 24, the third vibration element 23, the second vibration element 22, and the first vibration element 21. When the backpack 70 is seen from above while being carried by the user 90, the first to sixth vibration elements 21 to 26 vibrate in sequence clockwise in the fifth vibration pattern. By feeling the clockwise vibration, the user 90 can know that the target person has become unable to be tracked or that the target person has left the vicinity of the user 90. This clockwise vibration may be performed once or multiple times.
[0132] Furthermore, the first to fifth vibration patterns are not limited to those described above, and may be any patterns that allow the user 90 to distinguish the first to fifth vibration patterns from one another. For example, the first to fifth vibration patterns may be patterns that vibrate at different vibration periods from one another.
[0133] 2-3-7. Variations In the example of FIG. 17, the first to sixth vibration elements 21 to 26 of the vibration unit 2 are attached to the shoulder belts 71, 72 of the backpack 70. However, the present disclosure is not limited to this, and it is sufficient that the user 90 can distinguish the first to fifth vibration patterns from each other. For example, the first to sixth vibration elements 21 to 26 of the vibration unit 2 may be attached to the belt, wristband, wristwatch, hat, etc. of the user 90. The first to sixth vibration elements 21 to 26 of the vibration unit 2 may be incorporated into smart wear. Here, smart wear is, for example, a wearable device in which a sensor, an output device, etc. are attached to a wearable material such as clothing. The first to sixth vibration elements 21 to 26 of the vibration unit 2 are incorporated into smart textiles and materials constituting the smart wear.
[0134] 2-4. Actions and Effects As described above, the communication support device 100 according to the present embodiment includes the position acquisition unit 48, the camera 3, the storage unit 5, the category order setting unit 49, the target person detection unit 42, and the notification unit 10. The position acquisition unit 48 acquires position information indicating the position of the user 90. The camera 3 captures an image of the surrounding environment of the user 90 to acquire a captured image. The storage unit 5 stores a target person database 51 in which an image of the target person and a category indicating the nature of the target person are associated with the target person. The category order setting unit 49 sets a priority order to the category according to the position information acquired by the position acquisition unit 48. The target person detection unit 42 detects a target person belonging to a category in the captured image in the order of priority set by the category order setting unit 49. The notification unit 10 notifies the user 90 of information related to the target person detected by the target person detection unit 42.
[0135] In this way, the subject detection unit 42 detects subjects belonging to a category in the captured image in the order of priority set by the category ranking setting unit 49. For example, the communication support device 100 searches the subject database for subjects that match or are similar to the subject in the captured image, starting from the subject with the highest category ranking. This allows the communication support device 100 to reduce the time required to detect the subject. If the subject can be detected quickly, the notification unit 10 can also notify the user 90 of information quickly. Therefore, the user 90 can quickly obtain information about the subjects in his or her surroundings and can communicate with the subject smoothly or naturally.
[0136] The communication support device 100 may further include a distance measurement unit 45, a facial expression identification unit 46, and a motion identification unit 47. The distance measurement unit 45 measures the distance between the target person and the camera 3 based on the captured image. The facial expression identification unit 46 identifies the facial expression of the target person based on the captured image. The motion identification unit 47 identifies the motion of the target person based on the captured image. When the distance between the target person and the camera 3 measured by the distance measurement unit 45 is an exchange distance equal to or less than a first threshold, the notification unit 10 may notify the user 90 of identification information for identifying the target person by voice when the distance between the target person and the camera 3 is greater than the first threshold, the notification unit 10 may notify the user 90 of at least one of the facial expression information related to the facial expression identified by the facial expression identification unit 46 and the motion information related to the motion identified by the motion identification unit 47, and the identification information by voice.
[0137] With this configuration, when the distance between the target person and the camera 3 is greater than the first threshold, the communication support device 100 notifies the user 90 of at least one of the facial expression information and the motion information in addition to the identification information, thereby notifying the user 90 of as much information as possible. On the other hand, when the distance between the target person and the camera 3 is small, if the communication support device 100 reads out all the information detected by the communication support device 100, there is a possibility that the user 90 may pass the target person while reading out the information. In this case, the user 90 cannot communicate smoothly with the target person. Therefore, when the distance between the target person and the camera 3 is equal to or less than the first threshold, the communication support device 100 notifies the user 90 of only the identification information. In this way, the communication support device 100 omits the information to be notified when the distance between the target person and the camera 3 is small, thereby supporting the user 90 to communicate smoothly with the target person.
[0138] When the distance measured by the distance measurement unit 45 is a perceived distance that is longer than the first threshold value and equal to or smaller than the second threshold value, the notification unit 10 may notify the user 90 of one of the facial expression information and the action information, and the identification information. When the distance measured by the distance measurement unit 45 is an observation distance that is greater than the second threshold value, the notification unit 10 may notify the user 90 of the facial expression information, the action information, and the identification information.
[0139] In this way, by gradually reducing the amount of information communicated to the user 90 as the distance between the subject and the camera 3 becomes shorter, the user 90 can quickly learn appropriate information about the subject and can communicate smoothly with the subject.
[0140] When the distance between the subject and the camera 3 measured by the distance measurement unit 45 at the time of detection by the subject detection unit is the perceived distance, the notification unit 10 may notify the user 90 of the identification information by voice and then notify the user 90 of one of the facial expression information and the movement information by voice. When the distance between the subject and the camera 3 measured by the distance measurement unit 45 thereafter is the perceived distance, the notification unit 10 may notify the user 90 of the other of the facial expression information and the movement information by voice.
[0141] With this configuration, even if the target is detected for the first time at the perceptual distance, when the target does not approach to the interaction distance and remains at the perceptual distance, the user 90 can know both facial expression information and action information. In this way, by receiving an appropriate amount of information according to the situation, the user 90 can communicate smoothly with the target.
[0142] The notification unit 10 may further include a vibration unit 2 that notifies the user 90 of the detection result by the distance measurement unit 45 by vibration. The vibration unit 2 may change at least one of the pattern and the magnitude of the vibration according to the distance between the subject detected by the distance measurement unit 45 and the camera 3.
[0143] The vibration unit 2 allows the communication support device 100 to notify the user 90 of a part or all of the detection result in a shorter time than when the communication support device 100 notifies the user 90 of the result by voice. Furthermore, by combining the notification by voice and the notification by vibration, the communication support device 100 can convey more information to the user 90 in a shorter time than when the communication support device 100 uses only the notification by voice.
[0144] The camera 3 may capture a plurality of captured images by capturing an image of the surrounding environment of the user 90 in a time series. In this case, the communication support device 100 may further include a subject tracking unit 43 that tracks the subject detected by the subject detection unit 42 in the plurality of captured images. The vibration unit 2 may include two or more vibration elements that can vibrate. In this case, when the subject tracking unit 43 is tracking the subject, and the distance between the subject detected by the distance measurement unit 45 and the camera 3 is a predetermined feedback distance, the vibration unit 2 may vibrate each vibration element sequentially in a predetermined order. When the subject tracking unit 43 can no longer track the subject that it has been tracking, the vibration unit 2 may vibrate each vibration element sequentially in an order different from the predetermined order.
[0145] By having a vibration pattern in which two or more vibration elements vibrate in different orders, the communication support device 100 can inform the user 90 of various information by vibration. Due to the difference in the vibration pattern, the user 90 can recognize the difference between the fact that the distance with the target is the feedback distance and there is a possibility of communication, and the fact that the target that was being tracked can no longer be tracked.
[0146] The vibration elements may be three or more vibration elements arranged regularly. When the subject tracking unit 43 is tracking a subject, and the distance between the subject and the camera 3 detected by the distance measuring unit 45 is the feedback distance, the vibration unit 2 may vibrate each vibration element sequentially in the order of arrangement. When the subject tracking unit 43 can no longer track the subject that it has been tracking, the vibration unit 2 may vibrate each vibration element sequentially in the order of arrangement reverse to the order of arrangement.
[0147] By having a vibration pattern in which three or more vibration elements vibrate in different orders, the user 90 can more clearly distinguish and recognize the differences in information notified by vibration, compared to a case in which there are only two or less vibration elements.
[0148] The communication support device 100 may further include a subject position detection unit 44 that detects the position of the subject relative to the camera 3 based on the captured image. The vibration element may include a right vibration element and a left vibration element. The vibration unit 2 may vibrate the right vibration element when the position of the subject relative to the camera 3 detected by the subject position detection unit 44 is to the right of the optical axis of the camera 3, and may vibrate the left vibration element when the position is to the left of the optical axis of the camera 3.
[0149] With this configuration, the communication support device 100 can notify the user 90 of the relative position of the target person more clearly by vibration.
[0150] 3. Second embodiment 18 is a block diagram showing a configuration example of a communication support system 300 according to a second embodiment of the present disclosure. The communication support system 300 includes a communication support device 200, a server device 305, and a GPS receiver 7. The GPS receiver 7 is carried by a user 90. As described in the first embodiment, the GPS receiver 7 may be mounted in an information processing device such as a mobile phone or a smartphone carried by the user 90.
[0151] The server device 305 is an information processing device including a control unit 306, a storage unit 307, and a communication interface 308. The control unit 306 of the server device 305 is an information processing device similar to the control unit 4. The communication interface 308 is an example of a “transmitter” and a “receiver” in the present disclosure, and is, for example, an interface circuit similar to the communication interface 6.
[0152] The communication support device 200 includes a camera 3, a control unit 304, a communication interface 6, and a notification unit 10. Compared with the communication support device 100 in Fig. 2, the communication support device 200 in Fig. 18 does not include in its storage unit the category ranking information 50, the target person database 51, the facial expression database 52, and the action database 53. Instead, the category ranking information 50, the target person database 51, the facial expression database 52, and the action database 53 are stored in a storage unit 307 of a server device 305 that is communicatively connected to the communication support device 200 via a network 350.
[0153] The control unit 304 of the communication support device 200 includes an image acquisition unit 41, a subject detection unit 42, a subject tracking unit 43, a subject position detection unit 44, a distance measurement unit 45, a facial expression identification unit 46, and an action identification unit 47. Compared to the control unit 4 in Fig. 2, the control unit 304 in Fig. 18 does not include the position acquisition unit 48 and the category order setting unit 49. Instead, the control unit 306 of the server device 305 includes the position acquisition unit 48 and the category order setting unit 49.
[0154] The position acquisition unit 48 acquires position information indicating the position of the user 90 from the GPS receiver 7 via the network 350 and the communication interface 308 .
[0155] In the above example, the notification unit 10 is included in the communication support device 200, but the present disclosure is not limited to this. For example, the notification unit 10 may be installed in an information processing device such as a mobile phone or a smartphone carried by the user 90.
[0156] As described above, the server device 305 according to this embodiment includes the position acquisition unit 48, the subject database 51, the category rank setting unit 49, and the communication interface 308. The position acquisition unit 48 of the control unit 306 acquires position information indicating the position of the user 90, for example, from the GPS receiver 7. The control unit 306 transmits information in the subject database 51 and information indicating the priority order set by the category rank setting unit 49 to the communication support device 200 via the communication interface 308 and the network 350.
[0157] Communication support device 200 receives information in subject database 51 and information indicating the priority order set by category rank setting unit 49 from server device 305. Communication support device 200 includes a camera 3, a subject detection unit 42, and a notification unit 10. Camera 3 captures an image of the surrounding environment of user 90 to obtain a captured image. The subject detection unit 42 detects subjects belonging to a category in the captured image in the order of priority set by the category rank setting unit 49 of the server device 305. The notification unit 10 notifies the user 90 of information related to the subjects detected by the subject detection unit 42.
[0158] The communication support device 200 allows the user 90 to quickly obtain information about the target person in the vicinity, as in the first embodiment. Therefore, the user 90 can communicate with the target person smoothly or naturally. Furthermore, the server device 305 stores a database that may be large in capacity, and executes processing by the category ranking setting unit 49. This allows the communication support device 200 to retrieve necessary data via the network 350 only when necessary, and the processing load applied to the control unit 304 can be reduced. Furthermore, since the communication support device 200 does not need to be equipped with a processor having high processing power or a large-capacity storage device, it is possible to realize low cost and size reduction.
[0159] 4. Variations Although the embodiment of the present disclosure has been described in detail above, the above description is merely an example of the present disclosure in every respect. Various improvements and modifications can be made without departing from the scope of the present disclosure. For example, the following modifications are possible. In the following, the same reference numerals are used for the same components as in the above embodiment, and the description of the same points as in the above embodiment is omitted as appropriate. The following modifications can be combined as appropriate. In the following, modifications of the first embodiment of the present disclosure will be mainly described, but these modifications can also be applied to the second embodiment of the present disclosure.
[0160] 4-1.First modified example In FIG. 5 of the first embodiment, a processing example has been described in which step S91 of acquiring the position of the user 90 and step S92 of setting a category rank are executed before the image acquisition step S1. However, the embodiment of the present disclosure is not limited to this. For example, step S91 and step S92 may be executed as a process separate from the process after the image acquisition step S1 in FIG. 5. For example, step S91 of acquiring the position of the user 90 and step S92 of setting a category rank may be executed in parallel with the process after the image acquisition step S1 in FIG. 5.
[0161] Furthermore, the step of setting the category ranking may not be executed when the user 90 is not moving, including walking, running, etc. This makes it possible to omit the processing when the user 90 is not moving, and to reduce the load on the control unit 4.
[0162] 19 is a flowchart showing a category rank setting process S192 which is a first modified example of the step of setting such a category rank. The step S91 (see FIG. 5) of acquiring the position of the user 90 and the category rank setting process S192 may be repeatedly executed by the control unit 4 at a predetermined cycle, for example.
[0163] First, the category ranking setting unit 49 detects the movement of the user 90 (S193). For example, in order to detect the movement of the user 90, the communication support device 100 may further include a sensor such as an accelerometer or a speedometer. Alternatively, the movement of the user 90 may be detected by a sensor such as an accelerometer or a speedometer mounted on a portable device such as a smartphone of the user 90. In this case, the detection result is transmitted from the portable device such as a smartphone of the user 90 to the communication support device 100, for example, via a network or by wireless communication.
[0164] If no movement of the user 90 is detected (No in step S194), the category rank setting unit 49 does not update the category rank information 50 in the storage unit 5 (S200).
[0165] When movement of the user 90 is detected (Yes in step S194), the category order setting unit 49 starts measuring the period and distance from that time and position (S195). For example, the category order setting unit 49 resets the period and distance, and starts counting the period and distance. Regarding the distance, the category order setting unit 49 may store the position as a predetermined position in the storage unit 5.
[0166] Next, the position acquisition unit 48 acquires, from the GPS receiver 7, position information indicating the position of the communication support device 100 or the user 90 measured by the GPS receiver 7, and transmits the position information to the category rank setting unit 49 (S196). Step S196 may be the same as step S91 shown in FIG. 5.
[0167] Next, the category order setting unit 49 determines whether or not the communication support device 100 or the user 90 has moved a predetermined distance or more from the predetermined position in step S195 (S197). Here, the predetermined distance is, for example, a distance of 100 m or more, such as 500 m, 1 km, 2 km, 5 km, or 10 km.
[0168] When it is determined that the communication support device 100 or the user 90 has moved a predetermined distance or more (Yes in step S197), the category rank setting unit 49 updates the category rank information 50 in the storage unit 5 (S198).
[0169] If it is not determined in step S197 that the communication support device 100 or the user 90 has moved a predetermined distance or more (No in step S197), the category ranking setting unit 49 determines whether a predetermined period has elapsed from the time in step S195 (S199). Here, the predetermined period is, for example, the period required for the user 90 to walk 1 km. For example, the predetermined period is a period of 3 minutes or more, such as 5 minutes, 10 minutes, 12 minutes, 15 minutes, 30 minutes, etc.
[0170] If it is determined that the predetermined period has elapsed (Yes in step S199), the process returns to step S193. If it is not determined that the predetermined period has elapsed (No in step S199), the process returns to step S196.
[0171] In this way, by omitting the processing when the user 90 is not moving, it is possible to reduce the load on the control unit 4 including the category ranking setting unit 49. This allows the control unit 4 to allocate processing power to other processing such as the processing after the image acquisition step S1 in Fig. 5, for example, and improve the processing speed. Therefore, the control unit 4 can execute the communication support processing at a processing speed closer to real time, and support the realization of smooth communication by the user 90.
[0172] 4-2. Second modified example In the first embodiment, the communication support device 100 carried by the user 90 has been described. However, the present disclosure is not limited thereto. For example, the communication support device 100 may be mounted on a navigation robot that supports the movement of the user 90 by performing an operation such as guiding the user 90 to a destination. Such a navigation robot can support smooth communication by the user 90 in addition to supporting the movement of the user 90. Such a navigation robot may be, for example, a self-propelled robot that supports the movement of the user 90 by self-propelling and leading the user 90 to a place where the user 90 wants to go.
[0173] Alternatively, the communication support device according to the embodiment of the present disclosure may be applied to a case where the user is a robot. Fig. 20 is a schematic diagram for explaining an example of an operation of the communication robot 110 according to the second modified example. The communication robot 110 is equipped with the communication support device 100, 200 according to the above embodiment.
[0174] The communication robot 110 travels within the premises of, for example, a factory 400 and performs work while communicating with other people, other robots, etc. For example, the communication robot 110 is a transport robot that travels by itself to enter a building 401, receives a part manufactured in the building 401 from a worker, and delivers the part to a worker in a building 402. Alternatively, the communication robot 110 may be a work robot that travels within the buildings 401 to 406 and performs manufacturing work performed within the buildings 401 to 406. The communication robot 110 may be a collaborative robot that performs these tasks in cooperation with a person or a work robot. The communication robot 110 may also be a guide robot that guides visitors to the factory 400.
[0175] Since the communication robot 110 moves around multiple buildings and the grounds of the factory 400, the number of people with whom it can communicate is large, and the size of the target database becomes large. Therefore, it takes time for the communication robot 110 to search the target database for people or robots in the vicinity, and there is a risk that it will not be possible to smoothly communicate with the targets in real time.
[0176] Therefore, the communication robot 110 registers the name of the building to which each subject belongs as a category, as shown in Fig. 21, and updates the category ranking information according to the position of the communication robot 110 as shown in Fig. 22. Fig. 21 is a table showing an example of the subject database 51a in the second modified example. The subject database 51a stores information about the subject, such as the subject's identification number (ID), an image including the face, the name, and the building to which the subject belongs as a category.
[0177] 22 is a table showing an example of category ranking information stored in the storage unit 5. The category ranking information is updated in response to a change in the position of the communication robot 110. For example, when the communication robot 110 is inside or near the building 401, the category ranking information is set to the value shown in table 50f. Thereafter, when the communication robot 110 moves to inside or near the building 402, the category ranking information is updated to the value shown in table 50g.
[0178] Furthermore, if the site of a building is large, the number of subjects with whom the communication robot 110 can communicate within the building is large, and the size of the subject database becomes large. In such a case, it may take time for the communication robot 110 to search the subject database for nearby people or robots, and it may not be possible to smoothly communicate with the subjects in real time. Therefore, the categories may be further classified into smaller categories and assigned to multiple areas within the building.
[0179] In this way, by appropriately setting the category ranking information according to the position of the communication robot 110, the communication robot 110 can shorten the response time for detecting the target person, and can smoothly communicate with the target person.
[0180] 4-3.Third modified example FIG. 23 is a schematic diagram showing a modified example of the output mode of the communication support device 100. In the above embodiment, the output mode of the communication support device 100 that provides communication support to a visually impaired user 90 has been described. As such an output mode, the notification unit 10 including the audio output unit 1 that provides notification by audio and the vibration unit 2 that provides notification by vibration has been described. The output modes of the present disclosure are not limited to these, and may be any that can notify the user 90 of information. For example, the communication support device 100 may include a display 8 for visually notifying the user 90.
[0181] For example, the display 8 displays the image captured by the camera 3, the subject's name, the distance from the camera 3 or the user 90 to the subject, and the subject's position, movement, and facial expression. This allows the user 90 to quickly grasp the above information at a glance of the display 8, and to smoothly communicate with the subject. [Explanation of symbols]
[0182] 1 Audio output section 2 Vibration section 3 Camera (imaging unit) 4. Control section 5 Storage section 6 Communication Interface 7 GPS receiver (positioning unit) 8. Display 10. Notification Department 21 First Vibration Element 22 Second Vibration Element 23 The third vibration element 24 The Fourth Vibration Element 25 The fifth vibration element 26 The sixth vibration element 41 Image acquisition unit 42 Target detection unit 43 Subject Tracking Department 44 Target position detection unit 45 Distance measurement unit 46 Facial expression identification part 47 Operation specific part 48 Position acquisition part 49 Category ranking setting section 50 Category Ranking Information 51 Subject Database 52 Facial Expression Database 53 Action Database 70 Backpack 71 Left shoulder belt 72 Right shoulder belt 80 Target 90 users 100 Communication support device 110 Communication Robot 200 Communication Support Device 300 Communication Support System 304 Control Unit 305 Server device 306 Control Unit 307 Storage section 308 Communication Interface 350 Network 400 Factories 401~406 Building
Claims
1. A location acquisition unit that acquires location information indicating a user's location; An imaging unit that captures an image of a surrounding environment of the user and acquires a captured image; A storage unit that stores a subject database in which an image of a subject and a category indicating a characteristic of the subject are associated with the subject; a category ranking setting unit that sets priorities of the categories according to the location information; a subject detection unit that detects subjects belonging to the categories in the captured image in the order of priority set by the category ranking setting unit; a notification unit that notifies the user of information regarding the subject detected by the subject detection unit; a distance measurement unit that measures a distance between the subject and the imaging unit based on the captured image; a facial expression identification unit that identifies a facial expression of the subject based on the captured image; a motion identification unit that identifies a motion of the subject based on the captured image, The notification unit is When the distance measured by the distance measurement unit is an AC distance equal to or less than a first threshold, notifying the user of identification information for identifying the target person by voice; A communication support device that, when the distance measured by the distance measurement unit is greater than the first threshold value, notifies the user by voice of at least one of facial expression information regarding the facial expression identified by the facial expression identification unit and action information regarding the action identified by the action identification unit, and the identification information.
2. The notification unit is When the distance measured by the distance measurement unit is a perceptual distance that is longer than the first threshold value and equal to or less than a second threshold value, notifying the user of one of the facial expression information and the action information and the identification information; The communication support device according to claim 1 , wherein when the distance measured by the distance measurement unit is an observation distance greater than the second threshold, the facial expression information, the action information, and the identification information are notified to the user.
3. The notification unit is when the distance between the subject and the imaging unit measured by the distance measurement unit at the time of detection by the subject detection unit is the perceived distance, notifying the user of the identification information by voice, and then notifying the user of one of the facial expression information and the action information by voice; 3. The communication support device according to claim 2, wherein when the distance between the subject and the imaging unit measured by the distance measurement unit thereafter is the perceived distance, the other of the facial expression information and the action information is notified to the user by voice.
4. The notification unit further includes a vibration unit that notifies the user of the detection result by the distance measurement unit by vibration, The communication support device according to any one of claims 1 to 3, wherein the vibration unit changes at least one of a pattern and a magnitude of vibration depending on a distance between the subject and the imaging unit detected by the distance measurement unit.
5. the imaging unit captures an image of a surrounding environment of the user in time series to obtain a plurality of captured images; The communication support device further includes a subject tracking unit that tracks the subject detected by the subject detection unit in the plurality of captured images, The vibration unit includes two or more vibration elements that can vibrate. The vibration unit includes: When the subject tracking unit is tracking the subject, if a distance between the subject and the imaging unit detected by the distance measurement unit is a predetermined feedback distance, each vibration element is vibrated sequentially in a predetermined order; The communication support device according to claim 4 , wherein the target tracking unit, when it becomes unable to track the target that it has been tracking, vibrates each vibration element in sequence in a sequence different from the predetermined sequence.
6. The vibration elements are three or more vibration elements regularly arranged, The vibration unit includes: When the subject tracking unit is tracking the subject, if the distance between the subject and the imaging unit detected by the distance measurement unit is the feedback distance, vibrating each vibration element in sequence in the order of arrangement; The communication support device according to claim 5 , wherein the target tracking unit vibrates each of the vibration elements in sequence in a reverse order to the arrangement order when the target tracking unit is unable to track the target that it has been tracking.
7. A subject position detection unit detects a position of the subject with respect to the imaging unit based on the captured image, The vibration elements include a right vibration element and a left vibration element, 7. The communication support device according to claim 5 or claim 6, wherein the vibration unit vibrates the right vibration element when the position of the subject relative to the imaging unit detected by the subject position detection unit is to the right of the optical axis of the imaging unit, and vibrates the left vibration element when the position of the subject relative to the imaging unit detected by the subject position detection unit is to the left of the optical axis of the imaging unit.
8. A communication support device that receives data from a server device, The server device includes: A location acquisition unit that acquires location information indicating a user's location; a subject database in which an image of a subject and a category indicating a characteristic of the subject are associated with the subject; a category ranking setting unit that sets priorities of the categories according to the location information; a transmission unit that transmits, as the data, information in the target database and information indicating the priority order set by the category ranking setting unit to a communication support device that detects the target person based on the information and notifies the user; Equipped with The communication support device includes: An imaging unit that captures an image of a surrounding environment of the user and acquires a captured image; a subject detection unit that detects subjects belonging to the category in the captured image in the order of priority received from the server device; a notification unit that notifies the user of information regarding the subject detected by the subject detection unit; a distance measurement unit that measures a distance between the subject and the imaging unit based on the captured image; a facial expression identification unit that identifies a facial expression of the subject based on the captured image; a motion identification unit that identifies a motion of the subject based on the captured image, The notification unit is When the distance measured by the distance measurement unit is an AC distance equal to or less than a first threshold, notifying the user of identification information for identifying the target person by voice; A communication support device that, when the distance measured by the distance measurement unit is greater than the first threshold value, notifies the user by voice of at least one of facial expression information regarding the facial expression identified by the facial expression identification unit and action information regarding the action identified by the action identification unit, and the identification information.
9. A location acquisition unit acquires location information indicating a location of a user; An imaging unit captures an image of a surrounding environment of the user to obtain an image; storing a subject database in a storage unit in which an image of a subject and a category indicating a characteristic of the subject are associated with the subject; a category ranking setting unit setting a priority order for the categories according to the location information; a subject detection unit detecting, in the captured image, subjects belonging to the category in the order of priority set by the category ranking setting unit; a distance measurement unit measuring a distance between the subject and the imaging unit based on the captured image; A facial expression identification unit identifies a facial expression of the subject based on the captured image; a step of identifying a motion of the subject based on the captured image by a motion identifying unit, The communication support method includes: When the distance measured by the distance measurement unit is an exchange distance equal to or less than a first threshold, notifying the user of identification information for identifying the target person by voice, a step of notifying the user by voice of at least one of facial expression information related to the facial expression identified by the facial expression identification unit and action information related to the action identified by the action identification unit, and the identification information, when the distance measured by the distance measurement unit is greater than the first threshold value; Communication support methods.
10. A program for causing a computer to execute the communication support method according to claim 9.
Citation Information
Patent Citations
Intelligent auxiliary device and method for travelling of blind people
CN108836769A
Security system
JP2000126160A
Image recognition apparatus, image recognition processing method and image recognition program
JP2004127285A
