Information collection system, information collection method, and information collection program

Abstract

To identify a target to which a riding person pays attention appropriately and in real time while an automatically driving PMV is traveling within a site.SOLUTION: An information collection system 500 collects information in a movement area 800 in which a PMV 200 is moved. The information collection system 500 comprises a position sensor for acquiring positional information 41 of the PMV 200 and a rising person sensor for acquiring a visual line direction of a riding person riding on the PMV 200. The information collection system 500 further comprises an attention target identification unit for identifying an object or an event existing in the visual line direction of the riding person as an attention target to which the riding person pays attention based on the positional information of the PMV 200 and the visual line direction of the riding person.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information collection device, an information collection method, and an information collection program. In particular, it relates to a technique for collecting in real time information on objects that a passenger of an autonomous personal mobility vehicle (PMV) pays attention to during movement.

Background Art

[0002] When moving within the premises of various facilities such as theme parks, entertainment facilities, or elderly care facilities using an autonomous PMV, it is required to appropriately and in real time collect information on what the passengers of the autonomous PMV pay attention to. The autonomous PMV is a moving body such as, for example, an electric wheelchair or a senior car. By collecting information on objects that passengers pay attention to within the premises of various facilities and analyzing the collected information, it can be used to improve the comfort and safety of the autonomous PMV.

[0003] Patent Document 1 discloses a technique for reading the facial expressions of passengers and estimating whether the movement on the links during driving is comfortable.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, it is estimated whether the movement in the link during driving is comfortable based on the facial expression of the passenger. In Patent Document 1, it only estimates whether the link is comfortable or uncomfortable, and does not specify what the passenger is looking at to become comfortable or uncomfortable. Therefore, in Patent Document 1, information on the object that the passenger pays attention to during driving cannot be collected appropriately and in real time, and cannot be used to improve the comfort and safety of the automatic driving PMV.

[0006] In the present disclosure, it is an object to appropriately and in real time identify the object that the passenger of the automatic driving PMV is paying attention to during the driving of the automatic driving PMV.

Means for Solving the Problems

[0007] The information collection system according to the present disclosure is an information collection system that collects information in a moving area where a moving body moves. A position sensor that acquires the position information of the moving body, A passenger sensor that acquires the line-of-sight direction of a passenger riding on the moving body, Based on the position information of the moving body and the line-of-sight direction of the passenger, it includes a target identification unit that identifies an object or event existing in the line-of-sight direction of the passenger as a target object that the passenger is paying attention to.

Effects of the Invention

[0008] In the information collection system according to the present disclosure, the target identification unit identifies an object or event existing in the line-of-sight direction of the passenger as a target object that the passenger is paying attention to based on the position information of the moving body and the line-of-sight direction of the passenger. Therefore, according to the information collection system according to the present disclosure, it is possible to appropriately and in real time identify the object that the passenger of the moving body is paying attention to while the moving body is traveling in the moving area.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0010] Hereinafter, this embodiment will be described with reference to the drawings. In each figure, the same or corresponding parts are denoted by the same reference numerals. In the description of the embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate. Also, in the following figures, the relationship of the sizes of the respective components may be different from the actual ones. Further, in the description of the embodiment, directions or positions such as up, down, left, right, front, back, front side, and back side may be indicated. These notations are for convenience of explanation and do not limit the arrangement, direction, and orientation of the device, instrument, or parts, etc.

[0011] Embodiment 1. ***Description of Configuration*** FIG. 1 is a diagram showing an overall configuration example of an information collection system 500 according to this embodiment. The information collection system 500 is a system that collects various types of information in a moving area 800 where a moving body moves.

[0012] The PMV200 is an example of a moving body that moves in the moving area 800. The PMV200 is a PMV that performs autonomous driving. The PMV200 is also called an autonomous driving PMV. Although not shown in the figure, the passenger 30 is a person who rides in the PMV200 and moves. By setting a destination in the autonomous driving system mounted on the PMV200, the passenger 30 can automatically move to the destination without driving himself. Therefore, even while riding, the passenger 30 can move while viewing the surrounding scenery or the surrounding environment.

[0013] The PMV refers to a powered vehicle in which the passengers are not spatially enclosed like a car but are exposed to the environment as it is. Specifically, the PMV refers to a moving body such as an electric wheelchair or a senior car. In particular, those with an autonomous driving function in these moving bodies are called autonomous driving PMVs or simply PMVs.

[0014] The moving area 800 assumes an area where the PMV200 moves by automatic driving, such as within the site of a senior facility, within the site of an entertainment facility, or within the site of a theme park. Even when the moving object moves in a moving area such as a general road or a sidewalk, the present embodiment can be applied.

[0015] The information collection system 500 includes the PMV200 and the server device 300. In the present embodiment, the information collection device 100 is configured to be mounted on the PMV200. The PMV200 and the server device 300 communicate via a network. The server device 300 may be installed within the moving area 800 or outside the moving area 800.

[0016] FIG. 2 is a diagram showing an example of the PMV200 according to the present embodiment. The PMV200 is an example of a moving object. The PMV200 includes a position sensor 201, a passenger sensor 202, an imaging device 203, an automatic driving system 204, and an information collection device 100.

[0017] The position sensor 201 acquires the position information of the PMV200. More specifically, the position sensor 201 acquires the position information and time of the PMV200 during movement. The position sensor 201 is, for example, a GPS (Global Positioning System).

[0018] The passenger sensor 202 acquires the line-of-sight direction of the passenger 30 riding on the PMV200. The passenger sensor 202 also acquires emotion information indicating the type of emotion of the passenger 30 towards the object of attention from the facial expression of the passenger 30. The passenger sensor 202 is, for example, a smartphone or a tablet terminal with a camera function. The passenger sensor 202 acquires the line-of-sight direction of the passenger 30 by photographing the face of the passenger 30 with the camera function and analyzing the photographed face image. In addition, the passenger sensor 202 obtains emotion information representing the emotion of the passenger 30 by analyzing the facial image. The emotion information may be the facial expression. If the expression is a smile, it is considered to mean emotions such as comfortable, enjoyable, or happy. Also, if the expression is a severe expression, it means emotions such as discomfort, anger, or sadness. The passenger sensor 202 obtains emotion information representing the facial expression or emotion from the facial image.

[0019] The imaging device 203 includes an imaging device 203 capable of imaging in all circumferential directions. The imaging device 203 images, for example, the line-of-sight direction of the passenger 30. The imaging device 203 images the line-of-sight direction of the passenger 30 in response to a command from the information collection device 100.

[0020] The automatic driving system 204 includes a communication device that communicates with external devices via a network, an automatic driving control unit that performs automatic driving control, and a display device that displays a map and the like.

[0021] The information collection device 100 acquires information from the position sensor 201, the passenger sensor 202, or the imaging device 203, and identifies the object of attention that the passenger 30 is paying attention to while moving in the movement area 800. The information collection device 100 is a computer. In the present embodiment, the information collection device 100 is mounted on a smartphone or a tablet including the passenger sensor 202. However, the information collection device 100 may be mounted on a computer different from the passenger sensor 202. Alternatively, all of the position sensor 201, the passenger sensor 202, and the information collection device 100 may be mounted on one computer such as a smartphone or a tablet.

[0022] FIG. 3 is a diagram showing a configuration example of the information collection device 100 according to the present embodiment. The information collection device 100 is a computer. The information collection device 100 includes a processor 910 and other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. The processor 910 is connected to other hardware via signal lines and controls these other hardware.

[0023] As functional elements, the information collection device 100 includes an information acquisition unit 110, a sensor information reception unit 120, a target of interest identification unit 130, a display unit 140, and a storage unit 150. Map information 51 and a target of interest database 52 are stored in the storage unit 150.

[0024] The functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140 are realized by software. The storage unit 150 is provided in the memory 921. Note that the storage unit 150 may be provided in the auxiliary storage device 922, or may be distributed and provided in the memory 921 and the auxiliary storage device 922.

[0025] The processor 910 is a device that executes an information collection program. The information collection program is a program that realizes the functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140. The processor 910 is an IC (Integrated Circuit) that performs arithmetic processing. Specific examples of the processor 910 are a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit).

[0026] The memory 921 is a storage device that temporarily stores data. Specific examples of the memory 921 are SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The auxiliary storage device 922 is a storage device for storing data. A specific example of the auxiliary storage device 922 is an HDD. Also, the auxiliary storage device 922 may be a portable storage medium such as an SD (registered trademark) memory card, CF, NAND flash, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, or DVD. Note that HDD is an abbreviation for Hard Disk Drive. SD (registered trademark) is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash (registered trademark). DVD is an abbreviation for Digital Versatile Disk.

[0027] The input interface 930 is a port connected to an input device such as a mouse, keyboard, or touch panel. Specifically, the input interface 930 is a USB (Universal Serial Bus) terminal. Note that the input interface 930 may be a port connected to a LAN (Local Area Network).

[0028] The output interface 940 is a port to which a cable of an output device such as a display is connected. Specifically, the output interface 940 is a USB terminal or an HDMI (registered trademark) (High Definition Multimedia Interface) terminal. Specifically, the display is an LCD (Liquid Crystal Display). The output interface 940 is also referred to as a display interface.

[0029] The communication device 950 has a receiver and a transmitter. The communication device 950 is connected to a communication network such as a LAN, the Internet, or a telephone line. Specifically, the communication device 950 is a communication chip or a NIC (Network Interface Card).

[0030] The information collection program is executed in the information collection device 100. The information collection program is loaded into the processor 910 and executed by the processor 910. In the memory 921, not only the information collection program but also the OS (Operating System) is stored. The processor 910 executes the information collection program while executing the OS. The information collection program and the OS may be stored in the auxiliary storage device 922. The information collection program and the OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910. Note that part or all of the information collection program may be incorporated into the OS.

[0031] The information collection device 100 may include a plurality of processors that replace the processor 910. These multiple processors share the execution of the information collection program. Each processor is a device that executes the information collection program in the same way as the processor 910.

[0032] Data, information, signal values, and variable values used, processed, or output by the information collection program are stored in the memory 921, the auxiliary storage device 922, or registers or cache memories within the processor 910.

[0033] The "unit" of each of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140 may be read as "circuit", "process", "procedure", "processing", or "circuitry". The information collection program causes a computer to execute information acquisition processing, sensor information reception processing, target information identification processing, and display processing. The "processing" of the information acquisition processing, the sensor information reception processing, the target information identification processing, and the display processing may be read as "program", "program product", "computer-readable storage medium storing a program", or "computer-readable recording medium recording a program". Also, the information collection method is a method performed when the information collection device 100 executes the information collection program. The information collection program may be stored and provided on a computer-readable recording medium. Further, the information collection program may be provided as a program product.

[0034] FIG. 4 is a diagram showing a configuration example of the server device 300 according to the present embodiment. The server device 300 is a computer. The server device 300 includes a processor 910 and also includes other hardware such as a memory 921, an auxiliary storage device 922, an input interface 930, an output interface 940, and a communication device 950. The processor 910 is connected to other hardware via signal lines and controls these other hardware. In FIG. 4, for simplicity of explanation, the same reference numerals are given to the hardware having the same functions as those in FIG. 3, but the information collection device 100 and the server device 300 each include separate hardware.

[0035] The server device 300 includes, as functional elements, a map setting unit 310, a focus setting unit 320, an information distribution unit 330, a notification unit 340, and a storage unit 350. The storage unit 350 stores map information 51, a focus database 52, and notification conditions 53. The description of the hardware in the server device 300 is the same as that described for the information collection device 100.

[0036] ***Description of Operations*** Next, the operation of the information collection system 500 according to the present embodiment will be described. The operation procedure of the information collection system 500 corresponds to an information collection method. Further, the program for realizing the operation of the information collection system 500 corresponds to an information collection program.

[0037] <Focus Target Identification Process> FIG. 5 is a flowchart showing an example of the focus target identification process according to the present embodiment. In the present embodiment, the focus target identification process is performed by the information collection device 100 mounted on the PMV200.

[0038] In step S101, the information acquisition unit 110 of the information collection device 100 acquires the map information 51 and the target database 52 distributed from the server device 300 via the communication device 950. Then, the information acquisition unit 110 stores the distributed map information 51 and target database 52 in the storage unit 150. The server device 300 collects information regarding the target of interest from the PMV 200 moving in the moving area 800, and updates the map information 51 and the target database 52. Then, each time the server device 300 updates the map information 51 and the target database 52, or periodically, it distributes the latest map information 51 and target database 52 to the PMV 200. Therefore, the process of step S101 does not necessarily need to be performed at the beginning of the target-of-interest identification process, and it may be performed when the map information 51 and the target database 52 are distributed.

[0039] In step S102, the sensor information reception unit 120 of the information collection device 100 receives sensor information from various sensors mounted on the PMV 200.

[0040] Specifically, the sensor information reception unit 120 receives the position information 41 of the PMV 200 from the position sensor 201. In addition, the sensor information reception unit 120 receives the line-of-sight direction 42 of the passenger 30 riding in the PMV 200 from the passenger sensor 202. In addition, the sensor information reception unit 120 receives the emotion information 43 indicating the type of emotion of the passenger 30 from the passenger sensor 202. Note that the position information 41, the line-of-sight direction 42, and the emotion information 43 are associated with each other by time.

[0041] Next, the target-of-interest identification unit 130 of the information collection device 100 identifies, as a target of interest 45 that the passenger 30 is focusing on, an object or event existing in the line-of-sight direction 42 of the passenger 30 based on the position information 41 of the PMV 200 and the line-of-sight direction 42 of the passenger 30. For example, the target-of-interest identification unit 130 identifies the target of interest 45 from the map information 51 stored in the storage unit 150. Specifically, it is as follows.

[0042] First, in step S103, the target-of-interest specifying unit 130 converts the line-of-sight direction 42 of the passenger 30 into the absolute coordinate system. The position information 41 is acquired as the position in the absolute coordinate system by means of GPS or the like. Also, the line-of-sight direction 42 is acquired as the direction in the PMV coordinate system based on the traveling direction of the PMV200 from the image of the face of the passenger 30. The target-of-interest specifying unit 130 calculates the traveling direction of the PMV200 from the position information 41 of the PMV200. The traveling direction of the PMV200 is acquired as the direction of the absolute coordinate system. The target-of-interest specifying unit 130 calculates in which direction the line-of-sight direction 42 is directed with respect to the traveling direction of the PMV200, and converts the line-of-sight direction 42 of the passenger 30 into the absolute coordinate system.

[0043] Then, in step S104, the target-of-interest specifying unit 130 specifies, from the map information 51, an object or event existing in the line-of-sight direction 42 of the passenger 30 as the target of interest 45 based on the position of the PMV200 and the line-of-sight direction represented in the absolute coordinate system.

[0044] Note that the map information 51 is provided with peripheral environment information representing the peripheral environment on the map including the moving area 800. The peripheral environment information is an object or event that can be seen in the moving area 800, such as buildings, sculptures, attractions, plants, and animals set in the moving area 800. Also, events such as parades or fireworks held in the moving area 800, and events such as scenery that can be seen from the moving area 800 are also provided as the peripheral environment information in the map information 51. The target-of-interest specifying unit 130 specifies, as the target of interest 45, an object or event existing in the line-of-sight direction 42 of the passenger 30 from the peripheral environment information provided in the map information 51.

[0045] In step S105, the target-of-interest specifying unit 130 transmits target-of-interest information 40 including the position information 41 of the PMV 200, the target of interest 45, the line-of-sight direction 42, the position of the target of interest 45, and the emotion information 43 to the server device 300. Note that when the target of interest 45 is a scenery outside the moving area 800, the position of the target of interest 45 may not be included in the target-of-interest information 40. The position information 41 of the PMV 200, the target of interest 45, the line-of-sight direction 42, the position of the target of interest 45, and the emotion information 43 included in the target-of-interest information 40 are associated with a time. Therefore, it can be estimated that the emotion information 43 represents the emotion of the passenger with respect to the target of interest 45.

[0046] <Target setting process> FIG. 6 is a flowchart showing an example of the target setting process according to the present embodiment. The target setting unit 320 of the server device 300 sets the position of the target of interest 45 and the emotion information 43 of the passenger 30 with respect to the target of interest 45 in the target database 52.

[0047] In step S201, the target setting unit 320 receives the target-of-interest information 40 from the information collection device 100 via the communication device 950. In step S202, the target setting unit 320 sets the position information 41 of the PMV 200, the target of interest 45, the line-of-sight direction 42, the position of the target of interest 45, and the emotion information 43 included in the target-of-interest information 40 in the target database 52.

[0048] In step S203, the target setting unit 320 calculates a degree of attention 48 representing the degree of attention to the target of interest 45. For example, the target setting unit 320 totals the total number of passengers who have passed through a position where the target of interest 45 can be seen in the past, totals the number of passengers who have paid attention to the target of interest 45 among those numbers, and calculates the ratio of the passengers who have paid attention to the target of interest 45 among the total number as the degree of attention 48. The degree of attention 48 may represent the degree of attention to the target of interest 45, and may be calculated by other calculation methods. In step S204, the attention setting unit 320 sets the attention level 48 for the attention target 45 in the attention database 52.

[0049] FIG. 7 is a diagram showing an example of the attention database 52 according to the present embodiment. In the attention database 52, a position range 521 of the PMV200, an attention target 522, a line-of-sight direction 523, a position 524 of the attention target, emotion information 525, and an attention level 526 are set. The position range 521 of the PMV200 is a range within the movement area 800 where the attention target 522 can be seen. It is assumed that the movement area 800 is divided in a predetermined range in advance. Alternatively, the path or sidewalk on which the PMV200 moves in the movement area 800 may be divided in a predetermined range in advance. The attention setting unit 320 sets the position range 521 and the attention target 522 based on the attention target 45 and the position information 41 included in the attention target information 40. The line-of-sight direction 523 is a line-of-sight direction represented in the PMV coordinate system with reference to the traveling direction of the PMV200. The attention setting unit 320 sets the line-of-sight direction 523 based on the line-of-sight direction 42 included in the attention target information 40. The position 524 of the attention target is the position of the attention target included in the attention target information 40. The emotion information 525 is the emotion information 43 included in the attention target information 40. The attention level 48 is calculated in step S203.

[0050] In step S205, the information distribution unit 330 distributes the attention database 52 to the PMV200.

[0051] <Map setting process> The map setting unit 310 of the server device 300 adds the emotion information 43 and the attention level 48 of the passenger 30 with respect to the attention target 45 to the map information 51.

[0052] FIG. 8 is a flowchart showing an example of the map setting process according to the present embodiment. In step S301, the map setting unit 310 refers to the target database 52. In step S302, the map setting unit 310 extracts the information of the target 522 newly set in the target database 52 by the target setting process. The map setting unit 310 adds the target 522, the emotion information 525 of the passenger 30 with respect to the target 522, and the attention level 526 to the map information 51.

[0053] FIG. 9 is a diagram showing an example of the map information 51 according to the present embodiment. For example, in the target database 52 of FIG. 7, a case where the "model house" installed in the moving area 800 is newly specified as the target 45 will be described. At this time, in FIG. 7, the "model house" is set as the target 522, and the position range 521 where the "model house" can be seen and the line-of-sight direction 523 when the "model house" is specified as the target are set. Also, the position 524 of the "model house", the emotion information 525 for the "model house", which is a "smiling face", and the attention level 526 of the "model house" are set to "0.8". The map setting unit 310 adds a mark indicating that it is the target 45, a mark representing a "smiling face", and an attention level of "0.8" to the position of the "model house" in the map information 51.

[0054] <Recommended information display process> The display unit 140 of the information collection device 100 displays, on the display device of the PMV 200, recommended information for recommending to the passenger 30 to look at the target 45 to which good emotion information is assigned.

[0055] FIG. 10 is a flowchart showing an example of the recommended information display process according to the present embodiment. In step S401, the display unit 140 refers to the target database 52. In step S402, the display unit 140 determines whether the PMV 200 is approaching the position corresponding to the target of interest 45 to which good emotional information is assigned. When the display unit 140 determines that the PMV 200 is approaching the position corresponding to the target of interest 45 to which good emotional information is assigned, the display unit 140 displays on the display device recommendation information 46 that recommends looking at the target of interest 45.

[0056] <Warning notification process> When the notification unit 340 of the server device 300 satisfies the notification condition 53 for notifying the administrator of the movement area 800 of the warning 47, the notification unit 340 notifies the warning 47.

[0057] FIG. 11 is a flowchart showing an example of the warning notification process according to the present embodiment. In step S501, the notification unit 340 refers to the target database 52.

[0058] In step S502, the notification unit 340 determines whether there is a target of interest 45 that satisfies the notification condition 53 in the target database 52. For example, assume that the notification condition is bad emotional information and the degree of attention is 0.4 or more. At this time, the notification unit 340 refers to the target database 52 and searches for the target of interest 45 that has bad emotional information and a degree of attention of 0.4 or more. If the notification unit 340 finds a target of interest 45 that satisfies the notification condition 53, it proceeds to step S503; if not, it returns to step S501.

[0059] In step S503, the notification unit 340 notifies the administrator of the movement area 800 of the warning 47. The notification unit 340 may output the warning 47 to an output device such as the display device of the server device 300, or may send the warning 47 to a pre-registered email.

[0060] For example, as shown in FIG. 9, when there is a garbage can overflowing with garbage within the moving area 800, the passenger 30 of the PMV200 passing nearby is likely to look at the garbage can and show an unpleasant expression. In such a case, since a warning 47 is notified to the administrator in real time, the administrator can immediately take measures against the unpleasant factor within the moving area 800.

[0061] ***Other configurations*** <Modification Example 1> In the present embodiment, the target-of-interest identification unit 130 identifies the target of interest 45 from the map information 51. In Modification Example 1, an aspect of identifying the target of interest 45 by other methods will be described.

[0062] FIG. 12 is a flowchart showing an example of the target-of-interest identification process according to Modification Example 1 of the present embodiment. In FIG. 12, the processes of step S102 to step S103 and step S105 are the same as those in FIG. 5. Also, after step S103, a new process of step S106 is added, and then step S104a is implemented instead of step S104.

[0063] The PMV200 is provided with an imaging device 203 capable of imaging in all directions. The imaging device 203 images the line-of-sight direction of the passenger 30 in response to a command from the information collection device 100.

[0064] In step S106, the target-of-interest identification unit 130 transmits an instruction to image the line-of-sight direction converted into the absolute coordinate system in step S103 to the imaging device 203. The imaging device 203 images the line-of-sight direction of the passenger 30 and outputs the image obtained by imaging as the line-of-sight direction image 44. In step S104a, the target-of-interest identification unit 130 identifies the target of interest 45 from the line-of-sight direction image 44.

[0065] In this way, by identifying the target of interest 45 from the line-of-sight direction image 44 captured by the imaging device 203, there is an effect that the target of interest 45 that cannot be determined from the map information 51 can be identified.

[0066] <Modification Example 2> In the present embodiment, the information collection device 100 has been described as being mounted on a smartphone or a tablet included in the passenger sensor 202. However, the information collection device 100 may be mounted on a computer different from the passenger sensor 202. Alternatively, all of the position sensor 201, the passenger sensor 202, and the information collection device 100 may be mounted on a single computer such as a smartphone or a tablet. Also, some functions or all functions of the information collection device 100 may be mounted on the server device 300.

[0067] Also, in the present embodiment, in the passenger sensor 202, the line-of-sight direction and the emotion information are analyzed. However, the passenger sensor 202 may transmit only the face image of the passenger to the information collection device 100, and the information collection device 100 may analyze the line-of-sight direction and the emotion information of the passenger.

[0068] <Modification Example 3> FIG. 13 is a diagram showing a configuration example of the information collection device 100 according to Modification Example 3 of the present embodiment. FIG. 14 is a diagram showing an example of the PMV 200 according to Modification Example 3 of the present embodiment. In Modification Example 3, a safety assurance method for ensuring the safety of the passenger 30 will be described using the sensors mounted on the PMV. In the information collection device 100 according to Modification Example 3 of the present embodiment, in addition to the configuration of FIG. 3, an analysis unit 160 is provided. Also, in the PMV 200 according to Modification Example 3 of the present embodiment, in addition to the configuration of FIG. 2, a seating sensor 205 and a seat belt sensor 206 are provided.

[0069] <<Example 1 of Safety Assurance Method>> The passenger sensor 202 periodically transmits the face image 491. The analysis unit 160 analyzes the face image 491 and determines whether the health state of the passenger 30 is normal or abnormal. When the analysis unit 160 determines that the health state is abnormal, it notifies the administrator of the movement area 800 of the abnormality in the health state of the passenger 30. In the case of Example 1 of the safety assurance method, the seating sensor 205 and the seat belt sensor 206 may not be provided.

[0070] <<Example 2 of the safety assurance method>> The seating sensor 205 detects the seating state of the passenger 30. The seat belt sensor 206 detects the state of the seat belt worn by the passenger 30. Based on the sensing information 492 from the seating sensor 205 and the seat belt sensor 206, the analysis unit 160 determines whether the seating state of the passenger 30 is normal or abnormal. When it is determined that the seating state is abnormal, the analysis unit 160 notifies the administrator of the movement area 800 of the abnormality in the seating state of the passenger. Note that the abnormality in the seating state may be detected using either one of the seating sensor 205 and the seat belt sensor 206.

[0071] With the safety assurance method as described above, it is possible to detect the following dangers or abnormalities of the passenger. · Falling asleep, which is dangerous. · Not looking in the direction one should. For example, not looking at the signal even though in front of a crosswalk, or not looking in the direction of an approaching object that is likely to collide. · Obvious signs of poor condition such as a bad complexion, swelling, and bleeding. · The face is in an abnormal position, such as about to slip or about to fall sideways.

[0072] <Modification Example 4> In the present embodiment, the functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140 are realized by software. As a modification example 4, the functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140 may be realized by hardware. Specifically, the information collection device 100 includes an electronic circuit 909 instead of the processor 910.

[0073] FIG. 15 is a diagram showing a configuration example of the information collection device 100 according to Modification Example 4 of the present embodiment. The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140. Specifically, the electronic circuit 909 is a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

[0074] The functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140 may be realized by one electronic circuit or may be distributed and realized by a plurality of electronic circuits.

[0075] As another modification example, some of the functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140 may be realized by an electronic circuit, and the remaining functions may be realized by software. Also, some or all of the functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140 may be realized by firmware.

[0076] Each of the processor and the electronic circuit is also called a processing circuit. That is, the functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140 are realized by the processing circuit.

[0077] Note that, similar to the information collection device 100, the functions of the information acquisition unit 110, the sensor information reception unit 120, the target of interest identification unit 130, and the display unit 140 of the server device 300 may also be realized by hardware.

[0078] ***Explanation of the effects of this embodiment*** According to this embodiment, the following effects are obtained. · For theme parks or entertainment facilities, it is possible to identify what to pay attention to and what is disliked. · For tourist attractions, it is no longer the case that it is unknown what is popular. · It is possible to identify in real time what is received by a smiling face and whether there is any problem there due to a severe expression. · From this information, it is possible to identify a place where the expression changes when facing a specific direction when passing through a certain position.

[0079] Furthermore, according to this embodiment, the following effects are obtained. · If there is something unpleasant, a warning is sent to the center, making it easy to keep the moving area comfortable. · It is possible to obtain data such as how much attention is paid when a B signboard is placed.

[0080] Furthermore, according to this embodiment, it is possible to detect whether there is a bias in the position or time when a smiling face appears. When there is a bias in the position or time when a smiling face appears, by analyzing from what location, in what direction, and when looking at what makes a smiling face, it is possible to use it as basic data on what is received. · Conversely, when there is a bias in the position or time when a severe expression appears, it can be predicted that there is some problem there. Furthermore, according to this embodiment, it is possible to detect whether there is a bias in the facing direction. If there is a bias in the direction a person is facing at a certain position, it can be said that there is something with a high degree of attention in that direction. Conversely, if not, it can be said that the degree of attention is low. For example, when creating a large display object in an amusement park or the like, the degree of attention can be measured by how much the people around it are facing that direction. If it is exactly the same as when there is no object, it can be determined that the object is not being noticed at all. This can be used to obtain data such as how much attention is gathered when launching fireworks. In either case, when it is determined that there is a "bias", by installing a camera that images that direction, it is possible to simultaneously collect information on what made them look that way.

[0081] The safety assurance method in the information collection system 500 according to this embodiment is particularly effective when the seat belt is not fastened. As a demand for autonomous driving PMVs, it is predicted that drunk people or the elderly with declining cognitive functions will return home automatically. When such people are carried, even if they are initially seated correctly, after a while, their posture may collapse and there is a risk of falling out of the PMV. When the seating sensor detects that the posture has collapsed and the person is about to fall out, the following situation avoidance measures can be taken. · Stop at a place where it is considered relatively safe even if the person falls out. · Notify the person on board by sound, light, vibration, or a combination thereof that they are about to fall out. · Notify the administrator or caregiver by communication that the person is about to fall out.

[0082] Also, by the safety assurance method in the information collection system 500 according to this embodiment, it is also possible to identify places where such people are likely to board or such situations are likely to occur. As a result, it is possible to dispatch PMVs with more safety considerations and avoid problems in advance.

[0083] In the above-described Embodiment 1, each part of the information collection system has been described as an independent functional block. However, the configuration of the information collection system does not have to be the same as that of the above-described embodiment. As long as the functional blocks of the information collection system can realize the functions described in the above-described embodiment, any configuration may be used. Further, the information collection system may be composed of a plurality of devices instead of a single device. Also, in Embodiment 1, a plurality of parts may be combined and implemented. Alternatively, one part of this embodiment may be implemented. In addition, this embodiment may be implemented in any combination, either as a whole or partially. That is, in Embodiment 1, free combinations of each embodiment, or modifications of any constituent elements of each embodiment, or omissions of any constituent elements in each embodiment are possible.

[0084] Note that the above-described embodiments are essentially preferred examples and are not intended to limit the scope of the present disclosure, the scope of applications of the present disclosure, and the scope of uses of the present disclosure. The above-described embodiments can be variously modified as necessary.

Description of Reference Numerals

[0085] 40 Target information, 41 Location information, 42 Gaze direction, 43 Emotion information, 44 Gaze direction image, 45 Target of attention, 46 Recommended information, 47 Warning, 48 Degree of attention, 491 Face image, 492 Sensing information, 51 Map information, 52 Attention database, 53 Notification conditions, 30 Occupant, 100 Information collection device, 110 Information acquisition unit, 120 Sensor information reception unit, 130 Target of attention identification unit, 140 Display unit, 150 Memory unit, 160 Analysis unit, 200 PMV, 201 Position sensor, 202 Occupant sensor, 203 Imaging device, 204 Autonomous driving system, 205 Seating sensor, 206 Seat belt sensor, 300 Server device, 310 Map setting unit, 320 Attention setting unit, 330 Information distribution unit, 340 Notification unit, 350 Memory unit, 500 Information collection system, 521 Location range, 522 Target of attention, 523 Gaze direction, 524 Position, 525 Emotion information, 526 Degree of attention, 800 Movement area, 909 Electronic circuit, 910 Processor, 921 Memory, 922 Auxiliary storage device, 930 Input interface, 940 Output interface, 950 Communication device.

Claims

1. In an information collection system that collects information in a moving area where a moving body moves, a position sensor that acquires the position information of the moving body, a passenger sensor that acquires the line-of-sight direction of a passenger riding on the moving body, a target identification unit that identifies, as a target of interest, an object or event existing in the line-of-sight direction of the passenger based on the position information of the moving body and the line-of-sight direction of the passenger and comprising, the passenger sensor acquires emotion information representing the type of emotion of the passenger with respect to the target of interest from the expression of the face of the passenger, the information collection system is an information collection system comprising a target setting unit that sets the position of the target of interest and the emotion information of the passenger with respect to the target of interest in a target database.

2. The target setting unit aggregates the number of passengers who are paying attention to the target of interest set in the target database, calculates the degree of attention to the target of interest, and sets the degree of attention to the target of interest in the target database. The information collection system according to claim 1.

3. The information collection system comprises map information in which peripheral environment information representing the peripheral environment is added to a map including the moving area, the target identification unit identifies the target of interest from the map information. The information collection system according to claim 2.

4. The information collection system is an information collection system according to claim 3, comprising a map setting unit that adds the emotion information of the passenger with respect to the target of interest and the degree of attention to the map information.

5. The information collection system comprises an imaging device capable of imaging the entire circumferential direction of the moving body, the imaging device images the line-of-sight direction of the passenger and outputs the image obtained by imaging as a line-of-sight direction image, the target identification unit identifies the target of interest from the line-of-sight direction image. The information collection system according to claim 2.

6. The information collection system is an information collection system according to claim 5, comprising a map setting unit that sets the target of interest in map information in which peripheral environment information representing the peripheral environment is added to a map including the moving area, and sets the emotion information of the passenger with respect to the target of interest and the degree of attention.

7. The information collection system A display unit that displays, on a display device of the moving body, recommendation information for recommending that the passenger view a target of interest to which good emotional information is imparted. The information collection system according to any one of claims 1 to 6.

8. The information collection system is A notification unit that notifies the warning when the target of interest satisfies a notification condition for notifying a warning to the administrator of the moving area. The information collection system according to any one of claims 1 to 7.

9. The passenger sensor periodically transmits a face image, The information collection system is An analysis unit that analyzes the face image, determines whether the health state of the passenger is normal or abnormal, and when the health state is determined to be abnormal, notifies the administrator of the moving area of the abnormality in the health state of the passenger. The information collection system according to any one of claims 1 to 8.

10. The information collection system is A seating sensor that detects the seating state of the passenger, A seat belt sensor that detects the state of the seat belt worn by the passenger And The analysis unit is Based on the sensing information from the seating sensor and the seat belt sensor, determines whether the seating state of the passenger is normal or abnormal, and when the seating state is determined to be abnormal, notifies the administrator of the moving area of the abnormality in the seating state of the passenger. The information collection system according to claim 9.

11. In an information collection method used in an information collection system that collects information in a moving area where a moving body moves, Obtain the position information of the moving body, Obtain the line-of-sight direction of the passenger riding on the moving body, Based on the position information of the moving body and the line-of-sight direction of the passenger, identify an object or event existing in the line-of-sight direction of the passenger as a target of interest that the passenger pays attention to, Obtain emotional information representing the type of emotion of the passenger towards the target of interest from the facial expression of the passenger, An information collection method of setting the position of the target of interest and the emotional information of the passenger with respect to the target of interest in a attention database.

12. In an information collection program used in an information collection system that collects information in a moving area where a moving body moves, A position acquisition process for acquiring the position information of the moving body, A line-of-sight acquisition process for acquiring the line-of-sight direction of the passenger riding on the moving body, Based on the position information of the moving object and the line-of-sight direction of the passenger, a target identification process for identifying an object or event existing in the line-of-sight direction of the passenger as a target of interest that the passenger pays attention to, a process of obtaining emotion information representing the type of emotion of the passenger towards the target of interest from the facial expression of the passenger, a target setting process of setting the position of the target of interest and the emotion information of the passenger towards the target of interest in a target database An information collection program that causes a computer to execute.

13. In an information collection system that collects information in a moving area where a moving object moves, a position sensor that obtains the position information of the moving object, a passenger sensor that obtains the line-of-sight direction of a passenger riding in the moving object, a target identification unit that identifies an object or event existing in the line-of-sight direction of the passenger as a target of interest that the passenger pays attention to based on the position information of the moving object and the line-of-sight direction of the passenger, a display unit that displays, on a display device of the moving object, recommendation information for recommending that the passenger look at a target of interest to which good emotion information is assigned An information collection system comprising.

14. In an information collection system that collects information in a moving area where a moving object moves, a position sensor that obtains the position information of the moving object, a passenger sensor that obtains the line-of-sight direction of a passenger riding in the moving object, a target identification unit that identifies an object or event existing in the line-of-sight direction of the passenger as a target of interest that the passenger pays attention to based on the position information of the moving object and the line-of-sight direction of the passenger, a notification unit that notifies the warning when the target of interest satisfies a notification condition for notifying a warning to the administrator of the moving area An information collection system comprising.

15. In an information collection system that collects information in a moving area where a moving object moves, a position sensor that obtains the position information of the moving object, a passenger sensor that obtains the line-of-sight direction of a passenger riding in the moving object, a target identification unit that identifies an object or event existing in the line-of-sight direction of the passenger as a target of interest that the passenger pays attention to based on the position information of the moving object and the line-of-sight direction of the passenger comprising, the passenger sensor periodically transmits a face image, the information collection system, An information collection system comprising an analysis unit that analyzes the face image, determines whether the health condition of the passenger is normal or abnormal, and notifies the administrator of the moving area of the abnormality in the health condition of the passenger when it is determined that the health condition is abnormal.

16. In an information collection system that collects information in a moving area where a moving body moves, a position sensor that acquires the position information of the moving body, a passenger sensor that acquires the line-of-sight direction of a passenger riding on the moving body, a target identification unit that identifies an object or event existing in the line-of-sight direction of the passenger as a target of attention that the passenger pays attention to based on the position information of the moving body and the line-of-sight direction of the passenger, a attention setting unit that totals the number of passengers who are paying attention to the target of attention, calculates the degree of attention to the target of attention, and sets the degree of attention to the target of attention in an attention database An information collection system comprising.

17. The attention setting unit, The information collection system according to claim 16, wherein the total number of passengers passing through a position where the target of attention can be seen is counted, the number of passengers who have paid attention to the target of attention among the total number of passengers is counted, and the ratio of the number of passengers who have paid attention to the target of attention among the total number of passengers is calculated as the degree of attention to the target of attention.

Images