Information gathering device, information gathering server, and information gathering system

Abstract

The present invention provides an information gathering device, an information gathering server, and an information gathering system that enable the collection of information about events occurring at unspecified locations along the route of a moving object, such as roads, and the provision of such information to the moving object. [Solution] An information gathering device comprising: a passenger status information gathering unit that collects the status of a passenger on a mobile body as passenger status information; a status change detection unit that detects changes in the passenger's status based on the passenger status information and generates status change detection information; an external imaging unit that photographs the outside of the mobile body and generates an image; and an external status information transmission unit that generates external status information including the image in response to the status change detection information and transmits the external status information via a communication channel.

Description

Technical Field

[0001] The present invention relates to an information collection device, an information collection server, and an information collection system.

Background Art

[0002] Conventionally, a traffic information providing system has been studied that collects information from cameras installed at intersections and transmits the situation near the intersections to automobiles near the intersections (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a traffic information providing system as disclosed in Patent Document 1, only information on a specific location on a road such as an intersection where a camera is installed can be obtained. Therefore, for example, even if an event occurs at a location other than the specific location, the information cannot be collected and information provision based on the collected information cannot be performed. This is one example of the problem.

[0005] In view of the above points, the present invention has been made, and one of the problems is to provide an information collection device, an information collection server, and an information collection system that can collect information on an event occurring at an unspecified location on a route such as a road on which a moving object moves and provide the information to the moving object.

Means for Solving the Problems

[0006] The invention described in claim 1 of this application is an information gathering device comprising: a passenger status information gathering unit that collects the status of a passenger of a mobile body as passenger status information; a status change detection unit that detects changes in the status of the passenger based on the passenger status information and generates status change detection information; an external imaging unit that photographs the outside of the mobile body and generates an image; and an external status information transmission unit that generates external status information including the image in response to the status change detection information and transmits the external status information via a communication channel.

[0007] Furthermore, the invention described in claim 7 of this application is an information collection system comprising: a passenger status information collection unit provided on a mobile body for collecting the status of the passengers on the mobile body as passenger status information; a status change detection unit for detecting changes in the passenger's status based on the passenger status information and generating status change detection information; an external imaging unit provided on the mobile body for photographing the outside of the mobile body and generating an image; and an external status information generation unit for generating external status information including the image in response to the status change detection information.

[0008] Furthermore, the invention described in claim 10 of the present application is an information collection server comprising: an external situation information receiving unit that receives external situation information, including an image of the outside of a mobile body, generated in response to the detection of a change in the state of a passenger riding in the mobile body, via a communication channel; a determination unit that analyzes the image of the outside of the external situation information and determines the usefulness of the external situation information; and an external situation information distribution unit that distributes the external situation information determined to be useful via a communication channel. [Brief explanation of the drawing]

[0009] [Figure 1] This is an overall configuration diagram of the information gathering system of the present invention. [Figure 2] This diagram shows an example of how to install a terminal device. [Figure 3] This is a functional block diagram of the terminal device and server. [Figure 4] This is an example of the operating routine of the passenger monitoring unit. [Figure 5]This is an example of the operating routine for the image extraction unit. [Figure 6] This is an example of the operation routine for the self-positioning unit. [Figure 7] This is an example of the operating routine of the content selection unit. [Figure 8] This is an example of the operating routine for the information receiving unit. [Figure 9] This is an example of the operating routine of the content selection unit. [Figure 10] This is a functional block diagram of the terminal device and server. [Modes for carrying out the invention]

[0010] Preferred embodiments of the present invention will be described below. However, these may be modified and combined as appropriate. In the following description and accompanying drawings, substantially identical or equivalent parts will be denoted by the same reference numerals. In the following description, "image" will include both moving images and still images. In the following, "transmission or reception of information" will include the transmission or reception of information signals. [Examples]

[0011] Below, an information collection system 100, which is an embodiment of the present invention, will be described with reference to the attached drawings.

[0012] Figure 1 is an overall configuration diagram of the information collection system 100 of Embodiment 1. As shown in Figure 1, the information collection system 100 is configured by connecting a terminal device 10, which serves as an information collection device mounted on a vehicle M (which serves as a mobile body), and a server 50 via a network NW. In Figure 1, a vehicle M is shown as an example of a mobile body, but the mobile body may be anything other than a vehicle, such as a bicycle, motorcycle, airplane, ship, or a person in motion.

[0013] Figure 2 shows an example of the installation of the terminal device 10 on an automobile M. In Figure 2, the terminal device 10 is shown installed on the front seat area of ​​automobile M as an example of installation.

[0014] An in-vehicle camera 11 as an external imaging unit that captures the situation outside (surroundings) the vehicle M, that is, the outside (external environment) of the vehicle M is disposed, for example, on a dashboard DB. The in-vehicle camera 11 can be, for example, a wide-angle camera capable of capturing a wide area in front of the vehicle M through a windshield.

[0015] Note that the in-vehicle camera 11 may be provided anywhere as long as it can capture the situation outside the vehicle. For example, the in-vehicle camera 11 may be provided on the back of a rearview mirror RM, that is, on a surface facing a windshield FG. Also, the in-vehicle camera 11 may be provided outside the vehicle M, for example, on a front bumper or the like. Also, a plurality of in-vehicle cameras 11 may be provided so as to be able to capture multiple directions outside the vehicle.

[0016] An in-vehicle camera 13 as a passenger imaging unit is a camera that captures a situation such as an action of a passenger as passenger state information that is state information of the passenger. The in-vehicle camera 13 is provided, for example, at an upper end of a windshield FG or at a ceiling portion near the upper end. Note that the in-vehicle camera 13 may be provided anywhere as long as it can capture the situation inside the vehicle and of the passenger, and may be provided, for example, on a dashboard DB or on a rearview mirror RM or the like.

[0017] A microphone 15 that receives the sound inside the vehicle as passenger state information is disposed, for example, on a dashboard DB. The microphone 15 may be provided anywhere, such as on a rearview mirror RM or a steering wheel, as long as it can receive the sound inside the vehicle.

[0018] Note that the in-vehicle camera 13 and the microphone 15 are used as a passenger state information collection unit that collects passenger state information.

[0019] The GPS receiver 17, which receives signals (GPS signals) from GPS (Global Positioning System) satellites, is located, for example, on the dashboard DB. However, the GPS receiver 17 can be located anywhere as long as it can receive GPS signals.

[0020] The display 19, which serves as a means of displaying, or notifying, information received from external sources, is, for example, located on the dashboard DB. The display 19 only needs to be located in a place where it can be seen by the passenger. For example, the display 19 may be mounted on the center console CC within the dashboard DB.

[0021] The main unit 21 is connected to the exterior camera 11, the interior camera 13, the microphone 15, the GPS receiver 17, and the display 19, and includes a control unit (see Figure 3) that controls these components. The main unit 21 is mounted, for example, in the center console CC section of the dashboard DB.

[0022] Figure 3 shows the control unit 23 of the terminal device 10 and a functional block diagram of the server 50. First, the control unit 23 will be explained.

[0023] The passenger monitoring unit 25, acting as a state change detection unit, is capable of receiving information from the in-vehicle camera 13 and microphone 15. Based on the image or audio input transmitted from the in-vehicle camera 13 and microphone 15, the passenger monitoring unit 25 monitors the state of the passengers in the vehicle M, i.e., the interior of the passenger compartment, and determines whether a change in the passengers' state has been detected, for example, whether an abnormality or anomaly has occurred. If the passenger monitoring unit 25 determines that a change in the passengers' state has been detected, it transmits state change detection information to the image extraction unit 27, the self-positioning unit 29, and the information transmission unit 31.

[0024] The image extraction unit 27 is connected to the external camera 11 and receives images captured by the external camera 11. Based on the state change detection information from the passenger monitoring unit 25, the image extraction unit 27 extracts a portion of the image received from the external camera 11 and generates an extracted image. Once the image extraction unit 27 has generated an extracted image, it transmits the extracted image to the information transmission unit 31 as an image of the state change detected, that is, an event image capturing the event occurring outside the moving object when the state change was detected.

[0025] Furthermore, the external camera 11 does not necessarily take pictures continuously. Instead, the passenger monitoring unit 25 may transmit status change detection information to the external camera 11, and the external camera 11 may take pictures in response to receiving such information. For example, it may take pictures when it receives status change detection information, or it may take pictures for a certain period of time after receiving status change detection information. In this case, the captured image may be further partially extracted by the image extraction unit, or it may be transmitted to the information transmission unit 31 as an event image without going through the image extraction unit.

[0026] The self-positioning unit 29 is connected to the GPS receiver 17 and can output position information indicating the current location of the vehicle M on which the terminal device 10 is installed. When the self-positioning unit 29 receives status change detection information from the passenger monitoring unit 25, it determines the position of the vehicle M at the time the status change detection information was received using the GPS signal from the GPS receiver 17 and transmits the position information, which is the information of that position, to the information transmission unit 31. The self-positioning unit 29 can also transmit the position information to other parts such as the information receiving unit 33 as needed.

[0027] The self-positioning unit 29 may determine the position of the vehicle M based on attitude information of the vehicle M from a gyro device (not shown) or vehicle speed information obtained from the vehicle speed pulse of the vehicle M. Furthermore, the self-positioning unit 29 may store map information or make it available for external acquisition. That is, the position of the vehicle M may be determined by combining at least one of the following: GPS information from the GPS receiver 17, attitude information of the vehicle M from the gyro device, and vehicle speed information of the vehicle M, with the map information.

[0028] The information transmission unit 31, which functions as an external status information generation unit or an external status information transmission unit, is connected to a transmitter (not shown) capable of transmitting information signals to the server 50 via a network NW (see Figure 1).

[0029] When the information transmission unit 31 receives state change detection information from the passenger monitoring unit 25, it generates information content as external status information. The information transmission unit 31 receives event images from the image extraction unit 27 or the external camera 11, and also receives location information from the self-location identification unit 29. Based on this received information, it generates information content as external status information, which includes the time the state change detection information was received (for example, the time the event image was taken), the event image, and the location information. In other words, the information transmission unit 31 generates information content in response to the state change detection information.

[0030] When the information transmission unit 31 generates information content, it transmits the information content to the server 50. Note that even without directly receiving status change detection information from the passenger monitoring unit 25, the information transmission unit 31 may be deemed to have received status change detection information from the passenger monitoring unit 25 upon receipt of an event image or location identification information.

[0031] The information receiving unit 33 is connected to a receiver (not shown) capable of receiving information from an external source, such as a server 50, via a network NW. When the information receiving unit 33 receives information from the server 50, it can display information on the display 19 based on that information and inform passengers of the information based on that information.

[0032] Next, the server 50 will be described. The information receiving unit 51, acting as an external status information receiving unit, can receive information content from the information transmitting unit 31 via the network NW. When the information receiving unit 51 receives information content, it transmits it to the content selection unit 53. Although not shown in the diagram, the information receiving unit 51 also receives information content from multiple terminal devices 10, each installed in other automobiles M.

[0033] The content selection unit 53, which functions as a selection or determination unit, is a functional unit that receives information content and selects information content from it to be transmitted to the automobile M. The content selection unit 53 includes a content usefulness determination unit 55 that determines whether the information content is useful or significant from the perspective of whether it contributes to the selection of the automobile M's travel route.

[0034] The content usefulness determination unit 55, for example, when multiple information contents are received and stored, determines whether there are many (or more than a predetermined number) of information contents among the stored information contents that have location identification information indicating the same or nearby location (hereinafter also referred to as nearby content). If there are more than a predetermined number of nearby content, the content usefulness determination unit 55 determines that the group of nearby content is useful or significant, selects them, and transmits them to the representative content selection unit 57.

[0035] Furthermore, nearby content may be information content that has location identification information indicating the same or nearby location, and that has event images taken at the same time (for example, within 30 minutes before or after the time of shooting).

[0036] The content selection unit 53 includes a representative content selection unit 57 that selects information content containing the most useful, significant, or high-quality event image from among the group of nearby content as representative content. The representative content selection unit 57 may, for example, analyze the event images contained in the group of nearby content and select the information content containing the clearest event image as representative content. Alternatively, it may select information content containing the most recently captured event image from among the group of nearby content as representative content.

[0037] When the representative content selection unit 57 selects representative content, it transmits it to the information transmission unit 59.

[0038] The information transmission unit 59, acting as an external status information distribution unit, can transmit information signals to the terminal device 10 via the network NW. When the information transmission unit 59 receives representative content from the representative content selection unit 57, it transmits it to the terminal device 10.

[0039] As described above, when the information receiving unit 33 of the terminal device 10 receives information from the server 50, it can display information on the display 19 based on that information and make a notification. This display is done, for example, by displaying the location of the car M and the location identification information included in the information content, i.e., the location where the event occurred, on a map, and also by displaying the event image included in the information content.

[0040] Here, the information receiving unit 33 may, after receiving the representative content, determine whether to display information based on the representative content on the display 19 based on the location identification information and the time of capture of the event image contained in the representative content.

[0041] This determination may be made, for example, by determining whether the current location of the automobile M equipped with the terminal device 10 and the location indicated by the location identification information of the representative content are within a predetermined distance.

[0042] Furthermore, if the vehicle M is equipped with a navigation system, the determination may be made based on whether the route set in the navigation system passes through or near the location indicated by the location information of the representative content.

[0043] [Operating routines for each part] The operation routines for each of the above-mentioned parts are described below. Of the operation routines described below, the operation routines for each part within the control unit 23 of the terminal device 10 are started, for example, when the prime mover such as the engine or motor of the automobile M is started, and are repeatedly executed until the prime mover is stopped. Alternatively, the terminal device 10 may be provided with individual switches, and the operation routine may be started when the switch is turned ON and repeatedly executed until the switch is turned OFF.

[0044] Figure 4 shows an example of the operation routine of the passenger monitoring unit 25. First, the passenger monitoring unit 25 determines whether a change in state, such as an abnormality, has occurred in the passenger (step S1). In this determination, for example, it may be determined whether a change in state has occurred based on whether a change in the gaze state of the pilot or passenger has occurred. A change in the gaze state includes, for example, a change from a state of not looking to a state of looking, i.e., a change in the degree of gaze, or a change in the direction of gaze.

[0045] For example, when gaze analysis using images transmitted from the in-vehicle camera 13 detects that the driver has been staring in one direction for a certain period of time or longer, it may be determined that a change in state has occurred (been detected).

[0046] Furthermore, a change in state may be determined when it is detected that a large number of passengers, including the pilot, are looking in one direction for a certain period of time or longer. Alternatively, even without gaze analysis, it may be determined that the pilot or passengers are looking in one direction when the faces of the passengers, including the pilot, are facing in one direction for a certain period of time or longer.

[0047] Furthermore, a change in state may be determined when the movement of the operator's gaze direction (position) or line of sight exceeds a threshold, or when the gaze direction (position) or line of sight moves at a speed exceeding a threshold.

[0048] Furthermore, the system may perform speech recognition on the audio transmitted from microphone 15, and determine that an anomaly has occurred when specific keywords such as "accident," "construction," "difficult," or "what is it?" are recognized. The state change detection information may also include gaze direction information, which is information about the direction the pilot or passenger was looking at at the time the state change occurred. The state change detection information may also include the results of speech recognition.

[0049] If it is determined in step S1 that a state change has occurred (step S1: YES), the passenger monitoring unit 25 transmits the state change detection information to the image extraction unit 27, the self-positioning unit 29, and the information transmission unit 31 (step S2), and the operation routine ends. If it is determined in step S1 that no state change has occurred (step S1: NO), the operation routine then ends.

[0050] Figure 5 shows an example of the operation routine of the image extraction unit 27. First, the image extraction unit 27 determines whether state change detection information has arrived from the passenger monitoring unit 25 (step S1). If it is determined in step S1 that no state change detection information has arrived (step S1: NO), the operation routine then terminates.

[0051] In step S1, if it is determined that state change detection information has arrived, the image extraction unit 27 extracts an image from the external camera 11 and generates an extracted image (step S2). This generation of the extracted image may be performed by extracting a still image from the image of the external camera 11 at the time the state change detection information was received.

[0052] Alternatively, this may be done by extracting one or more still images or a video of a predetermined time range before and after the time the state change detection information was received from the image of the external camera 11. Furthermore, if the state change detection information includes the gaze direction information of the passenger as described above, a portion of the still image or video extracted based on the time the state change detection information was received may be further cropped (trimmed) based on the gaze direction information. In this case, the image extraction unit functions as an image cropping unit.

[0053] In this case, the region corresponding to the passenger's gaze direction may be extracted from the image or video extracted based on the time the state change detection information was received, based on the gaze direction information.

[0054] After step S2 is completed, the image extraction unit 27 transmits the extracted image to the information transmission unit 31 as an image of the state change detected, that is, an event image capturing the event occurring outside the moving object when the state change was detected (step S3), and then the operation routine ends.

[0055] As mentioned above, the external camera does not continuously record, and the passenger monitoring unit 25 may transmit status change detection information to the external camera 11, and the external camera 11 may record in response to receiving such information. For example, it may record when it receives status change detection information, or it may record for a certain period of time after receiving status change detection information.

[0056] In this case, the captured image may be further partially extracted by the image extraction unit 27 in a temporal manner and transmitted to the information transmission unit 31 as an event image. Alternatively, the captured image may be partially extracted by the image extraction unit 27, which functions as an image cropping unit, and transmitted to the information transmission unit 31 as an event image. The captured image may also be transmitted directly to the information transmission unit 31 as an event image without going through the image extraction unit 27.

[0057] Figure 6 shows an example of the operation routine of the self-positioning unit 29. First, the self-positioning unit 29 determines whether state change detection information has arrived from the passenger monitoring unit 25 (step S1). If it is determined in step S1 that no state change detection information has arrived (step S1: NO), the operation routine then terminates.

[0058] In step S1, when it is determined that state change detection information has arrived, the self-positioning unit 29 generates position identification information that identifies the position of the vehicle M on which the terminal device 10 is mounted (step S2). This position identification information may be position information identified by GPS signals from the GPS receiver 17. Alternatively, the self-positioning unit 29 may generate position identification information that identifies the position of the vehicle M based on attitude information of the vehicle M from a gyro device (not shown) or vehicle speed information based on vehicle speed pulses of the vehicle M.

[0059] Furthermore, the self-positioning unit 29 may either store map information or make it available for external acquisition. That is, the self-positioning unit 29 may generate positioning information to identify the position of the vehicle M by combining at least one of the following: GPS signals from the GPS receiver 17, attitude information of the vehicle M from the gyro device, and vehicle speed information of the vehicle M, with the map information.

[0060] After step S2 is completed, the self-positioning unit 29 transmits the generated positioning information to the information transmission unit 31 (step S3), and then the operation routine ends.

[0061] The event image and location information transmitted to the information transmission unit 31 are transmitted by the information transmission unit 31 to the information receiving unit 51 of the server 50 as information content including these. In other words, information content including the event image and location information is generated in the information transmission unit 31 and transmitted to the information receiving unit 51 of the server 50. When the information receiving unit 51 receives the information content, it transmits it to the content selection unit 53.

[0062] Figure 7 shows an example of the operation routine of the content selection unit 53, which consists of a content usefulness determination unit 55 and a representative content selection unit 57. In this example, the content selection unit 53 determines whether a certain number of information contents containing event images taken within a certain range have arrived in order to determine whether the information content is significant or useful. As mentioned above, the information receiving unit 51 receives information content from multiple terminal devices 10, each installed in multiple automobiles M.

[0063] First, the content selection unit 53 determines whether a predetermined number or more information content (hereinafter also referred to as neighboring content) having location identification information indicating the same or nearby location has arrived from the information receiving unit 51 (step S1). If it is determined in step S1 that a predetermined number or more of neighboring content have not arrived (step S1: NO), the operation routine then terminates.

[0064] In step S1, if it is determined that there are a predetermined number or more neighboring content items (step S1: YES), the content selection unit 53 selects one information content item from among the neighboring content items as the representative content item (step S2).

[0065] The selection of representative content may involve, for example, analyzing the event images held by each of the neighboring content, selecting the clearest event image as the representative image, and then selecting the information content holding that representative image as the representative content. Alternatively, the event image taken most recently among the event images held by each of the neighboring content may be selected as the representative image, and then selecting the information content holding that representative image as the representative content.

[0066] Furthermore, the information content may include vehicle speed pulses of the vehicle M at the time the event image was taken, or vehicle speed information based on those pulses. In such cases, the vehicle speed pulses or vehicle speed information may be used to select the clearest event image. For example, the vehicle speed pulses at the time the event image was taken may be analyzed, and the event image taken at the lowest speed may be designated as the representative image. The information content holding this representative image may then be designated as the representative content.

[0067] Furthermore, the information content may include information about the direction (position) or gaze direction of the passenger of the vehicle M at the time the passenger monitoring unit 25 transmits state change detection information (i.e., when the event image is taken). In this case, for example, the system may determine from the event image which lane the vehicle M was traveling in (or stopped in) on the road at the time the event image was taken, and then determine a representative image.

[0068] For example, if the occupant was looking to the left of car M when the state change detection information was transmitted, the event image taken in the leftmost lane may be used as the representative image. Similarly, if the occupant was looking to the right of car M, the event image taken in the rightmost lane may be used as the representative image.

[0069] When step S2 is completed, the content selection unit 53 transmits the representative content, which is information content including a representative image, to the information transmission unit 59 (step S3), and then the operation routine ends. When the information transmission unit 59 receives the information content from the content selection unit 53, it transmits it to the information receiving unit 33 of the terminal device 10.

[0070] Figure 8 shows an example of the operation routine of the information receiving unit 33. First, the information receiving unit 33 determines whether information content has arrived from the server (step S1). If it determines that no information content has arrived (step S1: NO), the operation routine then terminates.

[0071] In step S1, if it is determined that information content has arrived from the server (step S1: YES), the information receiving unit 33 determines whether the arrived information content is useful to the passengers of the automobile M in which the terminal device 10 is installed (step S2).

[0072] In this determination, for example, if the location indicated by the location information contained in the incoming information content is near the current location of the automobile M (within a predetermined distance), the information content may be determined to be useful.

[0073] Furthermore, for example, if a route is set in the navigation system installed in automobile M, it may be determined whether or not the information content that arrives based on that route is useful. That is, for example, the information content may be determined to be useful if the location indicated by the location information held by the arriving information content is on or near the set route.

[0074] In step S2, if it is determined that the incoming information content is useful (step S2: YES), the information receiving unit displays information based on the information content on the display 19 (step S3). This display may, for example, simply display the event image and location information contained in the information content. In addition, the location where the event image was taken may be displayed on a map based on the location information.

[0075] If, in step S2, it is determined that the incoming information content is not useful (step S2: NO), the operation routine then terminates. Also, if step S3 is completed, the operation routine then terminates.

[0076] According to the information system 100, terminal device 10, and server 50 described above, it is possible to collect information, including images, regarding events that occur at unspecified locations along the movement path of a mobile object, such as narrow roads or roads with little traffic, including routes where information collection devices are not installed. Furthermore, it is possible to distribute the collected information to other mobile objects.

[0077] Furthermore, according to the information system 100, terminal device 10, and server 50, information including images taken at the time of a change in state, based on changes in the state such as an anomaly, is generated for the passengers of a moving vehicle, and useful or significant information is selected and distributed by analyzing the information including the images, thereby enabling efficient distribution of only the necessary information. [Examples]

[0078] The system 200 of Embodiment 2 of the present invention will be described below with reference to the drawings. The system 200 of Embodiment 2 has the same configuration as the system 100 of Embodiment 1, except for the operating routine of the content selection unit 53. Therefore, explanations other than the operating routine of the content selection unit 53 will be omitted.

[0079] In Example 2, we will explain an example in which the content selection unit 53 of the system 200 analyzes an event image within the information content transmitted from the terminal device 10 to determine what event the event image depicts and to determine the significance or usefulness of each piece of information content. Specifically, we will explain an example in which the content selection unit 53 analyzes what event occurred when the event image was captured and to determine its significance or usefulness.

[0080] Figure 9 shows an example of the operation routine of the content selection unit 53 of the system 200 in Example 2. First, the content selection unit 53 determines whether information content has arrived from the terminal device 10 (step S1). If it is determined in step S1 that no information content has arrived (step S1: NO), the operation routine then terminates. As mentioned above, the information content arrives from multiple terminal devices 10, each installed in multiple automobiles M.

[0081] If it is determined in step S1 that information content has arrived (step S1: YES), the content selection unit 53 determines whether the arrived information content is useful or significant (step S2).

[0082] This determination is made, for example, by analyzing what each event image contained in the information content depicts, that is, what kind of event is occurring. For example, if the event image is analyzed to be a useful image such as "accident," "construction site," "traffic jam," "crowd," or "event," then the information content containing that event image may be determined to be useful or significant.

[0083] Specifically, for example, if an image of an event is analyzed and it is determined that a large number of cars have not moved for a certain period of time, it is determined to be a traffic jam. Also, if the word "construction" is recognized, or if the yellow and black diagonal stripe pattern seen at construction sites is recognized, it is determined to be a "construction site." Furthermore, if a large number of people are recognized, it is determined to be a "crowd" or an "event." If, in step S2, the information content is determined to be not useful or significant (step S2: NO), the routine then terminates.

[0084] In step S2, if the incoming information content is determined to be useful or significant (step S2: YES), the process proceeds to step S3. In step S3, the content selection unit 53 determines whether other information content has arrived that includes event images taken at the same time period (for example, within 30 minutes before or after the time of capture) and at the same location (or nearby location) as the event image held by the information content determined to be useful or significant (step S3).

[0085] In step S3, if it is determined that other information content, including event images taken at the same time and location, has arrived (step S3: YES), the content selection unit 53 selects one information content from among the information content as representative content (step S4).

[0086] As described in Example 1, the selection of representative content may involve, for example, analyzing the event images held by each piece of information content, selecting the clearest event image as the representative image, and then selecting the information content that holds that representative image as the representative content. Alternatively, the most recently captured event image among the event images held by the information content may be selected as the representative image, and then selecting the information content that holds that representative image as the representative content.

[0087] Furthermore, the information content may include vehicle speed pulses or vehicle speed information of the vehicle M at the time the event image was taken, in which case the vehicle speed pulses or vehicle speed information may be used to select the clearest event image. For example, the vehicle speed pulses at the time the event image was taken may be analyzed, the event image taken at the lowest speed may be designated as the representative image, and the information content holding that representative image may be designated as the representative content.

[0088] Furthermore, the information content may include information about the direction (position) or gaze direction of the occupant of the vehicle M at the time the state change detection information is transmitted from the occupant monitoring unit 25 (i.e., when the event image is taken). In this case, for example, it may be possible to determine which lane the vehicle M is traveling in (or stopped in) from the event image and then determine a representative image.

[0089] For example, if the occupant was looking to the left of car M when the state change detection information was transmitted, the event image taken in the leftmost lane may be used as the representative image. Alternatively, if the occupant was looking to the left of car M, the event image taken in the rightmost lane may be used as the representative image.

[0090] After step S4 is completed, and if step S3 determines that no other information content containing event images taken at the same time and location has arrived (step S3: NO), the content selection unit 53 transmits the information content containing the representative image to the terminal device 10 as is (step S5).

[0091] If, in step S3, it is determined that no other information content, including event images taken at the same time and location, has arrived, the information content determined to be useful or significant in step S2 is transmitted to the terminal device 10. After step S5 is completed, the operation routine ends.

[0092] In this case, the information content transmitted from the server 50 to the terminal device 10 may also include the results of the analysis of the event image performed in step S2. That is, the transmitted information content may include information about the event depicted in the event image (for example, "accident," "construction site," "traffic jam," "crowd," or "event").

[0093] When information regarding such an event is included in the information content and transmitted to the terminal device 10, the information receiving unit may determine whether or not to display information based on the information content on the display 19 based on the information regarding the event. For example, if the user of the terminal device 10 requests only information about an "accident," only information based on the information content regarding the "accident" may be displayed on the display 19.

[0094] Furthermore, if the vehicle M is equipped with a navigation system, information regarding the incident may be provided to the navigation system, prompting it to change its route. For example, based on the information content, information may be provided that an accident has occurred on the navigation system's set route, prompting it to change its route to avoid the accident.

[0095] Furthermore, for example, based on information content, the navigation system may provide information that an "event" such as a fireworks display is taking place near the pre-set route, and encourage the user to change the pre-set route to pass through the location of the "event." In addition, anticipating congestion and traffic jams caused by the "event," the system may also encourage the user to change the pre-set route to avoid the location of the "event."

[0096] According to the information system 200, terminal device 10, and server 50 described above, it is possible to collect information, including images, regarding events that occur at unspecified locations along the movement path of a mobile object, such as narrow roads or roads with little traffic, including routes where information collection devices are not installed. Furthermore, it is possible to distribute the collected information to other mobile objects.

[0097] Furthermore, according to the information system 200, terminal device 10, and server 50, information including images taken at the time of a change in state, based on changes in the state such as an anomaly, is generated for the passengers of a moving vehicle, and useful or significant information is selected and distributed by analyzing the information including the images, thereby enabling efficient distribution of only the necessary information.

[0098] Furthermore, the information system 200, terminal device 10, and server 50 can analyze images related to events captured and transmitted by a mobile device and distribute information including information about what kind of event is occurring. This makes it possible to distribute detailed information about the event that is occurring to other mobile devices such as automobiles. [Examples]

[0099] The system 300 of Example 3 will be described below with reference to Figure 10. As shown in Figure 10, in system 300, the passenger monitoring unit 25 and image extraction unit 27, which were provided on the terminal device 10 side in Examples 1 and 2, are provided on the server 50 side as the passenger monitoring unit 25S and image extraction unit 27S. In other words, it is a so-called cloud configuration in which the processing on the terminal device 10 side is shifted to the server side.

[0100] In system 300, images and audio from the external camera 11, the internal camera 13, and the microphone 15, as well as location information from the self-positioning unit 29, are constantly transmitted to the server 50 via the network NW by the information transmission unit 31. The transmitted images and audio are received by the information receiving unit 51 of the server 50. The above images, audio, and location information arrive from multiple terminal devices 10, each installed in multiple automobiles M.

[0101] The received images and audio are transmitted as follows: images and audio from the in-vehicle camera 13 and microphone 15 are sent to the passenger monitoring unit 25S, and images from the exterior camera 11 and location information from the self-positioning unit 29 are sent to the image extraction unit 27S. In this transmission, the images, audio, and location information may be transmitted in a manner that allows each transmitting vehicle M to be identified.

[0102] In system 300, the passenger monitoring unit 25S monitors the passenger's status by executing an operation routine similar to that of the passenger monitoring unit 25 in Embodiment 1 shown in Figure 4. The image extraction unit 27S also executes an operation routine similar to that of the image extraction unit 27 in Embodiment 1 shown in Figure 5. In step S3 of Figure 5, the image extraction unit 27S transmits the extracted event image along with the time and location information of the event image to the content selection unit 53.

[0103] The content selection unit 53 executes, for example, an operation routine similar to that in Embodiment 1 or 2 (for example, the operation routine shown in Figure 7 or Figure 9) to select information content to be transmitted to the terminal device 10. The selected information content is then transmitted from the information transmission unit 59.

[0104] The transmitted information content is received by the information receiving unit 33. The information receiving unit 33 executes an operation routine similar to that of Embodiment 1 or 2 (for example, the operation routine shown in Figure 8). The information receiving unit 33 displays the information on the display 19 as necessary, in a manner similar to that described in Embodiment 1 or 2.

[0105] According to the system 300 of Example 3, in addition to the advantages of Examples 1 or 2, it is possible to reduce the processing load on the terminal device 10 and decrease the power consumption of the terminal device 10. Furthermore, it is possible to miniaturize the terminal device 10.

[0106] In the above-described embodiment, a wide-angle camera was given as an example for the exterior camera 11, but various other cameras can also be used. For example, a camera mounted on an angle-adjustable stand and capable of changing the shooting direction may be used. In this case, based on the gaze direction information in the state change detection information transmitted by the passenger monitoring unit 25, the shooting direction of the exterior camera 11 may be changed to the gaze direction and shooting may be performed, and part or all of the captured image may be used as the event image.

[0107] Furthermore, in the above-described embodiment, the passenger monitoring unit 25 determines whether a change in the passenger's condition has occurred based on the image from the in-vehicle camera 13 and the audio from the microphone 15, which serve as the passenger condition information collection unit. However, the determination may also be made using other information.

[0108] For example, the passenger status information collection unit may be equipped with a biosensor that detects the passenger's biological information (vital signs), and based on the information from the biosensor (e.g., changes in heart rate or blood pressure), it may determine that a change in the passenger's status has occurred in the vehicle and transmit status change detection information. This biosensor may be attached to the passenger, or it may be installed on the steering wheel, seat, and seat belt.

[0109] Furthermore, the passenger status information collection unit may be equipped with means for collecting information related to the operation of the vehicle M. In this case, the passenger monitoring unit 25 may detect operations such as sudden acceleration, sudden braking, and sudden steering, and transmit status change detection information based on these operations.

[0110] Furthermore, in order to reduce the detection of unnecessary state changes, for example, when the shift lever is moved to the reverse position, the state change may be detected taking this situation into consideration. For example, when the shift lever is moved to the reverse position, the driver's gaze towards the rear of the vehicle M does not need to be detected as a state change.

[0111] Furthermore, in the above-described embodiment, the terminal device 10 was assumed to have a display 19 as a notification means, but it may also include a speaker as a notification means. In this case, notifications based on information content may be made by display on the display 19 and by sound from the speaker.

[0112] Furthermore, announcements based on information content may be made solely through audio. In the case of audio-only announcements, location-specific information included in the information content, namely the location where the event occurred, the time the event occurred, and the content of the event, may be announced via audio.

[0113] Furthermore, in the above embodiment, the content selection unit 53 selected one information content as representative content from among the nearby content, but multiple information content may be selected as representative content. In this case, for example, multiple information content may be selected as representative content in order of clarity of the included event images.

[0114] Furthermore, in the above-described embodiment, the terminal device 10 is provided with an information receiving unit 33 and a display 19, but the terminal device 10 does not necessarily have to include the information receiving unit 33 and the display 19. In other words, the terminal device 10 may be an information collection device that only transmits information to the server 50.

[0115] Furthermore, although the above embodiment was described using the example of transmitting representative content to all mobile bodies M, the representative content may also be transmitted to only some of the automobiles M. For example, the server 50 may hold the location information of each automobile M, and the content may be transmitted only to automobiles M located near the location indicated by the location identification information contained in the representative content, that is, within a certain distance from the location indicated by the location identification information.

[0116] Furthermore, although the above embodiment states that the information content includes location information indicating the location of the automobile M, location information does not necessarily have to be included. In this case, the server 50 may determine the location of the automobile M from information such as which base station of the mobile communication network the wireless communication from the automobile M is using. Alternatively, the location of an event image may be determined from an event image included in the information content.

[0117] Alternatively, the representative content may be transmitted only to automobile M located at a position where the time it takes to reach the location indicated by the location information contained in the representative content is within a certain time frame.

[0118] Furthermore, the information content from the server 50 may be received by devices other than the terminal device 10, such as smartphones, tablets, PDAs, and personal computers.

[0119] Furthermore, although the above embodiment described an example in which the terminal device 10 is mounted on an automobile M as an example of a mobile body, the mobile body is not limited to an automobile. For example, the mobile body on which the terminal device 10 is mounted may be a ship, airplane, motorcycle, bicycle, etc.

[0120] Alternatively, a person traveling on foot may carry the terminal device 10 with them, and the terminal device 10 may transmit information content to the server 50. In this case, the in-vehicle camera of the terminal device 10, which serves as a means for photographing passengers, will be a camera that photographs the person who is moving.

[0121] For example, the terminal device 10 may be implemented as a smartphone or tablet. In that case, the inner camera of the smartphone or tablet may correspond to the in-vehicle camera described in Examples 1 to 3, and the outer camera may correspond to the out-of-vehicle camera described in Examples 1 to 3.

[0122] Furthermore, although the configurations of the terminal device 10 and server 50 were illustrated and explained in the above embodiment with reference to Figures 3 and 10, the configurations of the terminal device 10 and server 50 are not limited to this example. For example, the passenger monitoring units 25, 25S, the image extraction units 27, 27S, and the content selection unit 53 may be located in either the terminal device 10 or the server 50.

[0123] The various configurations and routines described in the above-mentioned embodiments are merely examples and can be appropriately selected according to the application and other factors. [Explanation of symbols]

[0124] 100, 200, 300 Information Gathering Systems 10 Terminal devices 11. External camera 13 In-car camera 15 Mike 17 GPS receivers 19 displays 21 Main unit 23 Control Unit 25, 25S Passenger Monitoring Unit 27, 27S Image extraction section 29 Self-location part 31 Information Transmission Section 33 Information Receiving Unit 50 servers 51 Information Receiving Unit 53 Content Selection Department 55 Content Usefulness Determination Unit 57 Representative Content Selection Department 59 Information Transmission Section

Claims

1. A passenger status information collection unit collects the status of the passengers of a mobile vehicle as passenger status information, A state change detection unit detects a change in the state of the passenger based on the passenger state information and generates state change detection information including gaze direction information regarding the direction in which the passenger is looking, An external imaging unit that photographs the outside of the moving body and generates an image, An image extraction unit that extracts a portion of the captured image based on the gaze direction information, An external status information transmission unit generates external status information including an image of the partial region extracted by the image extraction unit in accordance with the state change detection information, and transmits the external status information via a communication channel. An information gathering device characterized by including [this].

2. The information gathering device according to claim 1, characterized in that the state change detection unit generates the state change detection information based on a change in the gaze state of the passenger.

3. The information acquisition device according to claim 1 or 2, characterized in that the external imaging unit performs imaging in accordance with the state change detection information.

4. The information gathering device according to any one of 1 to 3, further comprising an image extraction unit that extracts images within a predetermined time range from the captured images captured by the external imaging unit in response to the state change detection information, and the external status information transmission unit generates external status information including the extracted images and transmits the generated external status information.

5. The information gathering device according to any one of 1 to 4, characterized in that the captured image includes information on the location and time of capture.

6. The information gathering device according to any one of 1 to 5, characterized in that the image extraction unit extracts a region of the captured image corresponding to the gaze direction based on the gaze direction information.

7. A passenger status information collection unit is installed in the mobile body and collects the status of the passengers of the mobile body as passenger status information, A state change detection unit detects a change in the state of the passenger based on the passenger state information and generates state change detection information including gaze direction information regarding the direction in which the passenger is looking, An external imaging unit is provided on the moving body and captures images of the outside of the moving body to generate captured images, An image extraction unit that extracts a portion of the captured image based on the gaze direction information, An external status information generation unit generates external status information including an image of the partial region extracted by the image extraction unit in accordance with the state change detection information, An information gathering system characterized by including [this].

8. The information collection system according to claim 7, characterized in that the state change detection unit generates state change detection information based on a change in the gaze state of the passenger.

9. The information gathering system according to claim 7 or 8, characterized in that it includes an external status information distribution unit that distributes the aforementioned external status information via a communication channel.

10. The information gathering system according to claim 9, further comprising a selection unit that selects the external status information based on its usefulness, wherein the external status information distribution unit distributes the selected external status information via a communication channel.

11. An external status information receiving unit receives external status information via a communication channel, which is generated in response to the detection of a change in the state of a passenger on board the mobile vehicle and includes an extruded image obtained by cutting out a portion of an external image of the mobile vehicle based on gaze direction information relating to the direction the passenger is looking. A determination unit analyzes the cropped image included in the external situation information and determines the usefulness of the external situation information, An external status information distribution unit distributes the external status information that has been determined to be useful via a communication channel, An information gathering server characterized by including [this].

Images