Information processor, information processing method, and program

Abstract

To detect and notify an abnormal event.SOLUTION: An information processor includes a control unit that performs acquiring video taken by an onboard camera installed in a vehicle, analyzing the video so as to determine whether the video shows an abnormal event, and, when it is determined that the video shows an abnormal event, outputting a first indication, which indicates that an abnormal event has been detected, at a position in a map associated with a place where the video is taken.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system. [Background technology]

[0002] There is a technology for monitoring using a mobile object equipped with a camera. For example, Patent Document 1 discloses a system that uses a server device to manage the history of vehicle driving locations and identifies vehicles equipped with a drive recorder that are located near a caller. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-019792 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to detect and notify abnormal events. [Means for solving the problem]

[0005] One aspect of the present disclosure is The information processing device has a control unit that performs the following operations: acquiring video captured by an onboard camera mounted on a vehicle; analyzing the video to determine whether the video contains an abnormal event; and, if it is determined that the video contains the abnormal event, outputting a first display indicating that the abnormal event has been detected at a position on a map corresponding to the location where the video was acquired.

[0006] Furthermore, one aspect of the present disclosure is The information processing method includes the steps of acquiring video captured by an onboard camera mounted on a vehicle, analyzing the video to determine whether or not the video contains an abnormal event, and, if it is determined that the video contains the abnormal event, outputting a first display indicating that the abnormal event has been detected at a position on a map corresponding to the location where the video was acquired.

[0007] Other aspects include a method executed by the information processing device, a program for causing a computer to execute the method, or a computer-readable storage medium non-temporarily storing the program. [Effects of the Invention]

[0008] According to the present disclosure, abnormal events can be detected and notified. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram showing an outline of processing executed by the information processing system according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating components of the information processing system according to the first embodiment. [Figure 3] FIG. 10 is a sequence diagram of a process in which the server device collects vehicle data and images, and a process in which the server device provides a vehicle list. [Figure 4] 10 is a flowchart of a process performed by the server device to determine whether or not an abnormal event has occurred. [Figure 5] FIG. 10 is a sequence diagram of a process for playing back in-vehicle video. [Figure 6] FIG. 10 is a diagram showing an outline of processing executed by an information processing system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] (overview) In recent years, automobiles have become increasingly connected, with an increasing number of vehicles equipped with communication functions and in-vehicle cameras.

[0011] In relation to this, by making it possible for designated agencies to access footage captured by cameras (such as dashcams) mounted on vehicles, it can be useful in the initial response to incidents and accidents. For example, when a call is made to 110 or 119, relevant authorities can quickly confirm the situation at the scene by obtaining dashcam footage from a vehicle that was traveling near the relevant location around the time the incident occurred. As a system for this purpose, a system is known in which icons or the like representing the travel positions of a plurality of vehicles at a predetermined time in the past are mapped on a map and presented to an operator.

[0012] For example, the system obtains location information of vehicles traveling in a specified area around a time specified by the operator (e.g., five minutes before the time the call was made) and maps this information on a map. The operator selects a vehicle on the map that was traveling near a desired location (e.g., the location where the call was made) at any time close to the desired time (e.g., the time of the call). The system also requests that vehicle to transmit dashcam footage taken at the corresponding time. This allows the operator to view dashcam footage that may capture the scene of the call.

[0013] However, for emergency call receiving agencies, identifying the location of the incident and checking the dashcam footage after receiving the call can hinder a prompt initial response. In order for the agency to respond more smoothly, it is desirable for the system to be able to detect abnormal events that occur on the road and inform the operator of their location.

[0014] An information processing device according to one aspect of the present disclosure includes: The control unit performs the following operations: acquiring video captured by an onboard camera mounted on a vehicle; analyzing the video to determine whether or not the video contains an abnormal event; and, if it is determined that the video contains the abnormal event, outputting a first display indicating that the abnormal event has been detected at a position on a map corresponding to the location where the video was acquired.

[0015] An abnormal event is an unusual event that occurs on a road, such as a vehicle accident, that is included in the video captured by the in-vehicle camera. Specifically, the abnormal event may be a vehicle accident, an accident resulting in injury or death, luggage scattered on the roadway, the presence of some kind of obstacle, etc.

[0016] The position on the map corresponding to the location where the video was acquired is the position on the map indicating the location where the vehicle that acquired the video containing the abnormal event was traveling when the video was acquired.

[0017] The first indication is a mark pin, a pop-up, a mark, or the like that is displayed on a map. The first indication indicates that a video containing an abnormal event has been captured at a location corresponding to the position on the map to which the first indication is attached.

[0018] An information processing device according to the present disclosure analyzes video captured from a vehicle and determines whether the video contains an abnormal event. If the information processing device according to the present disclosure determines that the video contains an abnormal event, it outputs a display indicating that video containing an abnormal event has been captured at a position on a map corresponding to the location where the video was captured.

[0019] By analyzing the images captured from the vehicle as it moves, it is possible to detect abnormalities in real time. The information processing device according to the present disclosure can detect the abnormality based on the time and output the information to that effect on a map. This allows the information processing device according to the present disclosure to notify the operator of the location of the abnormality.

[0020] In addition, when a report is made to a specified administrative agency, the control unit may obtain report information, which is information indicating the content of the report, from a specified device in the administrative agency, identify the location where the event related to the report occurred based on the report information, and further output a second display indicating the location of occurrence at a position on the map corresponding to the location of occurrence.

[0021] That is, the information processing device according to the present disclosure can inform the operator of the location where the abnormality was detected as well as the location where the report was made, thereby enabling the operator to determine whether the detected abnormality occurred at the location related to the report, thereby enabling a smooth initial response by a fire department or the like.

[0022] In addition, the control unit may identify a congestion point where traffic is congested based on driving data acquired by multiple vehicles, and when the video including the abnormal event is acquired by a vehicle located near the congestion point, output the first display in an emphasized manner compared to when the video including the abnormal event is acquired by a vehicle that is not located near the congestion point.

[0023] There is a high possibility that traffic will be backed up at points where some kind of abnormality is occurring. Therefore, if an abnormal event is detected in a location where traffic is backed up, highlighting this makes it possible to show the location where the abnormal event is likely to have occurred, and to present information that is highly needed at the site.

[0024] Specific embodiments of the present disclosure will be described below with reference to the accompanying drawings. Unless otherwise specified, the hardware configuration, module configuration, functional configuration, etc. described in each embodiment are not intended to limit the technical scope of the disclosure to those configurations.

[0025] (First embodiment) [System Overview] An overview of an information processing system according to a first embodiment will be described with reference to Fig. 1. The information processing system according to this embodiment includes a plurality of vehicles 10, a server device 2, and a terminal device 3. The vehicles 10 are connected vehicles that can access a wireless communication network. The vehicles 10 can communicate with the server device 2 via the wireless communication network (for example, a cellular communication network).

[0026] The vehicle 10 is equipped with an in-vehicle device 1. The in-vehicle device 1 functions as a drive recorder and a wireless communication device. The in-vehicle device 1 has a function of acquiring and storing video images (hereinafter referred to as "video" or "in-vehicle video") of the area in front of the vehicle using an in-vehicle camera, and a function of providing the stored video to an external device (e.g., a server device 2) in response to a request from the external device. The vehicle 10 also has a function of periodically transmitting location information to the server device 2.

[0027] In this embodiment, the in-vehicle device 1 has a wireless communication function, but the vehicle 10 may have a data communication module (hereinafter referred to as DCM) for connecting the components of the vehicle to a network. That is, the in-vehicle device and the wireless communication device may be separate devices.

[0028] The server device 2 is configured to be able to communicate with a plurality of vehicles 10 via a network. The server device 2 periodically acquires location information from the plurality of vehicles 10 under its management, By referring to the database managed by the server device 2, it is possible to identify the locations and times where multiple vehicles 10 under the control of the system have traveled in the past.

[0029] Based on the information acquired from the server device 2, the terminal device 3 identifies a vehicle 10 that is presumed to have the desired vehicle-mounted video, and acquires the vehicle-mounted video from the vehicle 10.

[0030] Specifically, the terminal device 3 queries the server device 2 and obtains a list of vehicles that were traveling within a specified area at a specified time. The terminal device 3 then maps the corresponding vehicles on a map and accepts a vehicle selection from an operator. The terminal device 3 then requests the selected vehicle 10 to transmit on-board video. This allows the operator of the terminal device 3 to check the video captured by the target vehicle.

[0031] The terminal device 3 is placed in an institution that uses the video captured by the vehicle 10. Examples of such institutions include a police agency and a fire department. For example, when a fire department uses the terminal device 3, an operator is triggered by a call to 119 to identify a vehicle 10 that was traveling near the scene of the call around the time of the call, and acquires on-board video from the vehicle 10.

[0032] In this embodiment, the server device 2 periodically acquires location information from the traveling vehicle 10 as well as video captured by an on-board camera mounted on the vehicle 10. The server device 2 also automatically determines whether the acquired video includes any abnormal events such as vehicle accidents. The server device 2 also adds data indicating the results of the abnormal event determination to the vehicle list provided to the terminal device 3.

[0033] When displaying the vehicle position information acquired from the server device 2 on a map, the terminal device 3 refers to the abnormal event determination result made by the server device 2, and if it determines that an abnormal event is included in the video, highlights the position information of the corresponding vehicle 10. Highlighting may mean, for example, displaying an icon indicating the vehicle position in a different manner than usual.

[0034] On the screen of the terminal device 3, the operator selects any vehicle whose position is displayed on the map. Then, the terminal device 3 transmits a request to the selected vehicle 10 to transmit the video captured by the on-board camera to the server device 2.

[0035] As a result, the information processing device according to the present disclosure can map vehicles that can provide in-vehicle video on a map, detect abnormal events from video captured by the on-board camera of the vehicle 10, and highlight the location where the abnormal event was captured on the map. Note that although the information processing system is composed of the server device 2 and the terminal device 3, the information processing system may also be realized as a single information processing device. In other words, an operator may directly operate the server device 2 to obtain in-vehicle video from the vehicle and display it in a predetermined format on the display unit.

[0036] [Configuration of information processing system] Next, the hardware and software configurations of the devices that make up the system will be described. FIG. 2 is a diagram showing a schematic configuration of an in-vehicle device 1, a server device 2, and a terminal device 3 operated by an operator, all of which are installed in a vehicle 10. As shown in FIG.

[0037] The in-vehicle device 1 includes a processor (CPU, GPU, etc.), a main memory device (RAM, ROM, etc.), and an auxiliary memory device (EPROM, hard disk drive, removable media, etc.). The auxiliary storage device stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions (software modules) that meet predetermined purposes, as described below, can be realized. However, some or all of the functions may be realized as hardware modules using hardware circuits such as ASICs, FPGAs, etc.

[0038] The in-vehicle device 1 includes a control unit 11, a storage unit 12, a communication unit 13, a position information acquisition unit 14, and a camera 15.

[0039] The control unit 11 is a computing unit that executes predetermined programs to realize various functions of the in-vehicle device 1. The control unit 11 can be realized by, for example, a hardware processor such as a CPU. The control unit 11 may also be configured to include RAM, ROM (Read Only Memory), cache memory, etc.

[0040] In this embodiment, the control unit 11 of the in-car device 1 is configured to have three software modules: a photographing unit 111, a notification unit 112, and a provision unit 113. Each software module may be realized by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 12. Note that the information processing executed by the software modules is synonymous with the information processing executed by the control unit 11 (CPU, etc.).

[0041] While the device is in operation, the photographing unit 111 photographs moving images (in-vehicle images) using the camera 15 and stores the obtained data in the storage unit 12. The photographing unit 111 receives power supply from the vehicle and photographs continuously, and records the obtained moving image data in the storage unit 12.

[0042] Video data is made up of multiple video files. There is an upper limit to the length of a video corresponding to one file, and if the upper limit is exceeded, a new video file is generated. In this embodiment, a video file is generated every minute. If the storage capacity is insufficient, the shooting unit 111 deletes the oldest video file to secure free space and then continues shooting. In this embodiment, each video file is associated with an identifier (date and time ID) that identifies the date and time the video file was generated (i.e., the date and time when shooting started). The date and time ID may uniquely identify the year, month, day, hour, minute, and second.

[0043] When the image capturing unit 111 is activated, the in-vehicle device 1 functions as a drive recorder.

[0044] The notification unit 112 periodically generates data related to the traveling of the vehicle (hereinafter referred to as vehicle data) and transmits it to the server device 2. In this embodiment, the vehicle data includes the position information of the vehicle 10, the traveling direction of the vehicle 10, and a date and time ID indicating the date and time when the position information was acquired.

[0045] In this embodiment, the vehicle data includes an identifier (vehicle ID) that uniquely identifies the vehicle 10, the date and time when the vehicle data was generated, an identifier (date and time ID) that uniquely identifies the date and time, location information, and information indicating the direction of travel. The location information and direction of travel can be acquired from the location information acquisition unit 14, which will be described later.

[0046] The vehicle data is generated and transmitted at a predetermined cycle. The predetermined cycle may be measured independently, or may be linked to the image capturing unit 111. For example, the notification unit 112 may generate vehicle data at the timing when the image capturing unit 111 generates a new video file (for example, at one-minute intervals). In this case, the date and time indicated by the vehicle data will be the same as the date and time when the new video file was generated.

[0047] The providing unit 113 periodically transmits the video data generated by the imaging unit 111 to the server device 2. The video data transmitted to the server device 2 is used by the server device 2 to determine whether or not an abnormal event has occurred.

[0048] Furthermore, the providing unit 113 provides the terminal device 3 with a video file that was shot at a specified time, based on a request (a request to provide a video file) sent from the terminal device 3. For example, when the providing unit 113 receives a provision request including a date and time ID from the terminal device 3, the providing unit 113 obtains the video file corresponding to the date and time ID from the storage unit 22 and transmits it to the terminal device 3.

[0049] The storage unit 12 is a means for storing information, and is configured with storage media such as RAM, a magnetic disk, a flash memory, etc. The storage unit 12 stores programs executed by the control unit 11, data used by the programs, etc.

[0050] The communication unit 13 is a wireless communication interface for connecting the in-vehicle device 1 to an external network. The communication unit 13 is configured to be able to communicate with the server device 2 via, for example, a wireless LAN or a cellular communication network such as 3G, 4G, or 5G.

[0051] The position information acquisition unit 14 acquires position information of the vehicle 10. The position information acquisition unit 14 includes a GPS antenna and a positioning module for determining the position information. The GPS antenna is an antenna that receives positioning signals transmitted from positioning satellites (also referred to as GNSS satellites). The positioning module is a module that calculates position information based on the signals received by the GPS antenna. The position information acquisition unit 14 may determine the traveling direction of the vehicle 10 based on the transition of the position information.

[0052] The camera 15 is an optical unit including an image sensor for acquiring images, and is mounted, for example, facing forward of the vehicle 10.

[0053] Next, the server device 2 will be described.

[0054] Like the in-vehicle device 1, the server device 2 can be configured as a computer having a processor (CPU, GPU, etc.), a main memory device (RAM, ROM, etc.), and an auxiliary memory device (EPROM, hard disk drive, removable media, etc.).

[0055] The server device 2 includes a control unit 21, a storage unit 22, and a communication unit .

[0056] The control unit 21 is a computing unit that executes predetermined programs to realize various functions of the server device 2. The control unit 21 can be realized by, for example, a hardware processor such as a CPU. The control unit 21 may also be configured to include RAM, ROM, cache memory, etc.

[0057] In this embodiment, the control unit 21 of the server device 2 is configured to have two software modules: a data update unit 211 and an information providing unit 212. Each software module may be realized by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 22. Note that the information processing executed by the software module is synonymous with the information processing executed by the control unit 21 (CPU, etc.).

[0058] The data update unit 211 receives vehicle data from a plurality of vehicles 10 (on-vehicle devices 1) and stores the vehicle data in the storage unit 22. For example, the data update unit 211 receives vehicle data from a plurality of vehicles 10. The data may be stored in a table (referred to as a vehicle data table).

[0059] Furthermore, the data update unit 211 determines whether or not an abnormal event is included in the video data (video) received from the in-vehicle device 1. Specifically, the abnormal event here refers to a vehicle accident or the like. That is, the data update unit 211 determines whether or not an abnormal event such as a vehicle accident is captured in the video data (video). The data update unit 211 may use an image recognition technology such as deep learning to determine whether or not an abnormal event is included in the video data (video).

[0060] The information providing unit 212 provides information to the terminal device 3 based on the information stored in the storage unit 22. Specifically, the information providing unit 212 performs the following two types of processing.

[0061] (1) Providing information on vehicle location The vehicle data table stores location information of the vehicle 10 equipped with the in-vehicle device 1 that functions as a drive recorder, in association with date and time information. By referring to this, it is possible to extract vehicles that have traveled within a specified area within a specified time span.

[0062] When a request is received from the terminal device 3, the information providing unit 212 extracts vehicles that have traveled within a specified area within a specified time period, and transmits pairs of location information and date and time information to the terminal device 3. This allows mapping onto a map on the terminal device 3 side. The location information may include information indicating the direction of travel.

[0063] Furthermore, when the data update unit 211 determines that the video data (video) contains an abnormal event, the information providing unit 212 notifies the terminal device 3 of that fact. The information providing unit 212 may notify the terminal device 3 of the ID of the video data determined to contain an abnormal event, the location information of the vehicle that captured the video data, etc.

[0064] (2) In-car video broadcasting When the terminal device 3 selects a target vehicle for which in-vehicle video is requested, the information providing unit 212 receives a video playback request from the terminal device 3. The video playback request includes information identifying the target vehicle and the target date and time. The information providing unit 212 transfers the video playback request to the target vehicle based on the information. The information providing unit 212 also transfers the video file (or video stream) of the in-vehicle video transmitted from the target vehicle to the terminal device 3.

[0065] The storage unit 22 is a means for storing information, and is configured with storage media such as RAM, a magnetic disk, a flash memory, etc. The storage unit 22 stores programs executed by the control unit 21, data used by the programs, etc. The storage unit 22 also stores a table (vehicle data table) for storing data received from the in-vehicle device 1.

[0066] The communication unit 23 is a communication interface for connecting the server device 2 to a network. The communication unit 23 is configured to be able to communicate with the network via, for example, Ethernet (registered trademark), a wireless LAN, a cellular communication network, or the like.

[0067] Next, the terminal device 3 will be described. Like the in-vehicle device 1 and the server device 2, the terminal device 3 can be configured as a computer having a processor (CPU, GPU, etc.), a main memory device (RAM, ROM, etc.), and an auxiliary memory device (EPROM, hard disk drive, removable media, etc.).

[0068] The terminal device 3 includes a control unit 31, a storage unit 32, a communication unit 33, and an input / output unit .

[0069] The control unit 31 is a computing unit that executes predetermined programs to realize various functions of the terminal device 3. The control unit 31 can be realized by a hardware processor such as a CPU, for example. The control unit 31 may also be configured to include RAM, ROM, cache memory, etc.

[0070] In this embodiment, the control unit 31 of the terminal device 3 is configured to have two software modules: a display control unit 311 and a video acquisition unit 312. Each software module may be realized by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 32. Note that the information processing executed by the software module is synonymous with the information processing executed by the control unit 31 (CPU, etc.).

[0071] The display control unit 311 generates a user interface (GUI) for requesting acquisition of in-vehicle video and outputs it via the input / output unit 34, which will be described later. In this embodiment, the display control unit 311 generates and outputs a user interface including a road map image based on road map data stored or acquired in advance. The user interface includes a user interface component for specifying a target period.

[0072] Furthermore, the display control unit 311 draws an icon indicating the traveling position of the vehicle 10 based on the vehicle list notified by the information providing unit 212 of the server device 2. Then, when it is determined based on the information in the vehicle list that there is a vehicle 10 that captured video data including an abnormal event, the display control unit 311 displays the traveling position of the vehicle 10 that captured the video data (video) including the abnormal event at the time the video data was captured in a marked-up form on the map. In other words, the display control unit 311 displays the icon indicating the traveling position of the vehicle 10 that captured the video data including the abnormal event in a different form from the icon indicating the traveling position of a normal vehicle. The difference between the normal form and the marked-up form may be, for example, that the normal form is a display form set by default, and the marked-up form is a form that emphasizes the display form set by default.

[0073] When one of the mapped icons is selected by the operator, the video acquisition unit 312 transmits a request to play the in-vehicle video to the corresponding vehicle 10. The video acquisition unit 312 also plays the video file or video stream transmitted from the vehicle 10. In the following description, it is assumed that the playback request is transmitted to the vehicle 10 via the server device 2. It is also assumed that the video file or video stream is received via the server device 2.

[0074] The storage unit 32 is a means for storing information, and is configured with storage media such as RAM, a magnetic disk, a flash memory, etc. The storage unit 32 stores programs executed by the control unit 31, data used by the programs, etc.

[0075] The communication unit 33 is a wireless communication interface for connecting the terminal device 3 to an external network. The communication unit 33 is configured to be able to communicate with the server device 2 via, for example, a wireless LAN or a cellular communication network such as 3G, 4G, or 5G.

[0076] The input / output unit 34 is a unit that receives input from an operator of the device and presents information to the operator. Specifically, the input / output unit 34 is composed of a touch panel and its control means, and a liquid crystal display and its control means. In this embodiment, the touch panel and the liquid crystal display are composed of a single touch panel display.

[0077] 2 is an example, and all or part of the illustrated functions may be executed using a dedicated circuit. Furthermore, the information processing system of the present disclosure may store or execute programs using a combination of a main memory device and an auxiliary memory device other than those illustrated.

[0078] In addition, in the configuration shown in FIG. 2, the server device 2 and the terminal device 3 may be realized as a single information processing device.

[0079] [Information Processing System Processing] Next, the steps executed by each device and the flow of data exchanged between the devices will be described.

[0080] Figure 3 is a sequence diagram of the process of transmitting vehicle data and video data (video) from the in-vehicle device 1 to the server device 2, and the process of transmitting information for mapping the vehicle's location information from the server device 2 to the terminal device 3, and the process of the terminal device 3 drawing a map. First, the process (steps S11 to S14) of transmitting vehicle data and video data (images) from the in-vehicle device 1 to the server device 2 will be described. This process is executed at a predetermined cycle. The predetermined cycle may be determined by the notification unit 112, or may be determined by the notification unit 112 in response to a notification from the image capture unit 111. For example, the image capture unit 111 may transmit a notification to the notification unit 112 every time a video file is switched (for example, every minute), and in response to this, the notification unit 112 may start the process shown in the figure.

[0081] First, in step S11, the in-vehicle device 1 (notification unit 112) acquires vehicle data and video data. In this embodiment, the notification unit 112 acquires the position information and traveling direction of the vehicle from the position information acquisition unit 14, and generates vehicle data by combining the identifier of the vehicle with date and time information. Then, the notification unit 112 acquires the latest video data generated by the imaging unit 111 by capturing video using the camera 15. The acquired vehicle data and video data are transmitted to the server device 2 (step S12).

[0082] In step S13, the server device 2 (data update unit 211) receives the vehicle data and video data, and stores the vehicle data and video data in a vehicle data table in the storage unit 22. The data update unit 211 may update the vehicle data table with newly received vehicle data, etc.

[0083] Next, in step S14, the server device 2 (data update unit 211) determines whether or not the received video data includes an abnormal event. For example, if the received video data shows an abnormal event such as a vehicle accident, the data update unit 211 determines that the received video data includes an abnormal event. If it is determined that the received video data includes an abnormal event, a flag indicating that the video data includes an abnormal event may be added to the corresponding record in the vehicle data table.

[0084] 3 illustrates an embodiment in which the server device 2 determines whether or not an abnormal event is included in the video data, but the determination of whether or not an abnormal event is included in the video data may be performed by the in-vehicle device 1. In this case, the in-vehicle device 1 may transmit the result of the abnormality determination to the server device 2 together with the video data including the abnormal event.

[0085] By executing the above-described process for a plurality of vehicles 10, the server device 2 can collect and manage past position information of a plurality of vehicles under its management. Also, by referring to the vehicle data table stored in the server device 2, the server device 2 can search for vehicles that were traveling at a predetermined location on a predetermined date and time. Each vehicle is equipped with an in-vehicle video Because the vehicle is traveling while taking pictures, this makes it possible to search for vehicles that are likely to have captured footage of a particular event (for example, an event related to a report, such as a traffic accident).

[0086] Next, a description will be given of a process (steps S21 to S29) in which the terminal device 3 outputs a road map onto which vehicle position information is mapped. This process is started, for example, when a road map is displayed on the terminal device 3 (display control unit 311).

[0087] In step S21, the terminal device 3 (display control unit 311) determines whether or not it is necessary to redraw the icon. For example, if a road map is displayed for the first time, or if an operation to change the display range of the road map or an operation to change the target period has been performed, this step will result in a positive determination. For example, if an operation to display a road map is performed by the operator of the terminal device 3 and a target period is specified, it will be necessary to redraw the icon.

[0088] If the determination in this step is positive, the terminal device 3 generates a vehicle list generation request and transmits it to the server device 2 (step S22). The vehicle list generation request includes information indicating the position of the first area set by the device and information indicating the target period specified by the operator.

[0089] Next, in step S23, the data update unit 211 determines whether or not it has determined that the video data contains an abnormal event. That is, the data update unit 211 determines whether or not an abnormal event has been detected from the video. Note that if the in-car device 1 has determined whether or not the video data contains an abnormal event, in step S23, the server device 2 receives from the in-car device 1 the determination result indicating that an abnormal event has been captured and the video data, and then makes the determination in step S23.

[0090] If the determination in this step is positive, the server device 2 (information provider 212) generates a vehicle list indicating vehicles that have captured images of the abnormal event based on the received request and the determination result of the data updater 211 (step S24). Specifically, records of vehicles that have a history of traveling within the first area are extracted from the vehicle data table, and a vehicle list is generated that includes records of vehicles whose video data contains an abnormal event from among the extracted records. The vehicle list of vehicles that have captured images of the abnormal event is transmitted to the terminal device 3 (step S25).

[0091] In step S26, the terminal device 3 (display control unit 311) marks up and displays on the map the driving position of the vehicle that captured the image of the abnormal event. Displaying by marking up means that the terminal device 3 displays an icon that is highlighted, for example, compared with an icon that indicates the driving position of a normal vehicle. The display control unit 311 identifies the position where the vehicle 10 was driving when the video data containing the abnormal event was captured, based on the vehicle list that indicates the vehicles that captured the image of the abnormal event. Then, the display control unit 311 outputs an icon (first display) that is different from the icon that indicates the driving position of a normal vehicle, at a position on the map that corresponds to the identified position. The first display may be an icon, or may be, for example, a marking pin or a pop-up that indicates the driving position of the vehicle 10 that captured the image of the abnormal event.

[0092] If the display control unit 311 cannot identify a single vehicle driving position corresponding to a certain abnormal event based on the list, the display control unit 311 may display multiple vehicle driving positions as candidate positions. For example, the display control unit 311 may highlight on a map the driving position of the vehicle that captured the abnormal event, or display a pop-up at the position on the map.

[0093] The display control unit 311 displays the point identified as the traveling position of the vehicle corresponding to the abnormal event. , or may notify designated administrative agencies such as fire departments.

[0094] Furthermore, for example, if the location where a report has already been made to a specified administrative agency (e.g., a fire department system) is the same as the driving position of the vehicle that captured the abnormal event identified by the display control unit 311, the display control unit 311 may highlight that location on the map.

[0095] Furthermore, for example, when a report is made to a predetermined administrative agency, the display control unit 311 may acquire report information, which is information indicating the content of the report, from a predetermined device (e.g., a server device, etc.) in the administrative agency. Then, the display control unit 311 may identify the location where the event related to the report occurred based on the report information, and output a second display at a position on a map corresponding to the location of the occurrence. The second display may be, for example, a mark pin or a pop-up.

[0096] After step S25, the server device 2 (information provider 212) generates a vehicle list based on the received request (step S27). Specifically, records of vehicles that have a history of traveling within the first area during a specified period are extracted from the vehicle data table stored in the storage unit 22, and a vehicle list made up of the extracted records is generated. The vehicle list is a list of vehicles 10 that have not captured any video containing an abnormal event. The generated vehicle list is transmitted to the terminal device 3 (step S28). Steps S27 and S28 may be executed before step S26.

[0097] In step S29, the terminal device 3 (display control unit 311) draws a map based on the acquired vehicle list and maps an icon different from that in the first display to a corresponding point on the map. Step S29 may be executed simultaneously with step S26.

[0098] When the process of step S29 is completed, a plurality of icons are mapped on the map in the terminal device 3. The icons represent the positions of vehicles that have been traveling within the first area within the specified period.

[0099] In addition, when the server device 2 and the terminal device 3 are realized as a single information processing device, the processes performed by the server device 2 and the terminal device 3 may be realized as processes performed by each functional unit in the information processing device.

[0100] 4 is a flowchart illustrating the process of determining whether or not an abnormal event is included in video data, which is executed by the server device 2. The process in FIG. 4 is a detailed description of step S14 in FIG.

[0101] In step S141, the data update unit 211 analyzes the video data acquired from the in-car device 1 in step S12. The data update unit 211 may perform image analysis on the video data to recognize objects included in the video data. At this time, the data update unit 211 may analyze the video data using AI (Artificial Intelligence).

[0102] Next, in step S142, the data update unit 211 determines whether or not the video data (video) includes an abnormal event. The data update unit 211 may determine that the video data includes an abnormal event if the objects recognized by analyzing the video data include an object that represents an abnormal event such as a vehicle accident. If the data update unit 211 determines that the video data includes an abnormal event, the determination in this step is affirmative.

[0103] If the determination in this step is affirmative, the process proceeds to step S214.

[0104] If the determination in this step is negative, the process proceeds to step S215.

[0105] If the process proceeds to step S143, the data update unit 211 determines that the vehicle that captured the video data has detected an abnormal event. The data update unit 211 determines that the vehicle detected an abnormal event at the position where the vehicle was traveling when the video data was captured.

[0106] If the process proceeds to step S144, the data update unit 211 determines that the vehicle that captured the video data did not detect an abnormal event. The data update unit 211 determines that the vehicle did not detect an abnormal event at the position where the vehicle was traveling when the video data was captured.

[0107] 5 is a sequence diagram of a process in which the in-vehicle device 1 provides an image to the terminal device 3 based on a request from the terminal device 3. The illustrated process is started when an icon on a map is designated by the operator.

[0108] First, the terminal device 3 (video acquisition unit 312) generates a playback request (step S41). The playback request is data for requesting the in-vehicle device 1 to provide in-vehicle video, and includes an identifier (vehicle ID) of the in-vehicle device 1 (vehicle 10) and an identifier (date and time ID) of the date and time when the in-vehicle video was captured. The vehicle ID and date and time ID are included in the vehicle list that the terminal device 3 receives from the server device 2. When the operator selects an icon on the map, the terminal device 3 can identify the vehicle ID and date and time ID of the corresponding vehicle.

[0109] The playback request is transmitted from the terminal device 3 to the server device 2 (step S42), and is relayed from the server device 2 to the target in-vehicle device 1 (step S43). At this time, the server device 2 may read the vehicle ID included in the playback request and identify the in-vehicle device 1 to which the playback request should be relayed. For this reason, it is preferable that the server device 2 stores data associating the vehicle ID with the connection destination (e.g., network address, etc.) of the corresponding in-vehicle device 1. The playback request relayed by the server device 2 is received by the corresponding in-vehicle device 1.

[0110] Upon receiving the playback request, the in-car device 1 (providing unit 113) extracts the video file specified by the date and time ID from the stored video data (step S44).

[0111] The extracted video file is transmitted to the terminal device 3 via the server device 2 (steps S45 and S46). The terminal device 3 generates a playback screen and starts playing the received video file (step S47). If an operation to specify another video file is performed, the process shown in Fig. 5 is executed again, and a new playback request is generated.

[0112] In this example, a moving image file is the target of transmission, but the target of transmission may be a video stream or the like.

[0113] As explained above, in the information processing system according to this embodiment, the server device 2 determines whether or not an abnormal event is included in the video image acquired by the vehicle-mounted device 1, based on the video image acquired by the vehicle-mounted device 1. If the server device 2 determines that an abnormal event is included in the video image, the terminal device 3 marks up and displays on a map the traveling position of the corresponding vehicle at the time the video image was acquired. The operator of the terminal device 3 can visually identify the location where the abnormal event occurred on the map where the position is displayed, and can acquire the video image of the vehicle-mounted camera at that location. The markup If this is not done, after receiving a report, the operator needs to search for a vehicle that has captured the event related to the report, but according to this embodiment, the operator of the terminal device 3 can efficiently search for a vehicle that has captured the event. Note that the information processing system according to this embodiment may be realized as a single information processing device.

[0114] (Modification of the first embodiment) In a modification of the first embodiment, the display control unit 311 performs the following operation. When the data update unit 211 determines that video data captured by multiple vehicles 10 at the same location contains an abnormal event, the display control unit 311 may change the manner of the first display compared to when it is not determined that the video data contains an abnormal event. This is because, when an abnormal event is contained in each of the video data acquired from multiple vehicles 10 captured at the same location, the reliability that an abnormal event has actually occurred at that location increases. For example, when it is determined that video data captured by multiple vehicles 10 at the same location contains an abnormal event, the display control unit 311 may display the location in a more emphasized manner compared to the first display output when it is not determined that the video data contains an abnormal event.

[0115] (Second embodiment) In the first embodiment, the server device 2 receives vehicle data and video data from the in-vehicle device 1 and determines whether the video data contains an abnormal event. If the terminal device 3 determines that the video data contains an abnormal event, it marks up and displays the driving location of the vehicle that captured the video data. On the other hand, if video data containing an abnormal event is detected in a section with specific traffic conditions, such as traffic congestion, the reliability of the abnormal event actually occurring in that section increases. Therefore, in the second embodiment, the in-vehicle device 1 determines a traffic congestion section based on the vehicle data of the vehicle 10, and if an abnormal event is detected in the traffic congestion section, the terminal device 3 highlights the location compared to when an abnormal event is detected in a normal section.

[0116] The following describes in detail the processing executed by the in-car device 1 in the second embodiment. Fig. 6 is a diagram showing an outline of the processing executed by the information processing system in the second embodiment.

[0117] First, the in-vehicle device 1 acquires CAN data from the vehicle 10. The CAN data indicates, for example, running data of the vehicle 10, such as the speed, direction of travel, and acceleration. Then, the in-vehicle device 1 determines a section where traffic is stagnating (hereinafter referred to as a stagnating location) based on the CAN data. For example, the in-vehicle device 1 may determine, as a stagnating location, a section where the speed or acceleration of the vehicle 10 indicated by the CAN data is equal to or lower than a threshold.

[0118] Next, when the in-vehicle device 1 captures video data at a traffic congestion location, it transmits a flag indicating that the vehicle has traveled near the congestion location together with the vehicle data and video data to the server device 2. When the in-vehicle device 1 captures video data in a section where traffic is not stagnant, it transmits the vehicle data and video data to the server device 2 but does not transmit a flag.

[0119] The server device 2, which has received the vehicle data, video data, and flag, determines whether the video data contains an abnormal event, as in the first embodiment. If the video data captured at the stay location contains an abnormal event, the server device 2 adds information indicating that the corresponding vehicle captured the video data at the stay location to the vehicle list of the vehicle that captured the video data containing the abnormal event. The server device 2 then transmits the vehicle list to the terminal device 3.

[0120] Finally, the terminal device 3 (display control unit 311) that has received the vehicle list displays the position on the map corresponding to the traveling position of the vehicle that captured the video data containing the abnormal event at the stay location. The travel position is displayed on the map in a more emphasized manner than in normal cases. For example, the display control unit 311 outputs a larger mark pin than normal at a position on the map corresponding to the travel position. Alternatively, the display control unit 311 may output a display in a color different from normal at a position on the map corresponding to the travel position.

[0121] As described above, in the information processing system according to this embodiment, the in-vehicle device 1 determines a congestion location based on the CAN data of the vehicle 10. If the server device 2 determines that an abnormal event is included in the video captured at the congestion location, the terminal device 3 highlights on a map the traveling position of the corresponding vehicle at the time the video was acquired. The operator of the terminal device 3 can check the traffic conditions in the vicinity on the map displaying the position. As a result, the information processing system according to the present disclosure can realize a smoother initial response by fire departments and the like.

[0122] (Other variations) The above-described embodiment is merely an example, and the present disclosure can be implemented by appropriately modifying it within the scope that does not deviate from the gist thereof.

[0123] The present disclosure can also be realized by providing a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer via a non-transitory computer-readable storage medium connectable to the computer's system bus or via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (e.g., a floppy disk, a hard disk drive (HDD), etc.), an optical disk (e.g., a CD-ROM, a DVD disk, a Blu-ray disk), a read-only memory (ROM), a random-access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, or any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0124] 1...In-vehicle device 2. Server device 3. Terminal device 11, 21, 31...Control unit 12, 22, 32...Storage section 13, 23, 33...Communications Department 14...Location information acquisition unit 15. Camera 34...Input / output section 111···Photography Department 112...Notification Department 113...Providing Department 211...Abnormality determination section 212...Abnormality notification section 311 Display control unit 312...Video acquisition unit

Claims

1. Acquiring video captured by an on-board camera mounted on a vehicle; analyzing the video to determine whether the video contains an abnormal event; When it is determined that the video includes the abnormal event, outputting a first display indicating that the abnormal event has been detected at a position on a map corresponding to a location where the video was acquired; a control unit that executes the following: Information processing device.

2. The control unit When a report is made to a designated administrative agency, acquiring notification information indicating the content of the notification from a predetermined device in the administrative agency; Identifying the location of the event reported based on the report information; further outputting a second display showing the point of occurrence at a position on the map corresponding to the point of occurrence; The information processing device according to claim 1 .

3. The control unit Identifying a congestion point where traffic is stagnating based on the travel data acquired by the plurality of vehicles; When the video including the abnormal event is acquired by the vehicle located near the stagnation location, the first display is output in an emphasized manner compared to when the video including the abnormal event is acquired by the vehicle not located near the stagnation location.

3. The information processing device according to claim 1.

4. acquiring an image captured by an on-board camera mounted on a vehicle; analyzing the video to determine whether the video contains an abnormal event; outputting a first indication indicating that the abnormal event has been detected at a position on a map corresponding to a location where the image was acquired when it is determined that the abnormal event is included in the image; Including, Information processing methods.

5. A program for causing a computer to execute the information processing method according to claim 4.

Images