Information processing device

Abstract

The objective is to more efficiently identify and notify users of images containing specific content from the images captured by cameras, compared to conventional methods. [Solution] The information processing device displays multiple icons on a map, each indicating the location of multiple cameras at different times, based on the location history of multiple cameras. When any of the multiple icons is selected, the information processing device provides video footage captured by the camera corresponding to the selected icon. The device acquires video footage captured by each of the multiple cameras, analyzes the video footage, determines whether or not the video footage contains predetermined features, and if it determines that the video footage contains predetermined features, it causes the terminal to output a predetermined display.

Description

Technical Field

[0001] This disclosure relates to a monitoring system.

Background Art

[0002] There is a technology for performing monitoring using a moving body equipped with a camera. In this regard, for example, Patent Document 1 discloses a system that manages the history of the driving position of a vehicle by a server device and identifies a vehicle equipped with a drive recorder located near a reporter.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of this disclosure is to more efficiently identify and notify a video in which predetermined content is captured from videos captured by a camera.

Means for Solving the Problems

[0005] One aspect of this disclosure is an information processing device that displays a plurality of icons indicating the positions of each of the plurality of cameras at each time on a map based on the history of the positions of the plurality of cameras, and provides a video captured by the camera corresponding to the selected icon when any one of the plurality of icons is selected, the information processing device including acquiring videos captured by each of the plurality of cameras, analyzing the videos to determine whether the videos include a predetermined feature, and when it is determined that the videos include the predetermined feature, causing a terminal to output a predetermined display, and having a control unit that executes the above operations. It is an information processing device.

[0006] Furthermore, other embodiments include a program for causing a computer to execute an information processing method performed by the information processing device described above, or a computer-readable storage medium that non-temporarily stores the program. [Effects of the Invention]

[0007] According to this disclosure, it is possible to identify and notify, more efficiently than before, images captured by a camera that contain predetermined content. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram illustrating an overview of the processes performed by a system including a server device according to the first embodiment. [Figure 2] A diagram illustrating the components of a system including a server device according to the first embodiment. [Figure 3] A sequence diagram of the process of analyzing video data performed by a system including a server device according to the first embodiment. [Figure 4] A sequence diagram of the process of playing video data executed by a system including a server device according to the first embodiment. [Figure 5] An example of a screen displaying video including abnormalities on the road. [Figure 6] A sequence diagram of the process of analyzing video data performed by a system including a server device according to the second embodiment. [Modes for carrying out the invention]

[0009] (overview) In recent years, the connectivity of automobiles has advanced, and the number of vehicles equipped with communication functions has increased. Furthermore, the number of vehicles equipped with onboard cameras has also increased.

[0010] In this regard, making footage captured by cameras mounted on vehicles (such as dashcams) available for review by designated organizations can be useful in the initial response to incidents and accidents. For example, if a 110 (police) or 119 (fire / ambulance) call is made, the relevant authorities can quickly confirm the situation at the scene by obtaining dashcam footage from vehicles that were driving near the relevant location around the time the incident occurred. One known system for this purpose is one that maps icons representing the locations of multiple vehicles at predetermined times in the past onto a map and presents them to the operator.

[0011] The system acquires location information of vehicles that were traveling in a designated area around a time specified by the operator (for example, within 5 minutes prior to the time of the call) and maps it on a map. The operator selects vehicles on the map that were traveling near a desired location (for example, the location of the call) at a time close to the desired time (for example, the time of the call). The system also requests that the vehicle transmit the dashcam footage recorded at that time. This allows the operator to view dashcam footage that may have captured the scene of the call.

[0012] Incidentally, agencies that receive emergency calls have a need to detect and review video footage of accident or disaster scenes before a call is made, in order to facilitate a rapid initial response. To this end, it is desirable that the system be able to automatically recognize accident or disaster scenes from dashcam footage and notify the system user of the presence of an accident, in addition to providing regular dashcam footage.

[0013] An information processing device relating to one aspect of this disclosure is: Based on the history of the positions of a plurality of cameras, display a plurality of icons indicating the positions of each of the plurality of cameras at each time on a map, and when any one of the plurality of icons is selected, provide an information processing apparatus that provides the video captured by the camera corresponding to the selected icon, the information processing apparatus comprising: acquiring the video captured by each of the plurality of cameras; analyzing the video to determine whether the video contains a predetermined feature; and when it is determined that the video contains the predetermined feature, causing a terminal to output a predetermined display.

[0014] The control unit acquires the videos captured by the plurality of cameras and analyzes the videos. Then, the control unit determines whether the analyzed videos contain a predetermined feature. Subsequently, when the analyzed videos contain a predetermined feature, the control unit causes the terminal to output a predetermined display. Specifically, when the analyzed videos contain a predetermined feature, the control unit may display an icon indicating that at the corresponding position on the map.

[0015] According to such a configuration, the information processing apparatus according to the present disclosure can more efficiently identify and notify that there is a video in which a predetermined content is captured among the videos captured by the camera.

[0016] Further, the analysis of the video may include a process of textifying the video using machine learning. It may be.

[0017] Specifically, the control unit may textify the acquired video using a generation AI.

[0018] Thereby, the information processing apparatus according to the present disclosure can grasp the content of the video in a straightforward manner.

[0019] Further, the predetermined feature is an event related to an abnormality on the road, and when the control unit detects a word related to an abnormality on the road from the textified sentence of the video, the control unit may determine that the video contains an event related to an abnormality on the road. <000009

[0020] Road abnormalities refer to any event on the road that obstructs vehicle traffic, such as traffic accidents, the presence of fallen objects or obstacles on the road, road fires, road collapses, or road flooding.

[0021] Furthermore, the plurality of cameras may be cameras mounted on multiple vehicles, and the control unit may periodically acquire the video footage of a predetermined length from vehicles passing through a predetermined point.

[0022] As a result, the information processing device related to this disclosure can periodically observe specific locations, such as places on the road where abnormalities are likely to occur.

[0023] The aforementioned multiple cameras are cameras mounted on multiple vehicles, and the control unit is, A predetermined length of video footage may be acquired from a vehicle that has detected an abnormality in the road.

[0024] In other words, the vehicle may be equipped with a function to detect abnormalities in the road.

[0025] As a result, the information processing device relating to this disclosure can efficiently collect video footage of locations where road anomalies may be occurring.

[0026] The following describes specific embodiments of this disclosure with reference to the drawings. Unless otherwise specified, the hardware configurations, module configurations, functional configurations, etc., described in each embodiment are not intended to limit the technical scope of the disclosure to those configurations alone.

[0027] (First embodiment) [Overview of the processes performed by the system] An overview of the processing performed by a system including a server device according to the first embodiment will be described with reference to Figure 1. Figure 1 is a diagram showing an overview of the processing performed by a system including a server device 2 according to the first embodiment. An information processing device according to one aspect of this disclosure is implemented as a server device 2.

[0028] In the system including the server device 2 according to the first embodiment, the terminal device 3 outputs multiple icons on a map indicating the driving location of the vehicle providing video to the system. When a user clicks on one of these icons, the terminal device 3 acquires video footage captured at the location corresponding to that icon from the vehicle 10 and outputs it. This allows the user to view video footage captured by the vehicle at a desired location on the system.

[0029] In this embodiment, if the vehicle 10 detects any abnormality on the road, it transmits video to the server device 2. The server device 2 analyzes the video captured by the vehicle 10 that detected the abnormality and determines whether or not an abnormality on the road, such as a traffic accident (hereinafter also simply referred to as "abnormality"), has occurred. In this specification, "abnormality" refers to traffic accidents, incidents, fires, natural disasters, etc. This includes situations that would not normally occur.

[0030] First, if a traffic accident or other incident occurs on a road, a vehicle 10 traveling on that road detects the anomaly. There may be multiple vehicles 10. For example, a vehicle 10 may detect obstacles on the road using its mounted sensors, or it may detect that the image from its onboard camera is different from the image of a normal road. A vehicle 10 only needs to be able to detect that some kind of anomaly is occurring on the road; it does not need to determine the nature of the anomaly.

[0031] When vehicle 10 detects an abnormality, it transmits a predetermined length of video, including the moment the abnormality was detected, to the server device 2. Multiple vehicles 10 may each transmit a predetermined length of video they have captured to the server device 2.

[0032] The server device 2, which receives video from vehicle 10, analyzes the received video. Specifically, the server device 2 may convert the received video into text using a text generation AI. For example, if video of a traffic accident scene is received, the received video is converted into text as sentences using words such as "traffic accident," "person," and "vehicle."

[0033] The server device 2 then determines whether the received video contains predetermined features. For example, if the server device 2 converts the received video into text and analyzes it, the server device 2 determines whether the text generated from the received video contains a specific word. This specific word may be, for example, a word that describes some kind of abnormality on the road, such as "traffic accident."

[0034] If server device 2 determines that the received video contains predetermined features, it determines that an anomaly has occurred at the location where the video was captured. For example, if server device 2 converts the received video into text and analyzes it, and the text generated from the received video contains a specific word, server device 2 determines that an anomaly has occurred at the location where the video was captured.

[0035] Then, if the server device 2 determines that an abnormality has occurred at the location where the received video was captured, the server device 2 will output a predetermined display to the terminal device 3. For example, the server device 2 will output a display indicating the abnormality on the map at the location of the vehicle that captured the video in which the abnormality was determined to have occurred.

[0036] Terminal device 3 outputs an abnormality indicator, distinct from the multiple icons indicating the vehicle's location providing video footage to the system. By clicking on the abnormality indicator, the user can more easily identify and view video footage captured at the location where an abnormality such as a traffic accident has occurred.

[0037] As described above, the server device 2 analyzes the video footage acquired from the vehicle 10 that detected the anomaly using methods such as text conversion by a generating AI, and determines whether the video footage contains predetermined features such as words indicating a traffic accident. If the video footage contains the predetermined features, the server device 2 causes the terminal device 3 to output a predetermined display. This makes it possible for the system, including the server device 2, to identify and notify users of video footage containing predetermined content from among the video footage captured by the camera, more efficiently than before.

[0038] [System Configuration] Next, the hardware and software configuration of the system including the server device 2 will be described. Figure 2 is a diagram illustrating the components of the system including the server device 2 according to the first embodiment.

[0039] Figure 2 is a schematic diagram showing the configuration of the on-board device 1, server device 2, and terminal device 3 operated by the operator, which are mounted on the vehicle 10.

[0040] The in-vehicle device 1 can be configured as a computer having a processor (CPU, GPU, etc.), main memory (RAM, ROM, etc.), and auxiliary storage (EPROM, hard disk drive, removable media, etc.). The auxiliary storage contains an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions (software modules) that match a predetermined purpose, as described later, can be realized. However, some or all of the functions may be realized as hardware modules by hardware circuits such as ASICs and FPGAs.

[0041] The in-vehicle device 1 is composed of a control unit 11, a storage unit 12, a communication unit 13, a location information acquisition unit 14, and a camera 15.

[0042] The control unit 11 is a computing unit that realizes various functions of the in-vehicle device 1 by executing a predetermined program. The control unit 11 can be implemented by a hardware processor such as a CPU. The control unit 11 may also be configured to include RAM, ROM (Read Only Memory), cache memory, etc.

[0043] In this embodiment, the control unit 11 of the in-vehicle device 1 is configured with three software modules: an imaging unit 111, a notification unit 112, a provision unit 113, and an anomaly detection unit 114. Each software module may be implemented by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 12. The information processing performed by the software modules is synonymous with the information processing performed by the control unit 11 (CPU, etc.).

[0044] The shooting unit 111 uses the camera 15 to capture video (in-vehicle footage) while the device is in operation, and stores the obtained data in the storage unit 12. The shooting unit 111 continuously captures video using power supplied from the vehicle and records the obtained video data in the storage unit 12.

[0045] The video data consists of multiple video files. There is an upper limit to the length of a video corresponding to a single file, and if this limit is exceeded, a new video file is generated. In this embodiment, a video file is generated every minute. If the storage capacity becomes insufficient, the recording unit 111 deletes the oldest video file to free up space and then continues recording. In this embodiment, each video file is associated with an identifier (date and time ID) that identifies the date and time the video file was created (i.e., the date and time when recording started). The date and time ID may uniquely identify the year, month, day, hour, minute, and second.

[0046] When the camera unit 111 operates, the in-vehicle device 1 functions as a drive recorder.

[0047] The notification unit 112 periodically generates data related to the vehicle's movement (hereinafter referred to as vehicle data) and transmits it to the server device 2. In this embodiment, the vehicle data includes the location information of the vehicle 10, the direction of travel of the vehicle 10, and a date and time ID representing the date and time the location information was acquired.

[0048] The vehicle data in this embodiment consists of an identifier that uniquely represents the vehicle 10 (vehicle ID), the date and time the vehicle data was generated, an identifier that uniquely represents the date and time (date and time ID), location information, and information indicating the direction of travel. The location information and direction of travel are obtained using location information acquisition, which will be described later. It can be obtained from the acquisition part 14.

[0049] Vehicle data is generated and transmitted at predetermined intervals. These predetermined intervals may be measured independently or in conjunction with the imaging unit 111. For example, the notification unit 112 may generate vehicle data at the same time that the imaging unit 111 generates a new video file (for example, at 1-minute intervals). In this case, the date and time indicated by the vehicle data will be the same as the date and time when the video file was newly generated.

[0050] When the anomaly detection unit 114 (described later) detects an anomaly, the providing unit 113 extracts a predetermined length of video data from the video data generated by the shooting unit 111, including the moment when the anomaly detection unit 114 detected the anomaly, and transmits the video to the server device 2.

[0051] Furthermore, the provisioning unit 113 provides the terminal device 3 with a video file recorded at a specified time, based on a request (video file provision request) sent from the terminal device 3. For example, when the provisioning unit 113 receives a provision request from the terminal device 3 that includes a date and time ID, it retrieves the video file corresponding to that date and time ID from the storage unit 22 and sends it to the terminal device 3.

[0052] In other words, if an abnormality is detected on the vehicle side, the video data is spontaneously transmitted from the in-vehicle device 1 for analysis by the server device 2, and if no abnormality is detected on the vehicle side, the video data is transmitted based on a request from the terminal device 3.

[0053] The anomaly detection unit 114 detects the presence or absence of an anomaly based on detection data from sensors mounted on the vehicle 10, or images from an in-vehicle camera. For example, the anomaly detection unit 114 may determine that there is an anomaly on the road if it detects an obstacle on the road using sensors mounted on the vehicle 10, or if it detects, through image analysis or the like, that the images captured by the in-vehicle camera have characteristics different from normal road images.

[0054] The memory unit 12 is a means for storing information and is composed of storage media such as RAM, magnetic disks, and flash memory. The memory unit 12 stores programs executed by the control unit 11, data used by those programs, and so on.

[0055] The communication unit 13 is a wireless communication interface for connecting the in-vehicle device 1 to an external network. The communication unit 13 is configured to communicate with the server device 2 via a cellular communication network such as a wireless LAN, 3G, 4G, or 5G.

[0056] The location information acquisition unit 14 acquires location information of the vehicle 10. The location information acquisition unit 14 includes a GPS antenna and a positioning module for determining location information. The GPS antenna is an antenna that receives positioning signals transmitted from positioning satellites (also called GNSS satellites). The positioning module is a module that calculates location information based on the signals received by the GPS antenna. The location information acquisition unit 14 may also determine the direction of travel of the vehicle 10 based on the change in location information.

[0057] Camera 15 is an optical unit that includes an image sensor for acquiring images. Camera 15 is mounted, for example, facing forward of vehicle 10.

[0058] Next, we will describe server device 2.

[0059] Server device 2, like in-vehicle device 1, includes a processor (CPU, GPU, etc.), main memory (RAM, ROM, etc.), and auxiliary storage (EPROM, hard disk drive, remover). It can be configured as a computer that has (blue media, etc.).

[0060] The server device 2 is comprised of a control unit 21, a storage unit 22, and a communication unit 23.

[0061] The control unit 21 is a computing unit that realizes various functions of the server device 2 by executing a predetermined program. The control unit 21 can be implemented by a hardware processor such as a CPU. The control unit 21 may also be configured to include RAM, ROM, cache memory, etc.

[0062] In this embodiment, the control unit 21 of the server device 2 is configured with three software modules: an acquisition unit 211, an analysis unit 212, and an information provision unit 213. Each software module may be implemented by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 22. The information processing performed by the software modules is synonymous with the information processing performed by the control unit 21 (CPU, etc.).

[0063] The acquisition unit 211 receives vehicle data and video data transmitted when a vehicle 10 detects an abnormality from multiple vehicles 10 (in-vehicle devices 1), and stores them in the storage unit 22. For example, the acquisition unit 211 may store the vehicle data received from multiple vehicles 10 in a table (referred to as a vehicle data table).

[0064] The analysis unit 212 analyzes the video data acquired by the acquisition unit 211 and determines whether the video data contains predetermined features. For example, the analysis unit 212 may convert the video data acquired by the acquisition unit 211 into text using a generation AI or the like, and then determine whether the converted text contains specific words that indicate an anomaly, such as "traffic accident."

[0065] The information provision unit 213 provides information to the terminal device 3 based on the information stored in the storage unit 22. Specifically, the information provision unit 213 performs the following two types of processing.

[0066] (1) Provision of information regarding the vehicle's location The vehicle data table stores location information of vehicles 10 equipped with an in-vehicle device 1 that functions as a drive recorder, linked to date and time information. By referring to this, it is possible to extract vehicles that have traveled within a specified area within a specified time range.

[0067] When requested by the terminal device 3, the information provision unit 213 extracts vehicles that have traveled within a specified area within a specified time range and transmits pairs of location information and date / time information to the terminal device 3. The information provision unit 213 transmits these location information and date / time information pairs to the terminal device 3 as a vehicle list. This enables the terminal device 3 to map the vehicle's travel location onto a map. The location information may include information indicating the direction of travel. In addition, the vehicle list transmitted by the information provision unit 213 to the terminal device 3 includes location information and date / time information pairs of vehicles from which the analysis unit 212 has determined to have video data containing predetermined features.

[0068] Furthermore, if the analysis unit 212 determines that the video data (image) received from the in-vehicle device 1 contains predetermined features, the information provision unit 213 will display an icon indicating an anomaly on the map on the terminal device 3. The information provision unit 213 may also notify the terminal device 3 of the ID of the video data that was determined to contain predetermined features, the location information of the vehicle that captured the video data, etc.

[0069] (2) In-vehicle video relay When terminal device 3 selects a vehicle for which it requests in-vehicle video, information provision unit 213 receives a video playback request from terminal device 3. The video playback request includes information identifying the target vehicle and the target date and time. Based on this information, information provision unit 213 forwards the video playback request to the target vehicle. Information provision unit 213 also forwards the video file (or video stream) of the in-vehicle video transmitted from the target vehicle to terminal device 3.

[0070] (3) Transmission of video containing abnormalities If the terminal device 3 selects a display (e.g., an icon) indicating the driving position of the vehicle that captured the video data, which it has determined contains predetermined features, the information provision unit 213 transfers the video data acquired by the acquisition unit 211 to the terminal device 3.

[0071] The memory unit 22 is a means for storing information and is composed of storage media such as RAM, magnetic disks, and flash memory. The memory unit 22 stores programs executed by the control unit 21, data used by those programs, and so on. Furthermore, the storage unit 22 stores a table (vehicle data table) that stores data received from the in-vehicle device 1.

[0072] The communication unit 23 is a communication interface for connecting the server device 2 to a network. The communication unit 23 is configured to communicate with the network via, for example, Ethernet (registered trademark), wireless LAN, or cellular communication network.

[0073] Next, we will describe terminal device 3. Terminal device 3 can be configured as a computer having a processor (CPU, GPU, etc.), main memory (RAM, ROM, etc.), and auxiliary memory (EPROM, hard disk drive, removable media, etc.), similar to in-vehicle device 1 and server device 2.

[0074] The terminal device 3 is comprised of a control unit 31, a storage unit 32, a communication unit 33, and an input / output unit 34.

[0075] The control unit 31 is a processing unit that realizes various functions of the terminal device 3 by executing a predetermined program. The control unit 31 can be implemented by a hardware processor such as a CPU. The control unit 31 may also be configured to include RAM, ROM, cache memory, etc.

[0076] In this embodiment, the control unit 31 of the terminal device 3 is configured to have two software modules: a display control unit 311 and a video acquisition unit 312. Each software module may be implemented by the control unit 11 (CPU, etc.) executing a program stored in the storage unit 32. The information processing performed by the software modules is synonymous with the information processing performed by the control unit 31 (CPU, etc.).

[0077] The display control unit 311 generates a user interface (GUI) for acquiring and playing back in-vehicle video and outputs it via the input / output unit 34, which will be described later. In this embodiment, the display control unit 311 generates and outputs a user interface including a road map image based on stored or previously acquired road map data. The user interface includes a user interface component for specifying the target period.

[0078] Furthermore, the display control unit 311 draws an icon indicating the driving position of vehicle 10 based on the vehicle list notified by the information provision unit 213 of the server device 2. Also, if it is determined that there is a vehicle 10 that has captured video data containing predetermined characteristics based on the information in the vehicle list, A predetermined display indicating the vehicle's position at the time the video data (image) containing specific characteristics was captured is output on the map. The predetermined display may be a different type of display than the icon that normally indicates the vehicle's position.

[0079] The video acquisition unit 312 sends a playback request for in-vehicle video to the corresponding vehicle 10 when the operator selects one of the icons indicating the normal driving position of the vehicle 10, rather than a predetermined display corresponding to video data containing predetermined features. The video acquisition unit 312 also plays the video file or video stream transmitted from the vehicle 10. In the following description, the playback request will be assumed to be sent to the vehicle 10 via the server device 2. The video file or video stream will also be assumed to be received via the server device 2.

[0080] Furthermore, when a predetermined display corresponding to video data containing predetermined features is selected, the video acquisition unit 312 acquires the video data corresponding to the selected predetermined display from the server device 2 and plays the video data.

[0081] The memory unit 32 is a means for storing information and is composed of storage media such as RAM, magnetic disks, and flash memory. The memory unit 32 stores programs executed by the control unit 31, data used by those programs, and so on.

[0082] The communication unit 33 is a wireless communication interface for connecting the terminal device 3 to an external network. The communication unit 33 is configured to communicate with the server device 2 via a cellular communication network such as a wireless LAN, 3G, 4G, or 5G.

[0083] The input / output unit 34 is a unit that receives input from the operator of the device and presents information to the operator. Specifically, the input / output unit 34 consists of a touch panel and its control means, and a liquid crystal display and its control means. In this embodiment, the touch panel and liquid crystal display consist of a single touch panel display.

[0084] Note that the configuration shown in Figure 2 is just one example, and all or part of the illustrated functions may be performed using specially designed circuits. Furthermore, program storage and execution may be performed using combinations of main memory and auxiliary memory other than those shown.

[0085] [System Processing] Next, we will describe the specific details of the processing performed by a system including a server device 2 according to one embodiment of this disclosure. Figure 3 is a sequence diagram of the process of analyzing video data performed by a system including a server device 2 according to the first embodiment. In Figure 3, we will describe the process in which the system including the server device 2 analyzes video data, determines whether the video data contains abnormalities, and outputs an abnormality indicator if the text obtained from analyzing the video data contains words indicating an abnormality.

[0086] The in-vehicle device 1 periodically starts an operation that begins from step S11.

[0087] First, in step S11, the notification unit 112 of the in-vehicle device 1 acquires vehicle data from the vehicle 10, and the imaging unit 111 of the in-vehicle device 1 acquires video data from the camera 15. Here, vehicle data is information indicating the position of the vehicle 10 when the camera 15 captured the video data. The vehicle data corresponds to each video data.

[0088] Next, in step S12, the abnormality detection unit 114 of the in-vehicle device 1 determines whether or not it has detected an abnormal event based on detection data from sensors mounted on the vehicle 10, or images from an in-vehicle camera, etc. The system determines whether an abnormal event is captured in the video data acquired by the camera unit 111 through image analysis. For example, the system also determines whether a sensor mounted on the vehicle 10 has detected an obstacle on the road.

[0089] In this step, if the anomaly detection unit 114 detects an abnormal event, a positive determination is made.

[0090] If the result in this step is positive, the process proceeds to step S13.

[0091] If the result in this step is negative, the process proceeds to step S15.

[0092] If the process transitions to step S13, the supply unit 113 of the in-vehicle device 1 extracts video data captured when the abnormality detection unit 114 detected an abnormal event. The supply unit 113 extracts video data of a predetermined length that includes the moment when the abnormality detection unit 114 detected the abnormal event.

[0093] Next, in step S14, the provision unit 113 of the in-vehicle device 1 transmits vehicle data and video data to the acquisition unit 211 of the server device 2. The vehicle data is not limited to the vehicle data corresponding to the video data extracted in step S13, but all vehicle data acquired by the in-vehicle device 1 is transmitted.

[0094] Next, in step S16, the information provision unit 213 of the server device 2 updates the vehicle data table stored in the storage unit 22 based on the vehicle data acquired by the acquisition unit 211 of the server device 2.

[0095] If the process proceeds to step S15, that is, if no abnormal event is detected on the vehicle side, the supply unit 113 of the in-vehicle device 1 transmits vehicle data to the acquisition unit 211 of the server device 2. All vehicle data acquired by the in-vehicle device 1 is transmitted. After step S15, the process proceeds to step S16, similar to the process after step S14.

[0096] Furthermore, in step S21, the terminal device 3 determines whether or not to redraw the map showing the driving positions of the multiple vehicles 10. This step is affirmative if the terminal device 3 determines that it should redraw the map showing the driving positions of the multiple vehicles 10. For example, the terminal device 3 may redraw the map when the user refreshes the screen on the terminal device 3. The process in step S21 is started periodically.

[0097] If the result in this step is positive, the process proceeds to step S23.

[0098] If the result in this step is negative, the process will proceed back to step S21.

[0099] If the process proceeds to step S22, the display control unit 311 of the terminal device 3 generates a vehicle list generation request and sends it to the server device 2. The vehicle list generation request includes information representing a predetermined geographical area specified by the operator and information representing the period for which the vehicle list will be generated.

[0100] Next, in step S23, the information provision unit 213 of the server device 2 transmits a vehicle list to the terminal device 3 on a map. Specifically, the information provision unit 213 of the server device 2 extracts vehicles that have traveled within a specified geographical area during the period for which the specified vehicle list is generated, and transmits a pair of location information and date / time information for each vehicle to the display control unit 311 of the terminal device 3.

[0101] Next, in step S24, the analysis unit 212 of the server device 2 processes the data acquired by the acquisition unit 211. Image data is converted into text using machine learning. For example, the analysis unit 212 may use a generation AI to convert the video data into text.

[0102] Next, in step S25, the analysis unit 212 of the server device 2 determines whether the text generated in step S24 contains words that indicate an anomaly. For example, the analysis unit 212 performs text analysis on the text generated and determines whether it contains words that indicate a specific anomaly, such as "traffic accident".

[0103] In this step, if the analysis unit 212 determines that the converted text contains a word indicating an anomaly, it will make a positive determination.

[0104] If the result in this step is positive, the process proceeds to step S27.

[0105] If the result in this step is negative, the process proceeds to step S29.

[0106] If the process proceeds to step S26, the information provision unit 213 of the server device 2 transmits information for outputting an abnormality display to the display control unit 311 of the terminal device 3. The information for outputting an abnormality display is, for example, information for outputting a predetermined icon at the location where video data has been captured in which it has been determined that the text contains a word indicating an abnormality.

[0107] Next, in step S27, the display control unit 311 of the terminal device 3 outputs an indication of the abnormality at the driving position of the vehicle 10 that acquired the video data containing the abnormality. Here, the video data containing the abnormality refers to video data in which it has been determined that the transcribed text contains a word indicating the abnormality. The indication of the abnormality refers to, for example, an icon that is different in appearance from the normal icons that indicate the driving positions of multiple vehicles 10 displayed on this system. Alternatively, the indication of the abnormality may be a pop-up window of the screen that plays the video containing the abnormality.

[0108] If the process proceeds to step S28, the information provision unit 213 of the server device 2 sends a notification to the display control unit 311 of the terminal device 3 indicating that there are no abnormalities. Here, the notification indicating that there are no abnormalities means that the text generated from the acquired video data does not contain any words indicating an abnormality. If a notification indicating that there are no abnormalities is sent, the display control unit 311 of the terminal device 3 does not display any indication of an abnormality on the map.

[0109] After step S27 or step S28, in step S29, the display control unit 311 of the terminal device 3 displays the locations of all vehicles 10 corresponding to all acquired vehicle data on the map. The display control unit 311 displays a normal icon or the like at all locations of the vehicles 10 indicated by the acquired vehicle data.

[0110] Next, the process by which the system including server device 2 plays back video data will be described. Figure 4 is a sequence diagram of the process by which the system including server device 2 plays back video data according to the first embodiment.

[0111] Terminal device 3 periodically starts the process of step S41.

[0112] In step S41, the video acquisition unit 312 of the terminal device 3 determines whether or not the terminal device 3 has received an operation to play a video. For example, the video acquisition unit 312 may determine that the terminal device 3 has received an operation to play a video if any icon is selected on the map displayed by the terminal device 3. Any icon may be a normal icon indicating the vehicle's position, or an indicator of an anomaly that shows the location where video data containing an anomaly was acquired (for example). (It can be an icon different from the usual icon.)

[0113] In this step, the video acquisition unit 312 determines that the terminal device 3 has received an operation to play a video, resulting in a positive determination.

[0114] If the result in this step is positive, the process proceeds to step S42.

[0115] If the result in this step is negative, the process will return to step S41.

[0116] When the process transitions to step S42, the video acquisition unit 312 of the terminal device 3 sends a request for the corresponding video data to the information provision unit 213 of the server device 2. Here, the corresponding video data refers to the video data captured at the location indicated by the icon selected by the user on the map displayed by the terminal device 3. The request includes the video ID of the video data, the identifier of the vehicle 10 that captured the video, the date and time the video data was captured, or the location of the vehicle 10 when the video data was captured.

[0117] Next, in step S43, the information provision unit 213 of the server device 2 determines whether the corresponding video data exists in the server device 2. The presence of the corresponding video data in the server device 2 means that the storage unit 22 of the server device 2 stores the corresponding video data. This step results in a positive determination if the information provision unit 213 determines that the corresponding video data exists in the server device 2.

[0118] If the result in this step is positive, the process proceeds to step S46.

[0119] If the result in this step is negative, the process proceeds to step S44.

[0120] If the process proceeds to step S46, the information provision unit 213 of the server device 2 transmits the corresponding video data stored in the storage unit 22 to the video acquisition unit 312 of the terminal device 3. For example, if an abnormality indicator (for example, an icon different from the normal icon) is selected on the terminal device 3 in step S41, the information provision unit 213 provides the video data containing the abnormality, which was acquired in step S14 and stored in the storage unit 22 of the server device 2, to the video acquisition unit 312.

[0121] If the process proceeds to step S44, the information provision unit 213 of the server device 2 sends a request for the corresponding video data to the provision unit 113 of the in-vehicle device 1. This request is the same as the request in step S42.

[0122] Next, in step S45, the providing unit 113 of the in-vehicle device 1 identifies the corresponding video data. The in-vehicle device 1 of the vehicle 10 corresponding to the vehicle identifier included in the request sent in step S44 identifies the corresponding video data based on the location where the video data included in the request was captured, etc.

[0123] Next, in step S47, the supply unit 113 of the in-vehicle device 1 transmits the corresponding video data to the video acquisition unit 312 of the terminal device 3.

[0124] Next, in step S48, the display control unit 311 of the terminal device 3 plays back the corresponding video data acquired by the video acquisition unit 312. For example, the display control unit 311 may display a window on the screen for playing back the video data.

[0125] Specifically, if the corresponding video data is video data containing anomalies, the display control unit 3 11 may display a pop-up window on the map for playing video data containing anomalies. Figure 5 shows an example of a screen for displaying video containing anomalies. As shown in Figure 5, the case in which an anomaly indicator 81 is displayed on the map alongside the normal icon 80 will be described. When the anomaly indicator 81 is selected, a pop-up window 70 may be displayed near the anomaly indicator 81, and the video data containing anomalies may be played within the pop-up window 70.

[0126] As described above, the system including the server device 2 in this embodiment analyzes video data acquired from the vehicle 10 that has detected an abnormality and determines whether or not the video data contains predetermined features. If it is determined that the video data contains predetermined features, it outputs a predetermined display on the map. As a result, the system including the server device 2 can identify and notify video footage containing predetermined content from the video footage captured by the camera more efficiently than in the conventional method.

[0127] (Second Embodiment) In the first embodiment, when the vehicle's onboard device 1 detects an abnormality, the server device 2 acquires video data of a predetermined length, including the moment the onboard device 1 detects the abnormality, from the onboard device 1. However, in locations prone to accidents, there is a certain benefit to periodically monitoring whether an abnormality has occurred in the server device 2, even if the vehicle 10 has not detected an abnormality. Therefore, in the second embodiment, the server device 2 acquires and analyzes video data from a vehicle that has traveled to a designated location and determines whether or not an abnormality is contained in the video data. If the system determines that an abnormality is contained in the video data, it outputs a display indicating the abnormality. Figure 6 is a sequence diagram of the process of analyzing video data executed by the system including the server device 2 according to the second embodiment. In the explanation of Figure 6, the same parts as in Figure 3 will be omitted.

[0128] The in-vehicle device 1 periodically starts an operation that begins from step S51.

[0129] First, in step S51, the notification unit 112 of the in-vehicle device 1 acquires vehicle data from the vehicle 10, and the imaging unit 111 of the in-vehicle device 1 acquires video data from the camera 15. Here, vehicle data refers to information indicating the position of the vehicle 10 when the camera 15 captured the video data. The vehicle data corresponds to each video data.

[0130] Next, in step S52, the acquisition unit 211 of the server device 2 transmits a video data request and specified location data to the provision unit 113 of the in-vehicle device 1. The video data request is a request to transmit video data captured at a specified location. The specified location data is information indicating a pre-set specified location. Specifically, the specified location data may be location information such as latitude and longitude information indicating the specified location. Here, the specified location is, for example, a location where traffic accidents are likely to occur (such as a specific intersection). The video data request and specified location data may be transmitted to the in-vehicle device 1 in advance.

[0131] Next, in step S53, the supply unit 113 of the in-vehicle device 1 determines whether the vehicle 10 is located within a predetermined range from the designated point. In this step, if the supply unit 113 determines that the vehicle 10 is located within a predetermined range from the designated point, the determination is affirmative.

[0132] If the result in this step is positive, the process proceeds to step S54.

[0133] If the result in this step is negative, the process proceeds to step S56.

[0134] If the process proceeds to step S54, the supply unit 113 will output the video data captured by the camera 15. The data is cut to a predetermined length. The supply unit 113 extracts video data of a predetermined length, including the time when the vehicle traveled to a designated location, from the video data captured by the camera 15.

[0135] Next, in step S55, the provision unit 113 of the in-vehicle device 1 transmits vehicle data and video data to the acquisition unit 211 of the server device 2. The vehicle data is not limited to the vehicle data corresponding to the video data extracted in step S13, but all vehicle data acquired by the in-vehicle device 1 is transmitted.

[0136] Next, in step S57, the information provision unit 213 of the server device 2 updates the vehicle data table stored in the storage unit 22 based on the vehicle data acquired by the acquisition unit 211 of the server device 2.

[0137] If the process proceeds to step S56, the supply unit 113 of the in-vehicle device 1 transmits vehicle data to the acquisition unit 211 of the server device 2. All vehicle data acquired by the in-vehicle device 1 is transmitted. After step S56, the process proceeds to step S57, similar to the process after step S55.

[0138] Furthermore, in step S61, the terminal device 3 determines whether or not to redraw the map showing the driving positions of the multiple vehicles 10. This step is affirmative if the terminal device 3 determines that it should redraw the map showing the driving positions of the multiple vehicles 10. For example, the terminal device 3 may redraw the map when a user refreshes the screen on the terminal device 3. The process in step S61 is started periodically.

[0139] If the result in this step is positive, the process proceeds to step S62.

[0140] If the result in this step is negative, the process will proceed back to step S63.

[0141] If the process proceeds to step S62, the display control unit 311 of the terminal device 3 generates a vehicle list generation request and sends it to the server device 2. The vehicle list generation request includes information representing a predetermined geographical area specified by the operator and information representing the period for which the vehicle list will be generated.

[0142] Next, in step S63, the information provision unit 213 of the server device 2 transmits a vehicle list to the terminal device 3. Specifically, the information provision unit 213 of the server device 2 extracts vehicles that have traveled within a specified geographical area during the period for which the specified vehicle list is generated, and transmits a pair of location information and date / time information for those vehicles to the display control unit 311 of the terminal device 3.

[0143] Next, in step S64, the analysis unit 212 of the server device 2 converts the video data acquired by the acquisition unit 211 into text using machine learning. For example, the analysis unit 212 may use a generation AI to convert the video data into text.

[0144] Next, in step S65, the analysis unit 212 of the server device 2 determines whether the text generated in step S65 contains words that indicate an anomaly. For example, the analysis unit 212 performs text analysis on the text generated and determines whether it contains words that indicate a specific anomaly, such as "traffic accident".

[0145] In this step, if the analysis unit 212 determines that the converted text contains a word indicating an anomaly, it will make a positive determination.

[0146] If the result in this step is positive, the process proceeds to step S67.

[0147] If the result in this step is negative, the process proceeds to step S69.

[0148] If the process proceeds to step S66, the information provision unit 213 of the server device 2 transmits information for outputting an abnormality display to the display control unit 311 of the terminal device 3. The information for outputting an abnormality display is, for example, information for outputting a predetermined icon at the location where video data has been captured in which it has been determined that the text contains a word indicating an abnormality.

[0149] Next, in step S67, the display control unit 311 of the terminal device 3 outputs an indication of the abnormality at the driving position of the vehicle 10 that has acquired video data containing the abnormality. The indication of the abnormality refers to, for example, an icon that is different in form from the normal icons that indicate the driving positions of multiple vehicles 10 displayed on this system. Alternatively, the indication of the abnormality may be a pop-up window of the screen that plays the video containing the abnormality.

[0150] If the process proceeds to step S68, the information provision unit 213 of the server device 2 sends a notification to the display control unit 311 of the terminal device 3 indicating that there are no abnormalities. If a notification indicating that there are no abnormalities is sent, the display control unit 311 of the terminal device 3 does not display any indication of abnormalities on the map.

[0151] After step S67 or step S68, in step S70, the display control unit 311 of the terminal device 3 displays the locations of all vehicles 10 corresponding to all acquired vehicle data on the map. The display control unit 311 displays a normal icon or the like at all locations of the vehicles 10 indicated by the acquired vehicle data.

[0152] In step S52, the providing unit 113 of the in-vehicle device 1 determines whether the vehicle 10 is located within a predetermined range from the designated point. However, the providing unit 113 may also determine whether the vehicle 10 has previously traveled within a predetermined range from the designated point. In that case, in step S54, the providing unit 113 crops an image taken when the vehicle 10 previously traveled within a predetermined range from the designated point to a predetermined length.

[0153] As described above, the system including the server device 2 in this embodiment analyzes video data acquired from a vehicle 10 that has traveled to a designated location and determines whether or not the video data contains predetermined features. If it is determined that the video data contains predetermined features, it outputs a predetermined display on the map. As a result, the system including the server device 2 can focus on detecting and notifying the presence or absence of abnormalities at locations that are useful to monitor regularly, such as locations where traffic accidents are likely to occur.

[0154] (Other variations) The embodiments described above are merely examples, and this disclosure may be modified as appropriate without departing from its essence. For example, the processes and means described in this disclosure can be freely combined and implemented as long as no technical inconsistencies arise.

[0155] Furthermore, the abnormalities determined by the analysis unit 212 are not limited to traffic accidents, but may also include disasters that are more commonly known, such as heavy rain, floods, storms, tornadoes, earthquakes, lightning strikes, rockfalls, mudslides, pyroclastic flows, fires, and explosions.

[0156] This disclosure provides a computer program that implements the functions described in the above embodiments to a computer, and one or more processors in the computer read the program. This can also be achieved by execution. Such computer programs may be provided to a computer by a non-temporary computer-readable storage medium that can be connected to the computer's system bus, or by being provided to a computer via a network. Non-temporary computer-readable storage mediums include, for example, any type of disk such as magnetic disks (floppy disks, hard disk drives (HDDs), etc.), optical disks (CD-ROMs, DVDs, Blu-ray discs, etc.), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, flash memory, optical cards, and any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0157] 1...In-vehicle device 2. Server device 3. Terminal device 11, 21, 31... Control Unit 12,22,32...Storage section 13, 23, 33... Communications Department 14...Location information acquisition unit 15. Camera 34...Input / output section 111...Photography Department 112...Notification Department 113...Providing Department 114...Anomaly detection unit 211...Acquisition Department 212...Analysis Department 213...Information Department 311...Display Control Unit 312...Video Acquisition Unit

Claims

1. Based on the location history of multiple cameras, multiple icons indicating the time-based location of each of the multiple cameras are displayed on the map. An information processing device that, when one of the plurality of icons is selected, provides an image captured by the camera corresponding to the selected icon, The process involves acquiring images captured by each of the aforementioned multiple cameras, The process involves analyzing the aforementioned video to determine whether or not the video contains predetermined features, If it is determined that the aforementioned video contains the aforementioned predetermined features, the terminal will output a predetermined display, Having a control unit that performs the following: Information processing device.

2. The analysis of the aforementioned video is as follows: The process includes converting the aforementioned video into text using machine learning. The information processing apparatus according to claim 1.

3. The aforementioned predetermined characteristics are events related to abnormalities on the road, The control unit determines that the video contains events related to road abnormalities when it detects words related to road abnormalities from the text generated from the video. The information processing apparatus according to claim 2.

4. The aforementioned multiple cameras are cameras mounted on multiple vehicles, The control unit, The system periodically acquires video footage of a predetermined length from a vehicle passing through a predetermined point. The information processing apparatus according to claim 1 or 2.

5. The aforementioned multiple cameras are cameras mounted on multiple vehicles, The control unit, A predetermined length of video footage is acquired from a vehicle that has detected an abnormality in the road. The information processing apparatus according to claim 1 or 2.

Images