Video collection device and video collection system

Abstract

A video collection device provided with an arithmetic device that performs arithmetic processing and a storage unit that can be accessed by the arithmetic device, wherein the arithmetic device includes: an event recognition unit that recognizes an event occurring in the vicinity of a host vehicle and an event-related object involved in the event; and an event identification unit that protects, as an event video, a video portion that is included in a video captured of the surroundings of the host vehicle by an image capture device mounted on the host vehicle, and recorded by the arithmetic device, and that was recorded from an event cause occurrence time at which the event-related object appeared before the event occurred to the time when the event occurred.

Description

Video collection device and video collection system

[0001] The present invention relates to an image collection device and an image collection system for collecting images taken by a vehicle, and more particularly to a method for recording images before an event occurs.

[0002] A dashcam (hereafter abbreviated as "dashcam") is a device that records video of accidents and injuries involving the vehicle. It is installed in the vehicle and records video and audio around the vehicle. In particular, when a dashcam detects an event such as a sudden change in speed, the detected event is used as a trigger to protect the video, audio, and other data from being overwritten and deleted by new data. On the other hand, audio and video of an event are useful for the parties involved in an incident in accurately understanding the situation and clarifying the cause of the loss, so it is desirable to record the objects that caused the event in particular and keep many records of each event. Various methods for recording this event video have been considered, and the following literature is available.

[0003] Patent Document 1 (JP 2021-51628 A) describes a recording control device in which an event detection unit detects, as events, the occurrence of a save trigger event defined as a trigger for saving video data, and the occurrence of a non-save trigger event defined as an event different from the save trigger event and whose detection timing should be recorded. When a non-save trigger event is detected, the recording control unit records the detection timing. When a save trigger event is detected within a waiting period defined for the non-save trigger event after the detection of the non-save trigger event, the recording control device sets video data including the video at the detection timing of the non-save trigger event and the video at the detection timing of the save trigger event as an object to be saved.

[0004] Furthermore, Patent Document 2 (JP 2021-43846 A) describes an electronic device that includes an image acquisition unit that acquires captured image data from an imaging unit that is capable of capturing an image of a predetermined range, a storage control unit that stores the acquired image data in a storage unit, a detection unit that detects abnormal image data that is presumed to be an image of an abnormal event based on the acquired image data, and a protection processing unit that performs data protection for the detected abnormal image data so that the detected abnormal image data is not overwritten among the image data stored in the storage unit.

[0005] JP 2021-51628 A JP 2021-43846 A

[0006] The recording control device described in the aforementioned Patent Document 1 stores video between two events during video recording: a trigger event and a marking event. The trigger event is the accident itself, while the marking event is an event that occurred before the trigger event, such as an abnormal sound, the opening or closing of a door or window, a flashing light, a change in occupants, the vertical movement of the vehicle, or the behavior of surrounding vehicles. Therefore, if an abnormal sound or vertical movement of the vehicle that is unrelated to the accident is detected before the accident occurs, it may be recognized as a marking event, which may result in the protection of video unrelated to the accident and wasteful consumption of video storage space.

[0007] Furthermore, the electronic device described in Patent Document 2 appropriately protects, upon request, abnormal video data that is highly likely to represent an abnormal event among the captured video data. However, because the electronic device determines an abnormal event based solely on the video data, there is a possibility that the image may be mistakenly recognized and video of something other than an accident may be recorded.

[0008] The present invention aims to protect the minimum amount of video footage necessary from the time when the cause of an accident occurs until the accident occurs.

[0009] A representative example of the invention disclosed in the present application is as follows: That is, a video collection device includes a calculation device that executes calculation processing and a storage unit accessible by the calculation device, wherein the calculation device has an event recognition unit that recognizes events that have occurred in the vicinity of the host vehicle and event-related objects that have contributed to the events, and an event identification unit that protects, as event video, video that has been recorded around the host vehicle captured by an imaging device mounted on the host vehicle and that is recorded from an event-cause occurrence time at which the event-related object appears, going back from the time the event occurred, to the time the event occurred.

[0010] According to one aspect of the present invention, it is possible to protect the minimum amount of video footage from the time when the cause of the accident occurred until the accident occurred. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiment of the present invention.

[0011] 1 is a block diagram showing the physical configuration of a video collection system including a video collection device of an embodiment of the present invention. FIG. 2 is a block diagram showing the configuration of a vehicle sensor of this embodiment. FIG. 3 is a block diagram showing the configuration of an external sensor of this embodiment. FIG. 4 is a block diagram showing the configuration of a host vehicle information providing device of this embodiment. FIG. 5 is a block diagram showing the configuration of a vehicle video collection device of this embodiment. FIG. 6 is a block diagram showing the theoretical configuration of a vehicle video collection device of this embodiment. FIG. 7 is a flowchart of processing executed by a first event inference unit of this embodiment. FIG. 8 is a flowchart of processing executed by a second event inference unit of this embodiment. FIG. 9 is a flowchart of processing executed by an arbitration unit of this embodiment. FIG. 10 is a flowchart of processing executed by a video search unit of this embodiment.

[0012] First Embodiment FIG. 1 and FIGS. 2A to 2D are block diagrams showing the configuration of a video collection system including a video collection device according to an embodiment of the present invention.

[0013] The image collection system of this embodiment includes a vehicle sensor 1 , an external sensor 2 , a vehicle information providing device 3 , a vehicle image collection device 4 , and an external server 5 .

[0014] 2A , the vehicle sensor 1 is a group of sensors that detect the state of the host vehicle and measures information about the host vehicle, such as the host vehicle's speed, acceleration, angular velocity, and position, such as the latitude and longitude, of the host vehicle. The vehicle sensor 1 includes, for example, a vehicle speed sensor 11 that measures the host vehicle's speed, an acceleration sensor 12 that measures the host vehicle's acceleration, a gyro sensor 13 that measures the host vehicle's acceleration, an IMU (Inertial Measurement Unit) 14 that measures the host vehicle's acceleration and angular velocity and estimates the host vehicle's attitude, and a GNSS (Global Navigation Satellite System) 15 that measures the host vehicle's position using signals transmitted from artificial satellites and outputs position information, such as the latitude and longitude. The output of the vehicle sensor 1 is output to a host vehicle information providing device 3 and a vehicle image collecting device 4.

[0015] 2B , the external sensor 2 is a group of sensors that detects the environment around the host vehicle and measures the position, speed, type, and characteristics of surrounding objects, as well as lane markings and road markings around the host vehicle. The external sensor 2 includes, for example, at least one of a camera 21 that captures video of the environment around the host vehicle, a millimeter-wave radar 22 that detects objects around the host vehicle by measuring the reflection of irradiated electromagnetic waves, an ultrasonic sensor 23 that detects objects around the host vehicle by measuring the reflection of irradiated ultrasonic waves, a LiDAR 24 that detects objects around the host vehicle by measuring the reflection of irradiated laser light, and a V2X device 25 that acquires information about the environment detected by other vehicles, and preferably includes the camera 21. The output of the external sensor 2 is output to the host vehicle information providing device 3 and the vehicle video collecting device 4. The camera 21 may be, for example, a drive recorder, a monocular camera, a stereo camera, a multi-camera that captures images of the surroundings of the vehicle, an electronic mirror, or a far-infrared camera that can capture images in dark places as well as during the day.

[0016] As shown in FIG. 2C , the host vehicle information providing device 3 acquires vehicle speed, acceleration, attitude, and position information from the vehicle sensor 1. The host vehicle information providing device 3 includes a map information management unit 31, a locator 32, and a map information DB 33. The map information management unit 31 acquires, updates, and stores map information around the host vehicle based on the host vehicle's position information acquired from the GNSS 15, the V2X device 25, etc. At this time, the acquired map information is stored in the map information DB 33. The locator 32 calculates the position and orientation of the host vehicle based on the vehicle speed, acceleration, and attitude acquired from the vehicle sensor 1. The locator 32 may also correct the position of the host vehicle within the lane in which it is located using information on lane markings, road markings, etc. from the external sensor 2. The host vehicle information providing device 3 outputs map information including lane markings, road edges, road markings, and road signs on the road surface on which the host vehicle is traveling, as well as the position of the host vehicle.

[0017] 2D, the vehicle image collection device 4 includes an external environment information recognition unit 41, an event recognition unit 42, an event identification unit 43, and an external communication unit 44. Details of each unit will be described later.

[0018] The external server 5 receives the event video, event time, and target information transmitted from each vehicle. The external server 5 also has a video aggregation unit 51 and an aggregated video database 52, and collects information within a certain range and a certain time period.

[0019] The vehicle image collection device 4 is a control device having an arithmetic unit, a storage device, and a communication interface. The arithmetic unit is a processor (e.g., a microcomputer) that executes programs stored in the storage device. The arithmetic unit operates as a functional unit that provides various functions by executing predetermined programs. The storage device includes a non-volatile storage area and a volatile storage area. The non-volatile storage area includes a program area that stores programs executed by the arithmetic unit, and a data area that temporarily stores data used by the arithmetic unit when executing programs. The volatile storage area stores data used by the arithmetic unit when executing programs. The communication interface connects to other electronic control devices via a network such as CAN or Ethernet.

[0020] FIG. 3 is a block diagram showing the theoretical configuration of the vehicle image collecting device 4 of this embodiment.

[0021] The vehicle image collection device 4 includes an external environment information recognition unit 41 , an event recognition unit 42 , an event identification unit 43 , and an external communication unit 44 .

[0022] The external environment information recognition unit 41 receives the latitude, longitude, and speed of the host vehicle provided by the host vehicle information providing device, the host vehicle relative position and relative speed of targets around the host vehicle detected by the external environment sensor 2, and a target ID for uniquely identifying the target, converts the positions of the targets around the host vehicle into absolute positions expressed in latitude and longitude, and outputs target information around the host vehicle. When multiple detection results for the same target around the host vehicle are input, the external environment information recognition unit 41 may perform sensor fusion to integrate the multiple detection results into a single target. The target information around the host vehicle output from the external environment information recognition unit 41 includes, for example, a relative position, latitude, longitude, relative speed, and target ID. The relative position may be expressed as a position in a host vehicle coordinate system with the center of the host vehicle as the origin.

[0023] The event recognition unit 42 includes a first event inference unit 421, a second event inference unit 422, and an arbitration unit 423. The event recognition unit 42 receives target information output from the external environment information recognition unit 41 and frame images output from the camera 21 included in the external environment sensor 2, determines the occurrence of an event, and identifies event-related targets related to the determined event. In this embodiment, events are mainly automobile accidents, but other events such as vehicle vandalism, theft, wrong-way driving, vehicles traveling on sidewalks, bicycles traveling on motorways, and suspicious individuals are also treated as events. Automobile accidents include accidents involving two vehicles as well as accidents involving the subject vehicle and another vehicle. Note that the event recognition unit 42 includes a first event inference unit 421 and a second event inference unit 422 that infer events using different methods, but may also include a single event inference unit. Having multiple event inference units that infer events using different methods can prevent erroneous event determination.

[0024] The first event estimation unit 421 refers to each frame output from the camera 21, determines the occurrence of an event through image recognition using AI (Artificial Intelligence) with a recognition model configured by a neural network, and outputs event-related possibility target information. The event-related possibility target information is information on targets that may have been involved in the event. Furthermore, while image recognition is performed using AI, a recognition model based on machine learning or deep learning may also be used. This recognition model may be constructed by learning images of an event (e.g., an accident) labeled with targets involved in the event (e.g., an accident vehicle).

[0025] 4 is a flowchart of the process executed by the first event inferring unit 421 of this embodiment. The process shown in FIG. 4 will be described using a collision accident event as an example.

[0026] First, the first event inferring unit 421 acquires the video output from the camera 21 (S11).

[0027] Next, the first event estimation unit 421 uses the trained recognition model to determine whether any of the frame images in the acquired video corresponds to a collision accident (S12). A collision accident determined in step S12 is contact between two or more objects (e.g., two vehicles, a vehicle and a pedestrian, or a vehicle and a fixed object). If no collision is detected in the frame image (NO in S13), the process returns to step S12 and a collision accident is determined in the next frame image.

[0028] On the other hand, if a frame in which a collision accident has occurred is detected (YES in S13), the first event estimation unit 421 identifies targets involved in the collision accident from the frame, and calculates the position of each target on the frame image, treating the identified targets as a set (S14).

[0029] Then, the first event inferring unit 421 calculates the target position in the host vehicle coordinate system with the center of the host vehicle as the origin (S15).

[0030] Thereafter, the first event estimation unit 421 outputs information about the target object for which a collision has been determined as event involvement target information. The event involvement target information includes the relative position between the vehicle and the target object, the position of the target object on the frame image, and the event cause occurrence possibility time, which is the time of the first frame in which a collision has been determined (S16). This completes the processing of the first event estimation unit 421.

[0031] The second event inference unit 422 refers to the target information output by the external world information recognition unit 41, determines whether or not a target has collided with another target on a rule basis, and outputs event involvement possibility target information.

[0032] 5 is a flowchart of the process executed by the second event inferring unit 422 of this embodiment. The process shown in FIG. 5 will be described using a collision accident event as an example.

[0033] First, the second event estimation unit 422 acquires target information output from the external environment information recognition unit 41 (S21), and for each target represented in the host vehicle coordinate system, searches for a target closest to the target and creates a target pair (S22). For example, the center positions of two targets may be compared to search for targets that are close to the target. The center position of the target may be calculated geometrically by calculating the center of gravity from a polygon such as a rectangle that contains the target, or may be calculated using the size determined depending on the type of target and the outline of the target. Alternatively, targets that are close to the target may be searched for based on the distance between the outlines of polygons such as a rectangle that contain each target.

[0034] Next, the second event inferring unit 422 determines whether the outlines of the targets of each pair overlap (S23). For example, the outline of the target may be a polygon such as a rectangle that contains the target.

[0035] If the polygons representing the targets overlap (YES in S24), the second event estimation unit 422 determines that there is a possibility of a collision and calculates the speeds of the two targets (S25).The speeds of the overlapping targets are then compared during each processing cycle, and if the ratio between the absolute values ​​of the speeds of the two targets at the same time and the absolute value of the speed of one of the vehicles falls within a predetermined range close to 1 (for example, within 5%) (YES in S26), a collision is determined.In step S26, it is determined that the speeds of the two targets are close to each other, but a collision may also be determined based on a sudden change in the target speed that exceeds the range of maximum braking and maximum acceleration.

[0036] On the other hand, if the polygons representing the targets do not overlap (NO in S24), or if the polygons representing the targets do not overlap (NO in S26), the process returns to step S23, and overlap is determined for the next pair of targets.

[0037] If a collision is determined, the second event estimation unit 422 records the time when the collision was determined and outputs the objects determined to have collided as potential event-related objects. The potential event-related objects may include the relative position with respect to the vehicle, the position of the object (e.g., latitude and longitude), and the time and speed when the collision was determined (S27). The above is the processing of the second event estimation unit 422.

[0038] The arbitration unit 423 receives the output results of the first event inference unit 421 and the second event inference unit 422, determines whether an event (e.g., a collision accident) has occurred, and if it is determined that an event has occurred, integrates and outputs the event-related target information from the first event inference unit 421 and the second event inference unit 422.

[0039] FIG. 6 is a flowchart of the process executed by the arbitration unit 423 of this embodiment.

[0040] First, the arbitration unit 423 time-evolves the coordinates of the event-related object determined based on the rule base, relative to the center of the host vehicle, received from the second event estimation unit 422, to the time of the first frame in which the first event estimation unit 421 determined that a collision had occurred. The time evolution is, for example, a process of reflecting the amount of movement of the host vehicle during the time the host vehicle moves from when the sensor detects the object to when the object is processed within the host vehicle, on the position of the detected object. In step S31, if there is a time difference between the time of the collision determination output by the second event estimation unit 422 (not the time when the sensor detects the object) and the time of the first frame in which the collision was determined by the first event estimation unit 421, the arbitration unit 423 multiplies the speed of the host vehicle during that time difference by the time difference to calculate the amount of movement of the host vehicle. Then, the arbitration unit 423 adds the relative position of the vehicle output by the second event estimation unit 422 to the calculated amount of movement of the vehicle, thereby making it possible to compare the target information output by the second event estimation unit 422 with the target information output by the first event estimation unit 421 on the same time axis (S31).

[0041] Next, the arbitration unit 423 compares, for the same target, the position calculated by the first event inference unit 421 with the position calculated by the second event inference unit 422. For example, it is advisable to calculate the distance between the centers of the two targets and compare the positions (S32).

[0042] If the position difference between the two targets is equal to or less than a predetermined threshold (YES in S33), the arbitration unit 423 integrates the target information from the first event estimation unit 421 and the time-evolved target information from the second event estimation unit 422 (S34).

[0043] Finally, the arbitration unit 423 outputs, for each event-related object, the relative position between the host vehicle and the object, the absolute position (latitude and longitude) of the object, the position of the object in the frame, the time of the first frame in which the object was determined to have collided with the object, and the speed of the object (S35).

[0044] The event identification unit 43 includes a video management unit 431, a video search unit 432, a video database 433, and an output information generation unit 434. The event identification unit 43 continuously records video of the area around the vehicle, manages the video, and protects event video between specific times. Then, the event identification unit 43 outputs the event video, the event time, and target information to the external communication unit 44.

[0045] The video management unit 431 accumulates video of the area around the vehicle input from the camera 21 in the video database 433 and continuously records the video. Furthermore, when the video search unit 432 requests video from a specific time period, the video management unit 431 outputs frames from the specific time period to the video search unit 432. Furthermore, when the video search unit 432 requests protection of video from a specific time period, the video management unit 431 protects the video in the video database 433 from deletion. Furthermore, when the video search unit 432 requests protection of the video from a specific time period, the video management unit 431 releases protection for the video between the specified frames. For example, the external communication unit 44 requests removal of protection for video from a specific time period that has been transmitted to the external server 5. Furthermore, when the storage area of ​​the video database 433 becomes full with video data and no more video data can be written, the video database 433 may preferably have a function for deleting older video so that newer video can be written.

[0046] The video search unit 432 acquires from the video database 433 video of a specific time period from the specific time to a time a predetermined time prior, based on the event-related object information output from the arbitration unit 423 of the event recognition unit 42 .

[0047] FIG. 7 is a flowchart of the process executed by the video search unit 432 of this embodiment.

[0048] First, the video search unit 432 acquires event-related target information from the arbitration unit 423 (S41).

[0049] Next, the video search unit 432 acquires, from the video database 433 via the video management unit 431, video from the first frame in which a collision was determined in the event-related object information up to a predetermined time prior (S42). The predetermined time may be set to, for example, 20 minutes, or may be changed depending on the size of the storage capacity of the video database 433.

[0050] Thereafter, the video search unit 432 searches each of the frames of the acquired video for a frame in which the event-related object output from the arbitration unit 423 first appears, and identifies the time of that frame (S43).

[0051] The video search unit 432 then designates the video from the time when the event-related object first appeared to the time of the first frame in which a collision was determined as video to be protected, and issues an instruction to the video management unit 431 to request that the video to be protected be protected so that it is not deleted (S44). For example, the video to be protected may be flagged as non-erasable, or the video to be protected may be stored in a protected area in which data cannot be erased.

[0052] Then, the video search unit 432 outputs information related to the video to be protected to the output information generation unit 434 (S45), and this information is set as upload information. The information related to the video to be protected includes, for each target in the event-related target information output from the arbitration unit 423, the relative position with respect to the vehicle, the target position (latitude and longitude), the target speed, the time of the first frame in which a collision was determined, and the time when the event-related target first appeared. This completes the processing of the video search unit 432.

[0053] The external communication unit 44 outputs the upload information output from the output information generation unit 434 to the external server 5. If the upload is successful, the external communication unit 44 requests the video search unit 432 to release protection of the video to be protected. The data output to the external server 5 and the video protection release request complete the processing executed by the vehicle video collection device.

[0054] The external server 5 has a video aggregation unit 51 and an aggregated video database 52. The video aggregation unit 51 aggregates uploaded information that can be determined to be identical based on location information (latitude and longitude) included in the uploaded video, and records the information in the aggregated video database 52. The external server 5 may distribute the aggregated information not only to owners of vehicles involved in the event, but also to organizations that utilize accident information, such as the police and insurance companies.

[0055] The above is a description of the embodiment of the present invention.

[0056] As described above, according to the embodiments of the present invention, it is possible to identify accident-related targets from the video footage at the time of the accident occurrence, and to protect the minimum amount of video footage from the time the accident cause occurred until the accident itself, or to protect the video footage within an appropriate range that includes the minimum amount of video footage.

[0057] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0058] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0059] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0060] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. An image acquisition device, comprising: an arithmetic unit that executes arithmetic processing; and a storage unit accessible by the arithmetic unit, wherein the arithmetic unit includes: an event recognition unit that recognizes an event occurring around the host vehicle and an event-related object mark involved in the event; and an event specifying unit that protects, as event video, video recorded from the time when the event factor occurrence time at which the event-related object mark appears, retroactively from the time when the event occurred, to the time when the event occurred, among video recorded by an imaging device mounted on the host vehicle of the periphery of the host vehicle. The image acquisition device is characterized by having the above components.

2. The image acquisition device according to claim 1, wherein the event recognition unit includes: a first event estimation unit that estimates the occurrence of the event using a recognition model learned from an image in which the event is occurring; a second event estimation unit that estimates the occurrence of the event based on object mark information including at least the position and speed of an object mark acquired by an external sensor mounted on the host vehicle; and a mediation unit that outputs event-related possible object mark information integrated based on a first estimation result by the first event estimation unit and a second estimation result by the second event estimation unit, which is related to the event. The image acquisition device is characterized by having the above components.

3. The image acquisition device according to claim 1, wherein the event specifying unit includes: a video search unit that searches for the event factor occurrence time from the recorded video; and a video management unit that records and manages the video, protects the event video based on a protection request for the event video from the video search unit, and deletes the event video transmitted to a server provided outside the host vehicle. The image acquisition device is characterized by having the above components.

4. The image acquisition device according to claim 1, further comprising an external communication unit that transmits the event video protected by the event specifying unit to a server provided outside the host vehicle. The image acquisition device is characterized by having the above components.

5. A video collection system comprising a video collection device and a video collection server, wherein the video collection device has a first arithmetic unit that executes arithmetic processing and a first storage unit accessible by the first arithmetic unit, the first arithmetic unit has an event recognition unit that recognizes an event occurring around the host vehicle and an event-related object involved in the event, and the first arithmetic unit has an event specifying unit that protects, as event video, video recorded from the time when the event occurred to the time when the event occurred, among the video recorded by an imaging device mounted on the host vehicle, of the periphery of the host vehicle, from the time when the event-related object appeared, which is traced back from the time when the event occurred, to the time when the event occurred, and an external communication unit that transmits the event video protected by the event specifying unit to a server provided outside the host vehicle. A video collection system characterized by having the above.

6. The video collection system according to claim 5, wherein the server has a second arithmetic unit that executes arithmetic processing and a second storage unit accessible by the second arithmetic unit, and the second arithmetic unit has a video aggregation unit that aggregates the event videos transmitted from a plurality of vehicles including the host vehicle for each event at the same time and at the same location. A video collection system characterized by having the above.

Images