Information processing device

Abstract

Obtain appropriate analysis results. [Solution] A video is taken from the moving object during a predetermined period including the time of the accident that occurred to the moving object, driving data of the moving object during the predetermined period is obtained, the attributes of the other party object in the accident are determined based on an analysis video based on the video, the accident is analyzed based on the analysis video, the driving data, and the attributes of the other party object, and the percentage of fault in the accident is obtained.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus.

Background Art

[0002] Large Language Models (LLMs) that output responses described in natural language in response to inputs of instructions (prompts) described in natural language have been developed and used in various fields (for example, Patent Document 1). Also, multi-modal AI (Artificial Intelligence) that processes information based on a plurality of types of input data and prompts, including different types such as video, audio, and text data, and outputs analysis results described in natural language has been developed (for example, Non-Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

[0005] In multimodal AI, if the frame rate of the input video is too low, it may not be possible to obtain appropriate analysis results. On the other hand, if the frame rate of the video is too high, it may result in unnecessary computational load and long computation times. Furthermore, depending on the field in which the multimodal AI is performing the analysis, the AI ​​may not have been trained sufficiently, resulting in inappropriate analysis results.

[0006] One example of a problem that this invention aims to solve is obtaining appropriate analytical results. [Means for solving the problem]

[0007] To solve the above problems, the invention described in claim 1 is an information processing device comprising: a video information acquisition processing unit that acquires video footage taken from a moving object during a predetermined period including the time of an accident occurring to the moving object; a driving data acquisition processing unit that acquires driving data of the moving object during the predetermined period; an opposing party determination processing unit that determines the attributes of the opposing party object in the accident based on an analysis video based on the video; and an accident analysis processing unit that performs an analysis of the accident and acquires the percentage of fault in the accident based on the analysis video, the driving data, and the attributes of the opposing party object.

[0008] The invention described in claim 7 is an information processing method performed by a computer, comprising: a video information acquisition processing step of acquiring video footage taken from a moving object during a predetermined period including the time of an accident that occurred to the moving object; a driving data acquisition processing step of acquiring driving data of the moving object during the predetermined period; an opposing party determination processing step of determining the attributes of the opposing party object in the accident based on an analysis video based on the video footage; and an accident analysis processing step of performing an analysis of the accident and acquiring the percentage of fault in the accident based on the analysis video, the driving data, and the attributes of the opposing party object.

[0009] The invention described in claim 8 is an information processing program that causes a computer to execute the information processing method described in claim 7.

[0010] The invention described in claim 9 is a computer-readable storage medium storing the information processing program described in claim 8. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an accident analysis device 100 according to one embodiment of the present invention. [Figure 2] This is a diagram showing an example of the control unit 110. [Figure 3] This diagram illustrates an example of a video frame used for analysis. [Figure 4] This diagram illustrates an example of a detection result output by the sign / signal detection processing unit 113. [Figure 5] This figure shows an example of a processing operation performed in the control unit 110. [Figure 6] This figure shows an example of a processing operation performed in the control unit 110. [Figure 7] This figure shows an example of a processing operation performed in the control unit 110. [Figure 8] This figure shows an example of the priority road determination processing unit 119. [Figure 9] This diagram illustrates an example of an aerial image that includes the road R1 on which the moving object M traveled and the road R2 on which the other party's object involved in the accident with the moving object M traveled. [Figure 10] This figure shows an example of a processing operation performed in the control unit 110. [Figure 11] This figure shows an example of the processing operations performed in the priority road determination processing unit 119. [Figure 12] This figure shows an example of the sign / signal detection processing unit 113. [Figure 13] This figure shows an example of processing operations performed in the sign / signal detection processing unit 113. [Modes for carrying out the invention]

[0012] An information processing apparatus according to one embodiment of the present invention includes a video information acquisition processing unit that acquires a video captured from a moving object during a predetermined period including the occurrence time of an accident that occurred to the moving object, a driving data acquisition processing unit that acquires the driving data of the moving object during the predetermined period, a counterpart determination processing unit that determines the attribute of the counterpart object of the accident based on an analysis video based on the video, and an accident analysis processing unit that analyzes the accident based on the analysis video, the driving data, and the attribute of the counterpart object and acquires the negligence ratio of the accident. The counterpart determination processing unit may determine the attribute of the counterpart object based on the analysis video using an image analysis model. The attribute of the counterpart object may include whether the counterpart object corresponds to a child or an elderly person. Therefore, in this embodiment, it is possible to obtain a more appropriate negligence ratio of the accident considering the correction factors.

[0013] The information processing apparatus further includes an accident type information acquisition processing unit that acquires accident type information which is information regarding the accident type, and the accident analysis processing unit may acquire the negligence ratio of the accident based on the analysis video, the driving data, the accident type information, the states of traffic signs and traffic signals detected in each frame of the analysis video, and the attribute of the counterpart object using a natural language processing model capable of image analysis. The accident type information is a summary text of the accident type and may be generated based on an image of the accident type using an image caption generation model. By doing so, even if the accident analysis is performed using an accident analysis model that has not been specifically trained for the accident type as the image analysis model, it is possible to identify the accident type of the accident that occurred to the moving object M, and it is possible to obtain a more appropriate accident analysis result, particularly, a more appropriate negligence ratio of the accident.

[0014] The accident type information includes a correction factor for the fault ratio, and the correction factor includes a correction factor when the other object is a child or an elderly person. The attribute of the other object may include whether the other object corresponds to a child or an elderly person. By doing so, even if the accident analysis is performed using an accident analysis model that has not been specifically trained for the accident type as the image analysis model, it is possible to obtain a more appropriate fault ratio of the accident considering the correction factor.

[0015] An information processing method according to an embodiment of the present invention is an information processing method executed by a computer, including a video information acquisition processing step of acquiring a video captured from a moving body during a predetermined period including the occurrence time of an accident occurring in the moving body, a travel data acquisition processing step of acquiring travel data of the moving body during the predetermined period, an opponent determination processing step of determining the attribute of the opponent object of the accident based on an analysis video based on the video, and an accident analysis processing step of analyzing the accident and obtaining the fault ratio of the accident based on the analysis video, the travel data, and the attribute of the opponent object. Therefore, in this embodiment, it is possible to obtain a more appropriate fault ratio of the accident considering the correction factor.

[0016] An information processing program according to an embodiment of the present invention causes a computer to execute the above information processing method. Therefore, in this embodiment, it is possible to obtain a more appropriate fault ratio of the accident considering the correction factor.

[0017] A computer-readable storage medium according to an embodiment of the present invention stores the above information processing program. Therefore, in this embodiment, in addition to being incorporated into a device, the above information processing program can also be distributed alone, and it is possible to easily perform version updates and the like.

Example

[0018] <Accident Analysis Device 100> Figure 1 shows an accident analysis device 100 according to one embodiment of the present invention. The accident analysis device 100 analyzes an accident that occurred in a moving object M (for example, a vehicle) based on video footage taken from the moving object M.

[0019] The accident analysis device 100 includes a control unit 110, a storage unit 120, and a communication unit 130. The control unit 110 is an information processing device (computer) for processing information, and for example, includes a CPU (Central Processing Unit). The accident analysis device 100 is typically a server device. Alternatively, the accident analysis device 100 may be a PC (Personal Computer). The storage unit 120 is a storage device for storing information, and for example, includes a hard disk drive, a solid-state drive, or memory. The storage unit 120 stores various programs executed in the accident analysis device 100, such as an operating system and software. These various programs include programs for executing processing related to accident analysis in the accident analysis device 100. Part of the storage unit 120 may be a storage device such as a memory card that can be removed from the accident analysis device 100. The communication unit 130 is a communication device for sending and receiving information with other devices.

[0020] Figure 2 shows an example of the control unit 110. The control unit 110 includes a video information acquisition processing unit 111, a driving data acquisition processing unit 112, a sign / signal detection processing unit 113, and an accident analysis processing unit 114.

[0021] The video information acquisition processing unit 111 acquires video footage (accident video) taken from the mobile object M during a first period including the time of the accident that occurred to the mobile object M. The accident video is, for example, video footage taken by a drive recorder installed on the mobile object M. For example, a drive recorder is equipped with a camera that takes video footage and an impact sensor, and when the impact sensor detects an impact of a predetermined value or higher, it is considered that an accident has occurred, and the video footage taken by the camera during the first period including the timing of the accident is stored as the accident video. The first period is, for example, the period from a time one hour before the time of the accident that occurred to the mobile object M to a time two hours after the time of the accident. The first and second times are set as appropriate, and the first and second times may be the same time or different times.

[0022] The video information acquisition processing unit 111 may acquire accident videos from the storage unit 120 where the accident videos are stored. For example, if the accident analysis device 100 is a PC, the storage unit 120 may be a memory card on which the drive recorder stores the accident videos. Alternatively, the video information acquisition processing unit 111 may acquire accident videos from an external device using communication via the communication unit 130. For example, if the accident analysis device 100 is a server device, the video information acquisition processing unit 111 may acquire accident videos from the drive recorder via communication over the Internet using the communication unit 130.

[0023] The driving data acquisition processing unit 112 acquires driving data of the mobile body M during the first period described above. The driving data of the mobile body M includes, for example, the speed, acceleration, and driving history of the mobile body M, which are acquired based on the outputs of various sensors provided on the mobile body M. The driving history of the mobile body M includes the time-based position information of the mobile body M and information about the links (driving links) that the mobile body M traveled on (for example, the road type of the driving link). The driving data of the mobile body M may also include information about the operation of the mobile body M (for example, information about the operation of the accelerator, brakes, and turn signals of the mobile body M).

[0024] The driving data acquisition processing unit 112 may acquire driving data from the storage unit 120 where the driving data is stored. For example, if the accident analysis device 100 is a PC, the storage unit 120 may be a memory card on which the drive recorder stores the driving data. Alternatively, the driving data acquisition processing unit 112 may acquire driving data from an external device using communication via the communication unit 130. For example, if the accident analysis device 100 is a server device, the video information acquisition processing unit 111 may acquire driving data from the drive recorder via communication over the Internet using the communication unit 130.

[0025] The sign / signal detection processing unit 113 detects the status of traffic signs and signals in each frame of the analysis video based on the accident video acquired by the video information acquisition processing unit 111. The analysis video may be the accident video itself acquired by the video information acquisition processing unit 111, or, as detailed below, it may be a video generated by the analysis video generation processing unit 115 based on the accident video acquired by the video information acquisition processing unit 111.

[0026] Traffic signs include regulatory signs (for example, signs installed on the side of the road or above the road that notify of prohibitions, regulations, and restrictions, such as no entry, speed limits, and one-way streets), regulatory markings (for example, speed limits and lane divisions marked on the road) to indicate points where traffic caution is required (for example, signs installed on the side of the road or above the road that indicate the location of stop lines and pedestrian crossings), directional markings (for example, stop lines and pedestrian crossings marked on the road), and supplementary signs to supplement the main sign (for example, signs installed below the main sign that notify the date and time when the traffic regulations indicated by the main sign are in effect). Traffic signs may also include warning signs to draw attention (for example, signs installed on the side of the road or above the road that notify the presence of schools, kindergartens, or daycare centers nearby). For example, the sign / signal detection processing unit 113 detects the location of a sign and identifies the type of the detected sign, as detailed below.

[0027] The status of a traffic light includes the illumination status of the traffic light's lamps. The illumination status of the traffic light's lamps includes whether the red, yellow, or blue lamps are illuminated. The illumination status of the traffic light's lamps also includes the status of the arrow lights. For example, the sign / signal detection processing unit 113 detects the location of a traffic light and identifies the illumination status of the detected traffic light's lamps, as detailed below.

[0028] Figure 3 illustrates an example of a frame from a video for analysis. In the example shown in Figure 3, the sign / signal detection processing unit 113 detects the positions of traffic sign 1, traffic sign 2, traffic sign 3, traffic sign 4, traffic sign 5, traffic sign 6, traffic sign 7, traffic sign 8, traffic sign 9, and traffic light 1, and identifies that the type of traffic sign 1 is "Stop", the type of traffic sign 2 is "Stop line", the type of traffic sign 3 is "School, kindergarten, nursery school etc.", the type of traffic sign 4 is "Crosswalk", the type of traffic sign 5 is "Crosswalk ahead", the type of traffic sign 6 is "Crosswalk ahead", the type of traffic sign 7 is "Stop line", the type of traffic sign 8 is "Crosswalk", the type of traffic sign 9 is "Stop line", and the lighting state of traffic light 1 is "Green". Furthermore, the sign / signal detection processing unit 113 identifies the location of traffic signs and traffic lights by a rectangular area (bounding box) surrounding the traffic signs and traffic lights, for example, as shown in Figure 3. In the example shown in Figure 3, the bounding box is indicated by a thick dashed line.

[0029] Figure 4 illustrates an example of the detection results output by the sign / signal detection processing unit 113. In the example shown in Figure 4, it is an example of the detection results for the frame shown in Figure 3, and it describes the location and type of traffic signs 1, 2, 3, 4, 5, 6, 7, 8, and 9, as well as the location and illumination status of traffic light 1, detected in the frame shown in Figure 3. For example, for each frame of the analysis video, the sign / signal detection processing unit 113 obtains text data summarizing the location and type of the traffic signs detected in that frame, and the location and status of the traffic lights, as shown in Figure 4. In the example shown in Figure 4, the locations of the traffic signs and traffic lights are indicated by the coordinates of the upper left and lower right corners of the bounding boxes surrounding the traffic signs and traffic lights.

[0030] The accident analysis processing unit 114 analyzes the accident that occurred to the mobile vehicle M based on the analysis video, driving data, and the status of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113, and obtains the analysis results of the accident. The analysis results of the accident include, for example, a description of the accident situation, the cause of the accident, and the percentage of fault in the accident.

[0031] The accident analysis processing unit 114 uses, for example, a multimodal AI (also called an accident analysis model) that combines an image (video) analysis model and a natural language processing model to analyze an accident that occurred to a moving object M and obtain the analysis results of the accident. In other words, the accident analysis processing unit 114 inputs the accident analysis model with instructions (prompts) to cause the accident analysis model to analyze the accident captured in the analysis video based on the state of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113 and driving data, as well as the analysis video itself, and obtains the analysis results of the accident output from the accident analysis model. The accident analysis processing unit 114 may include an accident analysis model, or it may use an external accident analysis model.

[0032] In this embodiment, the sign and signal detection processing unit 113 detects the status of traffic signs and signals in advance, and the status of traffic signs and signals detected by the sign and signal detection processing unit 113, in addition to the video for analysis and driving data, is input to the accident analysis model. Therefore, in this embodiment, the accident analysis model performs accident analysis after the status of traffic signs and signals in each frame of the video for analysis has already been identified. In other words, in this embodiment, by combining the sign and signal detection processing unit 113, which can accurately detect the status of signs and signals, with an accident analysis model specialized for accident analysis, it is possible to obtain more appropriate accident analysis results.

[0033] Figure 5 shows an example of processing operations performed in the control unit 110. The video information acquisition processing unit 111 acquires video (accident video) taken from the mobile body M during a first period including the time of the accident that occurred to the mobile body M, and the driving data acquisition processing unit 112 acquires the driving data of the mobile body M during the first period (step S501). The sign / signal detection processing unit 113 detects the state of traffic signs and signals in each frame of the analysis video based on the accident video acquired by the video information acquisition processing unit 111 (step S502). The accident analysis processing unit 114 analyzes the accident that occurred to the mobile body M based on the analysis video, the driving data, and the state of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113, and obtains the analysis result of the accident (step S503).

[0034] <Analysis video generation processing unit 115, traffic light status identification processing unit 116> When the light source of a traffic light is an LED, the traffic light flashes according to the frequency of the AC power supply (50Hz or 60Hz). If the traffic light is filmed at a frame rate equivalent to the frequency of the AC power supply, or half of that frequency, depending on the timing, the traffic light may appear to be off even when it is actually lit. Therefore, the frame rate of many dashcams is set to, for example, 27.5fps. If every frame of a video filmed at 27.5fps is analyzed, it places a heavy computational load on the system and the calculation time becomes long.

[0035] Therefore, the control unit 110 may further include an analysis video generation processing unit 115, as shown in Figure 2. The analysis video generation processing unit 115 downsamples the frames of the accident video acquired by the video information acquisition processing unit 111 to generate an analysis video. This reduces the computational load during image analysis and shortens the computation time.

[0036] As the speed of the moving object M increases, the amount of movement of the object M per unit time increases, resulting in a greater change in the scenery captured by the camera between temporally consecutive frames in the accident video. Here, if the amount of frame downsampling from the accident video is kept constant regardless of the speed of the moving object M and a video for analysis is generated, frames containing information necessary for accident analysis may be omitted from the video for analysis, potentially reducing the accuracy of the accident analysis. Furthermore, the speed of the moving object M is generally proportional to the speed limit of the road, and the higher the speed limit of the road, the higher the speed of the moving object M traveling on that road.

[0037] Therefore, the analysis video generation processing unit 115 may generate an analysis video by downsampling frames from the accident video based on the driving data. In this case, the driving data may include at least one of the speed of the moving object M and the type of road on which the moving object M is traveling, and the analysis video generation processing unit 115 may determine the frame rate of the analysis video based on the speed of the moving object M or the road on which the moving object M is traveling. By doing so, it becomes possible to obtain appropriate analysis results while reducing the computational load.

[0038] In this case, the analysis video generation processing unit 115 may determine the frame rate of the analysis video to be larger as the speed of the moving object M increases. Alternatively, the analysis video generation processing unit 115 may determine the frame rate of the analysis video to be larger as the speed limit of the road on which the moving object M is traveling increases. In this case, the analysis video generation processing unit 115 may, for example, use a threshold of 1 or more to change the frame rate of the analysis video in steps according to the relationship between the threshold and the speed of the moving object M or the speed limit of the road on which the moving object M is traveling.

[0039] As mentioned above, when the light source of a traffic light is an LED, the traffic light flashes according to the frequency of the AC power supply (50Hz or 60Hz). Therefore, in some frames of the video for analysis, the traffic light may be recorded as being off or dimly lit, making it impossible to determine the state of the traffic light (the color of the light). In other words, there are frames in which the state of the traffic light detected by the sign / signal detection processing unit 113 is unknown (frames with unknown traffic light state).

[0040] Therefore, the control unit 110 may further include a traffic light state identification processing unit 116, as shown in Figure 2. The traffic light state identification processing unit 116 identifies the state of the traffic light in the frame with an unknown traffic light state based on the state of the traffic light in the frame before and after that frame in time. The accident analysis processing unit 114 may then analyze the accident that occurred to the moving object M based on the analysis video, the driving data, the state of traffic signs and traffic lights detected in each frame of the analysis video by the sign / signal detection processing unit 113, and the state of the traffic light identified by the traffic light state identification processing unit 116, and obtain the analysis results of the accident. For example, the traffic light state identification processing unit 116 may identify the state of the traffic light in the frame with an unknown traffic light state as the state of the traffic light in the frame with an unknown traffic light state, which is the frame in time closest to the frame with an unknown traffic light state among the frames in which the state of the traffic light detected by the sign / signal detection processing unit 113 is not unknown. By doing so, it becomes possible to identify the state of the traffic lights in every frame of the video being analyzed before conducting an accident analysis, thereby obtaining more accurate accident analysis results.

[0041] The sign / signal detection processing unit 113 obtains the reliability of the detected signal state, and the signal state identification processing unit 116 considers frames in which the reliability of the signal state detected by the sign / signal detection processing unit 113 is below a predetermined threshold as frames in which the signal state detected by the sign / signal detection processing unit 113 is unknown (unknown signal state frames), and frames in which the reliability of the signal state detected by the sign / signal detection processing unit 113 is equal to or greater than the predetermined threshold as frames in which the signal state detected by the sign / signal detection processing unit 113 is known (known signal state frames). Furthermore, the signal state identification processing unit 116 may identify the signal state in the frame that is temporally closest to the unknown signal state frame among the known signal state frames as the signal state in the unknown signal state frame.

[0042] Figure 6 shows an example of processing operations performed in the control unit 110. The video information acquisition processing unit 111 acquires video footage (accident video) taken from the mobile body M during a first period including the time of the accident that occurred to the mobile body M, and the driving data acquisition processing unit 112 acquires driving data of the mobile body M during the first period (step S601). The analysis video generation processing unit 115 generates an analysis video based on the accident video acquired by the video information acquisition processing unit 111 (step S602). The sign / signal detection processing unit 113 detects the state of traffic signs and signals in each frame of the analysis video generated by the analysis video generation processing unit 115 (step S603). The signal state identification processing unit 116 identifies the state of the signal in a frame where the state of the signal detected by the sign / signal detection processing unit 113 is unknown (unknown signal state frame), based on the state of the signal in the frames before and after the unknown signal state frame (step S604). The accident analysis processing unit 114 analyzes the accident that occurred to the moving vehicle M based on the analysis video, driving data, the status of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113, and the status of the signals identified by the signal status identification processing unit 116, and obtains the analysis results of the accident (step S605).

[0043] <Accident type information acquisition processing unit 117, opposing party determination processing unit 118> Generally, the percentage of fault in an accident is determined based on the type of accident. Therefore, the control unit 110 may further include an accident type information acquisition processing unit 117, as shown in Figure 2. The accident type information acquisition processing unit 117 acquires accident type information, which is information about the type of accident. Then, the accident analysis processing unit 114 may analyze the accident that occurred to the mobile vehicle M based on the analysis video, driving data, accident type information, and the state of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113, and obtain the analysis results of the accident, in particular, the percentage of fault in the accident. In this way, even if an accident analysis is performed using an accident analysis model that has not been specially trained on accident types, it becomes possible to identify the type of accident that occurred to the mobile vehicle M, and it becomes possible to obtain more appropriate accident analysis results, in particular, a more appropriate percentage of fault in the accident.

[0044] Accident type information should ideally be presented as a summary of the accident type. Hundreds of accident types exist as precedents in civil traffic litigation. These hundreds of accident types are compiled as images with accompanying text describing the accident circumstances, fault ratios, and factors modifying the fault ratio. Therefore, the summary texts for the accident types could be prepared in advance based on these accident type images. In this case, the summary texts could be generated using an image caption generation model, for example, based on these accident type images.

[0045] The percentage of fault in an accident may be modified based on the attributes of the other party. For example, if the other party is a vulnerable road user such as a child or an elderly person, the percentage of fault may be modified from the perspective of protecting vulnerable road users. Therefore, the accident type information may include elements for modifying the percentage of fault, and the control unit 110 may further have a counterparty determination processing unit 118, as shown in Figure 2. The counterparty determination processing unit 118 determines the attributes of the other party object involved in the accident with the moving object M based on the analysis video. If the other party object is a person, the counterparty determination processing unit 118 may determine, for example, whether the other party object is a child (e.g., a person between 6 and 13 years old) or an elderly person (e.g., a person 65 years or older), or whether it is an infant (a person under 6 years old) or a person with a disability. Also, if the other party object is a vehicle, the counterparty determination processing unit 118 may determine the type of vehicle (e.g., a large vehicle, a motorcycle, or a bicycle) as an attribute of the other party object. The accident analysis processing unit 114 may then analyze the accident that occurred to the moving object M based on the analysis video, driving data, accident type information, the status of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113, and the attributes of the opposing party determined by the opposing party determination processing unit 118, and obtain the analysis results of the accident, in particular, the percentage of fault in the accident. In this way, even if the accident analysis is performed using an accident analysis model that has not been specially trained on accident types, it becomes possible to obtain a more appropriate percentage of fault in the accident that takes corrective factors into account.

[0046] In particular, accident type information should include at least a modifying factor for the percentage of fault when the other party is a child or an elderly person, and the attributes of the other party in the accident should at least include whether or not it is a child or an elderly person.

[0047] The opposing party determination processing unit 118 uses, for example, an image (video) analysis model to determine the attributes of the opposing object involved in the accident with the moving object M based on the video for analysis. The image (video) analysis model may be a dedicated object detection model that has been specially trained on infants, children, the elderly, people with disabilities, and vehicle types; it may be a zero-shot object detection model; or it may be a multimodal AI (for example, a natural language processing model capable of analyzing images (videos)).

[0048] Figure 7 shows an example of processing operations performed in the control unit 110. The video information acquisition processing unit 111 acquires video (accident video) taken from the mobile body M during a first period including the time of the accident that occurred to the mobile body M, the driving data acquisition processing unit 112 acquires driving data of the mobile body M during the first period, and the accident type information acquisition processing unit 117 acquires accident type information, which is information about the type of accident (step S701). The sign / signal detection processing unit 113 detects the state of traffic signs and signals in each frame of the analysis video based on the accident video acquired by the video information acquisition processing unit 111, and the other party determination processing unit 118 determines the attributes of the other party object in the accident that occurred to the mobile body M based on the analysis video (step S702). The accident analysis processing unit 114 analyzes the accident that occurred to the moving object M based on the analysis video, driving data, accident type information, the status of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113, and the attributes of the opposing party determined by the opposing party determination processing unit 118, and obtains the analysis results of the accident, in particular the percentage of fault in the accident (step S703).

[0049] Between steps S702 and S703 of the processing operation shown in Figure 7, the signal status identification processing unit 116 identifies the status of the signal in the frame where the status of the signal detected by the sign / signal detection processing unit 113 is unknown (signal status unknown frame) based on the status of the signal in the frame before and after the unknown frame. In step S703 of the processing operation shown in Figure 7, the accident analysis processing unit 114 analyzes the accident that occurred to the moving object M based on the analysis video, driving data, accident type information, the status of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113, the attributes of the opposing party determined by the opposing party determination processing unit 118, and the status of the signal identified by the signal status identification processing unit 116, and obtains the analysis results of the accident, in particular the percentage of fault in the accident.

[0050] <Priority Road Determination Processing Unit 119> In determining the percentage of fault in an accident, it is also important to consider whether the road on which the moving object M traveled (traveling road) was a priority road relative to the travel road of the other party involved in the accident. Therefore, the control unit 110 may further include a priority road determination processing unit 119, as shown in Figure 2. The priority road determination processing unit 119 determines whether the road on which the moving object M traveled was a priority road relative to the travel road of the other party involved in the accident with the moving object M. The accident analysis processing unit 114 may then analyze the accident based on the analysis video, the travel data, the status of traffic signs and signals detected in each frame of the analysis video, and the determination result of the priority road determination processing unit 119, and obtain the analysis result of the accident, in particular, the percentage of fault in the accident. In this way, even if the accident analysis is performed using an accident analysis model that has not undergone special training for accident analysis, it becomes possible to obtain a more appropriate accident analysis result, in particular, a more appropriate percentage of fault in the accident.

[0051] The priority road determination processing unit 119 may also have a first priority road determination unit 1191, as shown in Figure 8. The first priority road determination unit 1191 uses an image (video) analysis model to determine, based on the video for analysis, whether the road traveled by the moving object M was a priority road relative to the road traveled by the other object. In this case, the image (video) analysis model may be a dedicated object detection model that has been specially trained on traffic signs, a zero-shot object detection model, or a multimodal AI (for example, a natural language processing model capable of analyzing images (videos)).

[0052] The priority road determination processing unit 119 may further include an aerial image acquisition processing unit 1192 and a second priority road determination unit 1193, as shown in Figure 8. If the first priority road determination unit 1191 could not determine whether the road traveled by the moving body M was a priority road to the road traveled by the other party object involved in the accident, the aerial image acquisition processing unit 1192 acquires an aerial image that includes the road traveled by the moving body M and the road traveled by the other party object involved in the accident with the moving body M. The aerial image is an image of the road as seen from above, and may be, for example, an aerial photograph, a satellite photograph, or a map image (for example, a map image that allows confirmation of road width). The second priority road determination unit 1193 uses an image caption generation model to determine, based on the aerial image acquired by the aerial image acquisition processing unit 1192, whether the road traveled by the moving body M was a priority road to the road traveled by the other party object involved in the accident with the moving body M. This approach makes it possible to determine which roads are priority even when it is not possible to determine priority roads based on the analysis video.

[0053] In this case, the prompt input to the image caption generation model should include an instruction to cause the image caption generation model to determine that the road on which the moving object M is traveling is a priority road to the road on which the other party's object is traveling if the road on which the moving object M is traveling satisfies at least one of the following determination conditions. The one or more determination conditions include, for example, "there is a 'priority road' traffic sign on the road on which the moving object M is traveling" (Condition 1), "there is a 'stop' traffic sign on the road on which the other party's object is traveling" (Condition 2), "the center line runs through the intersection on the road on which the moving object M is traveling" (Condition 3), "the width of the road on which the moving object M is traveling is wider than the width of the road on which the other party's object is traveling" (Condition 4), and "the road on which the moving object M is traveling does not satisfy any of conditions 1 to 4, and the moving object M is traveling from the left side of the other party's object" (Condition 5).

[0054] Figure 9 illustrates an example of an aerial image that includes the road R1 traveled by the moving object M and the road R2 traveled by the other party involved in the accident with the moving object M. In the example aerial image shown in Figure 9, traffic signs such as "priority road" and "stop" and the center line cannot be seen, but it can be seen that the width of the road R1 traveled by the moving object M is wider than the width of the road R2 traveled by the other party involved in the accident. Therefore, in the example shown in Figure 9, when the second priority road determination unit 1193 inputs the above prompt to the image caption generation model, the second priority road determination unit 1193 determines that the road R1 traveled by the moving object M is a priority road relative to the road R2 traveled by the other party involved in the accident.

[0055] Figure 10 shows an example of processing operations performed in the control unit 110. The video information acquisition processing unit 111 acquires video footage (accident video) taken from the mobile body M during a first period including the time of the accident that occurred to the mobile body M, and the driving data acquisition processing unit 112 acquires the driving data of the mobile body M during the first period (step S1001). The sign / signal detection processing unit 113 detects the status of traffic signs and signals in each frame of the analysis video based on the accident video acquired by the video information acquisition processing unit 111, and the priority road determination processing unit 119 determines whether the road on which the mobile body M was traveling was a priority road to the road on which the other party in the accident that occurred to the mobile body M was traveling (step S1002). The accident analysis processing unit 114 analyzes the accident that occurred to the mobile vehicle M based on the analysis video, driving data, the status of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113, and the determination result of the priority road determination processing unit 119, and obtains the analysis result of the accident (step S1003).

[0056] Between steps S1002 and S1003 of the processing operation shown in Figure 10, the signal status identification processing unit 116 may identify the signal status in the frame where the signal status detected by the sign / signal detection processing unit 113 is unknown (signal status unknown frame) based on the signal status in the frames before and after the unknown signal status frame. Furthermore, in step S1001 of the processing operation shown in Figure 10, the accident type information acquisition processing unit 117 may acquire accident type information, which is information related to the accident type, and in step S1002 of the processing operation shown in Figure 10, the other party determination processing unit 118 may determine the attributes of the other party object involved in the accident that occurred to the moving object M based on the analysis video. Furthermore, in step S1003 of the processing operation shown in Figure 10, the accident analysis processing unit 114 may analyze the accident that occurred to the moving body M based on the analysis video, the driving data, the status of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113, the determination result of the priority road determination processing unit 119, as well as the status of the signals identified by the signal status identification processing unit 116 and / or the attributes of the opposing party determined by the opposing party determination processing unit 118, and obtain the analysis result of the accident, in particular the percentage of fault in the accident.

[0057] Figure 11 shows an example of processing operations performed in the priority road determination processing unit 119. The first priority road determination unit 1191 uses an image (video) analysis model to determine, based on the video for analysis, whether the road traveled by the moving object M was a priority road relative to the road traveled by the other object (step S1101). If the first priority road determination unit 1191 can determine whether the road traveled by the moving object M is a priority road (step S1102, YES), the process ends. If the first priority road determination unit 1191 cannot determine whether the road traveled by the moving object M is a priority road (step S1102, NO), the aerial image acquisition processing unit 1192 acquires an aerial image that includes the road traveled by the moving object M and the road traveled by the other object involved in the accident with the moving object M (step S1103). The second priority road determination unit 1193 uses an image caption generation model to determine, based on the aerial image acquired by the aerial image acquisition processing unit 1192, whether the road on which the moving object M was traveling was a priority road relative to the road on which the other party object involved in the accident with the moving object M was traveling (step S1104).

[0058] <Signal detection processing unit 113> The sign / signal detection processing unit 113 detects, for example, the location of traffic signs and traffic lights, and identifies the type of the detected traffic sign and the lighting state of the detected traffic light. In this case, the sign / signal detection processing unit 113 may have a first sign / signal detection unit 1131, as shown in Figure 12. The first sign / signal detection unit 1131 uses a dedicated object detection model (e.g., YOLO) for detecting the state of traffic signs and traffic lights to detect the location of traffic signs and traffic lights in each frame of the analysis video, and detects the type of traffic sign and the state of the traffic light. The dedicated object detection model is, for example, an object detection model specialized for detecting traffic signs and traffic lights, which has been trained on images of various types of traffic signs and traffic lights. In this way, it becomes possible to recognize and detect the state of traffic signs and traffic lights with high accuracy. The sign / signal detection processing unit 113 may include a dedicated object detection model, or it may use an external dedicated object detection model.

[0059] While dedicated object detection models can accurately recognize pre-trained traffic signs and signals, their accuracy decreases if the training data is biased towards specific regions or countries, resulting in reduced recognition accuracy for traffic signs and signals in other regions or countries. Including images of traffic signs and signals from all regions and countries in the training data is costly.

[0060] Therefore, the sign / signal detection processing unit 113 may further include a second sign / signal detection unit 1132, as shown in Figure 12. The second sign / signal detection unit 1132 uses a zero-shot object detection model (e.g., Florence) to detect the positions of traffic signs and signals in each frame of the video for analysis, and to detect the type of traffic sign and the state of the signal. Since the zero-shot object detection model has the ability to recognize objects of classes that do not exist in the training data, it is possible to detect objects with designs of traffic signs and signals that do not exist in the training data as traffic signs and signals. The sign / signal detection processing unit 113 may include a zero-shot object detection model, or it may use an external zero-shot object detection model.

[0061] Furthermore, the appearance of traffic signs and traffic lights changes during adverse weather conditions such as rain, fog, and snow, as well as at night. Their appearance also changes if parts of them are obscured by obstacles (e.g., trees or trucks). If the training data does not account for these changes in the appearance of traffic signs and traffic lights, the accuracy of their recognition will decrease. Including all images of traffic signs and traffic lights that account for these changes in appearance in the training data would be very costly. Zero-shot object detection models enable flexible object recognition based on the context of the prompt.

[0062] Therefore, the prompts input to the zero-shot object detection model should include instructions to cause the model to consider changes in the appearance of traffic signs and signals under a given environment. Here, the given environment includes, for example, adverse weather conditions such as rain, fog, or snow, or environments where parts of traffic signs and signals are hidden by obstacles (e.g., trees or trucks).

[0063] The sign / signal detection processing unit 113 may further include a sign / signal classification unit 1133, as shown in Figure 12. The sign / signal classification unit 1133 uses a zero-shot classification model (e.g., CLIP) to classify the type of traffic sign and the state of the traffic light in the cropped image in each frame of the analysis video, based on the cropped image of the location of the traffic sign and traffic light (bounding box surrounding the detected traffic sign and traffic light) detected by the first sign / signal detection unit 1131 and the cropped image of the location of the traffic sign and traffic light (bounding box surrounding the detected traffic sign and traffic light) detected by the second sign / signal detection unit 1132. The sign / signal detection processing unit 113 may include a zero-shot classification model or use an external zero-shot classification model.

[0064] In other words, the sign / signal classification unit 1133 may use a zero-shot classification model to infer objects within the cropped image of the locations of traffic signs and signals detected by the first sign / signal detection unit 1131 and within the cropped image of the locations of traffic signs and signals detected by the second sign / signal detection unit 1132, and obtain classification results for the type of traffic sign and the state of the signal from the zero-shot classification model. The accident analysis processing unit 114 may use the classification results for the type of traffic sign and the state of the signal obtained by the sign / signal classification unit 1133 as the state of traffic signs and signals detected in each frame of the analysis video by the sign / signal detection processing unit 113. By doing so, it becomes possible to detect the type of traffic sign and the state of the signal more accurately, and to obtain more appropriate accident analysis results.

[0065] Figure 13 shows an example of processing operations performed in the sign / signal detection processing unit 113. The processing operations shown in Figure 13 are examples of processing operations performed by the sign / signal detection processing unit 113 in step S502 in Figure 5, step S603 in Figure 6, step S702 in Figure 7, and step S1002 in Figure 10. The first sign / signal detection unit 1131 detects the positions of traffic signs and signals in each frame of the analysis video using a dedicated object detection model for detecting the state of traffic signs and signals, and the second sign / signal detection unit 1132 detects the positions of traffic signs and signals in each frame of the analysis video using a zero-shot object detection model (step S1301). The sign / signal classification unit 1133 uses a zero-shot classification model to classify the type of traffic sign and the state of the traffic lights in the cropped image for each frame of the analysis video, based on the cropped image of the locations of traffic signs and traffic lights detected by the first sign / signal detection unit 1131 (locations of traffic signs and traffic lights detected using a dedicated object detection model) and the cropped image of the locations of traffic signs and traffic lights detected by the second sign / signal detection unit 1132 (locations of traffic signs and traffic lights detected using a zero-shot object detection model) (step S1302).

[0066] The present invention has been described above with reference to preferred embodiments. Although the present invention has been described with reference to specific examples, various modifications and changes can be made to these examples without departing from the spirit and scope of the invention as described in the claims. [Explanation of Symbols]

[0067] 100 Accident analysis device 110 Control Unit 111 Video Information Acquisition Processing Unit 112 Driving data acquisition processing unit 113 Sign / Signal Detection Processing Unit 1131 First sign / signal detection unit 1132 Second sign / signal detection unit 1133 Sign / Signal Classification Department 114 Accident Analysis and Processing Unit 115 Analysis and Video Generation Processing Unit 116 Traffic Signal Status Identification Processing Unit 117 Accident Type Information Acquisition Processing Unit 118 Opponent Determination Processing Unit 119 Priority Road Determination Processing Unit 1191 First priority road determination unit 1192 Aerial image acquisition processing unit 1193 Second priority road determination section 120 Storage section 130 Communications Department

Claims

1. A video information acquisition processing unit that acquires video footage taken from a moving object during a predetermined period including the time of the accident that occurred to the moving object, A driving data acquisition processing unit that acquires driving data of the moving body for the predetermined period, Based on the aforementioned video, an analysis video is used to determine the attributes of the other party's object in the accident, and An information processing device having an accident analysis processing unit that performs an analysis of the accident and obtains the percentage of fault in the accident based on the aforementioned video for analysis, the aforementioned driving data, and the attributes of the other party's object.

2. The aforementioned counterparty determination processing unit, The information processing apparatus according to claim 1, which uses an image analysis model to determine the attributes of the opposing object based on the video for analysis.

3. The information processing apparatus according to claim 1, wherein the attributes of the counterparty object include whether or not the counterparty object is a child or an elderly person.

4. It further includes an accident type information acquisition processing unit that acquires accident type information, which is information related to the type of accident. The aforementioned accident analysis processing unit, The information processing device according to claim 1, which uses an image-analyzable natural language processing model to obtain the percentage of fault in an accident based on the video for analysis, the driving data, the accident type information, the state of traffic signs and signals detected in each frame of the video for analysis, and the attributes of the other party's object.

5. The aforementioned accident type information includes factors for modifying the percentage of fault, The aforementioned modification element includes a modification element for when the other party object is a child or an elderly person, The information processing apparatus according to claim 4, wherein the attributes of the counterparty object include whether or not the counterparty object is a child or an elderly person.

6. The aforementioned accident type information is, This is a summary of accident types. The information processing apparatus according to claim 4, which generates images of the accident type based on an image caption generation model.

7. A method of information processing performed by a computer, A video information acquisition process that acquires video footage taken from a moving object during a predetermined period including the time of the accident that occurred to the moving object, A driving data acquisition process for acquiring driving data of the moving body during the predetermined period, Based on the aforementioned video for analysis, a process for determining the attributes of the other party's object in the accident is performed. An information processing method comprising: an accident analysis processing step that analyzes the accident based on the video for analysis, the driving data, and the attributes of the other party's object, and obtains the percentage of fault in the accident.

8. An information processing program that causes a computer to execute the information processing method described in claim 7.

9. A computer-readable storage medium storing the information processing program described in claim 8.

Images