Situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device

The system enhances situation detection by using machine learning to analyze video and measurement data from moving objects, enabling precise identification of object behaviors and interactions, addressing the limitations of existing systems in detecting complex situations.

JP7582833B2Active Publication Date: 2024-11-13YAZAKI ENERGY SYSTEM CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2020173655
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-10-15
Publication Date
2024-11-13
Estimated Expiration
2040-10-15

AI Technical Summary

Technical Problem

Existing situation detection systems, such as those described in Patent Document 1, lack the capability for detailed situation detection, particularly in identifying complex interactions and relationships between moving objects and objects in a moving image.

Method used

A system and method that utilizes a combination of object detection, behavior detection, and related behavior detection models, trained through machine learning, to analyze video data from moving objects, incorporating measurement data and distance data to identify and classify the type of behavior exhibited by objects and their interactions.

Benefits of technology

Enables precise detection of situations by identifying the type of behavior and interactions between objects, enhancing safety in applications like real-time driving diagnostics, crime prevention, and ride-hailing services by accurately recognizing dangerous or anomalous behaviors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582833000001
    Figure 0007582833000001
  • Figure 0007582833000002
    Figure 0007582833000002
  • Figure 0007582833000003
    Figure 0007582833000003
Patent Text Reader

Abstract

To provide a situation detection system, situation detection method, situation detection program, learned model generation system and situation detection device which can properly detect a situation.SOLUTION: A situation detection system, situation detection method, situation detection program, learned model generation system and situation detection device generate a learned object detection model M1, a learned action detection model M2, a learned relevant action detection model M3 and a learned type specification model M4 with learning, and uses the generated learned object detection model M1, learned action detection model M2, learned relevant action detection model M3 and learned type specification model M4 to specify the type of a behavior that is generated from the relation between the motion of a movable body and the motion of an object from moving image data D41 and measurement data D42.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a situation detection system, a situation detection method, a situation detection program, a trained model generation system, and a situation detection device. [Background technology]

[0002] For example, Patent Document 1 discloses an information processing device as a conventional technology for performing various detections from video images of a drive recorder or the like using artificial intelligence or deep learning. This information processing device includes an acquisition unit, a traffic light area recognition unit, a speed information acquisition unit, an acceleration information acquisition unit, and a judgment unit. The acquisition unit acquires a captured image captured by a photographing device mounted on a vehicle. The traffic light area recognition unit recognizes a red light area indicating a red light of a traffic light from the captured image acquired by the acquisition unit. The speed information acquisition unit acquires speed information indicating the speed of the vehicle. The acceleration information acquisition unit acquires acceleration information indicating the acceleration of the vehicle. The judgment unit judges whether or not the vehicle is driving while ignoring a red light based on the speed information or the acceleration information, the red light area, and red light ignoring identification information that is predetermined to identify that the vehicle is driving while ignoring a red light. In this case, the red light ignoring identification information is created in advance using a machine learning method using a support vector machine (SVM). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-072940 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the information processing device described in the above-mentioned Patent Document 1 has room for further improvement in terms of, for example, detecting a more detailed situation.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide a situation detection system, a situation detection method, a situation detection program, a trained model generation system, and a situation detection device that are capable of properly detecting a situation. [Means for solving the problem]

[0006] In order to achieve the above object, the situation detection system according to the present invention includes a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data, a second learning dataset including measurement data and the object detection features representing measurement values ​​corresponding to a state of the moving object measured while the video was being captured, teacher data related to moving object behavior detection features representing the movement of the moving object associated with a time change in the measurement data, and teacher data related to object behavior detection features representing the movement of the object associated with a time change in the object detection features, a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object, and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior, and a trained object detection model that extracts the object detection features from the video data using a plurality of the first learning datasets acquired by the preprocessing unit. an object detection model generation unit that generates a model by learning, the model being a training model for detecting a type of the behavior of the object ...and a type identification unit that identifies the type of the behavior from the video data and the measurement data input by the detection target input unit, using the learned type identification model generated by the type identification model generation unit.

[0007] Moreover, in the above situation detection system, the second learning data set includes, in addition to the measurement data and the object detection feature, distance data representing a distance between the moving object and a surrounding object of the moving object measured in conjunction with shooting the video, teacher data regarding the moving object behavior detection feature, and in addition to the teacher data regarding the object behavior detection feature, teacher data regarding a distance change feature representing a change in the distance associated with a change in the distance data over time; the third learning data set includes the moving object behavior detection feature, the object behavior detection feature, and the distance change feature, and teacher data regarding the related behavior feature; and the behavior detection model generation unit generates, by learning, the trained behavior detection model that extracts the moving object behavior detection feature, the object behavior detection feature, and the distance change feature from the measurement data, the object detection feature, and the distance data using the plurality of second learning data sets acquired by the preprocessing unit. the related behavior detection model generation unit generates, by learning, a trained related behavior detection model that extracts the related behavior features from the mobile object behavior detection features, the object behavior detection features, and the distance change features using the plurality of third learning data sets acquired by the preprocessing unit; the detection target input unit inputs the video data, the measurement data, and the distance data to be detected; and the type identification unit identifies the type of the behavior from the video data, the measurement data, and the distance data input by the detection target input unit, using the trained object detection model generated by the object detection model generation unit, the trained behavior detection model generated by the behavior detection model generation unit, the trained related behavior detection model generated by the related behavior detection model generation unit, and the trained type identification model generated by the type identification model generation unit.

[0008] In order to achieve the above object, the situation detection method according to the present invention includes the steps of acquiring a first training dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data, a second training dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object associated with a time change in the measurement data and teacher data related to object behavior detection features representing the movement of the object associated with a time change in the object detection features, a third training dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object, and a fourth training dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior, and the step of generating, by learning, a trained object detection model that extracts the mobile object behavior detection features and the object behavior detection features from the measurement data and the object detection features, using the plurality of acquired second training datasets; the step of generating, by learning, a trained related behavior detection model that extracts the related behavior features from the mobile object behavior detection features and the object behavior detection features, using the plurality of acquired third training datasets; the step of generating, by learning, a trained type identification model that identifies the type of the behavior from the related behavior features, using the plurality of acquired fourth training datasets; the step of inputting the video data and the measurement data to be detected; and the step of identifying the type of the behavior from the input video data and the measurement data, using the generated trained object detection model, the trained behavior detection model, the trained related behavior detection model, and the trained type identification model.

[0009] In order to achieve the above object, the situation detection program of the present invention acquires a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data, a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object associated with a time change in the measurement data and teacher data related to object behavior detection features representing the movement of the object associated with a time change in the object detection features, a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing a behavior occurring in relation to the movement of the moving object and the movement of the object, and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior, and the method for generating a trained object detection model by learning, the trained object detection model extracting the object detection feature from the measurement data and the object detection feature, the trained behavior detection model using the acquired multiple second training datasets, the trained behavior detection model extracting the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature, the trained related behavior detection model using the acquired multiple third training datasets, the trained related behavior detection model extracting the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature, the trained type identification model identifying the type of the behavior from the related behavior feature, the trained type identification model identifying the type of the behavior from the related behavior feature, the trained object detection model, the trained behavior detection model, the trained related behavior detection model, and the trained type identification model, the trained related behavior detection model, and the trained type identification model being generated as input of the video data and the measurement data to be detected.

[0010] In order to achieve the above object, the trained model generation system of the present invention includes a first training dataset including video data representing a video captured from a moving object and teacher data regarding object detection features representing an object detected in the video data, a second training dataset including measurement data representing measurement values ​​corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data regarding moving object behavior detection features representing the movement of the moving object associated with a time change in the measurement data and teacher data regarding object behavior detection features representing the movement of the object associated with a time change in the object detection features, a third training dataset including the moving object behavior detection features and the object behavior detection features, and teacher data regarding related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object, and a fourth training dataset including the related behavior features and teacher data regarding related behavior type data representing a type of the behavior. an object detection model generation unit that generates, by learning, a trained object detection model that extracts the object detection features from the video image data using the plurality of first training data sets acquired by the pre-processing unit; a behavior detection model generation unit that generates, by learning, a trained behavior detection model that extracts the mobile object behavior detection features and the object behavior detection features from the measurement data and the object detection features using the plurality of second training data sets acquired by the pre-processing unit; a related behavior detection model generation unit that generates, by learning, a trained related behavior detection model that extracts the related behavior features from the mobile object behavior detection features and the object behavior detection features using the plurality of third training data sets acquired by the pre-processing unit; and a type identification model generation unit that generates, by learning, a trained type identification model that identifies the type of the behavior from the related behavior features using the plurality of fourth training data sets acquired by the pre-processing unit.

[0011] In order to achieve the above object, the situation detection device of the present invention is characterized in that it comprises a detection target input unit that inputs video data representing a video captured from a moving object and measurement data representing measurement values ​​corresponding to a state of the moving object measured in conjunction with capturing the video data, a trained object detection model that has been trained to extract from the video data object detection features representing an object detected in the video data, a trained behavior detection model that has been trained to extract, from the measurement data and the object detection features, mobile object behavior detection features that represent the movement of the moving object associated with a time change in the measurement data and object behavior detection features that represent the movement of the object associated with a time change in the object detection features, a trained related behavior detection model that has been trained to extract, from the mobile object behavior detection features and the object behavior detection features, related behavior features that represent behavior that occurs in relation to the movement of the moving object and the movement of the object, and a type identification unit that identifies the type of the behavior from the video data and the measurement data input by the detection target input unit using a trained type identification model that has been trained to identify the type of the behavior from the related behavior features.

[0012] In order to achieve the above object, the situation detection system according to the present invention includes an object detection model generation unit that generates, by learning, a trained object detection model that extracts, from video data representing a video image captured from a moving object, object detection features that represent an object detected in the video data; a behavior detection model generation unit that generates, by learning, a trained behavior detection model that extracts, from measurement data representing measurement values ​​according to a state of the moving object measured in the capture of the video image and the object detection features, mobile object behavior detection features that represent the movement of the moving object accompanying a time change in the measurement data and object behavior detection features that represent the movement of the object accompanying a time change in the object detection features; and a behavior detection model generation unit that extracts, from the mobile object behavior detection features and the object behavior detection features, a related behavior feature that represents a behavior that occurs in relation to the movement of the moving object and the movement of the object. the learned object detection model generated by the object detection model generation unit, the learned behavior detection model generated by the behavior detection model generation unit, the learned related behavior detection model generated by the related behavior detection model generation unit, and the learned type identification model generated by the type identification model generation unit, Effect of the Invention

[0013] The situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device according to the present invention have the advantage of being able to properly detect a situation. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a situation detection system according to the first embodiment. [Diagram 2] FIG. 2 is a schematic diagram illustrating object detection in the situation detection system according to the first embodiment. [Diagram 3] FIG. 3 is a schematic diagram illustrating object detection in the situation detection system according to the first embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating behavior detection in the situation detection system according to the first embodiment. [Diagram 5] FIG. 5 is a schematic diagram illustrating related behavior detection in the situation detection system according to the first embodiment. [Figure 6] FIG. 6 is a schematic diagram showing the processing of the learning phase and the usage phase performed by the processing circuit of the situation detection system according to the first embodiment. [Figure 7] FIG. 7 is a block diagram illustrating a schematic configuration of the type identification unit of the situation detection system according to the first embodiment. [Figure 8] FIG. 8 is a schematic diagram illustrating an example of related behavior detection in the situation detection system according to the first embodiment. [Figure 9] FIG. 9 is a schematic diagram illustrating an example of related behavior detection in the situation detection system according to the first embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of processing performed by the processing circuit of the situation detection system according to the embodiment. [Figure 11] FIG. 11 is a block diagram showing a schematic configuration of a situation detection system according to a modified example. [Figure 12] FIG. 12 is a block diagram showing a schematic configuration of a situation detection system according to a modified example. [Figure 13] FIG. 13 is a block diagram illustrating a schematic configuration of a type identifying unit of a situation detection system according to the second embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of processing performed by a processing circuit of a situation detection system according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, the components in the following embodiments include those that are replaceable and easy for a person skilled in the art, or those that are substantially the same.

[0016] [Embodiment 1] 1 is a system for detecting a moving object V and its surrounding situation using images. The moving object V that is the subject of situation detection by the situation detection system 1 is typically a vehicle traveling on a road surface, and may be, for example, any of a private car, a rental car, a sharing car, a ride-sharing car, a bus, a taxi, a truck, a transport vehicle, a work vehicle, etc.

[0017] Here, examples of detection techniques using images include object detection and action detection.

[0018] Object detection, as shown in FIG. 2, is a method of detecting the type and position of an object from a still image that constitutes a moving image. More specifically, object detection is a method of detecting the static position and type of an object at a certain moment from a still image. Objects detected by object detection include, for example, vehicles, pedestrians, bicycles, obstacles, street lights, signs, utility poles, signs, stop lines, etc. FIG. 2 shows an example of a case where a "bicycle" and a "stop line" are detected as objects by object detection in a still image, and the positions of the detected "bicycle" and "stop line" are shown by rectangular frames. In addition, this object detection can also detect a change in the position of an object by comparing multiple still images that constitute a moving image, as shown in FIG. 3. FIG. 3 shows an example of a case where a "bicycle" and a "stop line" are detected as objects by object detection in multiple still images that constitute a moving image, and a change in the position of the "bicycle" from right to left is detected.

[0019] On the other hand, as shown in FIG. 4, behavior detection is to detect the movement (behavior) of an object alone from the change over time of the position, state, etc. of the object detected by object detection from multiple still images constituting a moving image. In other words, behavior detection is to detect the movement (behavior) of an object with a change over time. In this case, for example, the fact that a detected object is stationary at its position (in other words, not moving), such as a "stop line," is also included in the category of the concept of the movement of the object as an example of the movement (behavior) of the object. In behavior detection, for example, even if multiple objects are detected from still images constituting a moving image, the movement of each object alone is detected. FIG. 4 shows, as an example, a case where two "bicycles" are detected by object detection in multiple still images constituting a moving image, and then the movement of the two "bicycles" alone, that is, "each of them is running unsteadily," is detected by behavior detection.

[0020] This behavior detection detects the state or change of an object alone (for example, the unsteady movement of a "bicycle") and can predict danger to the driver of a moving object V, for example, but it cannot detect the meaning of behavior by relating multiple detected objects together. For this reason, for example, in the example of Figure 4, the two unsteady-moving "bicycles" will both be judged to be dangerous regardless of their relationship to other objects.

[0021] In contrast, the situation detection system 1 of this embodiment, in addition to object detection and action detection, further uses relational action detection to detect relationships between moving bodies V and objects detected from video images, and correlates these movements to detect the meaning (type) of their mutual behavior, thereby achieving more detailed situation detection.

[0022] The related behavior detection here is, as shown in FIG. 5, a detection of behavior (behavior) that occurs when multiple objects are related to each other and the relationship between objects from the movement of the object alone or the movement of the moving body V detected by behavior detection from multiple still images constituting a moving image. In other words, the related behavior detection is a detection of behavior that occurs when the movements of multiple objects including the moving body V are related to each other. The behavior that occurs when the movements of multiple objects are related to each other can be rephrased as, for example, "an event that occurs according to a change in the relationship between multiple objects." FIG. 5 shows, as an example, a case where two "bicycles" are detected by object detection in multiple still images constituting a moving image, and the movement of the two "bicycles" alone, that is, "each of them is running unsteadily," is detected by behavior detection. FIG. 5 also shows a case where the related behavior detection detects behavior in which the "bicycle" on the front side that is running unsteadily is heading in the direction of travel of the vehicle (moving body V) that is about to turn left, while the "bicycle" on the back side is heading in a direction different from the direction of travel of the vehicle.

[0023] The situation detection system 1 of this embodiment detects behavior (movement) that occurs due to the relationship between the movements of multiple objects including the moving body V using the above-mentioned related behavior detection, and identifies the meaning (type) of the behavior. As a result, for example, in the example of Fig. 5, the situation detection system 1 can determine that the "bicycle" in the foreground heading in the direction of travel of the vehicle is dangerous, while determining that the "bicycle" in the background, which is swaying but heading in a different direction from the vehicle, is not dangerous.

[0024] Here, as shown in Fig. 6, the situation detection system 1 of this embodiment has a learning phase in which a process for generating a trained model M is performed, and a usage phase in which the above-mentioned situation detection is actually performed using the trained model M. The situation detection system 1 is realized by various computer devices such as a personal computer, a workstation, and a tablet terminal. Hereinafter, each configuration of the situation detection system will be described in detail with reference to Figs. 1, 6, and 7.

[0025] Specifically, the situation detection system 1 includes an input device 10, an output device 20, a storage circuit 30, and a processing circuit 40. The input device 10, the output device 20, the storage circuit 30, and the processing circuit 40 are connected to each other so as to be able to communicate with each other via a network. The situation detection system 1 of this embodiment performs both the processes in the learning phase and the processes in the use phase. Here, the situation detection system 1 is configured by aggregating a trained model generation system that executes the processes in the learning phase and a situation detection device that executes the processes in the use phase into one system.

[0026] The input device 10 is a device that performs various inputs to the situation detection system 1. The input device 10 is realized, for example, by an operation input device that accepts various operation inputs from a user, a data input device that accepts data (information) input from other devices outside the situation detection system 1, etc. The operation input device is realized, for example, by a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touchpad, a touch screen, a non-contact input circuit, a voice input circuit, etc. The data input device is realized, for example, by a communication interface that transmits and receives various data to and from devices via communication, whether wired or wireless, a hard disk drive (HDD), a solid state drive (SSD), a flexible disk (FD), a magneto-optical disk (Magneto-Optical disk), a CD-ROM, a DVD, a USB memory, an SD card memory, a Flash memory, etc.

[0027] The input device 10 of this embodiment receives at least moving image data and measurement data from various devices.

[0028] The video data is data representing a video of the surroundings of a moving body V captured from the moving body V. The video of the surroundings of the moving body V is captured, for example, by a camera 50 mounted on the moving body V, and is input from the camera 50 to the input device 10 as video data.

[0029] The measurement data is data representing measurement values ​​corresponding to the state of the moving body V measured in conjunction with the capture of a moving image. The measurement values ​​are typically measured in the moving body V in synchronization with the capture of a moving image, and represent the state of the moving body V at the time of capture. The measurement values ​​are, for example, physical quantities such as the speed of the moving body V, the acceleration / deceleration of the moving body V, the rotation speed of the power source for running the moving body V, and the amount of change in direction of the moving body V. The measurement values ​​may also include the steering angle of the moving body V, the amount of operation of the accelerator pedal (amount of accelerator depression), the amount of operation of the brake pedal (amount of brake depression), and the like. The measurement values ​​corresponding to the state of the moving body V are measured by a measuring device 51 such as a sensor or detector mounted on the moving body V, and are input from the measuring device 51 to the input device 10 as measurement data.

[0030] The camera 50 and the measuring device 51 may be, for example, components of an in-vehicle device such as a drive recorder mounted on the moving object V.

[0031] The output device 20 is a device that performs various outputs from the situation detection system 1. The output device 20 is realized, for example, by a display that outputs and displays various image information, a speaker that outputs sound information, a data output device that outputs data (information) to other devices outside the situation detection system 1, etc. The data output device is realized, for example, by a communication interface that transmits and receives various data to devices via communication, whether wired or wireless, a recording medium interface that writes various data to a recording medium similar to the above, etc.

[0032] The above-mentioned input device 10 as a data input device and the output device 20 as a data output device may share a part or all of their configurations.

[0033] The memory circuit 30 is a circuit that stores various data. The memory circuit 30 is realized, for example, by a semiconductor memory element such as a random access memory (RAM), a flash memory, a hard disk, an optical disk, or the like. The memory circuit 30 stores, for example, a program for the situation detection system 1 to realize various functions. The programs stored in the memory circuit 30 include a program for making the input device 10 function, a program for making the output device 20 function, a program for making the processing circuit 40 function, and the like. In addition, the memory circuit 30 stores various data such as raw data D0 input via the input device 10, data required for various processes in the processing circuit 40, a learning data set D3 used for learning the learned model M, the learned model M, and type identification result data D5 output via the output device 20. These various data are read out from the memory circuit 30 by the processing circuit 40 or the like as necessary. The memory circuit 30 may be realized by a cloud server or the like connected to the situation detection system 1 via a network.

[0034] The processing circuit 40 is a circuit that realizes various processing functions in the situation detection system 1. The processing circuit 40 is realized by, for example, a processor. The processor means, for example, a circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). The processing circuit 40 realizes each processing function by, for example, executing a program read from the storage circuit 30.

[0035] The outline of the overall configuration of the situation detection system 1 according to the present embodiment has been described above. With such a configuration, the processing circuit 40 according to the present embodiment has a function for performing various processes for generating the trained model M in the learning phase. In addition, the processing circuit 40 according to the present embodiment has a function for performing various processes using the trained model M in the use phase.

[0036] Specifically, in order to realize the various processing functions described above, the processing circuitry 40 of this embodiment is configured to include, functionally and conceptually, a preprocessing unit 41, a model generating unit 42, a detection target input unit 43, a type identifying unit 44, and an output unit 45. The processing circuitry 40 realizes each of the processing functions of the preprocessing unit 41, the model generating unit 42, the detection target input unit 43, the type identifying unit 44, and the output unit 45 by, for example, executing a program read from the storage circuitry 30.

[0037] The preprocessing unit 41 is a part having a function capable of performing various preprocessing operations on data for training the trained model M in the learning phase. The preprocessing unit 41 in this embodiment is capable of performing a process of acquiring a training data set D3 including input data D1 serving as explanatory variables and teacher data D2 serving as objective variables.

[0038] The learning dataset D3 acquired by the preprocessing unit 41 is a teacher dataset used when generating the trained model M by machine learning. The learning dataset D3 is configured by linking input data D1 serving as explanatory variables and teacher data D2 serving as a response variable corresponding to the input data D1 as a set. More specifically, the learning dataset D3 is configured from the input data D1 quantified as an explanatory variable and the teacher data D2 quantified as a response variable.

[0039] The pre-processing unit 41 can, for example, perform various pre-processing operations on raw data D0 input from other devices outside the situation detection system 1, and create and acquire a learning dataset D3 by linking the input data D1 and the teacher data D2 corresponding to the input data D1 as a set.

[0040] Here, the raw data D0 to be preprocessed by the preprocessing unit 41 may be input from another device outside the situation detection system 1 via a data input device constituting the input device 10, or may be input by a user's operation via an operation input device constituting the input device 10. In this embodiment, the raw data D0 includes at least the above-mentioned moving image data and measurement data.

[0041] The preprocessing performed by the preprocessing unit 41 on the raw data D0 includes, for example, a process of collecting and combining the raw data D0, a process of extracting the input data D1 from the raw data D0 and quantifying it as an explanatory variable, a process of annotating the raw data D0 (a process of adding metadata to the data to make it valid as teacher data D2) to create teacher data D2 and quantifying it as a target variable, a process of linking and combining the quantified input data D1 and the quantified teacher data D2, etc. The preprocessing unit 41 may perform the preprocessing on the raw data D0 every time the raw data D0 is input, or may perform the preprocessing on the raw data D0 at an appropriate timing in response to a user's operation via an operation input device constituting the input device 10.

[0042] Furthermore, the preprocessing unit 41 may acquire a learning data set D3 created in advance from another device outside the situation detection system 1, for example, via a data input device constituting the input device 10. The other device outside the situation detection system 1 here is, for example, a learning data providing system that stores a large number of input data D1, teacher data D2, learning data set D3, etc. in a database.

[0043] In this embodiment, the training dataset D3 acquired by the preprocessing unit 41 includes a first training dataset D31, a second training dataset D32, a third training dataset D33, and a fourth training dataset D34. That is, the preprocessing unit 41 in this embodiment acquires the first training dataset D31, the second training dataset D32, the third training dataset D33, and the fourth training dataset D34 as the training dataset D3.

[0044] The first learning dataset D31 is a teacher dataset used when generating a learned object detection model M1 by machine learning as the learned model M. The learned object detection model M1 is a learned model M used to extract object detection features from video data. Here, the object detection features are features that represent an object detected in video data, and represent the static position and type of an object at a certain moment in a still image that constitutes the video. The process of extracting the object detection features from video data using the learned object detection model M1 corresponds to the above-mentioned process related to object detection. The first learning dataset D31 is configured by linking "video data" and "teacher data related to object detection features corresponding to the video data" as one set. In other words, the first learning dataset D31 is composed of the video data quantified as input data D1 (explanatory variable) and the teacher data related to the object detection features quantified as teacher data D2 (target variable).

[0045] The second learning data set D32 is a teacher data set used when generating the learned behavior detection model M2 by machine learning as the learned model M. The learned behavior detection model M2 is a learned model M used to extract a moving object behavior detection feature and an object behavior detection feature from measurement data and object detection feature. Here, the moving object behavior detection feature is a feature representing the movement (behavior) of the moving object V accompanying the time change of the measurement data, and represents the characteristics of the movement of the moving object alone accompanying the time change. The object behavior detection feature is a feature representing the movement (behavior) of the object accompanying the time change of the object detection feature, in other words, the movement of the object in a still image constituting a moving image, and represents the characteristics of the movement of the object alone accompanying the time change. The process of extracting the moving object behavior detection feature and the object behavior detection feature from the measurement data and object detection feature using the learned behavior detection model M2 corresponds to the process related to the behavior detection described above. The second learning dataset D32 is configured by linking together as a set "measurement data and object detection feature" and "teacher data on mobile object behavior detection feature corresponding to the measurement data and teacher data on object behavior detection feature corresponding to the object detection feature." In other words, the second learning dataset D32 is composed of the measurement data and the object detection feature quantified as input data D1 (explanatory variables), and the teacher data on the mobile object behavior detection feature and the teacher data on the object behavior detection feature quantified as teacher data D2 (target variable).

[0046] Here, the learned behavior detection model M2 is described as a learned model M that extracts both the moving object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature, but is not limited thereto, and may be composed of, for example, a plurality of sub-models. That is, the learned behavior detection model M2 may be composed of a moving object behavior detection sub-model that extracts the moving object behavior detection feature from the measurement data, and an object behavior detection sub-model that extracts the object behavior detection feature from the object detection feature. In this case, the second learning data set D32 may be composed of a moving object behavior detection learning sub-data set consisting of the measurement data and teacher data related to the moving object behavior detection feature representing the movement of the moving object V accompanying the time change of the measurement data, and an object behavior detection sub-data set consisting of the object detection feature and teacher data related to the object behavior detection feature representing the movement of the object accompanying the time change of the object detection feature.

[0047] The third learning data set D33 is a teacher data set used when generating the learned relational behavior detection model M3 by machine learning as the learned model M. The learned relational behavior detection model M3 is a learned model M used to extract relational behavior features from the moving object behavior detection feature and the object behavior detection feature. Here, the relational behavior feature is a feature that represents a behavior (behavior) that occurs in relation between the movement of the moving object V and the movement of an object in a still image that constitutes a moving image, and represents a feature of a behavior that occurs due to a change in the relationship associated with the time change of each of multiple objects including the moving object V. In other words, the relational behavior feature can also be said to be a feature that represents how the relationship between the movements of multiple objects including the moving object V changes over time. The process of extracting the relational behavior feature from the moving object behavior detection feature and the object behavior detection feature using the learned relational behavior detection model M3 corresponds to the process related to the relational behavior detection described above. The third learning dataset D33 is configured by linking together a set of "mobile object behavior detection features and object behavior detection features" and "teacher data on related behavior features corresponding to the mobile object behavior detection features and the object behavior detection features." In other words, the third learning dataset D33 is composed of the mobile object behavior detection features and the object behavior detection features quantified as input data D1 (explanatory variables) and the teacher data on the related behavior features quantified as teacher data D2 (target variables).

[0048] The fourth learning data set D34 is a teacher data set used when generating the learned type identification model M4 by machine learning as the learned model M. The learned type identification model M4 is a learned model M used to identify the type of behavior from the related behavior feature. Here, the type of behavior represents the type and meaning of the behavior (behavior) that occurs in relation between the movement of the moving body V and the movement of an object in a still image that constitutes a moving image, and represents the type and meaning of the feature of the behavior represented by the related behavior feature. The process of identifying the type of behavior from the related behavior feature using the learned type identification model M4 corresponds to the process of identifying the type of behavior that occurs in relation between the movement of the moving body V and the movement of an object in a still image that constitutes a moving image in the above-mentioned related behavior detection. The fourth learning data set D34 is configured by linking the "related behavior feature" and "teacher data related to related behavior type data that represents the type of behavior corresponding to the related behavior feature" as one set. That is, the fourth learning dataset D34 includes the relevant behavior feature amount quantified as the input data D1 (explanatory variable) and training data regarding the relevant behavior type data quantified as the training data D2 (objective variable).

[0049] The preprocessing unit 41 stores in the memory circuitry 30 the acquired multiple training data sets D3 (the first training data set D31, the second training data set D32, the third training data set D33, and the fourth training data set D34).

[0050] The model generation unit 42 is a part having a function capable of executing a process of generating a trained model M by machine learning, which identifies a type of behavior occurring in relation to the movement of the moving body V and the movement of an object from the video image data and the measurement data in the learning phase. The model generation unit 42 of this embodiment is capable of executing a process of generating the trained model M by machine learning, using a plurality of training data sets D3 acquired by the preprocessing unit 41. For example, the model generation unit 42 may train and generate the trained model M each time a new training data set D3 is acquired, or may train and generate the trained model M at an appropriate timing in response to a user's operation via an operation input device constituting the input device 10.

[0051] The model generation unit 42 generates a trained model M by performing machine learning based on various machine learning algorithms AL using a plurality of training data sets D3 as teacher data sets. Examples of the machine learning algorithms AL used include well-known algorithms such as deep learning, neural network, logistic regression, ensemble learning, support vector machine, random forest, and naive Bayes. The model generation unit 42 performs machine learning of the trained model M using the input data D1 of the training data set D3 as an explanatory variable and the teacher data D2 as an objective variable. As a result of the machine learning, the model generation unit 42 generates a trained model M in which machine learning has been performed to identify the type of the above-mentioned behavior from the video data and the measurement data.

[0052] The trained model M is realized by, for example, a neural network. As the neural network, for example, a neural network having any configuration such as a convolution neural network (CNN), a recurrent neural network (RNN), or a long short-term memory (LSTM) network can be adopted. In this case, the model generation unit 42 performs machine learning using a plurality of training data sets D3 to learn a learning weighting coefficient or the like used as weighting in the neural network, and generates the trained model M.

[0053] More specifically, the model generation unit 42 of the present embodiment is configured to include, in terms of functional concept, an object detection model generation unit 42a, a behavior detection model generation unit 42b, a related behavior detection model generation unit 42c, and a type identification model generation unit 42d.

[0054] The object detection model generation unit 42a is a part having a function capable of executing a process of generating a trained object detection model M1 from the trained model M by machine learning. The object detection model generation unit 42a is capable of executing a process of generating a trained object detection model M1 by machine learning using a plurality of first training data sets D31 acquired by the preprocessing unit 41. The object detection model generation unit 42a performs machine learning of the trained object detection model M1 using "video image data" from the first training data set D31 as input data D1 (explanatory variable) and "teacher data related to object detection feature amounts" as teacher data D2 (target variable) from the first training data set D31. The object detection model generation unit 42a can generate a trained object detection model M1 that has undergone machine learning to extract object detection feature amounts from video image data as a result of the machine learning.

[0055] The trained object detection model M1 generated by the object detection model generation unit 42a is a model in which the input is "video data" and the output is "object detection features." That is, the trained object detection model M1 is functionalized to receive input of video data and output object detection features from the video data. More specifically, the trained object detection model M1 causes the computer to perform a calculation based on the learning weighting coefficients in the neural network for the video data input to the input layer of the neural network, and to output the object detection features from the output layer of the neural network.

[0056] The behavior detection model generation unit 42b is a part having a function capable of executing a process of generating a learned behavior detection model M2 from the learned model M by machine learning. The behavior detection model generation unit 42b can execute a process of generating a learned behavior detection model M2 by machine learning using a plurality of second learning data sets D32 acquired by the preprocessing unit 41. The behavior detection model generation unit 42b performs machine learning of the learned behavior detection model M2 using the "measurement data and object detection feature amount" from the second learning data set D32 as input data D1 (explanatory variables) and the "teacher data regarding the moving object behavior detection feature amount and the teacher data regarding the object behavior detection feature amount" as teacher data D2 (target variable). The behavior detection model generation unit 42b can generate a learned behavior detection model M2 that has been subjected to machine learning for extracting the moving object behavior detection feature amount and the object behavior detection feature amount from the measurement data and the object detection feature amount as a result of the machine learning.

[0057] The learned behavior detection model M2 generated by the behavior detection model generation unit 42b is a model in which the input is "measurement data and object detection feature amount" and the output is "mobile object behavior detection feature amount and object behavior detection feature amount." That is, the learned behavior detection model M2 is functionalized to receive the input of the measurement data and the object detection feature amount and output the mobile object behavior detection feature amount and the object behavior detection feature amount from the measurement data and the object detection feature amount. More specifically, the learned behavior detection model M2 performs a calculation based on the learning weighting coefficient in the neural network for the measurement data and the object detection feature amount input to the input layer of the neural network, and causes the computer to function to output the mobile object behavior detection feature amount and the object behavior detection feature amount from the output layer of the neural network.

[0058] In addition, as described above, when the learned behavior detection model M2 is configured to be divided into multiple sub-models (mobile behavior detection sub-model, object behavior detection sub-model), the behavior detection model generation unit 42b may generate the multiple sub-models individually by machine learning as the learned behavior detection model M2.

[0059] The related behavior detection model generation unit 42c is a part having a function capable of executing a process of generating a learned related behavior detection model M3 by machine learning from the learned model M. The related behavior detection model generation unit 42c is capable of executing a process of generating a learned related behavior detection model M3 by machine learning using a plurality of third learning data sets D33 acquired by the preprocessing unit 41. The related behavior detection model generation unit 42c performs machine learning of the learned related behavior detection model M3 by using the "mobile object behavior detection feature and object behavior detection feature" from the third learning data set D33 as input data D1 (explanatory variables) and the "teacher data related to the related behavior feature" as teacher data D2 (target variable). The related behavior detection model generation unit 42c can generate a learned related behavior detection model M3 that has been subjected to machine learning for extracting a related behavior feature from the mobile object behavior detection feature and the object behavior detection feature as a result of the machine learning.

[0060] The learned relational behavior detection model M3 generated by the relational behavior detection model generation unit 42c is a model in which the input is "mobile object behavior detection feature and object behavior detection feature" and the output is "relevant behavior feature." That is, the learned relational behavior detection model M3 is functionalized to receive the input of the mobile object behavior detection feature and the object behavior detection feature and output the relevant behavior feature from the mobile object behavior detection feature and the object behavior detection feature. More specifically, the learned relational behavior detection model M3 performs a calculation based on the learning weighting coefficient in the neural network for the mobile object behavior detection feature and the object behavior detection feature input to the input layer of the neural network, and causes the computer to function to output the relevant behavior feature from the output layer of the neural network.

[0061] The type-specific model generation unit 42d is a part having a function capable of executing a process of generating a learned type-specific model M4 by machine learning among the learned model M. The type-specific model generation unit 42d is capable of executing a process of generating a learned type-specific model M4 by machine learning using a plurality of fourth learning data sets D34 acquired by the preprocessing unit 41. The type-specific model generation unit 42d performs machine learning of the learned type-specific model M4 by using the "related behavior feature amount" of the fourth learning data set D34 as input data D1 (explanatory variable) and the "teacher data related to related behavior type data" as teacher data D2 (objective variable) among the fourth learning data set D34. The type-specific model generation unit 42d can generate a learned type-specific model M4 that has been subjected to machine learning for identifying the type of the above-mentioned behavior from the related behavior feature amount as a result of the machine learning.

[0062] The learned type identification model M4 generated by the type identification model generation unit 42d is a model in which the input is a "related behavior feature" and the output is a "quantified value of the type of behavior occurring due to the relationship between the movement of the moving body V and the movement of the object." That is, the learned type identification model M4 is functionalized to receive the input of the related behavior feature and output a quantified value of the type of the behavior from the related behavior feature. More specifically, the learned type identification model M4 causes the computer to perform a calculation based on the learning weighting coefficient in the neural network for the related behavior feature input to the input layer of the neural network, and to output a quantified value of the type of the behavior from the output layer of the neural network.

[0063] The model generation unit 42 stores the trained models M (trained object detection model M1, trained behavior detection model M2, trained relational behavior detection model M3, trained type identification model M4) generated as described above in the storage circuit 30. At this time, if a previously generated trained model M has already been stored in the storage circuit 30, the model generation unit 42 replaces the stored trained model M with the newly generated trained model M.

[0064] The detection target input unit 43 is a part having a function capable of executing a process of inputting moving image data and measurement data to be detected in the use phase. Here, the moving image data and measurement data to be detected may be referred to as "detection target data (input data) D4." Furthermore, among the "detection target data D4," the moving image data may be referred to as "moving image data D41," and the measurement data may be referred to as "measurement data D42."

[0065] The detection target data D4 may be input from another device outside the situation detection system 1 via a data input device constituting the input device 10, or may be input by a user's operation via an operation input device constituting the input device 10. The detection target input unit 43 of this embodiment can execute a process of inputting the detection target data D4 (movie image data D41, measurement data D42) received via the input device 10 from the camera 50, the measurement device 51, etc. to the type identification unit 44. For example, the detection target input unit 43 may input the detection target data D4 in real time in accordance with the movement of the moving object V, or may input the detection target data D4 after the fact at an appropriate timing after the movement of the moving object V has ended. The detection target input unit 43 may temporarily store the input detection target data D4 in the memory circuit 30.

[0066] The type identification unit 44 is a part having a function capable of executing a process of identifying a type of behavior occurring in relation to the movement of the moving body V and the movement of an object in a still image constituting a moving image, using the trained model M, in the use phase. The type identification unit 44 of this embodiment is capable of executing a process of identifying a type of the behavior from the detection target data D4 that is a detection target input by the detection target input unit 43, using the trained model M generated by the model generation unit 42. More specifically, the type identification unit 44 is capable of executing a process of identifying a type of the behavior from the moving image data D41 and the measurement data D42 input by the detection target input unit 43, using the trained object detection model M1 generated by the object detection model generation unit 42a, the trained behavior detection model M2 generated by the behavior detection model generation unit 42b, the trained related behavior detection model M3 generated by the related behavior detection model generation unit 42c, and the trained type identification model M4 generated by the type identification model generation unit 42d.

[0067] Specifically, as shown in FIG. 7, the type identification unit 44 of this embodiment is configured, functionally and conceptually, to include an object detection feature extractor 44a, an integration unit 44b, a behavior detection feature extractor 44c, a related behavior feature extractor 44d, and a classifier 44e.

[0068] The object detection feature extractor 44a is a part having a function capable of executing a process of extracting an object detection feature D61 from video data D41 using the trained object detection model M1 generated by the object detection model generating unit 42a. The object detection feature extractor 44a inputs the "video data D41" of the detection target data D4 input by the detection target input unit 43 as input data to the trained object detection model M1, and outputs an "object detection feature D61" from the trained object detection model M1 in response to the input data. In this way, the object detection feature extractor 44a extracts, from the video data D41, an object detection feature D61 that represents an object detected in the video data D41. The process of extracting the object detection feature D61 from the video data D41 by the object detection feature extractor 44a corresponds to the above-mentioned process related to object detection. The object detection feature extractor 44a stores the output object detection feature D61 in the storage circuit 30, and also inputs it to the next stage, the integrating unit 44b.

[0069] The integration unit 44b is a part having a function capable of executing a process of integrating the object detection feature amount D61 and the measurement data D42. The integration unit 44b integrates the "object detection feature amount D61" input by the object detection feature amount extractor 44a and the "measurement data D42" of the detection target data D4 input by the detection target input unit 43 so as to be a combination of data at the same time in a chronological order, stores the combined data in the memory circuit 30, and inputs the combined data to the behavior detection feature amount extractor 44c in the next stage.

[0070] The behavior detection feature extractor 44c is a part having a function capable of executing a process of extracting a moving object behavior detection feature D62 and an object behavior detection feature D63 from the measurement data D42 and the object detection feature D61 using the learned behavior detection model M2 generated by the behavior detection model generation unit 42b. The behavior detection feature extractor 44c inputs the "measurement data D42 and object detection feature D61" input by the integration unit 44b at the same time in time series as input data to the learned behavior detection model M2, and outputs "moving object behavior detection feature D62 and object behavior detection feature D63" from the learned behavior detection model M2 in response to the input data. As a result, the behavior detection feature extractor 44c extracts the moving object behavior detection feature D62 representing the movement of the moving object V accompanying the time change of the measurement data D42 and the object behavior detection feature D63 representing the movement of the object accompanying the time change of the object detection feature D61 from the measurement data D42 and the object detection feature D61. The process of extracting the moving object behavior detection feature D62 and the object behavior detection feature D63 from the measurement data D42 and the object detection feature D61 by the behavior detection feature extractor 44c corresponds to the process related to behavior detection described above. The behavior detection feature extractor 44c stores the output moving object behavior detection feature D62 and object behavior detection feature D63 in the storage circuit 30, and also inputs them to the related behavior feature extractor 44d in the next stage.

[0071] In addition, when the learned behavior detection model M2 is configured to be divided into multiple sub-models (mobile object behavior detection sub-model, object behavior detection sub-model), the behavior detection feature extractor 44c inputs the measurement data D42 and the object detection feature D61 to each of the multiple sub-models, and outputs the mobile object behavior detection feature D62 and the object behavior detection feature D63, respectively.

[0072] The related behavior feature extractor 44d is a part having a function capable of executing a process of extracting a related behavior feature D64 from the moving object behavior detection feature D62 and the object behavior detection feature D63 using the learned related behavior detection model M3 generated by the related behavior detection model generation unit 42c. The related behavior feature extractor 44d inputs the "moving object behavior detection feature D62 and object behavior detection feature D63" input by the behavior detection feature extractor 44c as input data to the learned related behavior detection model M3, and outputs the "related behavior feature D64" from the learned related behavior detection model M3 in response to the input data. As a result, the related behavior feature extractor 44d extracts the related behavior feature D64 representing the behavior caused by the relationship between the movement of the moving object V and the movement of the object in the still image constituting the moving image from the moving object behavior detection feature D62 and the object behavior detection feature D63. The process of extracting the related behavior feature D64 from the mobile object behavior detection feature D62 and the object behavior detection feature D63 by the related behavior feature extractor 44d corresponds to the process related to the related behavior detection described above. The related behavior feature extractor 44d stores the output related behavior feature D64 in the memory circuit 30 and inputs it to the next classifier 44e.

[0073] The classifier 44e is a part having a function capable of executing a process of identifying the type of behavior occurring in relation between the movement of the moving body V and the movement of an object in a still image constituting a moving image from the related behavior feature D64, using the trained type-specific model M4 generated by the type-specific model generation unit 42d. The classifier 44e inputs the "related behavior feature D64" input by the related behavior feature extractor 44d as input data to the trained type-specific model M4 generated by the type-specific model generation unit 42d, and outputs a "quantified value of the type of the above behavior" from the trained type-specific model M4 in response to the input data. In this way, the classifier 44e identifies the type of the above behavior represented by the related behavior feature.

[0074] For example, in the example of Fig. 5 described above, the classifier 44e identifies "a dangerous behavior in which the bicycle on the near side, which is running unsteadily, is heading in the direction of travel of the host vehicle, which is trying to turn left, relative to the host vehicle, which is trying to turn left, from the relationship between the movements of the "bicycle on the near side" and "host vehicle (mobile body V)" as a type of behavior that occurs when the movements of the moving body V and the movements of an object are related, and identifies "a non-dangerous behavior in which the bicycle on the far side, which is running unsteadily, is heading in a direction different from the direction of travel of the host vehicle, relative to the host vehicle, which is trying to turn left, relative to the relationship between the movements of the "bicycle on the far side" and "host vehicle (mobile body V)" as a type of behavior that occurs when the movements of the moving body V and the movements of an object are related, respectively.

[0075] As another example, in the example of Fig. 8, the classifier 44e identifies, as a type of behavior occurring due to the relationship between the movement of the moving body V and the movement of an object, for example, "safe behavior of the own vehicle stopping on the stop line at a red light, which is not a traffic violation," from the relationship between the movement of "red light," "stop line," and "own vehicle (moving body V)." Meanwhile, in the example of Fig. 9, the classifier 44e identifies, as a type of behavior occurring due to the relationship between the movement of the moving body V and the movement of an object, for example, "dangerous behavior of the own vehicle not stopping on the stop line at a red light (ignoring a signal)," which is a traffic violation, from the relationship between the movement of "red light," "stop line," and "own vehicle (moving body V)."

[0076] The classifier 44e stores the quantified value of the output type of the behavior in the storage circuit 30 as type identification result data (output data) D5.

[0077] The output unit 45 is a part having a function capable of executing a process of performing an output based on the result of the identification of the type of behavior by the type identification unit 44. The output unit 45 of this embodiment is capable of executing a process of outputting the type identification result data D5 identified by the type identification unit 44 via the output device 20. The type identification result data D5 may be output as image information via a display constituting the output device 20, or may be output as sound information via a speaker constituting the output device 20. In addition, the type identification result data D5 may be output to another device outside the situation detection system 1 via a data output device constituting the output device 20. For example, the output unit 45 may output the type identification result data D5 in real time in accordance with the running of the vehicle, or may output the type identification result data D5 at an appropriate timing in response to a user's operation via an operation input device constituting the input device 10.

[0078] As an example, the output unit 45 may be configured to output an alert via the output device 20 when the type identification unit 44 identifies the type of behavior as “dangerous behavior”, i.e., when the type identification result data D5 represents “dangerous behavior”.

[0079] Next, the processing procedure of the situation detection method in the situation detection system 1 will be described with reference to the flowchart of FIG.

[0080] 10 includes an acquisition step (step S1), a model generation step (steps S2 to S5), an input step (step S6), a type identification step (step S7), an output step (step S8), a determination step (step S9), and an alarm step (step S10). Here, the processing for each of the above steps is executed by a processing circuit 40 of the situation detection system 1.

[0081] First, the preprocessing unit 41 of the processing circuit 40 executes a step (step S1) of acquiring the first training data set D31, the second training data set D32, the third training data set D33, and the fourth training data set D34 as the training data set D3. In this case, the preprocessing unit 41 may directly acquire the first training data set D31, the second training data set D32, the third training data set D33, and the fourth training data set D34 from another device outside the situation detection system 1 via the input device 10. In addition, the preprocessing unit 41 may create and acquire the first training data set D31, the second training data set D32, the third training data set D33, and the fourth training data set D34 by performing various preprocessing on the raw data D0 input from another device outside the situation detection system 1 via the input device 10. The preprocessing unit 41 stores in the memory circuitry 30 the acquired first training data set D31, the second training data set D32, the third training data set D33, and the fourth training data set D34.

[0082] Next, the model generation unit 42 of the processing circuit 40 executes a model generation step (steps S2 to S5) to generate a trained model M by machine learning using the multiple learning data sets D3 acquired in the acquisition step (step S1).

[0083] More specifically, the object detection model generation unit 42a of the model generation unit 42 executes an object detection model generation step (step S2) of generating a trained object detection model M1 by machine learning using a plurality of first learning datasets D31. The behavior detection model generation unit 42b of the model generation unit 42 executes a behavior detection model generation step (step S3) of generating a trained behavior detection model M2 by machine learning using a plurality of second learning datasets D32. The related behavior detection model generation unit 42c of the model generation unit 42 executes a related behavior detection model generation step (step S4) of generating a trained related behavior detection model M3 by machine learning using a plurality of third learning datasets D33. The type-specific model generation unit 42d of the model generation unit 42 executes a type-specific model generation step (step S5) of generating a trained type-specific model M4 by machine learning using a plurality of fourth learning datasets D34.

[0084] The model generation unit 42 stores the generated trained models M (trained object detection model M1, trained behavior detection model M2, trained relational behavior detection model M3, trained type identification model M4) in the storage circuitry 30. At this time, if a previously generated trained model M has already been stored in the storage circuitry 30, the model generation unit 42 replaces the stored trained model M with the newly generated trained model M.

[0085] Next, the detection target input unit 43 of the processing circuit 40 executes an input step (step S6) of inputting the moving image data D41 and the measurement data D42 as the detection target data D4 to be detected to the type identification unit 44 of the processing circuit 40. In this case, the detection target input unit 43 may input the detection target data D4 received from another device outside the situation detection system 1 via the input device 10, or may input the detection target data D4 received by a user operation via the input device 10. In addition, the detection target input unit 43 may temporarily store the input detection target data D4 in the storage circuit 30.

[0086] Next, the type identification unit 44 of the processing circuit 40 executes a type identification step (step S7) of identifying a type of behavior that occurs in relation between the movement of the moving object V and the movement of an object in a still image that constitutes the moving image from the moving image data D41 and the measurement data D42 (detection target data D4) input in the input step (step S6) using the trained object detection model M1, trained behavior detection model M2, trained relational behavior detection model M3, and trained type identification model M4 generated in the model generation step (steps S2 to S5). In this case, the type identification unit 44 identifies the type of the above-mentioned behavior from the moving image data D41 and the measurement data D42 by the object detection feature extractor 44a, the integration unit 44b, the behavior detection feature extractor 44c, the relational behavior feature extractor 44d, and the classifier 44e working together. The type identification unit 44 stores a value that quantifies the type of the identified behavior in the storage circuit 30 as type identification result data D5.

[0087] Next, the output unit 45 of the processing circuit 40 executes an output step (step S8) of outputting type identification result data D5 for the type of behavior identified in the type identification step (step S7). In this case, the output unit 45 may output the type identification result data D5 as image information or sound information via the output device 20, or may output the type identification result data D5 via the output device 20 to another device outside the situation detection system 1.

[0088] Here, the output unit 45 further executes a determination step (step S9) of determining whether or not the type of behavior identified by the type identification unit 44 is a behavior that requires an alarm, such as "dangerous behavior." If the output unit 45 determines that the type of behavior identified by the type identification unit 44 is not a behavior that requires an alarm (step S9: No), the output unit 45 ends the processing according to this flowchart. If the output unit 45 determines that the type of behavior identified by the type identification unit 44 is a behavior that requires an alarm (step S9: Yes), the output unit 45 executes an alarm step (step S10) of outputting an alarm via the output device 20, and ends the processing according to this flowchart.

[0089] The above-mentioned situation detection method can be realized by executing a prepared situation detection program on a computer such as a personal computer, a workstation, etc. This situation detection program causes the computer to execute each process of the above-mentioned acquisition step (step S1), model generation step (steps S2 to S5), input step (step S6), type identification step (step S7), output step (step S8), judgment step (step S9), and alarm step (step S10).

[0090] The situation detection system 1, the situation detection method, and the situation detection program described above can perform object detection by extracting "object detection features" representing objects detected in "video data" from the "video data" using the learned object detection model M1. Also, the situation detection system 1, the situation detection method, and the situation detection program can perform behavior detection by extracting "mobile object behavior detection features" representing the movement of a moving object and "object behavior detection features" representing the movement of an object from "measurement data and object detection features" using the learned behavior detection model M2. Then, the situation detection system 1, the situation detection method, and the situation detection program can perform related behavior detection by extracting "related behavior features" representing behaviors that occur in relation to the movement of a moving object and the movement of an object from the "mobile object behavior detection features and object behavior detection features" extracted as described above using the learned related behavior detection model M3. Then, the situation detection system 1, the situation detection method, and the situation detection program can identify "the type of the behavior" from this "related behavior feature" using the learned type identification model M4.

[0091] As a result, the situation detection system 1, the situation detection method, and the situation detection program can detect the relationship between the moving body V or the objects detected from the video image by using related behavior detection in addition to object detection and behavior detection, and can detect the meaning (type) of their mutual behavior by relating their movements. This allows the situation detection system 1, the situation detection method, and the situation detection program to perform appropriate situation detection, for example, more precisely.

[0092] The situation detection system 1, the situation detection method, and the situation detection program can apply the results of more precise situation detection obtained as described above to, for example, a real-time safe driving diagnosis system, an education system, a risk prediction system, an automatic driving system, etc. In this case, the situation detection system 1, the situation detection method, and the situation detection program can be used for various purposes, such as real-time driving warning, evaluation of driving skills, detection and evaluation of dangerous driving and traffic violations, guidance and education on risk suppression behavior, creation of a safe driving plan, and reflection in automatic driving.

[0093] For example, the situation detection system 1, the situation detection method, and the situation detection program can use related behavior detection technology to detect related behaviors such as prying open a vehicle door or breaking into a vehicle using a tool such as a bat or crowbar, and issue an alarm, thereby making it possible to utilize them for crime prevention, etc. Furthermore, the situation detection system 1, the situation detection method, and the situation detection program can be more suitably utilized for crime prevention, etc., by also using video data from security cameras installed on structures such as buildings and streetlights.

[0094] Furthermore, the situation detection system 1, the situation detection method, and the situation detection program can also be applied to a vehicle dispatch service system for taxis, etc., by using, for example, related behavior detection technology. In this case, the situation detection system 1, the situation detection method, and the situation detection program can detect the intention to ride in a taxi based on the behavior of a person on the roadside staring in the direction a vehicle is coming or the behavior of raising one's hand toward a vehicle, for example, by performing related behavior detection on the movements of people around the vehicle and the movement of the vehicle, and can be utilized for a vehicle dispatch service.

[0095] Furthermore, the situation detection system 1, the situation detection method, and the situation detection program can be applied to a parking lot vacant parking lot information service and vacant parking lot prediction information service by using related behavior detection technology, for example. Also, the situation detection system 1, the situation detection method, and the situation detection program can detect related behavior such as a person finishing loading luggage or getting into a car in a full parking lot by also using video image data from a camera or the like installed in the parking lot, and can identify a situation where a vacant parking lot is likely to become available in the next few minutes, and can be used for a guidance service for registered users, for example.

[0096] In the embodiment described above, an example has been described in which the situation detection system 1 performs both the learning phase and the usage phase in a consolidated manner in one system, but the embodiment is not limited to this.

[0097] For example, a situation detection system 1A according to a modified example illustrated in Fig. 11 differs from the above-described situation detection system 1 in that it is configured to be divided into a trained model generation system 100 that executes each process in the learning phase and a situation detection device 200 that executes each process in the usage phase. The situation detection device 200 may be mounted on a moving body V together with, for example, a camera 50, a measuring device 51, etc. On the other hand, the trained model generation system 100 may be installed in, for example, a business establishment that operates the moving body V.

[0098] The trained model generation system 100 includes an input device 110, an output device 120, a memory circuitry 130, and a processing circuitry 140, and performs processing to generate a trained model M by machine learning using a training dataset D3. The processing circuitry 140 conceptually includes a preprocessing unit 141 and a model generation unit 142 in order to realize the above-mentioned various processing functions.

[0099] The pre-processing unit 141 can execute a process of acquiring a training data set D3 (a first training data set D31, a second training data set D32, a third training data set D33, and a fourth training data set D34) in the same manner as the above-described pre-processing unit 41. The pre-processing unit 141 stores the acquired multiple training data sets D3 in the memory circuitry 130.

[0100] The model generation unit 142, like the above-mentioned model generation unit 42, can execute a process of generating a trained model M by machine learning using a plurality of training data sets D3 acquired by the preprocessing unit 141. Like the above-mentioned model generation unit 42, the model generation unit 142 is functionally configured to include an object detection model generation unit 142a, a behavior detection model generation unit 142b, a related behavior detection model generation unit 142c, and a type specific model generation unit 142d. The model generation unit 142 stores the generated trained models M (trained object detection model M1, trained behavior detection model M2, trained related behavior detection model M3, trained type specific model M4) in the storage circuit 130.

[0101] The situation detection device 200 includes an input device 210, an output device 220, a memory circuit 230, and a processing circuit 240, and performs processing to identify the type of behavior that occurs in relation to the movement of a moving body V and the movement of an object in a still image that constitutes a moving image, using a trained model M. The processing circuit 240 is functionally configured to include a detection target input unit 243, a type identification unit 244, and an output unit 245 in order to realize the various processing functions described above.

[0102] The detection target input unit 243, like the detection target input unit 43 described above, is capable of executing a process of inputting moving image data D41 and measurement data D42 that become detection target data D4 to be detected.

[0103] The type identification unit 244, like the above-mentioned type identification unit 44, can execute a process of identifying the type of behavior occurring in relation to the movements of multiple objects including the moving object V from the detection target data D4 input by the detection target input unit 243, using the trained model M. In this case, the type identification unit 244 can use, for example, the trained model M (trained object detection model M1, trained behavior detection model M2, trained relational behavior detection model M3, trained type identification model M4) previously stored in the storage circuit 230 via the output device 120 of the trained model generation system 100 and the input device 210 of the situation detection device 200. This trained model M is a model generated by the trained model generation system 100 as described above.

[0104] The output unit 245 is capable of executing a process of outputting the type identification result data D5 identified by the type identification unit 44 via the output device 220, similar to the output unit 45 described above.

[0105] The other configurations of the input devices 110, 210, the output devices 120, 220, the memory circuits 130, 230, and the processing circuits 140, 240 are substantially the same as those of the input device 10, the output device 20, the memory circuit 30, and the processing circuit 40 described above. In this modification, a moving image of the surroundings of the moving body V is captured, for example, by a camera 50 mounted on the moving body V, and input from the camera 50 to the input device 210 as moving image data. Similarly, a measurement value according to the state of the moving body V is measured, for example, by a measuring device 51 mounted on the moving body V, and input from the measuring device 51 to the input device 210 as moving image data.

[0106] Even in this case, the situation detection system 1A, the trained model generation system 100, and the situation detection device 200 can appropriately perform situation detection, similar to the situation detection system 1 described above, and can, for example, perform situation detection more precisely.

[0107] In addition, in the case of this modified example, the trained model M used in the situation detection device 200 is not limited to a model generated by the trained model generation system 100 as described above, but may be a trained model M generated by another system.

[0108] [Embodiment 2] The situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device according to the second embodiment further differ from the first embodiment in that they use distance data. In the following, the same components as those in the above-mentioned embodiments are given the same reference numerals, and duplicated explanations of the common configurations, actions, and effects are omitted as much as possible (the same applies below).

[0109] 12 and 13 differs from the above-described situation detection system 1 in that the situation detection system 1B of this embodiment performs related behavior detection using a distance change feature that indicates a change in distance between a moving body V and an object surrounding the moving body V, in addition to a moving body behavior detection feature and an object behavior detection feature. Here, the situation detection system 1B differs from the above-described situation detection system 1 in that the processing circuit 40 includes a model generation unit 42B and a type identification unit 44B instead of the model generation unit 42 and the type identification unit 44. The other configurations of the situation detection system 1B are substantially similar to those of the above-described situation detection system 1.

[0110] The input device 10 of this embodiment receives at least moving image data, measurement data, and distance data from various devices.

[0111] The distance data is data representing the distance between the moving body V and surrounding objects (e.g., vehicles, pedestrians, bicycles, obstacles, street lights, signs, utility poles, signs, stop lines, etc.) measured in conjunction with the capture of a moving image. The distance data is typically detected in the moving body V in synchronization with the capture of a moving image. The distance between the moving body V and the surrounding objects is detected, for example, by a distance sensor 52 mounted on the moving body V, and input from the distance sensor 52 to the input device 10 as distance data. The distance sensor 52 is, for example, mounted on the moving body V and is composed of various radars or sonars that detect distance using infrared rays, millimeter waves, ultrasonic waves, etc. In addition, the distance sensor 52 may be shared by, for example, the camera 50, and detect the distance by performing various processes on the moving image data captured by the camera 50.

[0112] The second learning dataset D32 of this embodiment includes the above-mentioned distance data in addition to the measurement data and the object detection feature, teacher data on the moving object behavior detection feature, and teacher data on the object behavior detection feature as well as the distance change feature. Here, the distance change feature is a feature representing a change in the distance between the moving object V and the surrounding objects with the time change of the distance data, and represents a feature of the change in the distance according to the movement of the moving object V and the surrounding objects with the time change. Here, the second learning dataset D32 is configured by linking "measurement data, object detection feature, and distance data" with "teacher data on the moving object behavior detection feature corresponding to the measurement data, teacher data on the object behavior detection feature corresponding to the object detection feature, and teacher data on the distance change feature corresponding to the distance data" as one set. In other words, the second learning dataset D32 of this embodiment consists of the measurement data, the object detection features, and the distance data quantified as input data D1 (explanatory variables), and teacher data regarding the moving object behavior detection features, teacher data regarding the object behavior detection features, and teacher data regarding the distance change features quantified as teacher data D2 (objective variables).

[0113] The third learning data set D33 of this embodiment is composed of a moving object behavior detection feature, an object behavior detection feature, a distance change feature, and teacher data related to a related behavior feature. The third learning data set D33 is composed of a set of "moving object behavior detection feature, object behavior detection feature, and distance change feature" and "teacher data related to the moving object behavior detection feature, the object behavior detection feature, and the related behavior feature corresponding to the distance change feature." That is, the third learning data set D33 of this embodiment is composed of the moving object behavior detection feature, the object behavior detection feature, and the distance change feature quantified as input data D1 (explanatory variable), and teacher data related to the related behavior feature quantified as teacher data D2 (target variable).

[0114] The model generation unit 42B of the present embodiment differs from the above-described model generation unit 42 in that it includes a behavior detection model generation unit 42Bb and a related behavior detection model generation unit 42Bc instead of the behavior detection model generation unit 42b and the related behavior detection model generation unit 42c. The other configurations of the model generation unit 42B are substantially the same as those of the above-described model generation unit 42.

[0115] The behavior detection model generation unit 42Bb generates a learned behavior detection model M2 by machine learning using a plurality of second learning data sets D32 acquired by the preprocessing unit 41. The behavior detection model generation unit 42Bb of this embodiment performs machine learning of the learned behavior detection model M2 using the "measurement data, object detection feature, and distance data" of the second learning data set D32 as input data D1 (explanatory variables) and the "teacher data on moving object behavior detection feature, teacher data on object behavior detection feature, and teacher data on distance change feature" as teacher data D2 (target variable). As a result of the machine learning, the behavior detection model generation unit 42Bb generates a learned behavior detection model M2 that has been subjected to machine learning for extracting moving object behavior detection feature, object behavior detection feature, and distance change feature from the measurement data, object detection feature, and distance data. The learned behavior detection model M2 generated by the behavior detection model generation unit 42Bb is a model whose inputs are “measurement data, object detection features, and distance data” and whose output is “mobile object behavior detection features, object behavior detection features, and distance change features.”

[0116] In addition, similar to the above-mentioned behavior detection model generation unit 42b, the behavior detection model generation unit 42Bb may use a plurality of second learning data sets D32 to individually generate a plurality of sub-models (a moving object behavior detection sub-model, an object behavior detection sub-model, and a distance change detection sub-model) by machine learning as the learned behavior detection model M2.

[0117] The relational behavior detection model generation unit 42Bc can execute a process of generating a learned relational behavior detection model M3 by machine learning using a plurality of second learning data sets D32 acquired by the preprocessing unit 41. The relational behavior detection model generation unit 42Bc of this embodiment performs machine learning of the learned relational behavior detection model M3 using the "moving object behavior detection feature, object behavior detection feature, and distance change feature" of the third learning data set D33 as input data D1 (explanatory variables) and the "teacher data regarding the relational behavior feature" as teacher data D2 (target variable). As a result of the machine learning, the relational behavior detection model generation unit 42Bc generates a learned relational behavior detection model M3 that has been machine-learned to extract a relational behavior feature from the moving object behavior detection feature, object behavior detection feature, and distance change feature. The learned relational behavior detection model M3 generated by the relational behavior detection model generation unit 42Bc is a model in which the input is the "moving object behavior detection feature, object behavior detection feature, and distance change feature" and the output is the "relational behavior feature."

[0118] In the use phase, the detection target input unit 43 of this embodiment inputs the moving image data D41, the measurement data D42, and the distance data D43 to the type identification unit 44B as detection target data D4 to be detected.

[0119] In the usage phase, the type identification unit 44B of this embodiment uses the learned object detection model M1 generated by the object detection model generation unit 42a, the learned behavior detection model M2 generated by the behavior detection model generation unit 42b, the learned related behavior detection model M3 generated by the related behavior detection model generation unit 42c, and the learned type identification model M4 generated by the type identification model generation unit 42d to identify the type of the above-mentioned behavior from the video data D41, the measurement data D42, and the distance data D43 input by the detection target input unit 43.

[0120] Specifically, the type identification unit 44B of the present embodiment includes an integration unit 44Bb, a behavior detection feature extractor 44Bc, and a related behavior feature extractor 44Bd, instead of the integration unit 44b, the behavior detection feature extractor 44c, and the related behavior feature extractor 44d. The other configurations of the type identification unit 44B are substantially the same as those of the type identification unit 44 described above.

[0121] The integration unit 44Bb integrates the "object detection feature D61" input by the object detection feature extractor 44a and the "measurement data D42" and "distance data D43" of the detection target data D4 input by the detection target input unit 43 so as to form a combination of data at the same time in a chronological order, stores the combination in the memory circuit 30, and also inputs it to the next-stage behavior detection feature extractor 44Bc.

[0122] The behavior detection feature extractor 44Bc uses the trained behavior detection model M2 generated by the behavior detection model generation unit 42Bb to extract a moving object behavior detection feature D62, an object behavior detection feature D63, and a distance change feature D65 from the measurement data D42, the object detection feature D61, and the distance data D43. The behavior detection feature extractor 44Bc inputs the "measurement data D42, object detection feature D61, and distance data D43" input by the integration unit 44Bb at the same time in time series as input data to the trained behavior detection model M2, and outputs the "moving object behavior detection feature D62, object behavior detection feature D63, and distance change feature D65" from the trained behavior detection model M2 in response to this. The behavior detection feature extractor 44Bc stores the output mobile object behavior detection feature D62, object behavior detection feature D63, and distance change feature D65 in the memory circuit 30, and also inputs them to the next-stage related behavior feature extractor 44Bd.

[0123] The related behavior feature extractor 44Bd extracts a related behavior feature D64 from the moving object behavior detection feature D62, the object behavior detection feature D63, and the distance change feature D65 using the learned related behavior detection model M3 generated by the related behavior detection model generation unit 42Bc. The related behavior feature extractor 44Bd inputs the "moving object behavior detection feature D62, object behavior detection feature D63, and distance change feature D65" input by the behavior detection feature extractor 44Bc as input data to the learned related behavior detection model M3, and outputs the "related behavior feature D64" from the learned related behavior detection model M3 in response to the input data. The related behavior feature extractor 44Bd stores the output related behavior feature D64 in the memory circuit 30 and inputs it to the next stage classifier 44e.

[0124] The classifier 44e inputs the “related behavior feature D64” input by the related behavior feature extractor 44Bd as input data to the learned type-specific model M4 generated by the type-specific model generation unit 42d, and in response to this, causes the learned type-specific model M4 to output a “quantified value of the type of the above-mentioned behavior.”

[0125] Even in this case, the situation detection system 1B, the situation detection method, and the situation detection program can appropriately detect the situation, similar to the situation detection system 1 described above, and can, for example, perform more precise situation detection.

[0126] Here, the situation detection system 1B, the situation detection method, and the situation detection program described above can extract "moving object behavior detection features, object behavior detection features, and distance change features" from "measurement data, object detection features, and distance data" using the learned behavior detection model M2. Then, the situation detection system 1, the situation detection method, and the situation detection program can extract "related behavior features" based on "moving object behavior detection features, object behavior detection features, and distance change features" using the learned related behavior detection model M3. Then, the situation detection system 1, the situation detection method, and the situation detection program can identify the type of the behavior from the related behavior features using the learned type identification model M4. As a result, the situation detection system 1, the situation detection method, and the situation detection program can perform higher-level situation detection based on the distance between the moving object V and the surrounding objects of the moving object V.

[0127] The situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device according to the above-described embodiments of the present invention are not limited to the above-described embodiments, and various modifications are possible within the scope of the claims.

[0128] For example, in the situation detection system 1 and the situation detection system 1B described above, the output unit 45 may be configured to determine whether or not an alarm is required based on the distance data input by the detection target input unit 43 as well.

[0129] In this case, the output unit 45 executes, after the output step (step S8), a determination step (step S9-1) of determining whether the type of behavior identified by the type identification unit 44, 44B is "dangerous behavior" or the like and whether there is a candidate for a warning target in the "dangerous behavior" or not, as illustrated in Fig. 14. When the output unit 45 determines that the type of behavior identified by the type identification unit 44 is not "dangerous behavior" or the like or that there is no candidate for a warning target in the "dangerous behavior" (step S9-1: No), the process according to this flowchart ends.

[0130] On the other hand, when the output unit 45 determines that the type of behavior identified by the type identification units 44, 44B is "dangerous behavior" or the like and that the "dangerous behavior" has a candidate that is the target of an alarm (step S9-1: Yes), the output unit 45 determines whether or not the distance between the moving object V and the candidate that is the target of an alarm is close based on the distance data (step S9-2). The output unit 45 can determine whether or not the distance between the moving object V and the candidate that is the target of an alarm (surrounding object) is close by, for example, determining whether or not the distance between the moving object V and the candidate that is the target of an alarm (surrounding object) is equal to or less than a preset determination threshold.

[0131] When the output unit 45 determines that the distance between the moving object V and the candidate for the warning is far (step S9-2: No), it determines that a warning is not necessary and ends the processing according to this flowchart. On the other hand, when the output unit 45 determines that the distance between the moving object V and the candidate for the warning is close (step S9-2: Yes), it determines that a warning is necessary and executes a warning step (step S10) of outputting a warning via the output device 20 and ends the processing according to this flowchart.

[0132] In this case, the situation detection systems 1 and 1B can more reliably suppress unnecessary warnings and can output warnings appropriately in accordance with the detected situation.

[0133] The model generation units 42, 42B, and 142 described above have been described as generating the trained model M by machine learning with teacher data, but this is not limiting, and the trained model M may be generated in part or in whole by machine learning without teacher data to the extent possible. For example, the behavior detection model generation units 42b, 42Bb, and 142b may use a so-called "LSTM Autoencoder" to generate the trained behavior detection model M2 by machine learning without teacher data that does not use teacher data related to mobile object behavior detection features.

[0134] The processing circuits 40, 140, and 240 described above have been described as having each processing function realized by a single processor, but this is not limited to the above. The processing circuits 40, 140, and 240 may be configured by combining multiple independent processors and having each processor execute a program to realize each processing function. The processing functions of the processing circuits 40, 140, and 240 may be appropriately distributed or integrated and realized in a single or multiple processing circuits. The processing functions of the processing circuits 40, 140, and 240 may be realized in whole or in any part by a program, or may be realized as hardware using wired logic or the like.

[0135] The programs executed by the processors described above are provided by being preinstalled in the storage circuits 30, 130, 230, etc. The programs may be provided by being recorded in a computer-readable storage medium in a format that can be installed in these devices or in a format that can be executed. The programs may also be provided or distributed by being stored on a computer connected to a network such as the Internet and downloaded via the network.

[0136] The situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device according to the present embodiment may be configured by appropriately combining the components of the embodiments and modified examples described above. [Explanation of symbols]

[0137] 1, 1A, 1B Situation Detection System 10, 110, 210 Input device 20, 120, 220 output device 30, 130, 230 memory circuit 40, 140, 240 Processing circuit 41, 141 Pretreatment section 42, 42B, 142 Model generation section 42a, 142a Object detection model generation unit 42b, 42Bb, 142b Behavior detection model generation unit 42c, 42Bc, 142c Related behavior detection model generation part 42d, 142d Type-specific model generation unit 43, 243 Detection target input section 44, 44B, 244 Type specific part 44a Object detection feature extractor 44b, 44Bb Integrated Section 44c, 44Bc Behavior detection feature extractor 44d, 44Bd Relational behavior feature extractor 44e classifier 45, 245 Output section 50 Camera 51 Measuring Instruments 52 Distance Sensor 100 Trained Model Generation System 200 Situation Detection Device AL Machine Learning Algorithm D0 Raw data D1 Input data D2 Teacher data D3 Training Dataset D31 First training dataset D32 Second training dataset D33 Third training dataset D34 4th training dataset D4 Detection target data D41 Video data D42 Measurement data D43 Distance Data D5 Type-specific result data D61 Object detection features D62 Mobile behavior detection features D63 Object and Action Detection Features D64 Relational behavior feature D65 Distance change feature M trained models M1 pre-trained object detection model M2 trained behavior detection model M3 trained relational behavior detection model M4 Trained species-specific model V Mobile

Claims

1. a pre-processing unit that acquires a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data; a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object accompanying a time change in the measurement data, and teacher data related to object behavior detection features representing the movement of the object accompanying a time change in the object detection features; a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object; and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior. an object detection model generation unit that generates, by learning, a trained object detection model that extracts the object detection feature from the video data, using the plurality of first training data sets acquired by the preprocessing unit; a behavior detection model generation unit that generates, by learning, a trained behavior detection model that extracts the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the second learning data sets acquired by the preprocessing unit; a related behavior detection model generation unit that generates, by learning, a trained related behavior detection model that extracts the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the third learning data sets acquired by the preprocessing unit; a type identification model generation unit that generates, by learning, a trained type identification model that identifies a type of the behavior from the related behavior feature by using the plurality of fourth learning data sets acquired by the preprocessing unit; a detection target input unit for inputting the moving image data and the measurement data to be detected; a type identification unit that identifies a type of the behavior from the video data and the measurement data input by the detection target input unit, using the trained object detection model generated by the object detection model generation unit, the trained behavior detection model generated by the behavior detection model generation unit, the trained related behavior detection model generated by the related behavior detection model generation unit, and the trained type identification model generated by the type identification model generation unit. Situation detection system.

2. the second learning data set includes, in addition to the measurement data and the object detection feature, distance data representing a distance between the moving object and a surrounding object of the moving object measured in conjunction with capturing the video, teacher data regarding the moving object behavior detection feature, and teacher data regarding a distance change feature representing a change in the distance associated with a change in the distance data over time in addition to the teacher data regarding the object behavior detection feature, the third learning data set includes the moving object behavior detection feature, the object behavior detection feature, the distance change feature, and training data regarding the related behavior feature, the behavior detection model generation unit generates, by learning, the trained behavior detection model that extracts the moving object behavior detection feature, the object behavior detection feature, and the distance change feature from the measurement data, the object detection feature, and the distance data, using the plurality of second learning data sets acquired by the preprocessing unit; the related behavior detection model generation unit generates, by learning, a trained related behavior detection model that extracts the related behavior feature from the moving object behavior detection feature, the object behavior detection feature, and the distance change feature, using the third learning data sets acquired by the preprocessing unit; the detection target input unit inputs the moving image data, the measurement data, and the distance data that are to be detected; the type identification unit identifies the type of the behavior from the video data, the measurement data, and the distance data input by the detection target input unit, using the trained object detection model generated by the object detection model generation unit, the trained behavior detection model generated by the behavior detection model generation unit, the trained related behavior detection model generated by the related behavior detection model generation unit, and the trained type identification model generated by the type identification model generation unit; The situation detection system according to claim 1 .

3. acquiring a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data; a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object associated with a time change in the measurement data, and teacher data related to object behavior detection features representing the movement of the object associated with a time change in the object detection features; a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object; and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior; generating, by learning, a trained object detection model that extracts the object detection feature from the video data using the acquired plurality of first training data sets; generating a trained behavior detection model by learning, the trained behavior detection model extracting the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the acquired second training data sets; generating a learned related behavior detection model by learning, the learned related behavior detection model extracting the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the acquired third learning data sets; generating a trained type identification model by learning, the trained type identification model identifying a type of the behavior from the related behavior feature by using the acquired plurality of fourth learning data sets; inputting the moving image data and the measurement data to be detected; and identifying a type of the behavior from the input video data and the measurement data using the generated trained object detection model, trained behavior detection model, trained relational behavior detection model, and trained type identification model. How to detect the situation.

4. obtain a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data; a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object associated with a time change in the measurement data, and teacher data related to object behavior detection features representing the movement of the object associated with a time change in the object detection features; a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object; and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior; generating, by learning, a trained object detection model that extracts the object detection feature from the video data using the acquired plurality of first training data sets; generating a trained behavior detection model by learning, the trained behavior detection model extracting the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the acquired plurality of second learning data sets; generating a trained related behavior detection model by learning, the trained related behavior detection model extracting the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the acquired third training data sets; generating, by learning, a trained type identification model that identifies a type of the behavior from the related behavior feature using the acquired plurality of fourth learning data sets; inputting the moving image data and the measurement data to be detected; identifying a type of the behavior from the input video data and the measurement data using the generated trained object detection model, trained behavior detection model, trained relational behavior detection model, and trained type identification model; Each process is executed by a computer. Situation detection program.

5. a pre-processing unit that acquires a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data; a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object accompanying a time change in the measurement data, and teacher data related to object behavior detection features representing the movement of the object accompanying a time change in the object detection features; a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object; and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior. an object detection model generation unit that generates, by learning, a trained object detection model that extracts the object detection feature from the video data, using the plurality of first training data sets acquired by the preprocessing unit; a behavior detection model generation unit that generates, by learning, a trained behavior detection model that extracts the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the second learning data sets acquired by the preprocessing unit; a related behavior detection model generation unit that generates, by learning, a trained related behavior detection model that extracts the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the third learning data sets acquired by the preprocessing unit; a type identification model generation unit that generates, by learning, a trained type identification model that identifies a type of the behavior from the related behavior feature using the plurality of fourth learning data sets acquired by the preprocessing unit. A trained model generation system.

6. a detection target input unit that inputs moving image data representing a moving image captured from a moving object and measurement data representing a measurement value corresponding to a state of the moving object measured in conjunction with capturing the moving image; the detection target input unit includes a trained object detection model that has been trained to extract from the video data an object detection feature that represents an object detected in the video data; a trained behavior detection model that has been trained to extract from the measurement data and the object detection feature a mobile object behavior detection feature that represents the movement of the mobile object accompanying a time change in the measurement data and an object behavior detection feature that represents the movement of the object accompanying a time change in the object detection feature; a trained related behavior detection model that has been trained to extract from the mobile object behavior detection feature and the object behavior detection feature a related behavior feature that represents a behavior occurring in relation to the movement of the mobile object and the movement of the object; and a type identification unit that identifies the type of the behavior from the video data and the measurement data input by the detection target input unit using a trained type identification model that has been trained to identify the type of the behavior from the related behavior feature, the trained object detection model is generated by learning using a plurality of first learning data sets including the video data and teacher data related to the object detection features; the trained behavior detection model is generated by learning using a plurality of second learning data sets including the measurement data and the object detection feature, teacher data regarding the mobile object behavior detection feature, and teacher data regarding the object behavior detection feature; the trained related behavior detection model is generated by learning using a plurality of third learning data sets including the mobile object behavior detection feature and the object behavior detection feature, and teacher data related to the related behavior feature; The learned type identification model is generated by learning using a plurality of fourth learning data sets including the related behavior feature amount and teacher data related to related behavior type data representing the type of the behavior. Situation detection device.

Citation Information

Patent Citations

  • Target state determination device for movable object and program

    JP2012118741A

  • Information processing device, information processing method and program

    JP2018072940A

  • Real time decision making for autonomous driving vehicle

    JP2020083309A