Information processing system, information processing method, and program

Abstract

To provide an abnormality detection system for flexibly performing a response after notification of abnormality information indicating an abnormality occurred in a vehicle.SOLUTION: An abnormality detection system 2 includes: an abnormality information acquisition section 202 for acquiring abnormality information which indicates an abnormality occurred in a vehicle; a driving state analysis section 205 for analyzing operation information which indicates an operation of a driver in the vehicle so as to generate a driving state based on the analysis result; and a control section 208 for controlling an output of a notification related to the abnormality based on the abnormality information and the driving state.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an anomaly detection system, an anomaly detection method, and a program recording medium. [Background technology]

[0002] Patent Document 1 discloses an in-vehicle monitoring device that notifies passengers about their safety based on both the passenger riding conditions and the vehicle's running conditions in order to prevent passengers from falling inside public transportation vehicles. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-62414 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology exemplified in Patent Document 1 does not assume a flexible response to an executed notification when the notification itself is based on a minor detection or a false detection.

[0005] In view of the above problems, an object of the present invention is to provide an anomaly detection system, an anomaly detection method, and a program recording medium that can provide a means for flexibly taking measures after a notification. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an anomaly detection system comprising an anomaly information acquisition unit that acquires anomaly information indicating an abnormality that has occurred within a vehicle, a driving status analysis unit that analyzes operation information indicating the driver's operations in the vehicle and generates a driving status based on the analysis results, and a control unit that controls the output of a notification regarding the abnormality based on the anomaly information and the driving status.

[0007] According to one aspect of the present invention, there is provided an anomaly detection method, which acquires anomaly information indicating an abnormality that has occurred within a vehicle, analyzes operation information indicating the driver's operation in the vehicle, generates a driving situation based on the analysis results, and controls the output of a notification regarding the abnormality based on the anomaly information and the driving situation.

[0008] According to another aspect of the present invention, there is provided a program recording medium having recorded thereon a program for causing a computer to execute the following processes: acquiring abnormality information indicating an abnormality that has occurred within a vehicle; analyzing operation information indicating the operation of the driver in the vehicle and generating a driving situation based on the analysis results; and controlling the output of a notification regarding the abnormality based on the abnormality information and the driving situation. Effect of the Invention

[0009] According to the present invention, it is possible to provide an anomaly detection system, an anomaly detection method, and a program recording medium that are capable of flexibly taking measures after a notification. [Brief description of the drawings]

[0010] [Figure 1] 1 is a functional block diagram illustrating an example of an anomaly detection system according to a first embodiment. [Diagram 2] 4 is an example of a flowchart showing a process flow of the first embodiment. [Diagram 3] 13 is a functional block diagram of an example of an anomaly detection system according to a second embodiment. FIG. [Figure 4] 13 is an example of a flowchart showing a process flow according to a second embodiment. [Diagram 5] FIG. 13 is a functional block diagram illustrating an example of an anomaly detection system according to a third embodiment. [Figure 6] 13 is an example of a flowchart showing a process flow according to a third embodiment. [Figure 7] 13 is an example of a flowchart showing a process flow according to a fourth embodiment. [Figure 8] 2 illustrates an example of a hardware configuration of an anomaly detection system in each embodiment. [Figure 9] 1 illustrates an example of the configuration of a video analysis system that cooperates with the anomaly detection system in each embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. In the drawings, similar or corresponding elements are designated by the same reference numerals, and descriptions thereof may be omitted or simplified. [First embodiment] The anomaly detection system 1 in this embodiment detects an abnormality occurring inside a vehicle, train, or other transportation facility in which a person is riding, and controls the output of a notification regarding the abnormality. Furthermore, the anomaly detection system in this embodiment cancels the notification in response to the driver's operation or the driving status of the vehicle.

[0012] An "anomaly" occurring inside the vehicle includes an event indicative of an accident. For example, an anomaly may indicate an accident such as a fall of a passenger, a blow, a collision between passengers, etc. In other examples, an anomaly may include an event such as a fire inside the vehicle, the presence of a suspicious object, etc.

[0013] Furthermore, "abnormality" may indicate an event that may be a sign of an accident. For example, it may be a state or behavior of a person not holding on to a handrail inside the vehicle, not sitting in a seat, standing near the boarding / alighting door, moving while the vehicle is moving, crouching, not wearing a seat belt, people standing close to each other, etc. Another example is a person leaving their luggage behind.

[0014] In addition, although the following description will be given using an example in which the anomaly detection system 1 is applied to a vehicle, the application of the anomaly detection system 1 is not limited to this. The anomaly detection system 1 can be applied to transportation means on which people board, such as buses, taxis, general vehicles, trains, airplanes, etc.

[0015] An anomaly detection system 1 according to the present embodiment will be described with reference to Fig. 1. Fig. 1 is a functional block diagram of the anomaly detection system 1 according to the present embodiment. The anomaly detection system 1 includes an anomaly information acquisition unit 101, a driving status analysis unit 102, and a control unit 103.

[0016] The anomaly information acquisition unit 101 acquires anomaly information that is information indicating an anomaly that has occurred inside the vehicle. The anomaly information includes information indicating the presence or absence of an accident or a sign of an accident, the type, the location where the accident occurred, the time, etc. The type of anomaly information may indicate the specific content of the accident or the sign, as exemplified above, in addition to information indicating whether the accident or the sign is an accident itself or a sign.

[0017] A vehicle is equipped with various sensors that detect the driver's operation of the driving device. The various sensors acquire information (operation information) that indicates the driver's operation of the various driving devices, such as the amount of displacement of the steering wheel, the amount of depression of the brake pedal, the operation of the shift lever, etc. If the vehicle control by the driving device is performed electronically, an electrical signal emitted from the driving device may be acquired as the operation information.

[0018] The driving situation analysis unit 102 reads out operation information obtained from the driving device and analyzes it to generate the current driving situation of the vehicle. The driving situation is information including information indicating how the vehicle is moving, such as when the vehicle itself is traveling at a constant speed, accelerating, decelerating, turning right, turning left, or reversing. The driving situation analysis unit 102 reads out the operation information at regular intervals and generates the latest driving situation. The driving situation analysis unit 102 may update the operation information and the driving situation to the latest information at regular intervals.

[0019] The control unit 103 controls the output of a notification regarding an abnormality based on the abnormality information and the driving situation. Specifically, the control unit 103 performs control so as to output a notification based on the abnormality information. Furthermore, in response to an update of the driving situation during the output of the notification, the control unit 103 performs control so as to stop the notification being output according to the updated driving situation. For example, if the driving situation indicates a right turn or acceleration of the vehicle while the notification is being output, it is highly likely that the driver intentionally prioritizes the running of the vehicle over the notification regarding the abnormality. Therefore, if the updated driving situation during the output of the notification indicates a right turn or acceleration, the control unit 103 may perform control so as to stop the notification being output.

[0020] Furthermore, the control unit 103 may control to stop the notification being output based on operation information instead of the driving situation. For example, if the depression amount of the accelerator pedal exceeds a threshold value while a notification is being output from among the operation information, it is highly likely that the driver ignores the notification regarding the abnormality and prioritizes the running of the vehicle. Therefore, when the updated operation information indicates depression of the accelerator pedal, the control unit 103 may control to stop the output of the notification.

[0021] The conditions for the control unit 103 to stop the notification being output are not limited to the above, and may be based on the driving situation indicating a left turn, deceleration, backing, etc., or on the amount of steering wheel displacement or shift lever operation in the operation information. The conditions for stopping the notification being output are not limited to the above examples.

[0022] The process flow of the anomaly detection system 1 in this embodiment will be described with reference to Fig. 2. The anomaly information acquisition unit 101 acquires anomaly information indicating an anomaly that has occurred in the vehicle (S101). Next, the driving situation analysis unit 102 analyzes the operation information acquired by the sensor and generates the driving situation of the vehicle (S102). Next, the control unit 103 controls the output of a notification regarding the anomaly based on the anomaly information and the driving situation (S103).

[0023] The anomaly detection system 1 in this embodiment includes a control unit 103 that controls the output of a notification based on the anomaly information and the driving situation. This is expected to have the effect of preventing the notification from interfering with the driver's driving by flexibly controlling the output of the notification in consideration of the driving situation when the anomaly detected by the sensor is minor or a false detection. [Second embodiment] Next, an anomaly detection system 2 in a second embodiment will be described with reference to Fig. 3. Fig. 3 is a functional block diagram of the anomaly detection system 2 in this embodiment. The anomaly detection system 2 in this embodiment includes an anomaly detection unit 201, an anomaly information acquisition unit 202, a driving information acquisition unit 204, a driving status analysis unit 205, a determination unit 207, a control unit 208, a notification unit 209, and a storage unit 211. Note that in the description of this embodiment, descriptions that overlap with the above-mentioned embodiments will be omitted.

[0024] The abnormality detection unit 201 includes a sensor 2011, and detects an abnormality occurring inside the vehicle based on information acquired by the sensor 2011. The sensor 2011 is, for example, a camera, an infrared sensor, an acceleration sensor, etc. Next, specific processing of the abnormality detection unit 201 will be described using an example in which the sensor 2011 is a camera.

[0025] A case where the sensor 2011 is a camera will be described. The sensor 2011 captures an image of a certain range inside the vehicle. The abnormality detection unit 201 estimates the posture of a person captured in an image captured by the camera. The posture can be estimated by using a known technique. In one example, the joint positions of the person captured in the image are detected, and the posture is estimated based on a change in the joint positions. In another example, a classifier is generated by performing machine learning using a histogram of a person image and a posture indicated by the person image as training data, and the posture of the person is estimated from an image in which the person is captured using the classifier. In addition, various methods for estimating the posture of a person captured in an image can be used for estimating the posture of the person. The abnormality detection unit 201 estimates the posture of a person captured in the camera, and detects the occurrence of an abnormality when the estimated posture is a posture indicating the occurrence of an accident or a predetermined posture leading to a sign of an accident.

[0026] The predetermined postures indicating the occurrence of an accident are postures such as falling, lying down, contact between persons, crouching, running, etc. The predetermined postures indicating a sign of an accident may also be postures such as standing without holding onto a handrail, walking, etc. The predetermined postures may be defined differently while the vehicle is moving and while the vehicle is stopped. For example, actions such as walking without holding onto a handrail are dangerous actions when performed while the vehicle is moving. Therefore, some of the predetermined postures, such as a posture indicating standing without holding onto a handrail and a posture indicating walking, may be detected as postures indicating a sign of an accident while the vehicle is moving.

[0027] Further, the abnormality detection unit 201 detects the presence of a person in a predetermined area such as a boarding / alighting door in the vehicle as an abnormality. More specifically, the abnormality detection unit 201 detects a person from an image captured by the sensor 2011 inside the vehicle. The abnormality detection unit 201 determines whether the detected person is in the predetermined area, and detects an abnormality when the person is in the predetermined area. The predetermined area is a predefined passageway, etc., in addition to the boarding / alighting door of the vehicle. The reason for this is that in a vehicle such as a bus, if a person is present near the boarding / alighting door, there is a risk of collision with a person getting on or off. Also, if a person stands in a passageway without a handrail in the vehicle, there is a risk of falling. The predetermined area is not limited to the above example, and may be a seat set as unavailable, etc., as long as it indicates a predefined position or space.

[0028] The sensor 2011 is not limited to a camera, and may be an infrared sensor, an acceleration sensor, an ultrasonic sensor, or the like. In this case, the sensor 2011 is installed around a predetermined area, and determines whether a person is present in the predetermined area and detects an abnormality. The sensor 2011 may also be installed on a handrail, a strap, or the like in a train, and detect an abnormality when the acceleration applied to the handrail, strap, or the like is equal to or greater than a threshold value.

[0029] The abnormality detection unit 201 may detect an abnormality that occurs inside the vehicle by using a method for determining the presence or absence of danger described in Patent Document 1 (JP Patent Publication 2016-62414 A). The method adopted by the abnormality detection unit 201 is not limited to the above, and other well-known techniques may be used.

[0030] When the abnormality detection unit 201 detects an abnormality, the abnormality information acquisition unit 202 generates abnormality information, which will be described later.

[0031] The abnormality information is information including the presence or absence of an accident or a sign of an accident, the type, the location where the abnormality occurred, the time, etc. The type of abnormality information may indicate the specific content of the accident or the sign, as exemplified above, in addition to information indicating whether the abnormality is an accident itself or a sign.

[0032] The driving information acquisition unit 204 acquires operation information indicating the driving operation of the vehicle, position information of the vehicle itself, and captured images showing the environment around the vehicle using various sensors.

[0033] The driving information acquisition unit 204 acquires operation information at regular intervals using various sensors that detect driving operations.

[0034] The operation information is information indicating the operation of various devices provided in the driving device of the vehicle, such as general driving operations such as turning the steering wheel to the right, turning the steering wheel to the left, depressing the accelerator pedal, depressing the brake pedal, changing the force with which the brake pedal or accelerator pedal is depressed in a direction to accelerate or decelerate the vehicle, and blinking the turn signal.

[0035] The driving device refers to a device that accepts the driver's operations to control the vehicle and controls various operations such as starting, stopping, turning right and left, etc. The various devices that the driving device is equipped with include the steering wheel, accelerator pedal, brake pedal, clutch pedal, gear, shift lever, as well as devices that accept operations to operate the turn signals, wipers, lights, etc.

[0036] The driving information acquisition unit 204 acquires vehicle position information using various positioning sensors. The driving information acquisition unit 204 may acquire the position information by wireless communication in addition to a GPS (Global Positioning System) sensor.

[0037] In addition, the driving information acquisition unit 204 acquires a captured image of the outside of the vehicle using a camera. The captured image is an image showing the surrounding environment such as people walking around the vehicle, other vehicles, signs, traffic lights, etc. The type of camera is not limited to a visible light camera, and may be an infrared camera, a stereo camera, a fisheye camera, etc.

[0038] The driving situation analysis unit 205 analyzes the operation information, position information, and captured images acquired by the driving information acquisition unit 204, and generates a driving situation including a moving situation and environmental information, which will be described later. In addition, the driving situation analysis unit 205 reads out the operation information from the driving information acquisition unit 204 at a constant cycle, and generates the driving situation, thereby updating the latest driving situation.

[0039] Next, the analysis process performed by the driving condition analysis unit 205 will be described.

[0040] The driving situation analysis unit 205 generates a movement situation, which is a part of the driving situation, by analyzing the operation information. The movement situation is information indicating how the vehicle itself is moving, for example, information indicating whether the vehicle itself is traveling at a constant speed, accelerating, decelerating, turning right, turning left, or reversing. The driving situation analysis unit 205 includes a classifier, and generates the movement situation by inputting the operation information to the classifier. The classifier may be generated by machine learning or may be constructed based on a rule base.

[0041] Next, an example of generating a movement situation by the driving situation analysis unit 205 will be described. In general, when a driver turns a vehicle left while driving, the driver performs the action of "turning on a left blinker", performs the action of "depressing the brake pedal", and then performs the action of "turning the steering wheel slightly to the left". Therefore, when the operation information indicates that the actions of "turning on a left blinker", "depressing the brake pedal", and "turning the steering wheel to the left" are performed consecutively or simultaneously, the driving situation analysis unit 205 generates a movement situation indicating "turning left" based on the operation information.

[0042] In another example, when intending to stop, the driver generally performs the action of "turning on the left blinker", "stepping on the brakes", and "turning the steering wheel sharply to the left". Therefore, when the operation information indicates that the actions of "turning on the left blinker", "stepping on the brakes", and "turning the steering wheel to the left" are performed consecutively or simultaneously, the driving situation analysis unit 205 generates a movement situation indicating "intent to stop" based on the operation information.

[0043] At this time, the driving status analysis unit 205 may obtain the amount of steering wheel operation and determine the movement status based on the amount of steering wheel operation in accordance with the above conditions. Comparing the driving operations for "left turn" and "stopping intention", the driver turns the steering wheel to the left significantly when making a "left turn" and to the left slightly when making a "stopping intention". Therefore, in the above example, the driving status analysis unit 205 may generate a movement status indicating "left turn" when the amount of steering wheel operation is relatively large, and may generate a movement status indicating "stopping intention" when the amount of steering wheel operation is relatively small. This allows for more detailed determination of the movement status.

[0044] In yet another example, when changing lanes to the right lane, the driver generally performs the action of "turning on the right turn signal", "pressing the accelerator pedal", and "turning the steering wheel to the right". Therefore, when the operation information indicates that the actions of "turning on the right turn signal", "pressing the accelerator pedal", and "turning the steering wheel to the right" are performed consecutively or simultaneously, the driving situation analysis unit 205 generates a movement situation indicating "changing lanes".

[0045] Examples of the driving situation analysis unit 205 generating a movement situation based on operation information are not limited to the above. For example, the above "stopping intention" is an example assuming stopping on the left side of the road, but depending on the country, stopping on the right side of the road is also possible. In that case, the operation information of "turning the steering wheel to the right" may be used as the basis for generating a movement situation indicating "stopping intention".

[0046] The driving situation analysis unit 205 analyzes the position information and the captured image, and outputs environmental information, which is a part of the driving situation. The environmental information is information indicating the surrounding environment in which the vehicle exists, such as intersections, entrances to expressways, closed areas, restricted areas, congested areas, surroundings of bus stops, etc.

[0047] For example, the driving situation analysis unit 205 compares the position information with existing map information, and outputs environmental information indicating whether the vehicle is located at an intersection, an entrance to a highway, or the like.

[0048] In addition, if the existing map information includes information such as congested areas, closed roads, and the presence or absence of lane restrictions in association with the location of roads, the driving situation analysis unit 205 generates environmental information indicating that the vehicle is in an area subject to congestion, road closure, or other restrictions based on the location information.

[0049] Furthermore, the driving situation analysis unit 205 analyzes objects captured in the captured image, and when a sign indicating a road closure or restriction is captured, generates environmental information indicating that the vehicle is in a restricted area. Furthermore, when a traffic light appears in the captured image, the driving condition analysis unit 205 generates environmental information indicating that the vehicle is located at an intersection.

[0050] The process of the driving situation analysis unit 205 is not limited to the above example, and may output environmental information based on both position information and captured images. For example, the driving situation analysis unit 205 may generate environmental information indicating "just before a crosswalk" based on position information indicating "a point on a single road with no intersection" and a captured image "showing a person crossing the road."

[0051] As described above, the driving situation analysis unit 205 generates a driving situation including a moving situation and environmental information by analyzing the operation information, the position information, and the captured image.

[0052] The determination unit 207 includes a classifier that identifies whether or not the driving situation corresponds to a situation in which notification is stopped. The determination unit 207 uses the classifier to determine whether or not the driving situation corresponds to a situation in which notification is stopped. The determination unit 207 makes a determination every time the driving situation is updated. Note that the classifier may be constructed based on a rule base, or may be one that has been trained by machine learning.

[0053] A case where the determination unit 207 uses a classifier constructed by a rule base will be described. A storage unit (not shown) stores a database in which abnormality information, a driving situation, and whether or not the target of notification suspension is required are associated. More specifically, the database stores information in which abnormality information, a moving situation, environmental information, and whether or not the target of notification suspension are associated. The classifier of the determination unit 207 refers to the database and determines whether or not the driving situation is a predetermined situation that is a target of notification suspension. The predetermined situation that is a target of notification suspension is specified by a combination of the moving situation and the environmental information. For example, the moving situation of the driving situation is a "right turn" and the environmental information indicates an "intersection", the moving situation is a "lane change" and the environmental information indicates an "entrance to a highway", or the moving situation is a "stop intention" and the environmental information indicates "near a bus stop". The predetermined situation may be a situation in which the moving situation of the driving situation indicates "deceleration" or "acceleration" regardless of the environmental situation, or a situation in which the environmental situation indicates an "entrance to a highway" regardless of the driving situation. The predetermined circumstances that are the subject of notification suspension are not limited to the above examples.

[0054] Next, a case where the determination unit 207 uses a classifier obtained by machine learning will be described. The determination unit 207 uses a classifier obtained by machine learning to determine whether or not the driving situation is a predetermined situation that is a target for notification suspension. The classifier is obtained, for example, by performing supervised learning using abnormality information, movement status, and environmental information as input data and whether or not the driving situation is a target for notification suspension as output data. The classifier may also be obtained by performing supervised learning using only one of the movement status or environmental information in addition to the abnormality information as input data and whether or not the driving situation is a target for notification suspension as output data. The determination unit 207 uses the classifier to input abnormality information and the driving situation and outputs whether or not the driving situation is a target for notification suspension.

[0055] The control unit 208 includes a notification control unit 2081. The notification control unit 2081 controls the notification unit 209 to output a notification based on the acquired abnormality information.

[0056] Furthermore, when the driving situation is updated during output of a notification, the notification control unit 2081 performs control so as to stop the notification being output in accordance with the updated driving situation.

[0057] Furthermore, the notification control unit 2081 may control to stop the notification being output based on operation information instead of the driving situation. For example, if the depression amount of the accelerator pedal exceeds a threshold value while the notification is being output from the operation information, the driver is likely to ignore the notification regarding the abnormality and prioritize the running of the vehicle. Therefore, when the updated operation information indicates depression of the accelerator pedal, the control unit 103 may control to stop the output of the notification. The operation information that is the condition for stopping the notification being output may be set appropriately.

[0058] The notification unit 209 outputs notifications such as video and audio under the control of the notification control unit 2081. The notification unit 209 may be a display that the driver can check, a speaker, a tablet, a smartphone, a wearable device, or a combination of these. When the notification unit 209 is the display device 104B such as a display, it displays abnormality information indicating the detected abnormality. For example, when the abnormality detected inside the vehicle is "a person's fall", the notification unit 209 displays information indicating "a person's fall", information indicating that the abnormality is "occurrence of an accident", and the position and time inside the vehicle where the abnormality occurred. Also, when the abnormality detected inside the vehicle is "the presence of a person standing without holding onto a handrail", the notification unit 209 displays information indicating "the presence of a person standing without holding onto a handrail", information indicating that the abnormality is "a sign of an accident", and the position and time inside the vehicle where the abnormality occurred.

[0059] The notification unit 209 may output, as sound, a part or all of the information displayed by the notification unit 209. The notification unit 209 may also output information that is the basis for detecting an abnormality and that is acquired by the sensor 2011. The information that is the basis for detecting an abnormality is video, images, sounds, and other sensor information acquired by the sensor 2011.

[0060] Next, the flow of processing of the anomaly detection system 2 will be described with reference to Fig. 4. The anomaly detection unit 201 detects an anomaly occurring inside the vehicle based on information acquired by the sensor 2011 (S201). In response to the anomaly detection unit 201 detecting an anomaly, the anomaly information acquisition unit 202 generates anomaly information (S202). Based on the anomaly information, the control unit 208 controls the notification unit 209 to output a notification (S203).

[0061] Next, the driving information acquisition unit 204 acquires operation information indicating the driving operation of the vehicle, position information of the vehicle itself, and captured images showing the environment around the vehicle using various sensors (S204). The driving situation analysis unit 205 analyzes the operation information, position information, and captured images acquired by the driving information acquisition unit 204, and outputs the driving situation including the movement situation and environmental information (S205).

[0062] The determination unit 207 uses a discriminator to determine whether or not the driving situation corresponds to a situation in which notification should be stopped (S206). The control unit 208 controls the notification unit 209 to stop the output notification based on the determination result of the determination unit 207 (S207). The notification unit 209 stops the notification under the control of the control unit 208.

[0063] According to the anomaly detection system 2 of this embodiment, even when a minor anomaly is detected or a false detection is made, the notification can be flexibly controlled in consideration of the driving situation. Furthermore, since the anomaly detection system 2 cancels the notification being output in response to the driver's driving operation or the driving situation of the vehicle itself, the driver is not required to perform an extra operation to cancel the notification, and the driving burden can be reduced. [Third embodiment] Next, an abnormality detection system 2 in a third embodiment will be described. The abnormality detection system 2 in this embodiment differs from the above-mentioned embodiments in that the control unit 208 includes a vehicle control unit 2082, and the vehicle control unit 2082 controls the operation of the vehicle so as to invalidate part of the driving operation based on the abnormality information and the judgment result of the judgment unit 207. Note that in the description of this embodiment, descriptions that overlap with the above-mentioned embodiments will be omitted.

[0064] 5 is a functional block diagram of the control unit 208 included in the anomaly detection system 2 in this embodiment. The control unit 208 further includes a vehicle control unit 2082. The vehicle control unit 2082 controls the operation of the vehicle so as to invalidate part of the driving operation based on the anomaly information. Invalidating the driving operation refers to controlling the operation of the vehicle so as not to accept part of the driving operation of the driver.

[0065] For example, if the generated abnormality information indicates "a person has fallen", the vehicle control unit 2082 controls the vehicle so as to disable acceleration caused by the driver's driving operation. Note that in this example, the driving operations to be disabled are limited to those related to acceleration, and deceleration caused by depressing the brake pedal, etc. may be set not to be disabled.

[0066] The abnormality information on which the vehicle control unit 2082 performs control is not limited to the above examples. In addition, it is preferable that the driving operation that the vehicle control unit 2082 disables is limited to minor control such as disabling acceleration. When the vehicle control unit 2082 disables acceleration, the method of disablement may be, in addition to simply disabling an acceleration command by the accelerator pedal, changing the gear of the vehicle to a neutral state, or changing the state to a clutch depressed state in the case of a manual vehicle. Examples of driving operations that are disabled are not limited to these.

[0067] Furthermore, the vehicle control unit 2082 cancels the ongoing vehicle control based on the determination result of the determination unit 207. In other words, the vehicle control unit 2082 executes control of the vehicle so as to invalidate a part of the driving operation of the driver when an abnormality occurs, and to cancel the invalidation of the driving operation when the driving situation satisfies a predetermined situation. Note that, as in the other embodiments, the determination result of the determination unit 207 indicates whether or not the driving situation satisfies a predetermined situation defined in advance for the acquired abnormality information.

[0068] As an example, consider a situation after the vehicle control unit 2082 has disabled acceleration caused by driving operations of the vehicle based on abnormality information. If the driver continuously depresses the accelerator pedal under this situation, there is a high possibility that the driver is trying to ignore the control of the vehicle control unit 2082. If the driver attempts a disabled driving operation, such as continuously depressing the accelerator pedal, despite the vehicle control unit 2082 having disabled acceleration caused by driving operations, the vehicle control unit 2082 controls the vehicle to cancel the disabled driving operations.

[0069] The conditions under which the vehicle control unit 2082 cancels the invalidation of the driving operation will be described in detail. For example, a storage unit (not shown) stores a database in which the contents of the driving operation to be invalidated by the vehicle control unit 2082 are associated with a predetermined situation indicating the condition for canceling the invalidation. The determination unit 207 determines whether or not the driving situation satisfies the predetermined situation by referring to the database. The vehicle control unit 2082 receives the determination result and cancels the invalidation of the driving operation. In other words, the determination unit 207 decides whether or not to invalidate the control of the vehicle by the driving operation of the driver based on the abnormality information and the driving situation. Note that the above is an example used for convenience of explanation, and the conditions for canceling the invalidation of a part of the driving operation are not limited to the above.

[0070] Next, a process flow of the abnormality detection system 2 in this embodiment will be described with reference to Fig. 6. Following step S202, the vehicle control unit 2082 controls the operation of the vehicle so as to invalidate part of the driving operation based on the abnormality information (S601).

[0071] Thereafter, similarly to other embodiments shown in FIG. 4 and the like, the driving information acquisition unit 204 acquires information indicating the driving information of the vehicle (S204). In addition, the driving situation analysis unit 205 analyzes the driving information acquired by the driving information acquisition unit 204 and generates a driving situation (S205). The determination unit 207 determines whether the output driving situation satisfies a predetermined condition (S206).

[0072] Following step S206, the vehicle control unit 2082 cancels the invalidation of the executed driving operation based on the determination result of the determination unit 207 (S603).

[0073] When the vehicle control unit 2082 cancels the disablement of the driving operation, the notification control unit 2081 included in the control unit 208 may perform control to stop the notification being output.

[0074] According to the anomaly detection system of the present embodiment, when an anomaly is detected, control is executed to disable a part of the driving operation, and in addition, the disablement of the driving operation is released depending on the driving situation. This allows the driver time to check the notification situation, and encourages an appropriate response to the detected anomaly. [Fourth embodiment] Next, an anomaly detection system 2 in a fourth embodiment will be described. The anomaly detection system 2 in this embodiment differs from the above-described embodiments in that the determination unit 207 determines the driving situation, and the notification control unit 2081 determines the timing of outputting a notification according to the determination result and the anomaly information. Note that in the description of this embodiment, descriptions that overlap with the above-described embodiments will be omitted.

[0075] The functional block diagram of the anomaly detection system 2 in this embodiment is the same as that in the second or third embodiment. The determination unit 207 determines whether or not the driving situation is a predetermined situation, as in the previous embodiment. It is preferable that the determination unit 207 determines whether or not the current driving situation corresponds to the predetermined situation at regular intervals.

[0076] The notification control unit 2081 controls the output of the notification based on the abnormality information as in the previous embodiment. Here, when the determination result of the determination unit 207 indicates that the driving situation is a predetermined situation, the notification control unit 2081 determines the timing of outputting the notification depending on whether the abnormality information indicates the occurrence of an accident itself or indicates a sign of an accident. The notification control unit 2081 controls the notification unit 209 to issue a notification at the determined timing.

[0077] A specific example will be described below. For example, assume that the driving device is accepting a complex driving operation such as lane change when an abnormality occurs inside the vehicle. In this case, the abnormality information acquisition unit 202 acquires abnormality information. The determination unit 207 determines that the driving situation corresponds to a predetermined situation of "lane change". The notification control unit 2081 determines whether to prioritize notification or "lane change" according to the content of the abnormality information, based on the determination result that the driving situation is the predetermined situation of "lane change". For example, if the abnormality information suggests the occurrence of an accident in the form of "a person crouching", there is a high possibility that an emergency has occurred. In this case, the notification control unit 2081 controls to immediately output a notification indicating the content of the abnormality before the determination unit 207 determines that the state of "lane change" has been resolved. On the other hand, if the abnormality information indicates "the presence of a person standing without holding onto a handrail", the urgency is low. In this case, the notification control unit 2081 controls to output a notification indicating the content of the abnormality after the determination unit 207 determines that the state of "lane change" has been resolved.

[0078] As described above, when the driving situation is a specified situation, the notification control unit 2081 changes the timing of outputting the notification, such as whether to output the notification before or after the specified situation is resolved, depending on the content of the abnormality information.

[0079] Next, the flow of processing of the anomaly detection system 2 in this embodiment will be described with reference to Fig. 7. The anomaly detection unit 201 detects an anomaly occurring inside the vehicle based on information acquired by the sensor 2011 (S401). In response to the anomaly detection unit 201 detecting an anomaly, the anomaly information acquisition unit 202 generates anomaly information (S402). The driving information acquisition unit 204 acquires information indicating driving information of the vehicle (S403). The driving situation analysis unit 205 analyzes the driving information acquired by the driving information acquisition unit 204 and generates a driving situation (S404).

[0080] Following step S404, the determination unit 207 determines whether the driving situation is a predetermined situation (S405). When the operation situation or the movement situation among the determination results satisfies the predetermined situation (S406, YES), the notification control unit 2081 determines the timing of outputting the notification depending on whether the abnormality information indicates the occurrence of an accident itself or indicates a sign of an accident (S407). When the determination result does not satisfy the predetermined situation (S406, NO), the process proceeds to step S408. The notification control unit 2081 controls the notification unit 209 to issue a notification (S408). Here, when the timing of outputting the notification has been determined in step S403, the notification control unit 2081 controls the notification to be output at the timing.

[0081] Note that steps S205 to S207 (FIG. 4) may be executed following step S408.

[0082] According to the abnormality detection system 2 in this embodiment, the timing of outputting a notification can be flexibly changed depending on the importance of the abnormality that has occurred and the driving situation in which the vehicle is placed. [Variations] Modifications applicable to the above-mentioned embodiments will be described. The anomaly detection system 2 in the fourth embodiment changes the timing of outputting a notification according to the driving conditions of the vehicle, but may also change the means of notification according to the driving conditions of the vehicle. For example, when the notification unit 209 is a display and a speaker, if the anomaly is a sign of an accident, notification is not performed by the speaker, but only by the display. Also, if the anomaly indicates the occurrence of an accident, the system may be designed to perform notification by both the display and the speaker. The above-mentioned display and speaker are merely examples, and examples of changing the means of notification are not limited to these.

[0083] Another modified example will be described. Even if the abnormality is a sign of an accident, if the sign is detected multiple times over a long period of time, the output of the notification may be changed. For example, in the specific example of the above fourth embodiment, if the abnormality information indicates "the presence of a person standing without holding onto the handrail", the notification control unit 2081 controls to output a notification indicating the content of the abnormality after the determination unit 207 determines that the state of "changing lanes" has been resolved. In contrast, if the "presence of a person standing without holding onto the handrail" has already been detected multiple times over a long period of time, the notification control unit 2081 controls to output a notification indicating the content of the abnormality before the determination unit 207 determines that the state of "changing lanes" has been resolved. Note that the above specific example is merely an example, and the abnormality information and driving conditions to be notified are not limited to those described above. [Variation 1] Next, a modified example applicable to the above-mentioned embodiment will be described. Anomaly detection system 2 in this modified example will be described. Anomaly detection system 2 in this modified example differs from the above-mentioned embodiment in that anomaly information acquisition unit 202 generates anomaly information including the reliability of anomaly detection, and a control unit 208 changes control based on the determination result and reliability of a determination unit 207. Note that in the description of this embodiment, descriptions that overlap with the above-mentioned embodiment will be omitted.

[0084] The abnormality information acquisition unit 202 generates abnormality information including a degree of certainty indicating the likelihood that an abnormality has occurred inside the vehicle.

[0085] The control unit 208 controls the driving device or the notification unit 209 differently based on the determination result and reliability of the determination unit 207. For example, when the reliability is less than a first threshold, the control unit 208 controls both the display and the speaker of the notification unit 209 to output a notification, but does not control the vehicle. When the reliability is less than a second threshold that is smaller than the first threshold, the control unit 208 controls only the display of the notification unit 209 to output a notification.

[0086] This enables flexible notification and vehicle control according to the certainty of abnormality detection.

[0087] The first and second thresholds can be changed by an operation of the driver or a system administrator. For example, the first and second thresholds may be changed by an input device (not shown) attached near the driver's seat such as a steering wheel. In this case, the first and second thresholds may be changeable for each type of abnormality to be detected, or the first and second thresholds may be changeable for each sensor 2011 that detects an abnormality. This makes it possible to maintain appropriate detection accuracy even when an abnormality is detected too sensitively while driving. [Variation 2] Other modified examples applicable to the above-described embodiment will be described. The control unit 208 may execute processing for deleting notifications, making announcements in the vehicle, making emergency calls, etc., by operating an input device (not shown). The input device may be, for example, a button installed near the driver's seat such as a steering wheel, a touch panel, a voice sensor, etc., but is not limited to these.

[0088] The input device (not shown) may be a device for inputting the driver's decision regarding the detected abnormality. For example, the input device is provided with the following three items. (A) Items indicating that the abnormality was detected normally and that the action was taken (B) An item indicating that the abnormality detection is normal and that rechecking will be performed after a certain period of time. (C) Items that indicate that anomaly detection is incorrect The driver receives the notification from the notification unit 209, judges which of the above items corresponds to the abnormality detection, and selects the corresponding item on the input device.

[0089] When item A or C is selected, notification unit 209 immediately stops the notification.

[0090] When item B is selected, the notification unit 209 executes notification again after a certain time and changes the content of the notification. For example, while the notification unit 209 previously only notified the content and time of the abnormality that occurred, when item B is selected, the notification unit 209 additionally notifies the location in the vehicle where the abnormality occurred.

[0091] Regardless of the input device, the notification unit 209 may be designed to change the content of the abnormality information to be notified when an abnormality continues to be detected or when a similar abnormality is detected again.

[0092] This provides the driver with a means for flexible response to the detection of an abnormality. [Hardware configuration example] Next, an example of a hardware configuration for implementing the anomaly detection system (1, 2) in each of the above-mentioned embodiments using one or more computers will be described. The anomaly detection system (1, 2) is equipped with a computer, and the function of the anomaly detection system (1, 2) can be implemented by having the computer execute a program. The anomaly detection system (1, 2) executes the anomaly detection method of the anomaly detection system (1, 2) using the program. The program can be recorded on a computer-readable program storage medium. Each functional unit of the anomaly detection system (1, 2) is implemented by any combination of hardware and software, centered on at least one CPU (Central Processing Unit) of any computer, at least one memory, a program loaded into the memory, at least one storage unit such as a hard disk that stores the program, an interface for network connection, and the like. Those skilled in the art will understand that there are various modifications of this implementation method and device. The storage unit can store programs stored before the shipment of the device, as well as programs downloaded from storage media such as optical disks, magneto-optical disks, and semiconductor flash memories, and from servers on the Internet.

[0093] FIG. 8 is a block diagram illustrating a hardware configuration of the anomaly detection system (1, 2). As shown in FIG. 8, the anomaly detection system (1, 2) has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, a communication interface 5A, and a bus 6A. The peripheral circuit 4A includes various modules. The anomaly detection system (1, 2) does not have to have the peripheral circuit 4A. The anomaly detection system (1, 2) may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices can have the above hardware configuration.

[0094] The bus 6A is a data transmission path for the processor 1A, memory 2A, input / output interface 3A, peripheral circuit 4A, and communication interface 5A to transmit and receive data to and from each other. The processor 1A is, for example, a processing device such as a CPU, a GPU (Graphics Processing Unit), or a microprocessor. The processor 1A can execute processes according to various programs stored in the memory 2A, for example.

[0095] The memory 2A is, for example, a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory), and stores programs and various data.

[0096] The input / output interface 3A includes interfaces for acquiring information from an input device, an external device, an external storage unit, an external sensor, a camera, etc., and interfaces for outputting information to an output device, an external device, an external storage unit, etc. Examples of the input device include a touch panel, a keyboard, a mouse, a microphone, a camera, etc. Examples of the output device include a display, a speaker, a printer, a lamp, etc.

[0097] The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0098] The communication interface 5A enables the anomaly detection system (1, 2) to communicate with an external device (not shown). Note that some of the functions of the anomaly detection system (1, 2) may be configured by a computer. [Additional Notes] The configurations of the above-described embodiments may be combined or some of the components may be replaced. Furthermore, the configuration of the present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present invention. For example, the configurations and processes disclosed in each embodiment and modification may be combined with each other. [Video analysis system configuration example] A configuration example of a video analysis system 1B including the configuration of the above embodiment will be described. The video analysis system 1B may include an anomaly detection system 101B including the configuration of the above embodiment, an analysis device 102B, a storage device 103B, a display device 104B, and a sensor device 105B. These various devices are connected to each other wirelessly or via wires and can communicate with each other.

[0099] First, a description will be given of cameras provided in the video analysis system 1B, cameras provided in various devices provided in the video analysis system 1B, or cameras capable of communicating with various devices provided in the video analysis system 1B.

[0100] The camera may be an IP (Internet Protocol) camera. The IP camera may include at least a part of the configuration or function of the server. For example, the IP camera may extract some data from a captured image and transmit the part of the data to the server. The IP camera may also extract and save or transmit only the image when an abnormality occurs, or may save or transmit the feature amount of a person extracted from the image.

[0101] The type of camera may be a visible light camera, an infrared camera, a stereo camera capable of measuring depth, a depth camera, an omnidirectional camera, a camera with a fisheye lens, etc. A plurality of cameras may be provided, and both infrared cameras and visible light cameras may be provided.

[0102] Next, optional functions of various devices included in the video analysis system 1B will be described.

[0103] The display device 104B displays an image captured by a camera or the like. The display device 104B may display a substitute image indicating the presence of a person in place of the person image captured in the image. Examples of the substitute image include an icon, skeletal information, a masked image, an image showing only the outline of a person, and the like. Note that various sensors such as an infrared sensor and a depth sensor may be used to obtain the substitute image. The substitute image may move on the image in accordance with the movement of the person captured by the camera.

[0104] When locating an object or person, analysis device 102B may use information obtained by a sensor other than the camera in addition to or instead of an image captured by a camera to identify the location. For example, the different sensor may be a microphone that acquires a sound emitted by the object or person. In another example, the different sensor may be a radio wave receiving device that acquires a signal transmitted by a beacon carried by the object or person. In this case, the radio wave receiving device acquires a relative distance and a spatial relative position with respect to the beacon. In another example, the different sensor may be an infrared sensor that detects a change in the infrared light received by the sensor. Analysis device 102B may acquire the position of a person or object shown in a 180-degree panoramic image captured by a camera or a 360-degree panoramic image captured by an omnidirectional camera.

[0105] Analysis device 102B may detect the following behaviors of people appearing in an image or video: Predetermined behaviors to be detected include, for example, high-fiving, putting one's arm around someone's shoulder, waving a towel, blowing a whistle, playing an instrument, waving a large flag, cheering involving group movements, and the like.

[0106] Analysis device 102B may detect, in an image or video captured by a camera, a person's actions of leaving something behind, staggering, pushing a stroller or wheelchair, stopping, leaning on a cane, carrying a suitcase, etc. Analysis device 102B may detect actions based on skeletal information of a person.

[0107] Analysis device 102B may detect, as an abnormal behavior, behavior that is different from that of other people captured at the same or different times in an image or video captured by a camera.

[0108] Anomaly detection system 101B may include some of the functions and configurations of the various devices (analysis device 102B, storage device 103B, display device 104B, sensor device 105B) included in video analysis system 1B. In addition, anomaly detection system 101B may incorporate the functions of the various devices by cooperating with each other to realize the present invention. The configuration of the present invention is not limited to the above-described embodiment, and various changes may be made within the scope of the gist of the present invention. [Explanation of symbols]

[0109] 1, 2, 101B Anomaly Detection System 101, 202 Abnormality information acquisition unit 102, 205 Driving Condition Analysis Section 103, 208 Control section 201 Anomaly detection unit 2011 Sensor 204 Driving Information Acquisition Department 207 Judgment section 2081 Notification control section 2082 Vehicle control unit 209 Notification Department 1A Processor 2A Memory 3A Input / Output Interface 4A Peripheral circuit 5A Communication Interface 6A Bus 1B Video analysis system 102B Analyzer 103B Storage device 104B Display device 105B Sensor device

Claims

1. an abnormality information acquisition unit that acquires abnormality information indicating an abnormality that has occurred in the vehicle; a driving situation analysis unit that analyzes operation information indicating an operation of a driver of the vehicle and generates a driving situation based on a result of the analysis; A control unit that controls output of a notification regarding the abnormality based on the abnormality information and the driving status, The driving situation analysis unit generates the driving situation including a moving situation obtained by analyzing the operation information, and environmental information indicating an environment around the vehicle obtained by analyzing position information of the vehicle and an image captured outside the vehicle. Anomaly detection system.

2. The control unit controls the notification unit to output a notification based on the abnormality information, and controls the notification unit to stop based on the operation information. The anomaly detection system according to claim 1 .

3. The control unit controls the notification unit to output the notification based on the abnormality information, and controls the notification unit to stop when the driving situation satisfies a predetermined situation. The anomaly detection system according to claim 1 or 2.

4. The control unit controls the vehicle so as to disable the driving operation of the driver based on the abnormality information and the driving situation. The anomaly detection system according to claim 1 .

5. the control unit changes a timing for outputting the notification depending on whether the abnormality information indicates a precursor of an accident or indicates the occurrence of an accident when the driving situation satisfies a predetermined situation. The anomaly detection system according to claim 1 .

6. an abnormality detection unit that detects the abnormality occurring in the vehicle; Further equipped with The anomaly information acquisition unit generates the anomaly information including a probability of occurrence of the anomaly, The control unit changes control of the vehicle based on the degree of accuracy. The anomaly detection system according to claim 4 or 5.

7. an abnormality information acquisition unit that acquires abnormality information indicating an abnormality that has occurred in the vehicle; a driving situation analysis unit that analyzes operation information indicating an operation of a driver of the vehicle and generates a driving situation based on a result of the analysis; A control unit that controls output of a notification regarding the abnormality based on the abnormality information and the driving status, the control unit changes a timing for outputting the notification depending on whether the abnormality information indicates a precursor of an accident or indicates the occurrence of an accident when the driving situation satisfies a predetermined situation. Anomaly detection system.

8. Acquire abnormality information indicating an abnormality occurring inside the vehicle, Analyzing operation information indicating an operation of a driver of the vehicle, and generating a driving situation based on a result of the analysis; Controlling output of a notification regarding the abnormality based on the abnormality information and the driving status; When generating the driving situation, the driving situation is generated including a moving situation obtained by analyzing the operation information, and environmental information indicating an environment around the vehicle obtained by analyzing position information of the vehicle and an image captured outside the vehicle. Anomaly detection methods.

9. On the computer, A process of acquiring abnormality information indicating an abnormality occurring within a vehicle; A process of analyzing operation information indicating an operation of a driver of the vehicle and generating a driving situation based on a result of the analysis; and controlling an output of a notification regarding the abnormality based on the abnormality information and the driving status. The process of generating the driving situation includes generating the driving situation including a moving situation obtained by analyzing the operation information, and environmental information indicating an environment around the vehicle obtained by analyzing position information of the vehicle and an image captured outside the vehicle. program.

Images