Warning system, warning device, warning method, and warning program

Abstract

A warning system according to the present disclosure includes: a character information generation unit that generates, by using a captured image of the inside or the outside of a mobile body, character information indicating a character representing an event appearing in the captured image; and a warning information notification unit that issues a notification of warning information corresponding to the character information generated by the character information generation unit. In a warning method according to the present disclosure, a computer: generates, by using a captured image of the inside or the outside of a mobile body, character information indicating a character representing an event appearing in the captured image; and issues a notification of warning information corresponding to the generated character information.

Description

Warning system, warning device, warning method, and warning program

[0001] The present disclosure relates to a warning system, a warning device, a warning method, and a warning program that provide warnings regarding moving objects.

[0002] Various technologies for issuing warnings regarding moving objects have been proposed. As an example of such a technology, Patent Literature 1 discloses an abnormality detection device that acquires a first image of a second vehicle captured by an in-vehicle camera of a first vehicle and detects an abnormal state of the driver of the second vehicle based on an image of the driver of the second vehicle included in the first image.

[0003] Japanese Patent Application Laid-Open No. 2020-95356

[0004] Yoshitaka Ushiku, "Image-Related Language Generation," [online], [March 7, 2024], Internet (URL: https: / / www.jstage.jst.go.jp / article / jjsai / 34 / 4 / 34_483 / _pdf / -char / ja) Koki Maeda and three others, "Automatic Evaluation Method for Image Caption Generation Using Visual Language Models Utilizing Visual Context," [online], [March 7, 2024], Internet (URL: https: / / www.anlp.jp / proceedings / annual_meeting / 2024 / pdf_dir / P7-10.pdf)

[0005] However, the anomaly detection device disclosed in Patent Document 1 has a problem in that it cannot issue a warning based on text information that indicates an event outside or inside the mobile object.

[0006] In view of the above-mentioned problems, one of the objectives of the present disclosure is to provide a warning system, a warning device, a warning method, and a warning program that are capable of issuing a warning based on text information representing an event outside or inside a moving body.

[0007] The warning system according to the present disclosure includes a character information generation unit that uses a captured image of the inside or outside of a moving body to generate character information indicating characters that describe an event captured in the captured image, and a warning information notification unit that notifies warning information corresponding to the character information generated by the character information generation unit.

[0008] The warning device according to the present disclosure includes a character information generation unit that uses a captured image of the inside or outside of a moving body to generate character information indicating characters that describe an event captured in the captured image, and a warning information notification unit that notifies warning information corresponding to the character information generated by the character information generation unit.

[0009] The warning method according to the present disclosure involves a computer using a captured image of the interior or exterior of a moving object to generate text information indicating characters that describe an event captured in the captured image, and notifying the user of warning information corresponding to the generated text information.

[0010] The warning program according to the present disclosure causes a computer to use a captured image of the inside or outside of a moving object to generate text information indicating characters describing an event captured in the captured image, and notify the computer of warning information corresponding to the generated text information.

[0011] The present disclosure makes it possible to provide a warning system, a warning device, a warning method, and a warning program that are capable of issuing a warning based on text information that describes an event outside or inside a moving object.

[0012] 1 is a diagram illustrating an example of a warning system according to the present disclosure. FIG. 1 is a diagram illustrating an example of a configuration of a monitoring device according to the present disclosure. FIG. 2 is a diagram illustrating an example of a monitoring target character table according to the present disclosure. FIG. 3 is a diagram illustrating an example of a danger state character table according to the present disclosure. FIG. 4 is a diagram illustrating an example of a danger level calculation table according to the present disclosure. FIG. 5 is a diagram illustrating an example of a warning information table according to the present disclosure. FIG. 6 is a diagram illustrating an example of a configuration of a management device according to the present disclosure. FIG. 7 is a diagram illustrating an example of a management information table according to the present disclosure. FIG. 8 is a diagram illustrating an example of an image according to the present disclosure. FIG. 9 is a diagram illustrating an example of processing performed by a monitoring device according to the present disclosure. FIG. 10 is a diagram illustrating an example of processing performed by a management device according to the present disclosure. FIG. 11 is a diagram illustrating main components of a warning system according to the present disclosure. FIG. 12 is a diagram illustrating another example of a configuration of a monitoring device according to the present disclosure. FIG. 13 is a diagram illustrating another example of processing performed by a monitoring device according to the present disclosure. FIG. 14 is a diagram illustrating an example of an image area according to vehicle speed. FIG. 15 is a diagram illustrating another example of a monitoring target character table according to the present disclosure. FIG. 16 is a diagram illustrating another example of a danger state character table according to the present disclosure. FIG. 17 is a diagram illustrating an example of an advice information table according to the present disclosure.

[0013] First Embodiment An exemplary embodiment will now be described with reference to the drawings. Fig. 1 is a diagram illustrating an example of a warning system 1 according to the present disclosure. The warning system 1 includes a monitoring device 10 and a management device 20. Although Fig. 1 illustrates a single monitoring device 10, the warning system 1 may include multiple monitoring devices 10.

[0014] The monitoring device 10 and the management device 20 communicate data with each other via a network 30. The network 30 may be configured to include a wide area network such as the Internet. The monitoring device 10 can send and receive data via wireless communication.

[0015] The monitoring device 10 is installed in various mobile objects such as automobiles, trains, and airplanes. The mobile objects are equipped with an imaging device capable of capturing images of the interior of the mobile object. The monitoring device 10 performs monitoring using captured images generated by the imaging device. The configuration of the monitoring device 10 will be described in detail below.

[0016] The management device 20 processes the information transmitted by the monitoring device 10. A specific example of the management device 20 is an information processing device such as a server. The configuration of the management device 20 will be described in detail later.

[0017] 2 is a schematic diagram showing an example of the configuration of a monitoring device 10 according to the present disclosure. The monitoring device 10 includes a computing device 11, a communication interface (I / F) 12, and a storage device 13.

[0018] The arithmetic device 11 is a device that performs overall control of the monitoring device 10. Specific examples of the arithmetic device 11 include processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). The arithmetic device 11 executes a program stored in the storage device 13 to perform a method defined by the program. Note that, in other embodiments, the functions performed by the arithmetic device 11 may also be performed by an integrated circuit such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). These devices correspond to computers.

[0019] The communication interface 12 is an interface for carrying out data communication between the monitoring device 10 and other devices.

[0020] The storage device 13 is a storage device that stores various data such as programs executed by the arithmetic device 11 and data processed by the arithmetic device 11. The storage device 13 stores a monitoring target character table, a danger state character table, a danger level calculation table, and a warning information table.

[0021] 3 is a diagram showing an example of a monitoring target character table. The monitoring target character table is a data table in which characters indicating monitoring targets and classification information of the characters indicating the monitoring targets are registered. Note that the characters registered in the monitoring target character table are not limited to those shown in FIG. 3, and characters indicating various monitoring target persons, etc. can be used.

[0022] 4 is a diagram showing an example of a dangerous state character table. The dangerous state character table is a data table in which characters indicating dangerous states are registered. Note that the characters registered in the dangerous state character table are not limited to those shown in FIG. 4, and characters indicating various dangerous states can be used.

[0023] 5 is a diagram showing an example of a risk calculation table. The risk calculation table is a data table for calculating the risk of an event captured in a captured image. The risk calculation table registers a monitored object, a character indicating a dangerous state, a state of the moving object, a speed of the moving object, and a risk level in association with each other.

[0024] 6 is a diagram showing an example of a warning information table. The warning information table is a data table in which identification information of the warning information, the warning information, time information, and the degree of danger are registered. The date and time information is information indicating the date and time when the warning information was registered in the warning information table.

[0025] The computing device 11 has an acquisition unit 110, a state determination unit 111, a character information generation unit 112, a character information determination unit 113, a risk calculation unit 114, a risk determination unit 115, a registration determination unit 116, a warning information management unit 117, a time determination unit 118, and a transmission unit 119. These functional units can be realized by a program.

[0026] The acquisition unit 110 acquires the speed information of the moving object, the control signal, and the captured image. The control signal includes, for example, a signal indicating that the accelerator pedal of the vehicle is being depressed, a signal indicating that the brake of the vehicle is being depressed, a signal indicating that the shift lever is in park, and the like.

[0027] The state determination unit 111 determines whether the mobile object is in a running state or a stopped state based on the speed information and control signal acquired by the acquisition unit 110. Specifically, the state determination unit 111 determines that the mobile object is in a running state when the speed information acquired by the acquisition unit 110 is greater than 0 km / h. On the other hand, when the speed information acquired by the acquisition unit 110 is 0 km / h, the state determination unit 111 determines that the mobile object is in a stopped state. Furthermore, the state determination unit 111 can determine that the mobile object is in a stopped state when the acquisition unit 110 acquires a control signal indicating that the shift lever is in park. In this case, the state determination unit 111 can determine that the mobile object is in a stopped state when the acquisition unit 110 acquires a control signal indicating that the brake is being depressed.

[0028] The text information generation unit 112 uses the captured image provided by the imaging device to generate text information indicating characters that describe the event captured in the captured image. The text information may be a natural language sentence that describes the event captured in the captured image. In this embodiment, the text information generation unit 112 can generate text information using a vision language model that uses the captured image as input information and text information as output information. It is preferable that the image language model specifies conditions for outputting text indicating a monitored object and text indicating a dangerous state as prompts. The image language model can be realized, for example, based on the image caption generation technology disclosed in Non-Patent Document 1 or Non-Patent Document 2.

[0029] For example, if the captured image shows the driver dozing off at the wheel, the text information generation unit 112 generates text information indicating that the driver is dozing off at the wheel. Also, if the captured image shows the driver operating a smartphone while driving, the text information generation unit 112 generates text information indicating that the driver is operating a smartphone while driving. Furthermore, if the captured image shows the driver looking away from the road, the text information generation unit 112 generates text information indicating that the driver is looking away from the road. Furthermore, if the captured image shows a passenger sitting down, the text information generation unit 112 generates text information indicating that the passenger is sitting down.

[0030] The character information determination unit 113 determines whether or not a predetermined character is included in the characters indicated by the character information generated by the character information generation unit 112. Specifically, the character information determination unit 113 refers to the monitoring target character table and determines whether or not a character indicating a predetermined monitoring target is included in the characters indicated by the character information generated by the character information generation unit 112. Furthermore, the character information determination unit 113 refers to the danger state character table and determines whether or not a character indicating a predetermined danger state is included in the characters indicated by the character information generated by the character information generation unit 112.

[0031] The risk calculation unit 114 calculates the risk of an event captured in a captured image. Specifically, the risk calculation unit 114 refers to a risk calculation table and calculates the risk corresponding to the characters indicating the monitored object and the characters indicating the dangerous state determined to be included in the character information generated by the character information generation unit 112. More specifically, the risk calculation unit 114 can calculate the risk based on the characters indicating the dangerous state, the state and speed of the moving object. For example, if the characters indicating the dangerous state for the operator (driver) are "distracted driving," the state of the moving object is "moving," and the speed is "less than 40 km," the risk calculation unit 114 calculates a risk of "9." Furthermore, if the characters indicating the dangerous state for the operator (driver) are "distracted driving" and the state of the moving object is "stopped," the risk calculation unit 114 calculates a risk of "6." Note that the method by which the risk calculation unit 114 calculates the risk is not limited to this. For example, the risk can also be calculated from the characters indicating the dangerous state without using the state and speed of the moving object.

[0032] The risk determination unit 115 determines whether the risk calculated by the risk calculation unit 114 is equal to or greater than a predetermined threshold. The threshold can be any value among the risk levels that can be calculated by the risk calculation unit 114.

[0033] The registration determination unit 116 determines whether the same warning information as the character information (hereinafter referred to as "warning information") that the character information determination unit 113 has determined to contain characters indicating a monitoring target and characters indicating a dangerous state is registered in the warning information management information table.

[0034] The warning information management unit 117 associates the identification information of the warning information, the warning information, the current date and time information, and the risk calculated by the risk calculation unit 114, and registers them in the warning information management information table.

[0035] The time determination unit 118 determines whether a predetermined time has elapsed from the time indicated by the time information registered in the warning information management information table, based on the time information associated with the same warning information registered in the warning information management information table and the current date and time information.

[0036] The transmitting unit 119 transmits various information to the management device 20 .

[0037] 7 is a schematic diagram showing an example of the configuration of the management device 20 according to the present disclosure. The management device 20 includes a computing device 21, a communication interface 22, and a storage device 23.

[0038] The arithmetic device 21 is a device that performs overall control of the management device 20. Specific examples of the arithmetic device 21 include processors such as a CPU, an MPU, and a GPU. The arithmetic device 21 executes a program stored in the storage device 23, thereby performing a method defined by the program.

[0039] The communication interface 22 is an interface for data communication between the management device 20 and other devices. The storage device 23 is a storage device for storing various data such as programs executed by the arithmetic device 21 and data processed by the arithmetic device 21. The storage device 23 stores a management information table.

[0040] FIG. 8 is a diagram showing an example of a management information table. The management information table is a data table in which management information is registered. The management information table registers warning information, a risk level, identification information of the moving body, information regarding blinking display, and information regarding audio output. The identification information of the moving body is identification information of the moving body in which the monitoring device 10 that transmitted the warning information is installed. The information regarding blinking display is information regarding the blinking display of the warning information. If the risk level related to the warning information is equal to or higher than a predetermined threshold, the information regarding the blinking display of the warning information is set to "yes". If the blinking display of the warning information is "yes", the warning information is displayed in a blinking manner. The information regarding audio output is information regarding the audio output of the warning information. For example, information regarding the audio output of warning information related to the warning information with the highest risk level related to the warning information is set to "yes". If the information regarding the audio output of warning information is "yes", the warning information is output as audio.

[0041] The computing device 21 has a reception determination unit 210, a warning information determination unit 211, a table update unit 212, a display control unit 213, a sound control unit 214, a request reception unit 215, a management information determination unit 216, and a captured image acquisition unit 217. These functional units can be realized by a program.

[0042] The reception determination unit 210 determines whether or not the warning information, the degree of danger, and the identification information of the moving object transmitted by the monitoring device 10 have been received.

[0043] The warning information determination unit 211 determines whether the same warning information related to the same moving body has been received. Specifically, the warning information determination unit 211 determines whether the same warning information and moving body identification information as the received warning information and moving body identification information are associated and registered in the management information table, thereby determining whether the same warning information and moving body identification information have been received from the same moving body. If the same warning information and moving body identification information as the received warning information and moving body identification information are associated and registered, the warning information determination unit 211 determines that the same warning information has been received from the same moving body. On the other hand, if the same warning information and moving body identification information as the received warning information and moving body identification information are not associated and registered, the warning information determination unit 211 determines that the same warning information has not been received from the same moving body.

[0044] The table update unit 212 updates the management information table.

[0045] The display control unit 213 displays an image on the display device based on the management information table. The display device is capable of data communication with the management device 20. Fig. 9 is a diagram showing an example of an image displayed on the display device. The image shown in Fig. 9 displays the warning content corresponding to the warning information, the risk level associated with the warning information, and the identification information of the mobile object on which the monitoring device 10 that transmitted the warning information is installed.

[0046] The audio control unit 214 controls the audio output device to output audio indicating the content of the warning based on the management information table. The audio output device is capable of data communication with the management device 20.

[0047] The request receiving unit 215 determines whether or not a request to delete management information registered in the management information table has been received.

[0048] The management information determination unit 216 determines whether or not management information is registered in the management information table.

[0049] The captured image acquisition unit 217 acquires captured images of a moving object specified by the user of the management device 20 from the monitoring device 10. The user of the management device 20 can specify the moving object from which the captured image is to be acquired based on an image such as that shown in Fig. 9 displayed on the display device. The display control unit 213 displays the captured images acquired by the captured image acquisition unit 217 on the display device.

[0050] Fig. 10 is a diagram showing an example of processing executed by the monitoring device 10. The processing shown in Fig. 10 is processing assuming that the moving body is a vehicle such as an automobile or a train.

[0051] In step S1, the acquisition unit 110 of the monitoring device 10 acquires vehicle speed information, a vehicle control signal, and an image captured inside the vehicle. In step S2, the state determination unit 111 determines whether the vehicle is in a running state or a stopped state based on the speed information and vehicle control signal acquired by the acquisition unit 110.

[0052] In step S3, the character information generation unit 112 uses the captured image provided by the photographing device installed in the vehicle to generate character information representing the event captured in the captured image. In step S4, the character information determination unit 113 refers to the monitoring target character table and determines whether the characters indicated by the character information generated by the character information generation unit 112 include characters indicating a predetermined monitoring target. If it is determined that characters indicating a monitoring target are not included (NO), the process returns to step S1. On the other hand, if it is determined that characters indicating a monitoring target are not included (YES), the process branches to step S5.

[0053] In step S5, the character information determination unit 113 refers to the danger state character table and determines whether the characters indicated by the character information generated by the character information generation unit 112 include characters indicating a predetermined danger state. If it is determined that characters indicating a danger state are not included (NO), the process returns to step S1. On the other hand, if it is determined that characters indicating a danger state are included (YES), the process branches to step S6.

[0054] In step S6, the risk calculation unit 114 refers to the risk calculation table and calculates the risk corresponding to the characters indicating the monitoring target and the characters indicating the dangerous state that are determined to be included in the character information generated by the character information generation unit 112. In step S7, the risk determination unit 115 determines whether the risk calculated by the risk calculation unit 114 is equal to or greater than a predetermined threshold. If it is determined that the risk is less than the predetermined threshold (NO), the process returns to step S1. On the other hand, if it is determined that the risk is equal to or greater than the predetermined threshold (YES), the process branches to step S8.

[0055] In step S8, the registration determination unit 116 determines whether the same warning information as the warning information, which is character information determined by the character information determination unit 113 to include characters indicating a monitoring target and characters indicating a dangerous state, is registered in the warning information management information table. If it is determined that the same warning information is not registered in the warning information management information table (NO), the process branches to step S9. In step S9, the warning information management unit 117 generates identification information for the warning information, associates the identification information, the warning information, the current date and time information, and the danger level calculated by the danger level calculation unit 114, and registers them in the warning information management information table, and the process returns to step S1. On the other hand, if it is determined that the same warning information is registered in the warning information management information table (YES), in other words, if the same dangerous state has been detected multiple times, the process branches to step S10.

[0056] In step S10, the time determination unit 118 determines whether a predetermined time has elapsed since the time indicated by the time information registered in the warning information management information table, based on the time information associated with the same warning information registered in the warning information management information table and the current date and time information. If it is determined that the predetermined time has not elapsed (NO), the process branches to step S1. On the other hand, if it is determined that the predetermined time has elapsed (YES), in step S11, the transmission unit 119 transmits the warning information, the risk level associated with the warning information, and the identification information of the vehicle in which the monitoring device 10 is installed to the management device 20, and the process of FIG. 10 ends.

[0057] 11 is a diagram showing an example of processing executed by the management device 20. In step S21, the reception determination unit 210 of the management device 20 determines whether or not the warning information, the risk level, and the identification information of the moving object transmitted by the monitoring device 10 have been received. If it is determined that this information has not been received (NO), the processing of step S21 is executed again. On the other hand, if it is determined that this information has been received (YES), the processing branches to step S22.

[0058] In step S22, the warning information determination unit 211 determines whether or not the same warning information related to the same moving object has been received. If it is determined that the same warning information related to the same moving object has not been received (NO), the process branches to step S23.

[0059] In step S23, the table update unit 212 updates the management information table. Specifically, the table update unit 212 associates and registers the received warning information, the risk level, and the identification information of the moving object, and also updates information related to flashing display and audio output. In step S24, the display control unit 213 updates the image displayed on the display device based on the management information table updated in step S23. At this time, the display control unit 213 flashes warning information whose risk level is equal to or greater than a threshold.

[0060] In step S25, the audio control unit 214 causes the audio output device to output audio indicating the content of the warning based on the management information table updated in step S23.

[0061] In step S26, the request receiving unit 215 determines whether a request to delete management information registered in the management information table has been received. If it is determined that a request to delete management information has not been received (NO), the process returns to step S21. On the other hand, if it is determined that a request to delete management information has been received (YES), the process branches to step S27.

[0062] In step S27, the table update unit 212 updates the management information table. Specifically, the table update unit 212 deletes the management information that is the target of the management information deletion request from the management information table, and updates the information related to the blinking display and audio output.

[0063] In step S28, the management information determination unit 216 determines whether or not management information is registered in the management information table. If it is determined that management information is registered in the management information table (YES), the process returns to step S24. On the other hand, if it is determined that management information is not registered in the management information table (NO), the process of FIG. 11 ends.

[0064] 12 is a diagram showing the main components of the warning system 1. The warning system 1 includes a character information generation unit 112 that uses a captured image of the outside or inside of a moving object to generate character information showing characters describing an event captured in the captured image, and a warning information notification unit 121 that notifies warning information corresponding to the character information generated by the character information generation unit. The warning information notification unit 121 corresponds to the display control unit 213 and the audio control unit 214. By employing this configuration, a warning can be issued based on character information describing an event outside or inside the moving object.

[0065] As described above, the monitoring device 10 includes a character information determination unit 113 that determines whether the characters indicated by the character information generated by the character information generation unit 112 include a predetermined character. If it is determined that the characters indicated by the character information include a predetermined character, the transmission unit 119 of the monitoring device 10 transmits the character information generated by the character information generation unit 112 to the management device 20. Typically, character information has a smaller amount of data than captured images. Therefore, compared to a configuration in which the monitoring device 10 transmits a large number of captured images to the management device 20 for monitoring the interior of a moving object, the amount of data transmitted from the monitoring device 10 to the management device 20 can be reduced.

[0066] Furthermore, the display control unit 213 of the management device 20 displays an image including text information transmitted by at least one monitoring device 10 and identification information of at least one mobile object on which the monitoring device 10 is installed. The captured image acquisition unit 217 of the management device 20 acquires, from the monitoring device 10, captured images of a mobile object specified by a user of the management device 20. The display control unit 213 of the management device 20 displays the captured images acquired by the captured image acquisition unit 217 on the display device. By employing this configuration, it is possible to display captured images of a mobile object specified by a user of the management device 20.

[0067] Furthermore, in the first embodiment, the management device 20 may perform operational intervention on the mobile object in which the monitoring device 10 that transmitted the warning information is installed, based on an instruction from the user of the management device 20 .

[0068] In the above embodiment, the warning information is displayed in a flashing manner when the risk level is equal to or greater than a predetermined threshold, but the display manner of the warning information is not limited to a flashing manner. For example, the warning information may be emphasized by displaying it in a predetermined color, a predetermined font, or enlarging it.

[0069] Second Embodiment In a second embodiment, a mobile body on which a monitoring device 10 is installed is equipped with an imaging device capable of capturing images of the outside of the mobile body. The second embodiment will be described below, focusing on the differences from the first embodiment. Note that a description of the same content as the first embodiment will be omitted.

[0070] The calculation device 11 has an acquisition unit 110, a state determination unit 111, a character information generation unit 112, a character information determination unit 113, an image area identification unit 120, a warning information notification unit 121, an advice information generation unit 122, and an advice information notification unit 123. These functional units can be realized by a program.

[0071] The image region identification unit 120 identifies an image region corresponding to the moving object's speed from a captured image showing the moving direction of the moving object, for example, the front of the moving object. For example, as shown in FIG. 14 , when the moving object's speed is slow, the image region identification unit 120 identifies an image region showing a foreground view based on the position of the moving object. When the moving object's speed is medium, the image region identification unit 120 identifies an image region showing a foreground view and a middle view based on the position of the moving object. When the moving object's speed is high, the image region identification unit 120 identifies an image region showing a foreground view, a middle view, and a background view based on the position of the moving object. When the moving object's speed is high, the moving distance of the moving object per unit time is large, so it is preferable to increase the target image region. Therefore, the faster the moving object's speed, the larger the image region identified by the image region identification unit 120. In other words, the slower the moving object's speed, the smaller the image region identified by the image region identification unit 120. Furthermore, in order to avoid collisions with moving objects, it is preferable that an image area including a foreground image be identified at any speed.

[0072] The warning information notification unit 121 notifies the warning information using at least one of audio and images. The warning information notification unit 121 can display the warning information on a display device with which the monitoring device 10 can communicate data. The display device that displays the warning information can be installed on a mobile body. Furthermore, when the mobile body is remotely controlled, the display device does not need to be installed on the mobile body. The warning information notification unit 121 can output audio indicating the warning information to an audio output device with which the monitoring device 10 can communicate data. The audio output device that notifies the warning information can be installed on the mobile body. Furthermore, when the mobile body is remotely controlled, the audio output device does not need to be installed on the mobile body.

[0073] The advice information generating unit 122 generates advice information regarding the operation of the mobile object corresponding to text information based on a captured image of the outside of the mobile object.

[0074] The advice information notification unit 123 notifies the user of the advice information generated by the advice information generation unit 122. The advice information notification unit 123 notifies the user of the advice information using at least one of a sound and an image.

[0075] Fig. 15 is a diagram showing an example of processing executed by the monitoring device 10 according to the second embodiment. The processing shown in Fig. 15 is processing assuming that the moving body is a vehicle such as an automobile or a train.

[0076] In step S31, the acquisition unit 110 of the monitoring device 10 acquires vehicle speed information and a captured image of the exterior of the vehicle. In step S32, the image area identification unit 120 identifies an image area in the captured image of the exterior of the vehicle according to the speed indicated by the speed information acquired by the acquisition unit 110.

[0077] In step S33, the text information generation unit 112 generates text information indicating an event captured in the captured image, using the image of the image area identified by the image area identification unit 120. For example, if the identified image area shows a vehicle ahead with its door open, the text information generation unit 112 generates text information indicating that the door of the vehicle ahead is open. Also, if the identified image area shows a child playing with a ball, the text information generation unit 112 generates text information indicating that a child is playing with a ball.

[0078] In step S34, the character information determination unit 113 refers to the monitoring target character table and determines whether a character indicating a predetermined monitoring target is included in the characters indicated by the character information generated by the character information generation unit 112. Fig. 16 is a diagram showing an example of the monitoring target character table according to the second embodiment.

[0079] If it is determined that the character indicating the monitoring target is not included (NO), the process returns to step S31. On the other hand, if it is determined that the character indicating the monitoring target is not included (YES), the process branches to step S35.

[0080] In step S35, the character information determination unit 113 refers to the danger state character table and determines whether a character indicating a predetermined danger state is included in the characters indicated by the character information generated by the character information generation unit 112. Fig. 17 is a diagram showing an example of the danger state character table according to the second embodiment.

[0081] If it is determined that the text does not contain characters indicating a dangerous state (NO), the process returns to step S31. On the other hand, if it is determined that the text contains characters indicating a dangerous state (YES), the process branches to step S36.

[0082] In step S36, the warning information notification unit 121 notifies the user of warning information, which is character information determined to include characters indicating the monitoring target and characters indicating a dangerous state.

[0083] In step S37, the advice information generation unit 122 generates advice information regarding the operation of the vehicle. Specifically, the advice information generation unit 122 can generate advice information by referring to an advice information table in which advice information associated with characters indicating dangerous conditions is registered. Fig. 18 is a diagram showing an example of the advice information table. Note that the advice information generated by the advice information generation unit 122 is not limited to that shown in Fig. 18, and various other advice information can be generated.

[0084] Furthermore, the advice information generation unit 122 can use captured images of the area around the vehicle to determine whether there is a space, such as an adjacent lane, into which the vehicle can move. In this case, the advice information generation unit 122 can determine whether there is a space into which the vehicle can move using a trained model that can recognize a space into which the vehicle can move. This model can be trained using training data in which captured images of a space into which the vehicle can move are used as input information and information indicating that there is a space into which the vehicle can move is output information. If the advice information generation unit 122 determines that there is a space into which the vehicle can move, it can generate advice information indicating that the vehicle should move into the space, such as "There are no cars nearby, so please change lanes."

[0085] In step S38, the advice information notification unit 123 notifies the driver of the driving advice information generated by the advice information generation unit 122, and the process of FIG. 15 ends.

[0086] As described above, in the second embodiment, the image area specifying unit 120 specifies an image area corresponding to the speed of the moving object from a captured image showing the moving direction of the moving object. Then, the character information generating unit 112 generates character information indicating characters that describe an event captured in the image area specified by the image area specifying unit 120.

[0087] By adopting this configuration, for example, when the speed of the moving body is high, it is possible to generate text information showing text representing events captured in image areas corresponding to the foreground, middle distance, and background. When the speed of the moving body is medium, it is possible to generate text information showing text representing events captured in image areas corresponding to the foreground and middle distance. When the speed of the moving body is low, it is possible to generate text information showing text representing events captured in the image area corresponding to the foreground. Therefore, it is possible to notify warning information according to the speed of the moving body.

[0088] Furthermore, when the warning information notification unit 121 notifies warning information based on a captured image of the outside of the moving object, the advice information generation unit 122 generates advice information regarding the operation of the moving object corresponding to the character information. Then, the advice information notification unit 123 notifies the operator of the moving object of the advice information generated by the advice information generation unit 122. This allows the operator of the moving object to receive advice regarding the operation of the moving object corresponding to the character information.

[0089] Furthermore, in the second embodiment, when the monitoring device 10 notifies warning information based on an image taken outside the mobile body, if the operator of the mobile body does not perform any operation to avoid danger, the monitoring device 10 may intervene in the operation of the mobile body on which the monitoring device 10 is installed.

[0090] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disk (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0091] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0092] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0093] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A warning system including: a character information generation unit that uses a captured image of the inside or outside of a moving object to generate character information indicating characters that describe an event captured in the captured image; and a warning information notification unit that notifies warning information corresponding to the character information generated by the character information generation unit. (Supplementary Note 2) The warning system according to Supplementary Note 1, wherein the character information generation unit is an image language model that uses the captured image as input information and the character information as output information. (Supplementary Note 3) The warning system according to Supplementary Note 1 or 2, wherein the warning system includes: a monitoring device including the character information generation unit; and a management device including the warning information notification unit, wherein the monitoring device includes a character information determination unit that determines whether the characters indicated by the character information generated by the character information generation unit include a predetermined character, and when it is determined that the characters indicated by the character information include the predetermined character, transmits the character information generated by the character information generation unit to the management device. (Supplementary Note 4) The warning system of Supplementary Note 3, wherein the warning information notification unit displays the character information transmitted by at least one of the monitoring devices and identification information of at least one moving object on which the monitoring device is installed, the management device includes a captured image acquisition unit that acquires, from the monitoring device, the captured image of the moving object specified by a user of the management device, and the warning information notification unit displays the captured image acquired by the captured image acquisition unit on a display device. (Supplementary Note 5) The warning system of Supplementary Note 1 or 2, wherein the warning information notification unit includes an image area identification unit that identifies, from a captured image showing the direction of movement of the moving object, an image area corresponding to the speed of the moving object, and the character information generation unit generates character information indicating characters that represent an event captured in the image area identified by the image area identification unit. (Supplementary Note 6) The warning system of Supplementary Note 1 or 2, including: an advice information generation unit that generates advice information regarding the operation of the moving object corresponding to the character information, and an advice information notification unit that notifies the advice information generated by the advice information generation unit.(Supplementary Note 7) A warning device comprising: a character information generation unit that uses a captured image of the inside or outside of a moving object to generate character information indicating characters that represent an event captured in the captured image; and a warning information notification unit that notifies warning information corresponding to the character information generated by the character information generation unit. (Supplementary Note 8) The warning device according to Supplementary Note 7, wherein the character information generation unit is an image language model that uses the captured image as input information and the character information as output information. (Supplementary Note 9) The warning device according to Supplementary Note 7 or 8, further comprising: an image area identification unit that identifies an image area corresponding to the speed of the moving object from a captured image that shows the direction of movement of the moving object, and the character information generation unit generates character information indicating characters that represent an event captured in the image area identified by the image area identification unit. (Supplementary Note 10) The warning device according to Supplementary Note 7 or 8, including: an advice information generation unit that generates advice information regarding the operation of the moving object that corresponds to the character information; and an advice information notification unit that notifies the advice information generated by the advice information generation unit. (Supplementary Note 11) A warning method in which a computer uses a captured image of the inside or outside of a moving object to generate character information indicating characters that represent an event captured in the captured image, and notifies warning information corresponding to the generated character information. (Supplementary Note 12) The warning method according to Supplementary Note 11, in which the computer generates the character information using an image language model that uses the captured image as input information and the character information as output information. (Supplementary Note 13) A warning method executed in a warning system including a monitoring device and a management device, in which the monitoring device uses a captured image of the inside or outside of a moving object to generate character information indicating characters that represent an event captured in the captured image, determines whether a predetermined character is included among the characters indicated by the generated character information, and if it is determined that the predetermined character is included among the characters indicated by the character information, transmits the generated character information to the management device, and the management device notifies warning information corresponding to the character information transmitted by the monitoring device.(Supplementary Note 14) The warning method according to Supplementary Note 13, wherein the management device displays the character information transmitted by at least one of the monitoring devices and identification information of at least one moving object on which the monitoring device is installed, acquires from the monitoring device the captured image of the moving object specified by a user of the management device, and displays the acquired captured image on a display device. (Supplementary Note 15) The warning method according to Supplementary Note 11 or 12, wherein the computer identifies an image area corresponding to the speed of the moving object from a captured image showing the direction of movement of the moving object, and generates character information indicating characters representing an event shown in the identified image area. (Supplementary Note 16) The warning method according to Supplementary Note 11 or 12, wherein the computer generates advice information regarding the operation of the moving object corresponding to the character information, and notifies the user of the generated advice information. (Supplementary Note 17) A warning program that causes a computer to generate character information indicating characters representing an event shown in the captured image using an image of the inside or outside of a moving object, and notify the user of the warning information corresponding to the generated character information. (Supplementary Note 18) The warning program according to Supplementary Note 17, which causes the computer to generate the text information using an image language model that uses the captured image as input information and the text information as output information. (Supplementary Note 19) The warning program according to Supplementary Note 17 or 18, which causes the computer to identify an image area corresponding to the speed of the moving object from a captured image that shows the direction of movement of the moving object, and generates text information showing text that represents an event that appears in the identified image area. (Supplementary Note 20) The warning program according to Supplementary Note 17 or 18, which causes the computer to generate advice information regarding the operation of the moving object that corresponds to the text information, and notify the computer of the generated advice information.

[0094] Some or all of the elements described in any appendix may be applied to a variety of hardware, software, recording means for recording software, systems, and methods.

[0095] 1: Warning system 10: Monitoring device 11: Calculation device 12: Communication interface 13: Storage device 20: Management device 21: Calculation device 22: Communication interface 23: Storage device 30: Network 110: Acquisition unit 111: Status determination unit 112: Character information generation unit 113: Character information determination unit 114: Risk calculation unit 115: Risk determination unit 116: Registration determination unit 117: Warning information management unit 118: Time determination unit 119: Transmission unit 120: Image area identification unit 121: Warning information notification unit 122: Advice information generation unit 123: Advice information notification unit 210: Reception determination unit 211: Warning information determination unit 212: Table update unit 213: Display control unit 214: Audio control unit 215: Request receiving unit 216: Management information determining unit 217: Photographed image acquiring unit

Claims

1. A warning system comprising: a character information generation unit that uses a photographed image of the inside or outside of a moving object to generate character information showing characters that describe an event captured in the photographed image; and a warning information notification unit that notifies warning information corresponding to the character information generated by the character information generation unit.

2. The warning system according to claim 1, wherein the character information generation unit generates the character information using an image language model that uses the captured image as input information and the character information as output information.

3. The warning system according to claim 1 or 2, wherein the warning system includes a monitoring device equipped with the character information generation unit and a management device equipped with the warning information notification unit, and the monitoring device includes a character information determination unit that determines whether the characters indicated by the character information generated by the character information generation unit include a predetermined character, and when it is determined that the characters indicated by the character information include the predetermined character, the monitoring device transmits the character information generated by the character information generation unit to the management device.

4. The warning system described in claim 3, wherein the warning information notification unit displays the text information transmitted by at least one of the monitoring devices and identification information of at least one mobile object on which the monitoring device is installed, the management device is provided with a captured image acquisition unit that acquires, from the monitoring device, the captured image of the mobile object designated by a user of the management device, and the warning information notification unit displays the captured image acquired by the captured image acquisition unit on a display device.

5. A warning system as described in claim 1 or 2, further comprising an image area identification unit that identifies an image area corresponding to the speed of the moving object from a captured image that shows the direction of movement of the moving object, and wherein the character information generation unit generates character information that shows characters that represent an event captured in the image area identified by the image area identification unit.

6. A warning system as claimed in claim 1 or 2, comprising: an advice information generation unit that generates advice information regarding the operation of the moving object corresponding to the character information; and an advice information notification unit that notifies the user of the advice information generated by the advice information generation unit.

7. A warning device comprising: a character information generation unit that uses a photographed image of the inside or outside of a moving object to generate character information showing characters that describe an event captured in said photographed image; and a warning information notification unit that notifies warning information corresponding to the character information generated by said character information generation unit.

8. The warning device according to claim 7, wherein the character information generation unit generates the character information using an image language model that uses the captured image as input information and the character information as output information.

9. A warning device as described in claim 7 or 8, further comprising an image area identification unit that identifies an image area corresponding to the speed of the moving object from a captured image that shows the direction of movement of the moving object, and wherein the character information generation unit generates character information that shows characters that represent an event captured in the image area identified by the image area identification unit.

10. A warning device as described in claim 7 or 8, comprising: an advice information generation unit that generates advice information regarding the operation of the moving object corresponding to the character information; and an advice information notification unit that notifies the advice information generated by the advice information generation unit.

11. A warning method in which a computer uses a photographed image of the inside or outside of a moving object to generate text information showing characters that describe an event captured in the photographed image, and notifies the user of warning information corresponding to the generated text information.

12. The warning method according to claim 11, wherein the computer generates the text information using an image language model that uses the captured image as input information and the text information as output information.

13. A method executed in a warning system including a monitoring device and a management device, wherein the monitoring device uses a captured image of the inside or outside of a moving object to generate character information indicating characters that describe an event captured in the captured image, determines whether the characters indicated by the generated character information include a predetermined character, and if it is determined that the characters indicated by the character information include the predetermined character, transmits the generated character information to the management device, and the management device notifies warning information corresponding to the character information transmitted by the monitoring device.

14. The warning method described in claim 13, wherein the management device displays the character information transmitted by at least one of the monitoring devices and identification information of at least one mobile object on which the monitoring device is installed, acquires from the monitoring device the captured image of the mobile object designated by a user of the management device, and displays the acquired captured image on a display device.

15. A warning method as claimed in claim 11 or 12, wherein the computer identifies an image area corresponding to the speed of the moving object from a captured image showing the direction of movement of the moving object, and generates text information showing characters representing an event captured in the identified image area.

16. A warning method according to claim 11 or 12, wherein the computer generates advice information regarding the operation of the moving object corresponding to the character information, and notifies the user of the generated advice information.

17. A warning program that causes a computer to use a photographed image of the inside or outside of a moving object to generate text information showing characters that describe an event captured in the photographed image, and notify warning information corresponding to the generated text information.

18. The warning program according to claim 17, which causes the computer to generate the text information using an image language model that uses the captured image as input information and the text information as output information.

19. A warning program as described in claim 17 or 18, which causes the computer to identify an image area corresponding to the speed of the moving object from a captured image showing the direction of movement of the moving object, and generates text information showing characters representing an event captured in the identified image area.

20. A warning program as claimed in claim 17 or 18, which causes the computer to generate advice information relating to the operation of the moving object corresponding to the character information, and notify the generated advice information.

Images