System and Program

Abstract

To provide a mechanism of determining whether or not a vehicle is able to safely travel, as viewed by a person.SOLUTION: A system (100) comprises: an acquisition part (131) for acquiring at least one of an image where the outside of a vehicle is captured and an image where the inside of the vehicle is captured, from the vehicle (2) equipped with a camera; a generation part (133) for obtaining a sentence that explains the content of an image by inputting the acquired image into a language model (122) as a generative AI; a determination part (134) for determining whether or not the content of the sentence indicates presence of a problematic event; and an operation part (135) for performing an operation to resolve the problematic event, when it is determined that the content of the sentence indicates the presence of the problematic event.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system and a program. [Background technology]

[0002] In recent years, with the lifting of the ban on "Level 4" autonomous driving, demonstration experiments of autonomous vehicles have been conducted across the country. In autonomous driving, a huge amount of data is collected from the vehicle, and this data is used by the vehicle's AI to recognize events, predict future events, plan trips, and improve (learn) the AI. For example, Non-Patent Document 1 below discloses the current status of demonstration experiments of such autonomous vehicles. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Autonomous driving demonstrations are becoming more active both domestically and internationally! What are the challenges in data management?” [online], October 15, 2020, Autonomous Driving Lab, [Retrieved July 7, 2023], Internet<URL:https: / / jidounten-lab.com / u_data-management-1> Summary of the Invention [Problem to be solved by the invention]

[0004] Although the safety of autonomous vehicles has improved, passengers and those remotely monitoring them may still feel uneasy. Therefore, there is a need for a system that can determine whether an autonomous vehicle is driving safely from a human perspective, separate from the actions (recognition, prediction, planning, and control) that the autonomous vehicle performs for its own driving. Such systems can be created using either rule-based or machine learning methods. However, in the case of rule-based systems, in order to represent all events, it is necessary to break down and recognize the images obtained from the cameras of the autonomous vehicle and the vast amount of data obtained from sensors into individual events, and then link them together. However, this type of processing is extremely complex, making it extremely difficult to build. Furthermore, when trying to expand functionality or add new functions, the logic must be rewritten, which is an equally difficult task. On the other hand, when using machine learning, a huge amount of training data must first be prepared for all problematic phenomena, and then the model must learn from this huge amount of training data. In other words, there were many difficulties in using conventional technology to build a system that would allow a human to determine whether an autonomous vehicle was driving safely. [Means for solving the problem]

[0005] In order to solve the above problem, a system according to one aspect of the present invention includes an acquisition unit that acquires at least one of an image showing the outside and an image showing the inside of a vehicle from a vehicle equipped with a camera; a generation unit that inputs the image acquired by the acquisition unit into a language model that generates a sentence explaining the content of the input image, and obtains a sentence explaining the content of the image; a judgment unit that determines whether the content of the sentence indicates the existence of a problematic phenomenon that is a problem in the situation of the vehicle; and an operation unit that performs an operation to resolve the problematic phenomenon when the judgment unit determines that the content of the sentence indicates the existence of the problematic phenomenon.

[0006] In addition, a program according to another aspect of the present invention causes a computer to execute the following steps: an acquisition step of acquiring at least one of an image showing the outside and an image showing the inside of a vehicle from a vehicle equipped with a camera; a generation step of inputting the image acquired in the acquisition step into a language model that generates a sentence explaining the content of the input image, thereby obtaining a sentence explaining the content of the image; a judgment step of determining whether the content of the sentence indicates a problematic phenomenon that is a problem in the situation of the vehicle; and an operation step of performing an operation to resolve the problematic phenomenon if it is determined in the judgment step that the content of the sentence indicates the problematic phenomenon. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram showing the functional configuration of an autonomous driving system incorporating a system according to an embodiment of the present invention. FIG. [Figure 2] 10 is a flowchart showing the flow of processing executed by the system. [Figure 3] 10A and 10B are diagrams showing an example of an image input to a language model by a generation unit included in the system, and an example of an explanatory sentence obtained from the language model. [Figure 4] FIG. 10 is a diagram showing an example of a method for determining whether the content of an explanatory text indicates the existence of a problem phenomenon, performed by a determination unit included in the system. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, one embodiment of the present invention will be described in detail.

[0009] <Configuration> As shown in FIG. 1 , the system 100 according to this embodiment is installed in the remote monitoring device 1 of a driving system including the remote monitoring device 1 and a vehicle 2 that communicates with the remote monitoring device 1. For this reason, the configuration of the system 100 will be described here as the configuration of the remote monitoring device 1. The location where the system 100 is installed is not particularly limited. For example, the system 100 may be installed in the vehicle 2, or in another device (not shown) that constitutes the driving system. The system 100 may also be incorporated in a distributed manner in multiple devices that constitute the driving system (for example, the remote monitoring device and the vehicle 2), or may be installed as a device separate from the driving system.

[0010] [vehicle] The vehicle 2 according to this embodiment is an autonomous vehicle. The vehicle 2 according to this embodiment is an autonomous vehicle of level 4 or higher that does not require a driver. For this reason, the vehicle 2 is equipped with at least one of a camera that captures the exterior of the vehicle 2 and a camera that captures the interior. The vehicle 2 recognizes its own situation based on at least one of an image capturing the exterior and an image capturing the interior of the vehicle 2. The vehicle 2 then predicts its own situation a predetermined time into the future based on the recognition results. The vehicle 2 then formulates a driving plan based on the predicted results and drives automatically according to the formulated driving plan. The vehicle 2 according to this embodiment also immediately transmits image data generated by the camera to the remote monitoring device 1. The vehicle 2 may be a connected car that does not have an autonomous driving function.

[0011] [Remote monitoring device] The remote monitoring device 1 is used by a remote monitor to remotely monitor the vehicle 2. The remote monitoring device 1 includes a communication unit 11, a memory unit 12, a calculation unit 13, and an output unit 14. Note that if the vehicle 2 does not have an autonomous driving function, the remote monitoring device 1 does not need to have a function to remotely control the vehicle 2.

[0012] [Communications Department] The communication unit 11 communicates with one or more vehicles 2. The communication unit 11 according to this embodiment is configured with a wireless communication module. That is, the communication unit 11 according to this embodiment communicates wirelessly with the vehicle 2. Note that, if at least a part of the system 100 is mounted on the vehicle 2, the communication unit 11 may communicate with the vehicle 2 via a wired connection.

[0013] [Storage section] The storage unit 12 stores a program 121 and a language model 122. The storage unit 12 according to this embodiment further stores a database 123. That is, the remote monitoring device 1 includes the database 123. The storage unit 12 according to this embodiment is configured with a semiconductor memory, a hard disk, etc.

[0014] The program 121 describes the operation of the remote monitoring device 1.

[0015] The language model 122 is a trained model (a type of generative AI (Artificial Intelligence)) that generates a sentence that explains the content of an input image. The language model 122 according to this embodiment is, for example, a trained model (e.g., GPT-4) constructed by MV-GPT (End to end Generative Pre-Training for Multimodal Video Captioning). This language model 122 outputs a description when at least an image is input. The "description" is a sentence that explains the content of the input image.

[0016] The database 123 stores operation information. Details of the operation information will be described later. The database 123 according to this embodiment further stores explanatory sentences generated by the language model 122 and example sentences related to problematic events. A "problematic event" is an event that becomes a problem in the situation of the vehicle 2. The "problematic event" includes, for example, at least one of the following (1) to (3): (1) An obstacle exists on the travel route of vehicle 2 in the direction of travel of vehicle 2. (2) Something is wrong with the person in vehicle 2. (3) Vehicle 2 broke down.

[0017] Furthermore, the "example sentences related to the problem phenomenon" are sentences that show examples of the problem phenomenon. The "example sentences" include, for example, sentences such as "There is a person lying on the road nearby" and "There is a person lying in a car." The database 123 may store the example sentences in the form of text or in the form of a set of word vectors.

[0018] The storage unit 12 may be divided into one that stores any one of the program 121, the language model 122, and the database 123, and one that stores the others. At least one of the program 121, the language model 122, and the database 123 may be stored in a storage device that is provided separately from the system 100 and communicates with the system 100. The database 123 may be divided into one that stores any one of the action information, the explanatory sentences, and the example sentences, and one that stores the others.

[0019] [Output section] The output unit 14 outputs various pieces of information required for remote monitoring of the vehicle 2 to the remote monitor. The output unit 14 according to this embodiment is configured as a monitor. That is, the output unit 14 according to this embodiment displays various pieces of information so that the remote monitor can visually recognize them. The output unit 14 may be configured as a speaker, or may be configured as a monitor and a speaker. If the output unit 14 is a speaker, the output unit 14 emits sounds so that the remote monitor can hear the various pieces of information.

[0020] [Arithmetic unit] The calculation unit 13 includes an acquisition unit 131, an output control unit 132, a generation unit 133, a determination unit 134, and an operation unit 135. The remote monitoring device 1 according to this embodiment is configured with a computer. Therefore, the calculation unit 13 according to this embodiment is configured with a processor. When this processor executes the program 121 stored in the storage unit 12 according to the flow shown in FIG. 2, the program 121 causes the calculation unit 13 to function as each of the control blocks 131 to 134.

[0021] (Acquisition Department) The acquisition unit 131 executes an acquisition step S1. In the acquisition step S1, the acquisition unit 131 acquires at least one of an image showing the outside of the vehicle 2 and an image showing the inside of the vehicle 2 from the vehicle 2. The "image" may be a video or a still image. The acquisition unit 131 according to this embodiment further acquires at least one of information relating to events outside the vehicle 2 and information relating to events inside the vehicle 2, which is different from the image. The "information different from the image" includes, for example, audio, detection data generated by various sensors equipped in the vehicle 2, and the like.

[0022] (Output control section) The output control unit 132 displays the image acquired by the acquisition unit 131 on the output unit 14 as one of various pieces of information required for remote monitoring. This allows the remote monitor to grasp at least one of the conditions outside and inside the vehicle 2 in real time.

[0023] (Generation part) After acquiring the image, the generation unit 133 executes a generation step S2. In the generation step S2, the generation unit 133 inputs the image acquired by the acquisition unit 131 into the language model 122 and obtains a description. When the generation unit 133 inputs an image such as that shown on the left side of FIG. 3 into the language model 122, the generation unit 133 obtains a description such as "A car is parked on the left side" from the language model 122. When the generation unit 133 inputs an image such as that shown in the center of FIG. 3 into the language model 122, the generation unit 133 obtains a description such as "A person is sitting on the sidewalk" from the language model 122. When the generation unit 133 inputs an image such as that shown on the right side of FIG. 3 into the language model 122, the generation unit 133 obtains a description such as "A person in the car is lying down" from the language model 122. After generating a description, the generation unit 133 according to this embodiment stores the generated description in the database 123.

[0024] As described above, the language model according to this embodiment is a trained model constructed using MV-GPT. Therefore, the generation unit 133 according to this embodiment obtains explanatory text using a multimodal video caption technique. That is, the generation unit 133 according to this embodiment is configured to input the image acquired by the acquisition unit 131 and information other than the image into the language model 122 and obtain explanatory text that explains the content of the image and the information. This allows the language model 122 to generate explanatory text that more accurately describes the situation by taking into account information other than the image, thereby enabling the determination unit 134 (described later) to more accurately determine whether the content of the explanatory text indicates a problematic event.

[0025] (Judgment Department) After generating the explanatory sentence, the determination unit 134 executes a determination step S3. In the determination step S3, the determination unit 134 determines whether the content of the explanatory sentence indicates the existence of a problematic phenomenon that is problematic in the situation of the vehicle 2. The determination unit 134 according to this embodiment makes a determination based on the similarity of the content of the explanatory sentence and the example sentence. Specifically, as shown in FIG. 4, the determination unit 134 first converts each word constituting the explanatory sentence into a word vector. Then, the determination unit 134 calculates difference information. The difference information is information indicating the difference between the word vector of each word in the explanatory sentence and the word vector of each word constituting the example sentence related to the problematic phenomenon that is prepared in advance (stored in the database 123). The "difference information" may be cosine similarity or Euclidean distance. The "cosine similarity" is the angle (cosine value) between the word vector of each word in the sentence and the word vector of each word in the example sentence. The "Euclidean distance" is the distance between the word vector of each word in the sentence and the word vector of each word in the example sentence. Then, when there is an example sentence containing multiple words (synonyms) corresponding to the word vectors whose difference indicated by the difference information is equal to or less than a predetermined value, the determination unit 134 determines that the content of the explanatory sentence indicates the presence of a problematic phenomenon. As a result, even if the words in the explanatory sentence and the words in the example sentence do not completely match, if the meaning of the explanatory sentence is close to that of the example sentence, the operation unit 135, which will be described later, will operate. This makes it possible to prevent problematic phenomena from being overlooked.

[0026] (Operating part) When the determination unit 134 determines that the content of the explanation indicates the existence of a problematic phenomenon, the operation unit 135 executes operation step S4. In operation step S4, the operation unit 135 performs an operation to resolve the problematic phenomenon. The "operation to resolve the problematic phenomenon" includes, for example, at least one of the following operations (1) and (2). (1) The output unit 14 outputs an alert (one of the various types of information required for remote monitoring). (2) Transmitting a control signal to vehicle 2 instructing it to slow down, stop, or change direction.

[0027] The operation unit 135 according to this embodiment accumulates operation information in the database 123 when the determination unit 134 determines that the content of the explanatory text indicates the presence of a problematic event. The "operation information" is information about the operation of the operation unit 135 when the determination unit 134 determines that the content of the explanatory text indicates the presence of a problematic event. The "information about the operation" includes at least one of the content of the alert output by the output unit 14 and the content of the control signal. This makes it possible to use the operation information to verify a decision made by the system 100 later, or as a criterion for changing a decision made once.

[0028] (Determination and operation part 2) Furthermore, each time the determination unit 134 determines whether the content of the explanatory text indicates the existence of a problem, it refers to the operation information stored in the database 123 and compares the latest determination with the previous (most recent) determination. If the previous determination was that the content of the explanatory text indicates the existence of a problem (when operation information has been stored based on the previous determination), and the latest determination is that the content of the explanatory text does not indicate the existence of a problem, the operation unit 135 stops the operation to resolve the problem. "Stop" includes, for example, at least one of the following operations (1) and (2). This makes it possible to prevent the operation to resolve a problem that is no longer necessary from continuing even after the problem has been resolved. (1) Stop the alert being output by the output unit 14. (2) A control signal instructing acceleration is transmitted to vehicle 2.

[0029] (Determination and operation part 3) Furthermore, the determination unit 134 according to this embodiment refers to the operation information stored in the database 123 and determines whether the determination that the content of the explanatory text indicates the existence of a problematic event has been made a predetermined number of times within a predetermined period. If the determination unit 134 determines that the determination that the content of the explanatory text indicates the existence of a problematic event has been made a predetermined number of times within a predetermined period, the operation unit 135 performs a second operation. The "second operation" is an operation that is more effective than the operation performed before the predetermined number of determinations that the content of the explanatory text indicates the existence of a problematic event were made. The "highly effective operation" includes, for example, at least one of the operations (1) and (2) below. This makes it possible to strengthen the operation to resolve the problematic event if the problematic event remains unresolved. (1) An alert indicating the possibility of collision with an obstacle is output to the output unit 14 (if an alert indicating that an obstacle is approaching has been output to the output unit 14 up until that point). (2) Transmit a control signal instructing the vehicle to apply sudden braking (if a control signal instructing the vehicle to slow down and / or change direction has been transmitted up until that point).

[0030] <Action and effect> The explanatory text that explains the content of an image output by the language model 122 of the system 100 described above corresponds to a link between multiple recognized events. Therefore, the system 100 eliminates the need for the conventional complex process of breaking down data into individual events, recognizing them, and linking them together. Furthermore, the system 100 determines whether or not a problematic event exists based on the content of the generated explanatory text. Therefore, the system 100 does not require the language model 122 to learn special training data for autonomous driving. Therefore, the system 100 does not require the work of preparing and learning massive amounts of training data for autonomous driving. As a result, it is possible to easily build a mechanism for determining whether or not the vehicle 2 is traveling safely from a human perspective.

[0031] Furthermore, such effects will also contribute to achieving the United Nations' Sustainable Development Goals (SDGs), such as Goal 3 "Ensure good health and well-being for all" and Goal 11 "Ensure sustainable cities and communities."

[0032] <Modification> The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, the program 121 may be stored in one or more computer-readable storage media, rather than being stored temporarily. The storage media may or may not be included in the device. In the latter case, the program 121 may be supplied to the remote monitoring device 1 via any wired or wireless transmission medium.

[0033] In addition, some or all of the functions of the control blocks 131 to 134 can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0034] Furthermore, embodiments obtained by appropriately combining the technical means disclosed in the different embodiments are also included in the technical scope of the present invention.

[0035] <Summary> The system according to aspect 1 of the present invention comprises an acquisition unit that acquires at least one of an image showing the outside of a vehicle and an image showing the inside of the vehicle from a vehicle equipped with a camera; a generation unit that inputs the image acquired by the acquisition unit into a language model that generates a sentence explaining the content of the input image, and obtains a sentence explaining the content of the image; a judgment unit that determines whether the content of the sentence indicates the existence of a problematic phenomenon that is a problem in the situation of the vehicle; and an operation unit that performs an operation to resolve the problematic phenomenon when the judgment unit determines that the content of the sentence indicates the existence of the problematic phenomenon.

[0036] A system according to aspect 2 of the present invention may be configured such that, in aspect 1 above, the acquisition unit further acquires at least one of information relating to events outside the vehicle and information relating to events inside the vehicle, which information is different from the image, and the generation unit is configured to input the image and information acquired by the acquisition unit into the language model and obtain sentences describing the contents of the image and the information.

[0037] A system according to aspect 3 of the present invention may be configured such that, in aspect 1 or 2 above, the judgment unit converts each word that constitutes the sentence into a word vector, calculates difference information indicating the difference between the word vector of each word in the sentence and the word vector of each word that constitutes pre-prepared example sentences related to the problematic phenomenon, and if there is an example sentence that contains a word corresponding to a word vector whose difference indicated by the difference information is less than a predetermined value, judges that the content of the sentence indicates the existence of the problematic phenomenon.

[0038] A system according to aspect 4 of the present invention may be configured such that, in aspect 3 above, the difference is the cosine similarity between the word vector of each word in the sentence and the word vector of each word in the example sentence, and the judgment unit judges that the content of the sentence indicates the existence of the problem phenomenon when there is an example sentence containing a word corresponding to a word vector whose cosine similarity is greater than or equal to a predetermined value.

[0039] The system according to aspect 5 of the present invention may be configured in any of aspects 1 to 4 above, further comprising a database that stores information regarding the operation of the operating unit when the judgment unit judges that the content of the sentence indicates the existence of the problem phenomenon.

[0040] A system according to aspect 6 of the present invention may be configured such that, in aspect 5 above, the judgment unit refers to the information stored in the database to determine whether a judgment that the content of the sentence indicates the existence of the problematic event has been made a predetermined number of times within a predetermined period of time, and if the judgment unit determines that a judgment that the content of the sentence indicates the existence of the problematic event has been made a predetermined number of times within a predetermined period of time, the operation unit performs an operation that is more effective than the operation performed before the predetermined number of judgments that the content of the sentence indicates the existence of the problematic event were made.

[0041] A system according to aspect 7 of the present invention may be configured such that, in aspect 5 or 6 above, the judgment unit, each time it determines whether the content of the sentence indicates the existence of the problematic event, refers to the information stored in the database and compares the latest judgment with the previous judgment, and if the previous judgment was that the content of the sentence indicates the existence of the problematic event and the latest judgment was that the content of the sentence does not indicate the existence of the problematic event, the operating unit stops operation to resolve the problematic event.

[0042] The program according to aspect 8 of the present invention is configured to cause a computer to execute the following steps: an acquisition step of acquiring at least one of an image showing the outside and an image showing the inside of a vehicle from a vehicle equipped with a camera; a generation step of inputting the image acquired in the acquisition step into a language model that generates a sentence explaining the content of the input image, thereby obtaining a sentence explaining the content of the image; a judgment step of determining whether the content of the sentence indicates a problematic phenomenon that is a problem in the situation of the vehicle; and an operation step of performing an operation to resolve the problematic phenomenon if it is determined in the judgment step that the content of the sentence indicates the problematic phenomenon. [Explanation of symbols]

[0043] 100 systems 1. Remote monitoring device 11 Communications Department 12 Storage section 121 Programs 122 language models 123 databases 13 Arithmetic section 131 Acquisition Department 132 Output control section 133 Generation part 134 Judgment Department 135 Operating unit 14 Output section 2 vehicles S1 Acquisition step S2 generation step S3 Decision Step S4 Operation step

Claims

1. an acquisition unit that acquires at least one of a video showing the outside of a vehicle and a video showing the inside of the vehicle from a moving vehicle equipped with a camera; a generation unit that inputs the video acquired by the acquisition unit into a language model that generates sentences that explain the content of the input video, and obtains sentences that explain the content of the video; a determination unit that determines whether the content of the sentence indicates the existence of a problematic event that may cause a problem in the running of the vehicle; an operation unit that, when the determination unit determines that the content of the sentence indicates the existence of the problem, performs an operation to resolve the problem; Equipped with A system characterized by:

2. the acquisition unit further acquires at least one of information regarding an event outside the vehicle and information regarding an event inside the vehicle, the information being different from the video; The generation unit is configured to input the video and the information acquired by the acquisition unit into the language model and obtain sentences that explain the contents of the video and the information.

2. The system of claim 1.

3. An acquisition unit that acquires at least one of an image showing the outside of a vehicle and an image showing the inside of the vehicle from a vehicle equipped with a camera; a generation unit that inputs the image acquired by the acquisition unit into a language model that generates a sentence that explains the content of the input image, and obtains a sentence that explains the content of the image; a determination unit that determines whether the content of the sentence indicates the existence of a problematic event that is a problem in the situation of the vehicle; an operation unit that, when the determination unit determines that the content of the sentence indicates the existence of the problem, performs an operation to resolve the problem; Equipped with The determination unit Converting each word constituting the sentence into a word vector; Calculating difference information indicating the difference between the word vector of each word in the sentence and the word vector of each word constituting a pre-prepared example sentence relating to the problem phenomenon; If there is an example sentence containing a word corresponding to a word vector whose difference indicated by the difference information is equal to or smaller than a predetermined value, it is determined that the content of the sentence indicates the existence of the problem phenomenon. A system characterized by:

4. the difference information is a cosine similarity between a word vector of each word in the sentence and a word vector of each word in the example sentence; the determination unit determines that the content of the sentence indicates the existence of the problem phenomenon when there is an example sentence including a word corresponding to a word vector having a cosine similarity equal to or greater than a predetermined value.

4. The system of claim 3.

5. The system further includes a database that stores information regarding the operation of the operation unit when the determination unit determines that the content of the sentence indicates the existence of the problem event.

2. The system of claim 1.

6. An acquisition unit that acquires at least one of an image showing the outside of a vehicle and an image showing the inside of the vehicle from a vehicle equipped with a camera; a generation unit that inputs the image acquired by the acquisition unit into a language model that generates a sentence that explains the content of the input image, and obtains a sentence that explains the content of the image; a determination unit that determines whether the content of the sentence indicates the existence of a problematic event that is a problem in the situation of the vehicle; an operation unit that, when the determination unit determines that the content of the sentence indicates the existence of the problem, performs an operation to resolve the problem; a database that stores information about the operation of the operation unit when the determination unit determines that the content of the sentence indicates the existence of the problem event; Equipped with The determination unit refers to the information stored in the database and determines whether or not a determination that the content of the sentence indicates the existence of the problem event has been made a predetermined number of times within a predetermined period of time, When the determination unit determines that the content of the sentence indicates the existence of the problem phenomenon a predetermined number of times within a predetermined period, the operation unit performs an operation that is more effective than the operation that was performed before the predetermined number of determinations that the content of the sentence indicates the existence of the problem phenomenon were made. A system characterized by:

7. An acquisition unit that acquires at least one of an image showing the outside of a vehicle and an image showing the inside of the vehicle from a vehicle equipped with a camera; a generation unit that inputs the image acquired by the acquisition unit into a language model that generates a sentence that explains the content of the input image, and obtains a sentence that explains the content of the image; a determination unit that determines whether the content of the sentence indicates the existence of a problematic event that is a problem in the situation of the vehicle; an operation unit that, when the determination unit determines that the content of the sentence indicates the existence of the problem, performs an operation to resolve the problem; a database that stores information about the operation of the operation unit when the determination unit determines that the content of the sentence indicates the existence of the problem event; Equipped with the determination unit, each time determining whether the content of the sentence indicates the existence of the problem phenomenon, refers to the information accumulated in the database and compares the latest determination with the previous determination; If the previous determination was that the content of the sentence indicates the existence of the problem event and the latest determination is that the content of the sentence does not indicate the existence of the problem event, the operation unit stops the operation for resolving the problem event. A system characterized by:

8. The problem event includes: There is an obstacle on the travel route of the vehicle in the direction of travel of the vehicle; There is something wrong with the person in the vehicle, that the vehicle has broken down; At least one of the following is included: The system of claim 1 .

9. On the computer, an acquisition step of acquiring at least one of a video showing the outside of a vehicle and a video showing the inside of the vehicle from a moving vehicle equipped with a camera; a generation step of inputting the video acquired in the acquisition step into a language model that generates a sentence that explains the content of the input video, and obtaining a sentence that explains the content of the video; a determining step of determining whether the content of the sentence indicates a problematic event that may cause a problem in the running of the vehicle; an action step of performing an action to resolve the problematic event when it is determined in the determination step that the content of the sentence indicates the problematic event; Execute A program characterized by:

Images