System and program

Abstract

To provide a mechanism of determining whether or not a vehicle is able to safely travel, as viewed by a person.SOLUTION: A system (100) comprises: an acquisition part (131) for acquiring at least one of an image where the outside of a vehicle is captured and an image where the inside of the vehicle is captured, from the vehicle (2) equipped with a camera; a generation part (133) for obtaining a sentence that explains the content of an image by inputting the acquired image into a language model (122) as a generative AI; a determination part (134) for determining whether or not the content of the sentence indicates presence of a problematic event; and an operation part (135) for performing an operation to resolve the problematic event, when it is determined that the content of the sentence indicates the presence of the problematic event.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a system and a program. [Background technology]

[0002] In recent years, with the lifting of the ban on "Level 4" autonomous driving, demonstration experiments of autonomous vehicles are being carried out all over the country. In autonomous driving, a huge amount of data is collected from the vehicle, and this data is used by the vehicle's AI to recognize events, predict future events, plan driving, improve (learn) the AI, etc. For example, Non-Patent Document 1 below discloses the current status of demonstration experiments of such autonomous vehicles. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Autonomous driving demonstrations are becoming more active both domestically and internationally! What are the challenges in data management?”, [online], October 15, 2020, Autonomous Driving Lab, [Retrieved July 7, 2023], Internet<URL:https: / / jidounten-lab.com / u_data-management-1> Summary of the Invention [Problem to be solved by the invention]

[0004] Although the safety of autonomous vehicles has improved, passengers in autonomous vehicles and those who monitor autonomous vehicles remotely may still feel uneasy. Therefore, there is a demand for a mechanism to determine whether an autonomous vehicle is driving safely from a human perspective, separate from the actions (perception, prediction, planning, and control) that autonomous vehicles perform for their own driving. Such mechanisms can be created in two ways: rule-based and machine learning. However, in the case of rule-based systems, in order to represent all events, it is necessary to break down and recognize the images obtained from the cameras of autonomous vehicles and the vast amount of data obtained from sensors into individual events, and then link them together. However, this processing is extremely complex, and building such a system is extremely difficult. In addition, when trying to expand functions or add new functions, it is necessary to rewrite the logic. This rewriting is also an equally difficult task. On the other hand, when using machine learning, a huge amount of training data must be prepared for all problematic phenomena, and the model must learn from the huge amount of training data. In other words, there were many difficulties involved in using conventional technology to build a system that could determine from a human perspective whether an autonomous vehicle was driving safely. [Means for solving the problem]

[0005] In order to solve the above problems, a system according to one embodiment of the present invention includes an acquisition unit that acquires at least one of an image showing the outside and an image showing the inside of a vehicle from a vehicle equipped with a camera, a generation unit that inputs the image acquired by the acquisition unit into a language model that generates a sentence explaining the content of the input image, and obtains a sentence explaining the content of the image, a judgment unit that judges whether the content of the sentence indicates the existence of a problematic phenomenon that is a problem in the situation of the vehicle, and an operation unit that performs an operation to resolve the problematic phenomenon when the judgment unit judges that the content of the sentence indicates the existence of the problematic phenomenon.

[0006] In addition, a program related to another aspect of the present invention causes a computer to execute the following steps: an acquisition step of acquiring at least one of an image showing the outside and an image showing the inside of a vehicle from a vehicle equipped with a camera; a generation step of inputting the image acquired in the acquisition step into a language model that generates a sentence explaining the content of the input image, and obtaining a sentence explaining the content of the image; a judgment step of determining whether the content of the sentence indicates a problematic phenomenon that is an issue in the situation of the vehicle; and an operation step of performing an operation to resolve the problematic phenomenon if it is determined in the judgment step that the content of the sentence indicates the problematic phenomenon. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing a functional configuration of an autonomous driving system in which a system according to an embodiment of the present invention is incorporated. [Diagram 2] 2 is a flowchart showing the flow of processing executed by the system. [Diagram 3] 1A to 1C are diagrams illustrating an example of an image that is input to a language model by a generation unit of the system, and an example of an explanatory sentence obtained from the language model. [Figure 4] FIG. 13 is a diagram showing an example of a method for determining whether the contents of an explanatory text indicate the existence of a problem phenomenon, performed by a determination unit of the system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, one embodiment of the present invention will be described in detail.

[0009] <Configuration> As shown in FIG. 1, the system 100 according to the present embodiment is mounted on the remote monitoring device 1 of a driving system including the remote monitoring device 1 and a vehicle 2 that communicates with the remote monitoring device 1. For this reason, the configuration of the system 100 will be described here as the configuration of the remote monitoring device 1. The location where the system 100 is provided is not particularly limited. For example, the system 100 may be mounted on the vehicle 2, or may be mounted on another device (not shown) that constitutes the driving system. The system 100 may be incorporated in a distributed manner in a plurality of devices (e.g., the remote monitoring device and the vehicle 2) that constitute the driving system, or may be provided as a device separate from the driving system.

[0010] [vehicle] The vehicle 2 according to this embodiment is an autonomous vehicle. The vehicle 2 according to this embodiment is an autonomous vehicle of level 4 or higher that does not require a driver. For this reason, the vehicle 2 is equipped with at least one of a camera that captures the outside of the vehicle 2 and a camera that captures the inside. The vehicle 2 recognizes its own situation based on at least one of an image showing the outside of the vehicle 2 and an image showing the inside of the vehicle 2. The vehicle 2 predicts its own situation a predetermined time ahead based on the recognition result. The vehicle 2 then formulates a driving plan based on the predicted result and automatically drives according to the formulated driving plan. The vehicle 2 according to this embodiment immediately transmits image data generated by the camera to the remote monitoring device 1. The vehicle 2 may be a connected car that does not have an autonomous driving function.

[0011] [Remote monitoring device] The remote monitoring device 1 is used by a remote monitor to remotely monitor the vehicle 2. The remote monitoring device 1 includes a communication unit 11, a storage unit 12, a calculation unit 13, and an output unit 14. Note that if the vehicle 2 does not have an autonomous driving function, the remote monitoring device 1 does not need to have a function to remotely control the vehicle 2.

[0012] [Communications Department] The communication unit 11 communicates with one or more vehicles 2. The communication unit 11 according to this embodiment is configured with a wireless communication module. That is, the communication unit 11 according to this embodiment communicates wirelessly with the vehicle 2. Note that, when at least a part of the system 100 is mounted on the vehicle 2, the communication unit 11 may communicate with the vehicle 2 via a wired connection.

[0013] [Storage section] The storage unit 12 stores a program 121 and a language model 122. Moreover, the storage unit 12 according to this embodiment further stores a database 123. That is, the remote monitoring device 1 includes the database 123. The storage unit 12 according to this embodiment is configured with a semiconductor memory, a hard disk, and the like.

[0014] The program 121 describes the operation of the remote monitoring device 1 .

[0015] The language model 122 is a trained model (a type of generative AI (Artificial Intelligence)) that generates a sentence that explains the contents of an input image. The language model 122 according to this embodiment is, for example, a trained model (for example, GPT-4) constructed by MV-GPT (End to end Generative Pre-Training for Multimodal Video Captioning). This language model 122 outputs a description when at least an image is input. The "description" is a sentence that explains the contents of the input image.

[0016] The database 123 accumulates operation information. Details of the operation information will be described later. The database 123 according to this embodiment further accumulates explanatory sentences generated by the language model 122 and example sentences related to problematic events. A "problematic event" is an event that is problematic in the situation of the vehicle 2. The "problematic event" includes, for example, at least one of the following (1) to (3). (1) An obstacle exists on the travel path of vehicle 2 in the direction of travel of vehicle 2. (2) Something is wrong with the person in vehicle 2. (3) Vehicle 2 broke down.

[0017] Moreover, the "example sentences related to the problem phenomenon" are sentences showing examples of the problem phenomenon. The "example sentences" include, for example, sentences such as "a person has collapsed on the road nearby" and "a person has collapsed in a car." The database 123 may store the example sentences in the form of text or in the form of a collection of word vectors.

[0018] The storage unit 12 may be divided into one that stores any one of the program 121, the language model 122, and the database 123, and one that stores the others. At least any one of the program 121, the language model 122, and the database 123 may be stored in a storage device that is provided separately from the system 100 and communicates with the system 100. The database 123 may be divided into one that stores any one of the action information, the explanatory sentence, and the example sentence, and one that stores the others.

[0019] [Output section] The output unit 14 outputs various pieces of information required for remote monitoring of the vehicle 2 to the remote monitor. The output unit 14 according to this embodiment is configured with a monitor. That is, the output unit 14 according to this embodiment displays various pieces of information so that the remote monitor can visually confirm them. The output unit 14 may be configured with a speaker, or may be configured with a monitor and a speaker. When the output unit 14 is a speaker, the output unit 14 emits sounds so that the remote monitor can hear the various pieces of information.

[0020] [Calculation section] The calculation unit 13 includes an acquisition unit 131, an output control unit 132, a generation unit 133, a judgment unit 134, and an operation unit 135. The remote monitoring device 1 according to this embodiment is configured with a computer. Therefore, the calculation unit 13 according to this embodiment is configured with a processor. This processor executes the program 121 stored in the storage unit 12 in the flow shown in FIG. 2, whereby the program 121 causes the calculation unit 13 to function as each of the control blocks 131 to 134.

[0021] (Acquisition Department) The acquisition unit 131 executes an acquisition step S1. In the acquisition step S1, the acquisition unit 131 acquires at least one of an image showing the outside of the vehicle 2 and an image showing the inside of the vehicle 2 from the vehicle 2. The "image" may be a video or a still image. The acquisition unit 131 according to this embodiment further acquires at least one of information related to events outside the vehicle 2 and information related to events inside the vehicle 2, which is different from the image. The "information different from the image" includes, for example, audio, detection data generated by various sensors equipped in the vehicle 2, and the like.

[0022] (Output control section) The output control unit 132 causes the image acquired by the acquisition unit 131 to be displayed on the output unit 14 as one of various pieces of information required for remote monitoring. This allows the remote monitor to grasp at least one of the external and internal conditions of the vehicle 2 in real time.

[0023] (Generation part) After acquiring the image, the generating unit 133 executes a generating step S2. In the generating step S2, the generating unit 133 inputs the image acquired by the acquiring unit 131 to the language model 122 and acquires an explanatory sentence. When the generating unit 133 inputs an image, for example, as shown in the left side of FIG. 3, to the language model 122, the generating unit 133 acquires an explanatory sentence, for example, "A car is parked on the left side" from the language model 122. When the generating unit 133 inputs an image, for example, as shown in the center of FIG. 3, to the language model 122, the generating unit 133 acquires an explanatory sentence, for example, "A person is sitting on the sidewalk" from the language model 122. When the generating unit 133 inputs an image, for example, as shown in the right side of FIG. 3, to the language model 122, the generating unit 133 acquires an explanatory sentence, for example, "A person in the car is lying down" from the language model 122. When the generating unit 133 according to this embodiment generates an explanatory sentence, it accumulates it in the database 123.

[0024] As described above, the language model according to this embodiment is a trained model constructed by MV-GPT. Therefore, the generation unit 133 according to this embodiment obtains an explanatory text using a multimodal video caption technique. That is, the generation unit 133 according to this embodiment is configured to input the image acquired by the acquisition unit 131 and information other than the image to the language model 122, and obtain an explanatory text that explains the contents of the image and the information. This allows the language model 122 to generate an explanatory text that more accurately explains the situation by taking into account information other than the image, so that the determination unit 134 described later can more accurately determine whether the contents of the explanatory text indicate a problematic event.

[0025] (Judgment Department) After generating the explanatory sentence, the judgment unit 134 executes a judgment step S3. In the judgment step S3, the judgment unit 134 judges whether the content of the explanatory sentence indicates the presence of a problematic phenomenon that is a problem in the situation of the vehicle 2. The judgment unit 134 according to the present embodiment makes a judgment based on the similarity of the content of the explanatory sentence and the example sentence. Specifically, the judgment unit 134 first converts each word constituting the explanatory sentence into a word vector as shown in FIG. 4. Then, the judgment unit 134 calculates difference information. The difference information is information indicating the difference between the word vector of each word of the explanatory sentence and the word vector of each word constituting the example sentence related to the problematic phenomenon that is prepared in advance (stored in the database 123). The "difference information" may be cosine similarity or Euclidean distance. The "cosine similarity" is the angle (cosine value) between the word vector of each word of the sentence and the word vector of each word of the example sentence. The "Euclidean distance" is the distance between the word vector of each word of the sentence and the word vector of each word of the example sentence. Then, the judgment unit 134 judges that the content of the explanatory sentence indicates the presence of a problematic phenomenon when there is an example sentence containing multiple words (synonyms) corresponding to the word vectors whose difference indicated by the difference information is equal to or smaller than a predetermined value. As a result, even if the words in the explanatory sentence and the words in the example sentence do not completely match, if the meaning of the explanatory sentence is close to that of the example sentence, the operation unit 135 described later will operate. This makes it possible to prevent problematic phenomena from being overlooked.

[0026] (Operating part) When the determination unit 134 determines that the content of the explanation indicates the presence of a problematic phenomenon, the operation unit 135 executes operation step S4. In operation step S4, the operation unit 135 performs an operation for solving the problematic phenomenon. The "operation for solving the problematic phenomenon" includes, for example, at least one of the following operations (1) and (2). (1) The output unit 14 outputs an alert (one of the various pieces of information required for remote monitoring). (2) Sending a control signal to vehicle 2 instructing it to at least one of slow down, stop, and change direction.

[0027] The operation unit 135 according to this embodiment accumulates operation information in the database 123 when the determination unit 134 determines that the contents of the explanatory text indicate the presence of a problematic event. The "operation information" is information related to the operation of the operation unit 135 when the determination unit 134 determines that the contents of the explanatory text indicate the presence of a problematic event. The "information related to the operation" includes at least one of the contents of the alert output by the output unit 14 and the contents of the control signal. This makes it possible to use the operation information to verify a decision made by the system 100 later, or to use it as a criterion for changing a decision that has already been made.

[0028] (Judgment section / operation section 2) Furthermore, each time the judgment unit 134 judges whether the contents of the explanatory text indicate the presence of a problematic event, it refers to the operation information stored in the database 123 and compares the latest judgment with the previous (most recent) judgment. If the previous judgment was that the contents of the explanatory text indicate the presence of a problematic event (if operation information has been stored from the previous judgment) and the latest judgment is that the contents of the explanatory text do not indicate the presence of a problematic event, the operation unit 135 stops the operation for solving the problematic event. "Stopping" includes, for example, at least one of the following operations (1) and (2). This makes it possible to prevent the operation for solving a problematic event that is no longer necessary from continuing even after the problematic event has been solved. (1) Stop the alert being output by the output unit 14. (2) A control signal instructing vehicle 2 to accelerate is transmitted to vehicle 2.

[0029] (Judgment section / operation section 3) Furthermore, the determination unit 134 according to this embodiment refers to the operation information stored in the database 123 and determines whether the determination that the contents of the explanatory text indicate the existence of a problematic event has been made a predetermined number of times within a predetermined period. Then, when the determination unit 134 determines that the determination that the contents of the explanatory text indicate the existence of a problematic event has been made a predetermined number of times within a predetermined period, the operation unit 135 performs a second operation. The "second operation" is an operation that is more effective than the operation performed before the determination that the contents of the explanatory text indicate the existence of a problematic event was made the predetermined number of times. The "highly effective operation" includes, for example, at least one of the following operations (1) and (2). This makes it possible to strengthen the operation for solving the problematic event when the problematic event continues to remain unsolved. (1) An alert notifying the user of the possibility of a collision with an obstacle is output to the output unit 14 (if, up until that point, the output unit 14 has been caused to output an alert notifying the user of an approaching obstacle). (2) Send a control signal to apply sudden braking (if previously a control signal was sent to slow down and / or change direction).

[0030] <Action and effect> The explanatory text that explains the contents of the image output by the language model 122 of the system 100 described above corresponds to a link between multiple recognized events. Therefore, the system 100 does not require the complicated process of breaking down data into events, recognizing them, and linking them, as in the conventional system. In addition, the system 100 judges whether or not there is a problematic event from the contents of the generated explanatory text. Therefore, the system 100 does not require the language model 122 to learn special training data for autonomous driving. Therefore, the system 100 does not require the work of preparing a huge amount of training data for learning in autonomous driving and learning the data. As a result, it is possible to easily build a mechanism for judging whether or not the vehicle 2 is traveling safely from a human perspective.

[0031] Furthermore, such effects will contribute to achieving, for example, Goal 3 of the Sustainable Development Goals (SDGs) advocated by the United Nations, "Ensure good health and well-being for all," and Goal 11, "Sustainable cities and communities."

[0032] <Modification> The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, the program 121 may be recorded in one or more computer-readable recording media, not temporarily. The recording media may or may not be included in the device. In the latter case, the program 121 may be supplied to the remote monitoring device 1 via any wired or wireless transmission medium.

[0033] In addition, some or all of the functions of the control blocks 131 to 134 can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0034] Furthermore, embodiments obtained by appropriately combining the technical means disclosed in the different embodiments are also included in the technical scope of the present invention.

[0035] <Summary> The system according to aspect 1 of the present invention comprises an acquisition unit that acquires at least one of an image showing the outside and an image showing the inside of a vehicle from a vehicle equipped with a camera; a generation unit that inputs the image acquired by the acquisition unit into a language model that generates a sentence explaining the content of the input image, and obtains a sentence explaining the content of the image; a judgment unit that judges whether the content of the sentence indicates the existence of a problematic phenomenon that is a problem in the situation of the vehicle; and an operation unit that performs an operation to resolve the problematic phenomenon when the judgment unit judges that the content of the sentence indicates the existence of the problematic phenomenon.

[0036] A system according to aspect 2 of the present invention may be configured in the above-mentioned aspect 1, wherein the acquisition unit further acquires at least one of information relating to events outside the vehicle and information relating to events inside the vehicle, the information being different from the image, and the generation unit is configured to input the image and the information acquired by the acquisition unit into the language model to obtain sentences describing the contents of the image and the information.

[0037] A system according to aspect 3 of the present invention may be configured in such a way that, in aspect 1 or 2 above, the judgment unit converts each word that constitutes the sentence into a word vector, calculates difference information indicating the difference between the word vector of each word in the sentence and the word vector of each word that constitutes pre-prepared example sentences related to the problem phenomenon, and if there is an example sentence containing a word corresponding to a word vector whose difference indicated by the difference information is less than a predetermined value, judges that the content of the sentence indicates the existence of the problem phenomenon.

[0038] A system according to aspect 4 of the present invention may be configured such that, in aspect 3 above, the difference is the cosine similarity between the word vector of each word in the sentence and the word vector of each word in the example sentence, and the judgment unit judges that the content of the sentence indicates the existence of the problem phenomenon when there is an example sentence containing a word corresponding to a word vector whose cosine similarity is a predetermined value or greater.

[0039] The system of aspect 5 of the present invention may be configured in any of aspects 1 to 4 above, further comprising a database that stores information regarding the operation of the operating unit when the judgment unit judges that the content of the sentence indicates the existence of the problem phenomenon.

[0040] A system according to a sixth aspect of the present invention may be configured in such a way that, in the fifth aspect above, the judgment unit refers to the information stored in the database to judge whether a judgment that the content of the sentence indicates the existence of the problematic event has been made a predetermined number of times within a predetermined period of time, and if the judgment unit judges that the content of the sentence indicates the existence of the problematic event has been made a predetermined number of times within a predetermined period of time, the operation unit performs an operation that is more effective than the operation performed before the predetermined number of judgments were made that the content of the sentence indicates the existence of the problematic event.

[0041] The system of aspect 7 of the present invention may be configured such that, in aspect 5 or 6 above, the judgment unit, each time it determines whether the content of the text indicates the existence of the problem event, refers to the information stored in the database and compares the latest judgment with the previous judgment, and if the previous judgment was that the content of the text indicates the existence of the problem event and the latest judgment was that the content of the text does not indicate the existence of the problem event, the operating unit stops operation to resolve the problem event.

[0042] The program according to aspect 8 of the present invention is configured to cause a computer to execute the following steps: an acquisition step of acquiring at least one of an image showing the outside and an image showing the inside of a vehicle from a vehicle equipped with a camera; a generation step of inputting the image acquired in the acquisition step into a language model that generates a sentence explaining the content of the input image, and obtaining a sentence explaining the content of the image; a judgment step of determining whether the content of the sentence indicates a problematic phenomenon that is an issue in the situation of the vehicle; and an operation step of performing an operation to resolve the problematic phenomenon if it is determined in the judgment step that the content of the sentence indicates the problematic phenomenon. [Explanation of symbols]

[0043] 100 Systems 1 Remote monitoring device 11 Communications Department 12 Storage section 121 Programs 122 Language Models 123 Database 13 Arithmetic section 131 Acquisition Department 132 Output control section 133 Generation part 134 Judgment Department 135 Operating Part 14 Output section 2 Vehicles S1 Acquisition step S2 Generation Step S3 Decision Step S4 Operation Step

Claims

1. An acquisition unit that acquires at least one of an image showing the outside of a vehicle and an image showing the inside of the vehicle from a vehicle equipped with a camera; a generation unit that inputs the image acquired by the acquisition unit into a language model that generates a sentence that explains the content of the input image, and obtains a sentence that explains the content of the image; A determination unit that determines whether the content of the sentence indicates the presence of a problematic event that is a problem in the situation of the vehicle; an operation unit that performs an operation to resolve the problematic event when the determination unit determines that the content of the text indicates the presence of the problematic event; Equipped with A system characterized by:

2. The acquisition unit further acquires at least one of information regarding an event outside the vehicle and information regarding an event inside the vehicle, the information being different from the image; The generation unit is configured to input the image and the information acquired by the acquisition unit into the language model to obtain a sentence explaining the contents of the image and the information.

2. The system of claim 1 .

3. The determination unit is Convert each word constituting the sentence into a word vector; Calculating difference information indicating a difference between a word vector of each word in the sentence and a word vector of each word constituting a previously prepared example sentence related to the problem phenomenon; When there is an example sentence including a word corresponding to a word vector whose difference indicated by the difference information is equal to or smaller than a predetermined value, it is determined that the content of the sentence indicates the existence of the problem phenomenon.

3. The system according to claim 1 or 2.

4. the difference information is a cosine similarity between a word vector of each word in the sentence and a word vector of each word in the example sentence; the determination unit determines that the content of the sentence indicates the presence of the problem phenomenon when there is an example sentence including a word corresponding to a word vector having a cosine similarity equal to or greater than a predetermined value; 4. The system of claim 3.

5. a database that accumulates information regarding the operation of the operation unit when the determination unit determines that the content of the sentence indicates the existence of the problem phenomenon; 2. The system of claim 1 .

6. The determination unit refers to the information stored in the database and determines whether or not a determination that the content of the text indicates the existence of the problem phenomenon has been made a predetermined number of times within a predetermined period of time, When the determination unit determines that the content of the sentence indicates the existence of the problem phenomenon a predetermined number of times within a predetermined period of time, the operation unit performs an operation that is more effective than the operation performed before the determination that the content of the sentence indicates the existence of the problem phenomenon was performed a predetermined number of times.

6. The system of claim 5.

7. the determination unit refers to the information stored in the database and compares the latest determination with the previous determination each time the determination unit determines whether the content of the text indicates the existence of the problem event; When the previous determination was that the content of the sentence indicates the existence of the problem phenomenon and the latest determination is that the content of the sentence does not indicate the existence of the problem phenomenon, the operation unit stops the operation for solving the problem phenomenon.

6. The system of claim 5.

8. On the computer, An acquisition step of acquiring at least one of an image showing the outside of a vehicle and an image showing the inside of the vehicle from a vehicle equipped with a camera; a generation step of inputting the image acquired in the acquisition step into a language model that generates a sentence explaining the content of the input image, and obtaining a sentence explaining the content of the image; a determination step of determining whether the content of the sentence indicates a problematic event that is a problem in the situation of the vehicle; an action step of performing an action for solving the problem phenomenon when it is determined in the determination step that the content of the sentence indicates the problem phenomenon; Execute the A program characterized by:

Images