Text generation system, text generation program, and method for producing report database

Abstract

A work assistance system (1000) comprises: a vehicle image analysis device (1) that uses generative artificial intelligence (AI) to generate situation-explaining text from an image captured by a camera and that stores the situation-explaining text in a database; a scene designation unit (1001) that designates a scene to be explained; a scene retrieval unit (1002) that retrieves, from situation-explaining text stored in the database, situation-explaining text related to the scene to be explained which has been designated by the scene designation unit (1001); a summary condition setting unit (1003) that saves a time and an explanation subject according to the scene to be explained which has been designated by the scene designation unit (1001); and a report creation device (1004) that, from among the situation-explaining text retrieved by the scene retrieval unit (1002), uses situation-explaining text corresponding to the saved time and explanation subject to generate, via generative AI, a report pertaining to the explanation subject.

Description

Text generation system, text generation program, and report database production method

[0001] The present invention relates to a text generation system, a text generation program, and a method for producing a report database.

[0002] A technology for extracting desired scenes from a video is disclosed in Patent Document 1. Patent Document 1 discloses a method for generating explanatory text, particularly for sports video content. In particular, Patent Document 1 discloses a method for creating a video summary with explanatory text added to video sections that are of interest to the user.

[0003] Japanese Patent Application Laid-Open No. 2005-109566

[0004] However, Patent Document 1 does not take into consideration the development of autonomous vehicles. For example, in the development of autonomous vehicles, developers may prepare reports explaining the results of test drives. For example, developers of autonomous vehicles may analyze videos captured by sensors such as cameras mounted on the autonomous vehicles, identify scenes that should be reported, and prepare reports so that the results of the test drives can be used in subsequent development. If an incident such as an accident occurs during a test drive, a report explaining the situation before and after the accident may be required. Furthermore, a test drive that lasts for an entire day may require reports covering several hours.

[0005] The present invention has been made in consideration of the above circumstances, and its main object is to improve the efficiency of the work of creating reports based on images.

[0006] In order to solve the above problems, the text generation system of the present invention has the following features: An image analysis unit that uses generative AI (artificial intelligence) to generate a situation description from an image captured by a camera and stores the generated situation description in a database, a scene designation unit that designates a target scene that is the subject of the description, a scene search unit that searches the situation descriptions stored in the database for a situation description related to the target scene designated by the scene designation unit, a summary condition setting unit that saves a period and an object to be described according to the target scene designated by the scene designation unit, and a report creation unit that uses the generative AI to generate a report on the object to be described, using the situation description corresponding to the time and object to be described saved by the summary condition setting unit from the situation descriptions searched by the scene search unit.

[0007] The text generation program of the present invention causes a computer to execute the following steps: generating a situation description from an image captured by a camera using a generation AI; storing the situation description in a database; specifying a target scene to be described; searching for a situation description related to the specified target scene from the situation descriptions stored in the database; saving a period and an object to be described according to the specified target scene; and using a generation AI to generate a report on the object to be described using a situation description corresponding to the time and object to be described according to the target scene from the searched situation description.

[0008] The method for producing a report database of the present invention includes the steps of generating a situation description using a generating AI from an image captured by a camera, storing the situation description in a database, a scene designation unit designating a target scene to be described, a scene search unit searching the situation descriptions stored in the database for a situation description related to the target scene designated by the scene designation unit, a summary condition setting unit saving a period and an object to be described according to the designated target scene, generating a report on the object to be described using the generating AI using the situation description corresponding to the time and object to be described stored by the summary condition setting unit among the situation descriptions searched by the scene search unit, and generating a database including the report. Other features will be described later.

[0009] According to the present invention, the work of creating a report based on an image can be made more efficient.

[0010] 13 is a diagram showing the configuration of a work support system according to the present embodiment. FIG. 14 is a flowchart of a report creation process according to the present embodiment. FIG. 15 is a diagram showing details of a driving log database according to the present embodiment. FIG. 16 is a diagram showing an example of still image data that is part of vehicle driving video data according to the present embodiment. FIG. 17 is a diagram showing the configuration of a vehicle image analysis device according to the present embodiment. FIG. 18 is a diagram showing the configuration of a necessity table according to the present embodiment. FIG. 19 is a diagram showing the configuration of a recognition table according to the present embodiment. FIG. 19 is a flowchart of a caption generation unit according to the present embodiment. FIG. 19 is a diagram showing the hardware configuration of a work support system according to the present embodiment. FIG. 19 is a detailed flowchart of an image caption generation process according to the present embodiment. FIG. 19 is a flowchart showing an example of a specific operation of the image caption generation process described in FIG. 10 according to the present embodiment. FIG. 19 is a detailed flowchart of an image caption generation process for general roads according to the present embodiment. FIG. 20 is a table showing intermediate data resulting from executing the image caption generation process on the still image data of FIG. 4 according to the present embodiment. FIG. 21 is a table showing output data resulting from deleting unnecessary data by the caption generation unit from the intermediate data of FIG. 13 according to the present embodiment. FIG. 21 is a diagram showing details of a GPS caption generation process according to the present embodiment. FIG. 22 is a diagram showing details of a control caption generation process according to the present embodiment. FIG. 23 is a table showing examples of the captions generated in FIGS. 15 and 16 according to the present embodiment. 21 is a table showing instructions to a large-scale language model unit used in the generation process of a traffic condition description related to this embodiment. FIG. 22 is a table showing an example generated from an image different from the image description of FIG. 14 related to this embodiment. FIG. 23 is a flowchart showing the processing of a search unit related to this embodiment. FIG. 24 is a diagram showing an image search interface of an input / output unit related to this embodiment. FIG. 25 is a playback screen when the search result (scene 1) of FIG. 21 related to this embodiment is clicked. FIG. 26 is a playback screen when the search result (scene 2) of FIG. 21 related to this embodiment is clicked. FIG. 27 is a flowchart showing search processing. FIG. 28 is a diagram explaining a period including a scene. FIG. 29 is a diagram explaining a table recording a period including a scene. FIG. 30 is a diagram explaining a table in which an explanation object is set. FIG. 31 is a diagram explaining a program startup status description. FIG. 32 is a diagram explaining an object detection status description.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] 1 is a configuration diagram of a work support system 1000. The work support system 1000 includes a driving log database 11, a vehicle image analysis device 1, a scene designation unit 1001, a scene search unit 1002, a summary condition setting unit 1003, and a report creation device 1004. This work support system 100 operates as a document generation system that supports the creation of reports.

[0013] The vehicle image analysis device 1 includes a first generation AI (Artificial Intelligence) 1101. The vehicle image analysis device 1 functions as an image analysis unit that generates a traffic condition description 1006 from an image captured by a camera using the first generation AI 1101 and stores the description in the description database 15 of FIG.

[0014] The scene designation unit 1001 includes a second generation AI 1102 and designates a scene to be explained using the second generation AI 1102. The scene search unit 1002 includes the second generation AI 1102 and searches for a traffic condition description 1006 related to the scene to be explained designated by the scene designation unit 1001 from the traffic condition description stored in the description database 15 of Figure 4.

[0015] When a period before and after a scene and an object to be explained (e.g., the other party in the accident or the point of sudden braking) are specified, the summary condition setting unit 1003 saves the period before and after the specified scene to be explained and the object to be explained. The report creation device 1004 includes a third generation AI 1103. The first generation AI 1101, the second generation AI 1102, and the third generation AI 1103 may be separate generation AIs or may be a single generation AI. When the report creation device 1004 creates a report 1005, it appropriately creates a report database 1007 and stores the report 1005.

[0016] The first generation AI 1101 includes a model trained using images to identify scenes from images captured by a camera. The second generation AI 1102 and the third generation AI 1103 include models trained using text. The first generation AI 1101, the second generation AI 1102, or the third generation AI 1103 may use existing services such as GPT-4, CLIP, BLIP, and BLIP-2.

[0017] FIG. 2 is a flowchart of the report creation process according to this embodiment. In this embodiment, the following four steps are included in order to create the report 1005 shown in FIG. 1 . Each step will be described in detail below. In step S11, the vehicle image analysis device 1 uses an image captured by a sensor such as a camera to generate a traffic condition description 1006 using a first generation AI 1101. In step S12, the scene search unit 1002 searches the traffic condition description 1006 generated in step S11 for scenes necessary for creating the report 1005 using a second generation AI 1102. In step S13, the summarization condition setting unit 1003 specifies a summarization period and an object to be described during this summarization period. In step S14, the report creation device 1004 creates the report 1005 using a third generation AI 1103 based on the results of the search process, the summarization period, and the object to be described during the summarization period. This improves the efficiency of the image-based report creation process. In step S15, report creation device 1004 stores the created report 1005 and the corresponding driving log in report database 1007, and the processing in FIG. 2 ends.

[0018] Step S11: Details of the Description Generation Process The vehicles 91-93 shown in FIG. 1 are so-called connected cars that can communicate with the work assistance system 1000 via a communication line 8. Each vehicle 91-93 transmits various measurement data acquired by the vehicle to a driving log database 11 via the communication line 8. Each vehicle 91-93 is equipped with an onboard camera that captures images. The vehicle image analysis device 1 generates a traffic condition description for the images captured by the onboard camera. The communication line 8 can be a general public line network, whether wired or wireless, such as a fifth-generation mobile communication system (5G), which enables multiple simultaneous connections and ultra-low latency. Furthermore, by taking advantage of the features of new mobile phone systems beyond 5G, effects such as online (real-time while driving) description generation can be expected.

[0019] 3 is a diagram showing details of the driving log database 11. The driving log database 11 stores measurement data from each vehicle 91 to 93 by data type as follows: Vehicle driving video data 11A is video data captured by an on-board camera (not shown) while the vehicle is driving or stopped. Vehicle driving GPS data 11B is on-board GPS (Global Positioning System) data. Vehicle driving control data 11C is vehicle control data such as speed, acceleration / deceleration, and steering angle, which is vehicle driving log data obtained from an on-board ECU (Electronic Control Unit) or the like. Note that GPS is an example of a satellite positioning system.

[0020] 4 is a diagram showing an example of still image data 111, which is part of the vehicle driving video data 11A. The still image data 111 is an example of data extracted from the vehicle driving video data 11A captured by each of the vehicles 91 to 93 driving on a highway.

[0021] FIG. 5 is a configuration diagram of the vehicle image analysis device 1. In addition to the driving log database 11 described in FIG. 3, the vehicle image analysis device 1 includes an explanation generation unit 12, a large-scale language model unit 13, an explanation target setting unit 14, an explanation database (image database) 15, a search unit 16, and an input / output unit 17. The explanation target setting unit 14 includes a necessity table 14A and a recognition table 14B. The large-scale language model unit 13 includes a VQA (Visual Question Answer) unit 13A and a summary generation unit 13B. Note that the various data stored in the vehicle image analysis device 1 (the driving log database 11, the necessity table 14A, and the recognition table 14B) may be stored in a storage device (not shown) external to the vehicle image analysis device 1 and may be configured to be accessible from the storage device via a network.

[0022] The description generation unit 12 analyzes the driving situation of each vehicle using the large-scale language model unit 13 and the explanation target setting unit 14 based on the data in the driving log database 11, and generates natural sentences explaining the driving situation of the vehicle according to the analysis results. To this end, the description generation unit 12 analyzes whether the driving situation of the vehicle corresponds to any of the traffic scenes classified in advance.

[0023] The large-scale language model unit 13 is called by the explanation generation unit 12. The large-scale language model unit 13 is realized by LAVIS (LAnguage VISion) or the like, which uses a natural language conversational interface for exchanging questions and answers, and has the following processing units.

[0024] The VQA unit 13A responds to queries about images in natural language in natural language. To this end, the VQA unit 13A prepares an image recognition model such as a convolutional neural network (CNN) using training data in advance, and inputs a query to the image recognition model to obtain a description of the corresponding image.

[0025] The summary generation unit 13B responds by summarizing (integrating multiple sentences) the content of the input natural language (prompt) as a traffic situation description. The summary generation unit 13B may use existing services such as GPT-4, CLIP, BLIP, and BLIP-2 as a text generation AI service that performs summarization and translation processing.

[0026] The explanation object setting unit 14 sets explanation objects corresponding to traffic scenes in the explanation generation unit 12 by referring to the following tables, and stores them in the explanation database 15. The necessity table 14A (FIG. 6) associates explanation objects with each traffic scene and defines the degree of importance of whether or not to mention each explanation object in the explanation. The recognition table 14B (FIG. 7) defines the detailed content to be mentioned in the explanation for each explanation object.

[0027] The explanation database 15 stores the traffic condition explanations generated by the explanation generation unit 12 .

[0028] In this way, the description generator 12 executes the following steps (1) to (3): (1) Identifying the scenes captured in the images received from the camera, and for each identified scene, reading the recognition necessity information for the objects in the image from the necessity table 14A; (2) Recognizing the objects in the image that are designated as requiring recognition in the read recognition necessity information, and generating a description for each object from the recognition results; (3) Generating a traffic condition description for the image based on the identified scenes and the descriptions for each object, and storing the traffic condition description for the image in association with the image in the description database 15.

[0029] FIG. 6 is a diagram illustrating the configuration of the necessity table 14A. The explanation object setting unit 14 sets explanation objects for each traffic scene, such as a highway, a general road, and a parking lot. For example, the combination of a traffic scene "highway" and an explanation object "pedestrian" is "necessary / necessary." The recognition necessity information "necessary" on the left side of this "necessary / necessary" notation indicates that the explanation object needs to be recognized from the image. Furthermore, the explanation necessity information "necessary" on the right side of the "necessary / necessary" notation indicates that an explanation object that is recognized or not recognized from the image needs to be explained in the explanation text. Therefore, in the traffic scene "general road," recognition of pedestrians is "necessary," and even if pedestrians are not recognized, an explanation is set as "necessary." On the other hand, in the traffic scene "general road," recognition of pedestrian crossings is "necessary," but an explanation is not required.

[0030] In this way, the description generator 12 reads the explanation necessity information for an object in the image from the explanation necessity table 14A, and if the object specified as requiring explanation in the read explanation necessity information cannot be recognized in the image, the description generator 12 generates, as an explanation for each object, a description indicating that the object is not present in the image. This allows for generating an explanation only for objects that are not normally present in the traffic scene in which the vehicle is traveling, and for objects that are normally present in the traffic scene in which the vehicle is traveling, generating an explanation even when the object is not present. This has the effect of generating more natural traffic situation descriptions and improving the search accuracy for user search queries.

[0031] 7 is a diagram illustrating the configuration of the recognition table 14B. The recognition table 14B defines, as detailed items, items to be analyzed by the VQA unit 13A for the recognized explanation object and items to include the analysis results in the explanation. For example, when a pedestrian is recognized, the VQA unit 13A analyzes the location, color of clothing, and movement of the pedestrian.

[0032] In this way, the description generator 12 reads detailed items about the objects in the image from the recognition table 14B and generates a description for each object based on the read detailed items about the objects recognized in the image. This makes it possible to individually set the information to be added depending on the object being described, and generates natural traffic situation descriptions. This improves the search accuracy for user search queries.

[0033] 8 is a flowchart of the description generation unit 12. As an image description generation process (step S21), the description generation unit 12 generates an image description by causing the VQA unit 13A to perform image analysis on still image data 111 extracted from the vehicle driving video data 11A. The extraction process of the still image data 111 is a process of extracting images continuously captured at regular intervals, such as every 10 seconds, or extracting 10 images at equal time intervals from one video file.

[0034] In the GPS description generation process (step S22), the description generator 12 generates a GPS description based on the vehicle driving GPS data 11B. At this time, the target GPS data (positioning data) is the GPS data from the same time as the still image data 111 of S21, or the GPS data from the closest time. That is, the description generator 12 adds a description relating to at least one of information on the time period when the image was captured and information on the driving position when the image was captured to the traffic condition description of the image, based on the GPS data read from the onboard GPS of the vehicle 91.

[0035] The description generator 12 generates a control description based on the vehicle driving control data 11C as a control description generation process (step S23). At this time, the target control data is the control data that is generated at the same time as the still image data 111 of S21 or the closest time. That is, the description generator 12 adds a description relating to at least one of speed information, acceleration / deceleration information, and steering angle information to the traffic condition description of the image based on the vehicle driving control data read from the on-board ECU (Electronic Control Unit) of the vehicle 91.

[0036] In the traffic condition description generation process (step S24), the description generator 12 generates a traffic condition description by having the summary generator 13B summarize the image description generated in step S21, the GPS description generated in step S22, and the control description generated in step S23. The description generator 12 associates the generated traffic condition description with the still image data 111 that is the description target of the traffic condition description, and stores them in the description database 15. That is, the summary generator 13B generates a summary in accordance with the input prompt. The description generator 12 then generates a traffic condition description for the image by inputting a prompt to the summary generator 13B, the prompt including information to be included in the traffic condition description, information to be excluded from the traffic condition description, and an instruction for generating a traffic condition description based on information on example sentences of the traffic condition description, as well as a description for each object.

[0037] FIG. 9 is a hardware configuration diagram of the vehicle image analyzing device 1. The vehicle image analyzing device 1 is configured as a computer 900 having a CPU 901, a RAM 902, a ROM 903, a HDD 904, a communication I / F 905, an input / output I / F 906, and a media I / F 907. The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from a recording medium 917 and installs a document generation program stored in the recording medium 917. Furthermore, the CPU 901 embodies each processing unit in FIG. 1 by executing the document generation program (also called an application or an app for short) loaded into the RAM 902. This document generation program can also be distributed via a communication line or recorded on a recording medium 917 such as a CD-ROM.

[0038] 10 is a detailed flowchart of the image caption generation process (step S21 in FIG. 8). The caption generation unit 12 classifies traffic scenes captured in the still image data 111 by querying the VQA unit 13A (step S211). The VQA unit 13A receives input of still image data 111 and a traffic scene query (e.g., "Where is this scene? For example, is it a road, a highway, or a parking lot?") and sends an answer (e.g., a highway) to the explanation generator 12 (line A01 in FIG. 13 ). Note that the explanation generator 12 reads the traffic scene query set in advance by the administrator in the vehicle image analysis device 1, so the user of the vehicle image analysis device 1 does not need to generate the traffic scene query themselves. Hereinafter, the explanation generator 12 executes the loop processing of steps S212 to S217, sequentially selecting explanation objects registered in the necessity table 14A (pedestrians, bicycles, automobiles, etc.). Hereinafter, the explanation object selected in this loop processing will be referred to as the selected object.

[0039] The description generation unit 12 determines whether or not the selected object needs to be recognized in the traffic scene identified in step S221 by referring to the necessity table 14A (step S212). If the answer is Yes (necessary) in step S212, the process proceeds to step S213; if the answer is No, the process proceeds to step S217. The description generation unit 12 queries the VQA unit 13A of the large-scale language model unit 13 to determine whether or not the selected object exists in the still image data 111. This query may be, for example, "Are there any pedestrians in this scene?" (line A02 in FIG. 13). The description generation unit 12 recognizes the object designated as requiring recognition from within the image based on the response to this query (object recognition result). As a result of this recognition, the description generation unit 12 determines whether or not the selected object exists in the still image data 111 (step S213). If the answer is Yes (exists) in step S213, the process proceeds to step S214; if the answer is No, the process proceeds to step S215.

[0040] The description generation unit 12 acquires detailed items about the selected object present in the still image data 111 (step S214). To this end, the description generation unit 12 references the recognition table 14B to acquire detailed items corresponding to the selected object. Then, the description generation unit 12 queries the VQA unit 13A of the large-scale language model unit 13 for detailed items about the selected object in the still image data 111 one by one. For example, since the selected object is a car and the detailed items corresponding to the car in the recognition table 14B include "color," the description generation unit 12 generates a query such as "What color is the car?"

[0041] The explanation generator 12 determines whether an explanation (mention) of the selected object is necessary by referring to the necessity table 14A (step S215). If the answer is Yes (necessary) in step S215, the process proceeds to step S216; if the answer is No, the process proceeds to step S217. The explanation generator 12 generates an explanation for the selected object (step S216) using one of the following methods: If the answer is Yes in step S213, the explanation for the selected object is generated from a combination of a query about detailed items about the selected object acquired in step S214 and its response. If the answer is No in step S213, the explanation generator 12 generates an explanation for the selected object indicating that the selected object was not recognized in the still image data 111 (e.g., row D07 in Figure 19). The explanation generator 12 determines whether processing for all selected objects has been completed by finishing processing the current selected object (step S217). If the answer is Yes (completed) in step S217, the process ends; if the answer is No, the process switches to an unprocessed selected object and the process returns to step S212. This allows for the generation of natural traffic situation descriptions that include explanations of surrounding objects that should be noted and confirmed according to the traffic scene in which the vehicle is traveling, thereby improving the search accuracy for user search queries.

[0042] Fig. 11 is a flowchart showing an example of a specific operation of the image caption generation process described in Fig. 10. The caption generation unit 12 branches into processing for each traffic scene as follows (step S301) depending on the result of the process of classifying traffic scenes captured in the still image data 111 (step S211 in Fig. 10). If the classification result is an ordinary road, an image caption for ordinary roads is generated (step S302). If the classification result is an expressway, an image caption for expressways is generated (step S303). If the classification result is a parking lot, an image caption for parking lots is generated (step S304). If the classification result is other, an image caption specified for other uses is generated (step S305).

[0043] FIG. 12 is a detailed flowchart of the image description generation process for general roads (S302 in FIG. 11 ). The description generation unit 12 generates a traffic scene question and answer (step S211 in FIG. 10 ) and a traffic light question and answer (step S311). As the determination process (step S213) in FIG. 10 , the description generation unit 12 asks whether a pedestrian is present in the image (step S312). If a pedestrian is present, the process proceeds to step S313. If a pedestrian is not present, the process proceeds to step S314. The description generation unit 12 generates a question and answer regarding the presence of a pedestrian, a pedestrian location question and answer, a pedestrian color question and answer, and a pedestrian action question and answer (step S313) as answers to questions regarding pedestrian details (step S214 in FIG. 10 ). The description generation unit 12 generates a question indicating the absence of a pedestrian (step S314).

[0044] As part of the determination process (step S213) of FIG. 10 , the description generator 12 asks whether or not a car is present in the image (step S315). If a car is present, the process proceeds to step S316; if not, the process proceeds to step S317. The description generator 12 generates, as questions and answers about the car's details (step S214 of FIG. 10 ), a response indicating the presence of a car, a question and response about the car's location, a question and response about the car's model, a question and response about the car's color, and a question and response about the car's operation (step S316). As part of the determination process (step S213) of FIG. 10 , the description generator 12 asks whether or not a bicycle is present in the image (step S317). If a bicycle is present, the process proceeds to step S318; if not, the process proceeds to step S319. The explanation generation unit 12 generates, as answers to questions about the bicycle's detailed items (step S214 in FIG. 10), an answer sentence indicating the existence of a bicycle, a question and answer sentence about the bicycle's location, a question and answer sentence about the bicycle's color, and a question and answer sentence about the bicycle's operation (step S318).

[0045] As part of the determination process (step S213) in Fig. 10, the description generator 12 asks whether a pedestrian crossing is present in the image (step S319). If present, the process proceeds to step S320, and if not, the process ends. The description generator 12 generates a response indicating the presence of a pedestrian crossing (step S320). While the process of generating an image description for an ordinary road has been described above with reference to Fig. 11, the process of generating an image description for other traffic scenes (steps S303 to S305) is similar.

[0046] FIG. 13 is a table showing intermediate data resulting from the image description generation process (step S21) performed on the still image data 111 of FIG. 4. In this table, each row of the image description is composed of a question and an answer to that question. For example, row A01 is the result of the process of classifying traffic scenes (step S211). Rows A02 to A04 are the results of the process of determining whether a selected object exists in the still image data 111 (step S213). Rows A05 to A13 are the results of the process of acquiring detailed items about the selected object (step S214).

[0047] Fig. 14 is a table showing output data resulting from the explanation generation unit 12 deleting unnecessary data from the intermediate data of Fig. 13. The difference from Fig. 13 is that the explanation for pedestrians (row A02), bicycles (row A03), and toll booths (row A13), which are deemed unnecessary when not recognized in the necessity table 14A of the explanation target setting unit 14, have been deleted in Fig. 14.

[0048] FIG. 15 is a diagram showing details of the GPS description generation process (step S22). The description generator 12 generates a description that classifies the capture time of the still image data 111 into morning, afternoon, evening, night, etc., according to the time period of the GPS information obtained from the vehicle travel GPS data 11B (step S221). The description generator 12 generates a description that identifies the capture location of the still image data 111 as a major trunk road or a city name according to the latitude and longitude of the GPS information obtained from the vehicle travel GPS data 11B (step S222). This allows the generation of a traffic condition description that includes information about the vehicle's travel time and location. This improves the search accuracy for user search queries.

[0049] FIG. 16 is a diagram showing details of the control explanation generation process (step S23). The explanation generator 12 generates an explanation regarding driving speed control from the vehicle driving control data 11C (step S231). The explanation may be, for example, a low speed for a speed of 20 km / h or less, a medium speed for a speed of 20 km / h to 60 km / h, and a high speed for a speed of 60 km / h or more. The explanation generator 12 generates an explanation regarding acceleration / deceleration control from the vehicle driving control data 11C (step S232). The explanation may be, for example, an acceleration state when acceleration is above a certain level, a deceleration state when deceleration is above a certain level, and a constant speed state otherwise. The explanation generator 12 generates an explanation regarding steering control from the vehicle driving control data 11C (step S233). The explanation may be, for example, a left turn state or a right turn state when a steering angle is above a certain level, and a straight-ahead state otherwise. This allows for the generation of a more natural traffic situation explanation that includes an explanation of the vehicle's control state and behavior. This has the effect of improving the accuracy of searches performed by users.

[0050] Figure 17 is a table showing an example of the explanations generated in Figures 15 and 16. In row B01, an explanation of the time period is generated in step S221. In row B02, an explanation of the driving location is generated in step S222. In row B03, an explanation of the driving speed is generated in step S231. In row B04, an explanation of the acceleration / deceleration state is generated in step S232. In row B05, an explanation of the steering state is generated in step S233.

[0051] 18 is a table showing instructions to the large-scale language model unit 13 used in the traffic condition description generation process (step S24). Line C01 lists an instruction to generate a traffic condition description. Line C02 lists information to be included in the traffic condition description. Line C03 lists information to be excluded from the traffic condition description. Line C04 lists an example of a traffic condition description.

[0052] Then, in the traffic condition description generation process (step S24), the description generation unit 12 generates a prompt by sequentially combining the following texts (1) to (4). Note that at least one of (2) and (3) may be omitted: (1) Instructions to the large-scale language model unit 13 (FIG. 18); (2) GPS description (lines B01 and B02 in FIG. 17); (3) Control description (lines B03, B04, and B05 in FIG. 17); (4) Image description (FIG. 14).

[0053] The description generation unit 12 inputs the generated prompt into the large-scale language model unit 13 to obtain a traffic situation description written in natural language. An example of a traffic situation description (corresponding to traffic situation description 732 in FIG. 22 described later) generated by the large-scale language model unit 13 from the prompt is shown below: Traffic situation description = "It is noon and you are driving straight on Metropolitan Expressway Route 5 while slowing down at high speed. In this scene, there are solid orange lane lines on the expressway. Furthermore, a white truck is driving ahead on the road ahead of you." This generates natural traffic situation descriptions that suit the purpose of use and the user's preferences, thereby improving the search accuracy for user search queries.

[0054] FIG. 19 is a table showing an example generated from an image different from the image caption in FIG. 14 . The image caption in FIG. 19 targets an image captured from a vehicle traveling on a public road. Therefore, since the combination of a public road and a pedestrian is "required" in the necessity table 14A, the information "no pedestrians were recognized" is entered in row D07. The traffic situation caption generated by the large-scale language model unit 13 from the prompt including the image caption in FIG. 19 through the traffic situation caption generation process (step S24) is as follows: Traffic situation caption = "You are currently traveling at a slow and steady speed on a city road in Mito City at night, preparing to turn left. There are no pedestrians in this scene."

[0055] The creation of a database of traffic condition descriptions (the process up to storing the data in the description database 15) has been described above with reference to up to FIG. 19 . An example of utilizing the databased information will be described below. FIG. 20 is a flowchart showing the process of the search unit 16. The search unit 16 searches for images that match an input search statement from among the images stored in the description database 15. Specifically, the search unit 16 outputs, as search results, images that have a high similarity between the input search statement and the traffic condition descriptions stored in the description database 15. Note that the search unit 16 may, for example, list the images in the search results in order of similarity and output the top X images as the "highly similar images" (i.e., a relative similarity determination), or may output search results whose similarity is higher than a predetermined reference value (threshold Y) (i.e., an absolute similarity determination). The process of the search unit 16 will be described in detail below.

[0056] The search unit 16 receives a search statement entered by a user from the input / output unit 17 (step S61). The user in this example is, for example, a commentator at a traffic control center that manages expressways. The search statement, for example, requests a database to collect images of situations similar to an accident that occurred at a specific location on the expressway at a specific time. The commentator plans to edit the image materials obtained from the database to produce a news program about the accident. The search unit 16 evaluates the similarity between the user's search statement and the descriptions stored in the description database 15 (step S62), and retrieves video information (image information) associated with the description with a high similarity from the description database 15. The similarity evaluation may be performed using a cosine similarity search based on document vectorization.

[0057] In addition to the driving video and traffic condition description acquired in step S62, the search unit 16 searches for and acquires related information about the driving condition (such as weather information that is not included in the description database 15 but can be acquired from a weather database by specifying the location and date of the traffic condition description), and generates a response based on these results (step S63). That is, the search unit 16 may also output related information acquired from a database other than the description database 15 based on information included in the traffic condition description corresponding to an image with a high degree of similarity as a search result. The search unit 16 transmits the response obtained in step S63 to the input / output unit 17 (step S64). This allows the user to quickly search for the desired video when searching for video data containing a traffic condition description similar to the natural language search query entered by the user.

[0058] 21 is a diagram showing an image search interface 71 of the input / output unit 17. The image search interface 71 is composed of a search statement input section 72, which is an input field for the search statement in step S61, and a search result display section 73, which is a display field for the answer statement in step S64. The search result display section 73 displays multiple search results (scenes 1 to 4) as icons or thumbnail images.

[0059] Figure 22 shows the playback screen when the search result (scene 1) in Figure 21 is clicked. This playback screen displays the following information from top to bottom: - Image 731 of the search result. - Traffic condition description 732 of image 731 was extracted as having a high similarity to the search statement. - Additional information 733 such as GPS description (time, location), control description (vehicle type, driving speed), and weather.

[0060] Figure 23 shows the playback screen when the search result (scene 2) in Figure 21 is clicked. As in Figure 22, this playback screen displays a search result image 741, its traffic condition description 742, and additional information 743, in the same manner as in Figure 22. This allows the user to input a search query in natural language and then search for videos by referring to traffic condition descriptions, videos, and related information that are similar to the search query, thereby improving search efficiency.

[0061] According to the description generation process described above, when generating a description of a vehicle image, the description generator 12 refers to the necessity table 14A to generate a natural description that includes the presence or absence of peripheral objects that should be noted based on the traffic scene that the vehicle is in. This allows a database to be created of natural descriptions that mention necessary peripheral objects but do not mention unnecessary peripheral objects, thereby improving the accuracy of database searches using search queries entered by humans.

[0062] Step S12: Details of Search Processing Next, we will explain the details of the search processing by the scene designation unit 1001. The search processing is a process of extracting image descriptions related to a specific scene from the traffic condition descriptions generated in step S11.

[0063] As described above, the scene designation unit 1001 includes the second generation AI 1102. Therefore, even if the worker is not an experienced worker, a specific scene can be extracted through the dialogue between the worker and the second generation AI 1102 shown in Fig. 24. The flowchart in Fig. 24 will be described below with reference to Fig. 1 as appropriate.

[0064] In step S71, the scene designation unit 1001 accepts a prompt input by the operator. The prompt may be, for example, "Please tell us about incidents that occurred during driving between 10:00 AM and 8:00 PM."

[0065] In step S72, the scene designation unit 1001 provides a prompt to the second generation AI 1102, causing the second generation AI 1102 to access the explanatory sentence database 15. The second generation AI 1102 generates a response sentence to the prompt (step S73). An example of the response is, "The incident that occurred while driving between 10:00 a.m. and 8:00 p.m. was a collision with a pedestrian."

[0066] In step S73, the scene designation unit 1001 accepts a prompt input by the operator. The prompt may be, for example, "Please search for traffic situation descriptions related to collisions with pedestrians."

[0067] In step S74, the scene designation unit 1001 provides a prompt to the second generation AI 1102, causing the second generation AI 1102 to access the description database 15 and extract the relevant traffic condition description 1006. When the processing of step S74 is completed, the processing of Figure 24 is completed.

[0068] The prompts input by the worker include, for example, the vehicle location, such as "highway," "general road," or "parking lot," as shown in FIG. 6. The prompts input by the worker also include, for example, incidents, such as "sudden braking," "collision with another vehicle," "lane departure," or "failure of equipment within the vehicle." The prompts input by the worker also include times, such as "10:00 AM to 8:00 PM" or "3:00 PM to 3:10 PM."

[0069] Furthermore, the present invention does not necessarily require the second generation AI 1102. The scene designation unit 1001 includes an interface for the operator to designate a specific scene, and can directly access the description database 15 and extract related traffic condition descriptions 1006 based on the natural language input by the operator, similar to step S11, without going through the second generation AI 1102.

[0070] Step S13: Details of Condition Setting Process Next, details of the condition setting process by the summarization condition setting unit 1003 will be described with reference to Fig. 1. The condition setting process is a process for specifying traffic condition descriptions 1006 that meet the conditions from the search results of step S12 and specifying the objects to be described in the report 1005 in order to generate the report 1005. The summarization condition setting unit 1003 stores, for example, two conditions.

[0071] The first condition is a period that includes a scene. The period that includes a scene means, for example, from time (-T2) to time T1 in FIG. 25. Depending on the scene, information after time T0 may be important, or information before time T0 may be important. Therefore, time T1 and time (-T2) may take the same value, or they may take different values.

[0072] 26 is a diagram illustrating a table recording a period including a scene. As shown in FIG. 26, for example, the period including a scene, including how much of the period before the scene and how much of the period after the scene is included, can be set for each scene, and can be saved as table 2401 in summary condition setting unit 1003. Note that it is also within the scope of the disclosure of this embodiment to create report 1005, which will be described later, using only either time T1 or time (-T2).

[0073] Note that scenes include both incidents and non-incidents. Examples of incidents include collisions between the vehicle and a pedestrian, collisions between the vehicle and another vehicle, and collisions between the vehicle and a structure. Examples of non-incidents include sudden braking, lane departure, vehicle breakdown, violations of road traffic laws, and speeding.

[0074] 27 is a diagram illustrating a table in which an explanation object is set for each scene. The second condition is the explanation object. The explanation object is an object mentioned in the report 1005. As shown in FIG. 27, for example, the explanation object can be set for each scene and stored in the summary condition setting unit 1003 as a table 2501.

[0075] The first and second conditions can be set in advance by the worker. As will be described later, the report 1005 is created according to the conditions specified in step S13. Therefore, the conditions in step S13 function as a prompt to the third generation AI 1103. In other words, the prompt to the third generation AI 1103 is instructed to "Please include buildings, intersections, pedestrians, and crosswalks as the objects to be described in the report." By storing standard prompt information in a table in advance in this way, it is possible to give instructions to the generation AI without error.

[0076] Step S14: Details of the report creation process Next, we will explain the details of the report creation process by the report creation device 1004. In step S14, the third generation AI 1103 in the report creation device 1004 creates a report 1005, which is a summary of the traffic situation description 1006 that meets the conditions in step S13.

[0077] For example, if the worker selects a pedestrian collision in step S12, the third generation AI 1103 generates a report 1005. The generated report 1005 is stored in the explanatory sentence database 15 in a format that the worker can search in natural language. The report may be, for example, as follows: "The video begins with you driving through a city street, capturing a typical driving scenario in an urban environment. As you progress, new high-rise buildings become more prominent along your route, suggesting you're in a developed area of ​​the city. Approaching an intersection near this landmark is a common location for increased traffic and pedestrian activity. Soon after, while driving near a crosswalk, a pedestrian appears on your right, indicating a potential collision or distraction point. The situation quickly escalates, and in the next moment, you see a person in your vehicle's windshield, a sudden and startling scene suggesting a pedestrian collision. In the final scene, you're driving through a crosswalk and a person is seen lying in the street, indicating a serious accident likely resulting from an earlier interaction with a pedestrian. Overall, the video captures the progression from normal city driving to a serious incident involving a pedestrian, suggesting a serious accident at a crosswalk."

[0078] As described above, the description generator 12 generates a control description based on the vehicle driving control data 11C as part of the control description generation process (step S23). The description generator 12 can also add a description related to vehicle control to the traffic condition description 1006 for the image based on the vehicle driving control data read from an on-board ECU (Electronic Control Unit) of the vehicle 91. Therefore, the report 1005 can also reflect information from the on-board ECU.

[0079] The information of the on-board ECU includes, for example, information about sensors such as cameras. The information about the sensors includes information suggesting whether a program for detecting objects such as pedestrians by the sensors such as cameras was running, and, if the program was running, information suggesting whether an object such as a pedestrian was detected. Furthermore, the information of the on-board ECU includes, for example, information about vehicle behavior. The information about vehicle behavior includes information about speed, acceleration / deceleration, acceleration, jerk information which is the time derivative of acceleration, yaw rate, brake hydraulic pressure, and / or steering angle.

[0080] For example, if report 1005 states that a program for detecting objects such as pedestrians was not running, the developer can improve the program so that it will run in similar scenes. Also, if report 1005 states that the program was running but did not detect an object, the developer can improve the program so that it can detect objects. Also, if report 1005 states that the object was detected, the developer can know that there is room for improvement in vehicle control.

[0081] In addition, similar to the image description, GPS description, and control description, it is also within the scope of the disclosure of this embodiment that the description generation unit 12 creates the program startup status description 2801 in Figure 28 and the object detection status description 2901 in Figure 29 and inputs them to the third generation AI 1103 as one of the prompts so that the third generation AI 1103 can more reliably refer to the program startup status and / or object detection status.

[0082] 28 and the object detection status description 2901 in Fig. 29, it is also within the scope of the disclosure of this embodiment to collect the program startup status and / or object detection status from at least one of the vehicles 91 to 93 in association with time information and store the collected information in the driving log database 11. Furthermore, it is also within the scope of the disclosure of this embodiment to output the program startup status description 2801 and / or the object detection status description 2901 in association with an image as one of the additional information 733, 743.

[0083] According to this embodiment, it is possible to improve the efficiency of the work of creating the report 1005. By making it possible to specify the time before and after and / or the subject of explanation, it is possible to avoid creating a redundant report 1005. The present invention is particularly effective in the development of autonomous vehicles.

[0084] The configuration and effects of the present invention will be described below.

[0085] [1] An image analysis unit (vehicle image analysis device 1) that generates a situation description from an image taken by a camera using a generation AI (artificial intelligence) (first generation AI 1101) and stores the generated situation description in a database; a scene designation unit (1001) that designates a scene to be described that is the subject of the description; a scene search unit (1002) that searches the situation descriptions stored in the database for a situation description related to the scene to be described designated by the scene designation unit (1001); a summary condition setting unit (1003) that saves a time and an object to be described according to the scene to be described designated by the scene designation unit (1001); and a report creation unit (report creation device 1004) that uses a generation AI (third generation AI 1103) to create a report (1005) related to the object to be described using a situation description corresponding to the time and the object to be described stored by the summary condition setting unit (1003) from the situation descriptions searched by the scene search unit (1002). A text generation system (task support system 1000) characterized by having the above.

[0086] This allows for the efficient creation of reports based on images.

[0087] [2] The text generation system according to claim 1, characterized in that the scene designation unit (1001) designates an explanation target scene that is the subject of the explanation by a generation AI (second generation AI 1102).

[0088] By using the generation AI, it is easy to specify the scene to be explained.

[0089] [3] The sentence generation system according to claim 1, wherein the camera is an in-vehicle camera.

[0090] This makes it possible to create reports in the development of autonomous vehicles in an ideal manner.

[0091] [4] The document generation system according to claim 3, wherein the report (1005) includes information related to an in-vehicle ECU.

[0092] This makes it possible to create reports in the development of autonomous vehicles in an ideal manner.

[0093] [5] The sentence generation system according to claim 4, wherein the information related to the in-vehicle ECU includes information related to sensors connected to the in-vehicle ECU and information related to vehicle behavior.

[0094] This makes it possible to create reports in the development of autonomous vehicles in an ideal manner.

[0095] [6] A text generation program for causing a computer to execute the following steps: generating a situation description from an image captured by a camera using a generation AI (first generation AI 1101); storing the situation description in a database; specifying a target scene to be described; searching for a situation description related to the specified target scene from the situation descriptions stored in the database; saving the time and object to be described according to the specified target scene; and generating a report on the object to be described using a generation AI (third generation AI 1103) using a situation description corresponding to the time and object to be described according to the target scene from the searched situation description.

[0096] This allows for the efficient creation of reports based on images.

[0097] [7] A method for producing a report database, comprising: a step of generating a situation description using a generation AI (first generation AI 1101) from an image taken by a camera; a step of storing the situation description in a database; a step by a scene designation unit (1001) designating a target scene to be described; a step by a scene search unit (1002) searching for a situation description related to the target scene designated by the scene designation unit (1001) from the situation descriptions stored in the database; a step by a summary condition setting unit (1003) saving a time and an object to be described according to the designated target scene; a step of generating a report (1005) related to the object to be described using a generation AI (third generation AI 1103) using a situation description corresponding to the time and object to be described stored by the summary condition setting unit (1003) from the situation descriptions searched by the scene search unit (1002); and a step of generating a database (1007) including the report (1005).

[0098] This allows for the efficient creation of reports based on images.

[0099] The present invention can be applied to applications other than the development of autonomous vehicles. For example, the present invention can be applied to the following fields: (1) In the logistics field, the present invention can be applied to creating daily reports for drivers. (2) In the education field, the present invention can be applied to summarizing related content based on keywords from online lectures, seminars, and academic conferences. It can also be applied to summarizing the content of lecture questioners.

[0100] (3) In the field of crime prevention, the present invention can be applied to reports summarizing security camera footage, which can be used to detect, for example, vehicle theft, snatching, shoplifting, and suspicious individuals during late-night hours.

[0101] (4) In the retail sector, the present invention can be applied to the creation of reports using anti-theft camera data. The reports can be used to search for theft scenes. The present invention can also be applied to the creation of reports using camera data from unmanned stores and self-checkout registers.

[0102] (5) In the broadcasting field, the present invention can be applied to summarizing video before and after sports scoring scenes and program highlights. (6) In the medical field, the present invention can be applied to summarizing surgical videos. The created report functions as a textbook. Therefore, by referring to the report, it is possible to confirm procedures. It is also possible to search for desired procedures. As a result, training can be made more efficient.

[0103] (7) In the fields of factory, construction site, and warehouse management, the present invention can be applied to the creation of accident reports. Accidents can include objects falling or tipping over. Reports can be created from videos of the site. From the reports, it becomes possible to search for signs of failure. The present invention can also be applied to the output of work reports with videos, the creation of inspection and maintenance reports, and the creation of procedure manuals that describe maintenance procedures.

[0104] (8) In the field of manufacturing, the present invention makes it possible to summarize the skills of skilled craftsmen and other skilled technicians. (9) In the field of building and elevator management, the present invention makes it possible to create a report summarizing the number of people entering and exiting buildings by time period. The report can be used for persona analysis.

[0105] (10) In the field of home appliances, the present invention can also create consumption reports from consumption records obtained from refrigerators equipped with sensors such as cameras. (11) The present invention can also be applied to summarizing cooking videos, summarizing product and service introduction videos, and summarizing the behavioral history of pets and children.

[0106] Furthermore, the present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit of the present invention as defined in the claims. For example, the above-described embodiments provide a detailed and specific description of the configuration of the vehicle image analysis device 1 in order to clearly explain the present invention, and the present invention is not necessarily limited to a configuration including all of the described components. Furthermore, it is possible to replace part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. It is also possible to add, replace, or delete other components from part of the configuration of each embodiment.

[0107] Furthermore, some or all of the above-described configurations, functions, processing units, etc. may be implemented in hardware, for example, by designing them as integrated circuits. Broadly defined processor devices such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits) may also be used as hardware. Furthermore, the components of the vehicle image analysis device 1 according to the above-described embodiment may be implemented in any hardware as long as the respective hardware can transmit and receive information to and from each other via a network. Furthermore, the processing performed by a certain processing unit may be implemented by a single piece of hardware, or may be implemented by distributed processing using multiple pieces of hardware.

[0108] DESCRIPTION OF SYMBOLS 1 Vehicle image analysis device (image analysis unit) 8 Communication line 11 Driving log database 11A Vehicle driving video data 11B Vehicle driving GPS data 11C Vehicle driving control data 12 Description generation unit 13 Large-scale language model unit 13A VQA unit 13B Summary generation unit 14 Description target setting unit 14A Necessity table 14B Recognition table 15 Description database (image database) 16 Search unit 17 Input / output unit 91-93 Vehicle 100 Image description system 111 Still image data 732, 742 Situation description 901 CPU 902 RAM 903 ROM 904 HDD 905 Communication I / F 906 Input / output I / F 907 Media I / F 1000 Work support system 1001 Scene designation unit 1002 Scene search unit 1003 Summary condition setting unit 1004 Report creation device 1005 Report 1006 Traffic condition description 1101 First generation AI 1102 Second generation AI 1103 Third generation AI

Claims

1. A text generation system comprising: an image analysis unit that uses generative AI (Artificial Intelligence) to generate a situation description from an image taken by a camera and stores the generated description in a database; a scene designation unit that designates a target scene that is the subject of the description; a scene search unit that searches the situation descriptions stored in the database for a situation description related to the target scene designated by the scene designation unit; a summary condition setting unit that saves a period and an object to be described according to the target scene designated by the scene designation unit; and a report creation unit that uses generative AI to generate a report on the object to be described using a situation description corresponding to the time and object to be described saved by the summary condition setting unit from the situation descriptions searched by the scene search unit.

2. The text generation system according to claim 1, characterized in that the scene designation unit designates an explanation target scene that is the subject of the explanation using a generation AI.

3. The sentence generation system according to claim 1, characterized in that the camera is an in-vehicle camera.

4. The text generation system according to claim 3, wherein the report includes information about an in-vehicle ECU.

5. The sentence generation system according to claim 4, wherein the information relating to the vehicle ECU includes information relating to sensors connected to the vehicle ECU and information relating to vehicle behavior.

6. A text generation program that causes a computer to execute the following steps: generating a situation description from an image taken by a camera using a generation AI; storing the situation description in a database; specifying a target scene to be described; searching for a situation description related to the specified target scene from the situation descriptions stored in the database; saving the time and object to be described according to the specified target scene; and generating a report on the object to be described using a generation AI using a situation description corresponding to the time and object to be described according to the target scene from the searched situation descriptions.

7. A method for producing a report database, comprising the steps of: generating a situation description using a generating AI from an image taken by a camera; storing the situation description in a database; a scene designation unit designating a target scene to be described; a scene search unit searching for a situation description related to the target scene designated by the scene designation unit from the situation descriptions stored in the database; a summary condition setting unit saving a period and an object to be described according to the designated target scene; generating a report on the object to be described using a generating AI using a situation description corresponding to the period and object to be described stored by the summary condition setting unit from among the situation descriptions searched by the scene search unit; and generating a database including the report.

Images