Descriptive text generating system and production method of descriptive text data-base

Abstract

To make a data-base of images associated with a natural descriptive text such that it matches a retrieval sentence entered by a person.SOLUTION: A descriptive text generating part 12 of a vehicle image analyzer 1 identifies a scene captured in an image received from a camera, reads recognition necessity information on a target in the image for each of the identified scene from a necessity table 14A, identifies the target being specified as necessary to be recognized in the read recognition necessity information, generates a descriptive text for each target from the recognition result, generates a situation descriptive text of the image on the basis of the identified scene and the descriptive text for each target, and memories the image associated with the situation descriptive text of the image into a descriptive text DB 15.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image description system, an image analysis device, and an image description method. [Background technology]

[0002] Conventionally, a technique known as image caption generation has been disclosed that recognizes an image or video and generates an explanatory text for the image (for example, Patent Document 1). The technology described in Patent Document 1 recognizes surrounding objects captured in images captured by an onboard camera, and outputs text including the positional relationship between the vehicle and the surrounding objects, which is provided as an explanation of the surrounding situation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-214320 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies such as Patent Document 1 do not provide a system suitable for creating a database of vehicle images collected from connected cars and searching the database for images that match the user's natural language instructions. When constructing such an image database, it is desirable to associate each image in the database with a description of the image in advance, and to assign a description in the database that is similar to the search text entered by the user.

[0005] On the other hand, in the conventional technology such as Patent Document 1, no particular consideration was given to generating a natural description of the surrounding situation that resembles the search sentence entered by the user. For example, simply detecting an object in an image and using the name of the object as the description may not mention the presence or absence of a surrounding object that should be noted according to the traffic scene in which the vehicle is placed, and as a result, it may not be possible to match the search sentence entered by the user.

[0006] The present invention has been made in consideration of the above circumstances, and its main objective is to create a database of images associated with natural explanatory text that matches search text entered by humans. [Means for solving the problem]

[0007] In order to solve the above problems, the image description system of the present invention has the following features. The present invention provides an image description system having a camera mounted on a vehicle for taking images, and a vehicle image analysis device for generating a situation description of an image taken by the camera, The vehicle image analysis device includes an explanatory sentence generation unit, The explanation generation unit, Identifying a scene appearing in an image received from the camera, and for each identified scene, reading recognition necessity information for an object in the image from a necessity table; Recognizing from within the image the objects designated as requiring recognition in the read recognition necessity information, and generating an explanatory text for each object from the recognition result; A situation description for the image is generated based on the specified scene and a description for each object, and the situation description for the image and the image are associated with each other and stored in an image database. Other features will be described later. Effect of the Invention

[0008] According to the present invention, it is possible to create a database of images associated with natural explanatory text that matches a search text entered by a person. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a configuration diagram of an image description system according to an embodiment of the present invention. [Diagram 2] FIG. 4 is a diagram showing details of a driving log DB according to the present embodiment. [Diagram 3] 4 is a diagram showing an example of still image data that is a part of vehicle driving video data according to the embodiment; FIG. [Figure 4] 1 is a configuration diagram of a vehicle image analysis device according to an embodiment of the present invention. [Diagram 5] FIG. 13 is a diagram showing the configuration of a necessity table according to the present embodiment. [Figure 6] FIG. 4 is a diagram showing the configuration of a recognition table according to the present embodiment. [Figure 7] 10 is a flowchart of an explanation generation unit according to the present embodiment. [Figure 8] FIG. 2 is a hardware configuration diagram of the vehicle image analysis device according to the embodiment. [Figure 9] 10 is a detailed flowchart of the image description generation process according to the present embodiment. [Figure 10] 10 is a flowchart showing an example of a specific operation of the image caption generation processing described with reference to FIG. 9 according to the present embodiment. [Figure 11] 11 is a detailed flowchart of a process for generating an image description for a general road according to the present embodiment. [Figure 12] 4 is a table showing intermediate data resulting from execution of an image caption generation process for the still image data of FIG. 3 according to this embodiment. [Figure 13] 13 is a table showing output data resulting from an explanation generation unit deleting unnecessary data from the intermediate data in FIG. 12 according to this embodiment. [Figure 14] 11 is a diagram showing details of a process for generating a GPS description according to the present embodiment. FIG. [Figure 15] 11A to 11C are diagrams illustrating details of a process for generating a control explanatory text according to the present embodiment. [Figure 16]16 is a table showing an example of the explanatory text generated in FIG. 14 and FIG. 15 according to the present embodiment. [Figure 17] 11 is a table showing instruction sentences for a large-scale language model unit used in the process of generating a driving situation description according to the present embodiment. [Figure 18] 14 is a table showing an example of the image caption generated from an image different from that of FIG. 13 according to this embodiment. [Figure 19] 6 is a flowchart showing a process of a search unit according to the present embodiment. [Figure 20] FIG. 4 is a diagram showing an image search interface of an input / output unit according to the embodiment. [Figure 21] This is a playback screen when the search result (scene 1) in FIG. 20 related to this embodiment is clicked. [Figure 22] This is a playback screen when the search result (scene 2) in FIG. 20 related to this embodiment is clicked. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, one embodiment of the present invention will be described with reference to the drawings.

[0011] FIG. 1 is a configuration diagram of an image description system 100. As shown in FIG. In the image description system 100, vehicles 91-93 exist as vehicles (connected cars) capable of communicating with the vehicle image analysis device 1 via a communication line 8. Each vehicle 91-93 transmits various measurement data measured by the vehicle to the driving log DB 11 of the vehicle image analysis device 1 via the communication line 8. Each vehicle 91-93 is equipped with an on-board camera that captures images. The vehicle image analysis device 1 generates a situation description for the image captured by the on-board camera. The communication line 8 can be a general public line network, whether wired or wireless, such as a fifth generation mobile communication system that enables "multiple simultaneous connections" and "ultra-low latency," or so-called 5G (5th Generation). Furthermore, by taking advantage of the characteristics of new mobile phone systems beyond 5G, effects such as online (real-time while driving) description generation can be expected.

[0012] FIG. 2 is a diagram showing details of the driving log DB 11. As shown in FIG. The driving log DB 11 stores the measurement data from each vehicle 91-93 by data type as follows. Vehicle driving video data 11A, which is video data captured by an on-board camera (not shown) while the vehicle is driving or stopped. Vehicle driving GPS data 11B, which is on-board GPS (Global Positioning System) data. Vehicle driving control data 11C, which is vehicle control data such as speed, acceleration / deceleration, and steering angle, as vehicle driving log data obtained from the on-board ECU (Electronic Control Unit) etc. It should be noted that GPS is an example of a satellite positioning system.

[0013] FIG. 3 is a diagram showing an example of still image data 111 which is a part of the vehicle driving video data 11A. The still image data 111 is an example of data extracted from vehicle travel video data 11A captured by each of the vehicles 91-93 traveling on the expressway.

[0014] FIG. 4 is a configuration diagram of the vehicle image analyzing device 1. 2, the vehicle image analysis device 1 has an explanation generation unit 12, a large-scale language model unit 13, an explanation target setting unit 14, an explanation DB (image database) 15, a search unit 16, and an input / output unit 17. The explanation target setting unit 14 has a necessity table 14A and a recognition table 14B. The large-scale language model unit 13 has a VQA (Visual Question Answer) unit 13A and a summary generation unit 13B. The various data stored in the vehicle image analysis device 1 (driving log DB11, necessity table 14A, and recognition table 14B) may be stored in a storage device (not shown) outside the vehicle image analysis device 1 and configured to be accessible from the storage device via a network from the vehicle image analysis device 1.

[0015] The explanatory sentence generation unit 12 analyzes the driving situation of each vehicle using the large-scale language model unit 13 and the explanation target setting unit 14 based on the data in the driving log DB 11, and generates natural sentences explaining the driving situation of the vehicle according to the analysis results. For this purpose, the explanatory sentence generation unit 12 analyzes whether the driving situation of the vehicle corresponds to any of the traffic scenes classified in advance. The large-scale language model unit 13 is called by the explanatory sentence generation unit 12. The large-scale language model unit 13 is realized by LAVIS (LAnguage VISion) or the like, which uses a natural language conversational method as an interface for exchanging questions and answers, and has the following processing units. The VQA unit 13A responds to inquiries about images in natural language in natural language. To this end, the VQA unit 13A prepares an image recognition model such as a Convolutional Neural Network (CNN) using learning data in advance, and inputs an inquiry text to the image recognition model to obtain an explanatory text for the corresponding image. The summary generation unit 13B responds by summarizing (integrating multiple sentences) the contents of the input natural language (prompt) as a traffic situation explanation. The summary generation unit 13B may use existing services such as GPT-4, CLIP, BLIP, and BLIP-2 as a text generation AI service that performs summarization and translation processing.

[0016] The explanation object setting unit 14 refers to the table below, sets explanation objects corresponding to the traffic scene in the explanation sentence generating unit 12, and stores them in the explanation sentence DB 15. The necessity table 14A (FIG. 5) associates an object to be explained with each traffic scene, and defines the degree of importance of whether or not to mention the explanation for each object to be explained. The recognition table 14B (FIG. 6) defines the detailed contents to be mentioned in the description for each individual explanatory object. The explanatory sentence DB 15 stores the driving situation explanatory sentences generated by the explanatory sentence generating unit 12.

[0017] In this manner, the explanation generating unit 12 executes the following (Process 1) to (Process 3). (Process 1) Scenes appearing in an image received from the camera are identified, and for each identified scene, recognition necessity information for an object in the image is read from the necessity table 14A. (Process 2) Recognize from within the image any objects designated as requiring recognition in the read recognition necessity information, and generate an explanatory text for each object from the recognition results. (Process 3) A situation description of the image is generated based on the identified scene and the description for each object, and the situation description of the image and the image are associated with each other and stored in the description DB 15.

[0018] 5 is a configuration diagram of the necessity table 14A. The explanation object setting unit 14 sets explanation objects for each traffic scene, such as an expressway, a general road, and a parking lot. For example, the combination of a traffic scene "highway" and an explanation object "pedestrian" is "required / required." The recognition necessity information "required" on the left side of this "required / required" notation indicates that the explanation object needs to be recognized from the image. Also, the explanation necessity information "required" on the right side of the "required / required" notation indicates that the explanation object that was or was not recognized from the image needs to be explained in the explanation text. Therefore, in the traffic scene "general road", pedestrian recognition is "necessary", and even if pedestrians are not recognized, an explanation is "necessary". On the other hand, in the traffic scene "general road", crosswalk recognition is "necessary", but an explanation is not required.

[0019] In this way, the explanation generation unit 12 reads explanation necessity information for an object in the image from the necessity table 14A, and if an object specified as requiring explanation in the read explanation necessity information cannot be recognized from within the image, an explanation is generated for each object to the effect that the object is not present in the image. This allows for the generation of descriptions of objects that are not usually present in accordance with the traffic scene in which the vehicle is traveling, only when they are present, and for objects that are usually present in accordance with the traffic scene in which the vehicle is traveling, descriptions can be generated even when they are not present. This has the effect of generating more natural traffic situation descriptions and improving the search accuracy for user search queries.

[0020] 6 is a diagram showing the configuration of the recognition table 14B. The recognition table 14B defines, as detailed items, items to be analyzed by the VQA unit 13A for the recognized explanation object and items to include the analysis results in the explanation. For example, when a pedestrian is recognized, the VQA unit 13A analyzes the location, color of clothing, and movement of the pedestrian. In this way, the description generator 12 reads detailed items of the object in the image from the recognition table 14B, and generates a description of the read detailed items of the object in the image as a description for each object. This makes it possible to individually set information to be added depending on the object to be described, and a natural traffic situation description can be generated. This improves the search accuracy for the user's search query.

[0021] FIG. 7 is a flowchart of the explanation generating unit 12. As an image description generation process (S21), the description generation unit 12 generates an image description by having the VQA unit 13A execute image analysis on the still image data 111 extracted from the vehicle driving video data 11A. The extraction process of the still image data 111 is a process of extracting images continuously captured at a fixed interval, for example, every 10 seconds, or extracting 10 images at equal time intervals from one video file.

[0022] The description generator 12 generates a GPS description based on the vehicle driving GPS data 11B as a GPS description generation process (S22). At this time, the target GPS data (positioning data) is the GPS data of the same time as the still image data 111 of S21 or the GPS data of the closest time. In other words, based on the GPS data read from the vehicle's onboard GPS 91, the description generation unit 12 adds a description regarding at least one of the following information: the time period when the image was taken and the driving position when the image was taken, to the image situation description.

[0023] The explanatory text generation unit 12 generates a control explanatory text based on the vehicle driving control data 11C as a control explanatory text generation process (S23). At this time, the target control data is the control data that is generated at the same time as the still image data 111 of S21 or the control data that is closest in time to the still image data 111 of S21. In other words, the description generation unit 12 adds a description regarding at least one of speed information, acceleration / deceleration information, and steering angle information to the situation description of the image based on the vehicle driving control data read from the vehicle 91's on-board ECU (Electronic Control Unit).

[0024] As a generation process (S24) of a driving situation description, the description generator 12 generates a driving situation description by having the summary generator 13B summarize the image description generated in S21, the GPS description generated in S22, and the control description generated in S23. The description generator 12 associates the generated driving situation description with the still image data 111 that is the description subject of the driving situation description, and stores them in the description DB 15. That is, summary generation unit 13B generates a summary sentence according to the input prompt. Then, explanation generation unit 12 generates a situation explanation sentence for the image by inputting a prompt to summary generation unit 13B, the prompt including an instruction sentence for generating a situation explanation sentence based on information to be included in the situation explanation sentence, information to be excluded from the situation explanation sentence, and information of example sentences of the situation explanation sentence, and an explanation sentence for each object.

[0025] FIG. 8 is a diagram showing the hardware configuration of the vehicle image analyzing device 1. As shown in FIG. The vehicle image analysis device 1 is configured as a computer 900 having a CPU 901 , a RAM 902 , a ROM 903 , a HDD 904 , a communication I / F 905 , an input / output I / F 906 , and a media I / F 907 . The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from a recording medium 917. Furthermore, the CPU 901 executes a program (also called an application or its abbreviation, an app) loaded into the RAM 902, thereby improving and controlling each processing unit. This program can also be distributed via a communication line, or recorded on a recording medium 917 such as a CD-ROM and distributed.

[0026] FIG. 9 is a detailed flowchart of the image caption generation process (S21 in FIG. 7). The description generation unit 12 classifies the traffic scenes captured in the still image data 111 by querying the VQA unit 13A (S211). The VQA unit 13A receives the still image data 111 and a query for the traffic scene (e.g., "Where is this scene? For example, is it a road, a highway, or a parking lot?") and responds with an answer (e.g., a highway) to the description generation unit 12 (line A01 in FIG. 12). Note that the description generation unit 12 reads the query for the traffic scene set in advance by the administrator in the vehicle image analysis device 1, so that the user of the vehicle image analysis device 1 does not need to generate the query for the traffic scene by himself / herself. Hereafter, the explanation generating unit 12 executes the loop processing of S212 to S217 by sequentially selecting explanation objects registered in the necessity table 14A (pedestrians, bicycles, automobiles, ...). Hereafter, the explanation object selected in this loop processing is referred to as the selected object.

[0027] The explanation generator 12 determines whether or not recognition of the selected object is necessary in the traffic scene identified in S221 by referring to the necessity table 14A (S212). If Yes (necessary) in S212, proceed to S213, and if No, proceed to S217. The explanatory text generation unit 12 queries the VQA unit 13A of the large-scale language model unit 13 as to whether or not the selected object exists in the still image data 111. This query is, for example, "Are there any pedestrians in this scene?" (line A02 in FIG. 12). The explanatory text generation unit 12 recognizes the object designated as requiring recognition from within the image, based on the response sentence (object recognition result) to this query. As the recognition result, the explanatory text generation unit 12 determines whether or not the selected object exists in the still image data 111 (S213). If the answer is Yes (exists) in S213, the process proceeds to S214, and if the answer is No, the process proceeds to S215.

[0028] The explanatory text generation unit 12 acquires detailed items for the selected object present in the still image data 111 (S214). For this purpose, the explanatory text generation unit 12 acquires detailed items corresponding to the selected object by referring to the recognition table 14B. Then, the explanatory text generation unit 12 queries the VQA unit 13A of the large-scale language model unit 13 for detailed items for the selected object in the still image data 111 one by one. For example, since the selected object is a car and the detailed items corresponding to the car in the recognition table 14B include "color", the explanatory text generation unit 12 generates a query sentence such as "What is the color of the car?".

[0029] The explanation generating unit 12 judges whether an explanation (mention) of the selected object is necessary or not by referring to the necessity table 14A (S215). If Yes (necessary) in S215, the process proceeds to S216, and if No, the process proceeds to S217. The explanation generating unit 12 generates an explanation for the selected object by any of the following methods (S216). If the answer is Yes in S213, an explanatory text for the selected object is generated from a combination of the inquiry text for the detailed items of the selected object acquired in S214 and the answer text. If the result of S213 is No, a text explaining the selected object is generated to the effect that the selected object was not recognized in the still image data 111 (eg, line D07 in FIG. 18). The explanation generation unit 12 judges whether or not the processing for all selected objects has been completed by finishing the processing for the current selected object (S217). If the answer is Yes (completed) in S217, the processing ends, and if the answer is No, the processing switches to an unprocessed selected object and returns to S212. This makes it possible to generate natural traffic situation descriptions that include descriptions of surrounding objects that should be noted and confirmed according to the traffic scene in which the vehicle is traveling, thereby improving the search accuracy for user search queries.

[0030] FIG. 10 is a flowchart showing an example of a specific operation of the image caption generation processing described with reference to FIG. The caption generating unit 12 branches into processing for each traffic scene as follows (S301) depending on the result of the processing for classifying the traffic scenes captured in the still image data 111 (S211 in FIG. 9). If the classification result is that of a general road, an image description for a general road is generated (S302). If the result is a highway classification, generate an image description for the highway (S303). If the result is a parking lot classification, an image description for the parking lot is generated (S304). If the classification result is "other," generate an image description specified for "other" (S305).

[0031] FIG. 11 is a detailed flowchart of the process of generating an image caption for an ordinary road (S302 in FIG. 10). The explanation generator 12 generates a traffic scene question and answer (S211 in FIG. 9), and a question about the presence or absence of a traffic light and an answer (S311). As the determination process (S213) of FIG. 9, the caption generating unit 12 asks whether or not there is a pedestrian in the image (S312), and if there is, the process proceeds to S313, and if there is no pedestrian, the process proceeds to S314. The explanation generation unit 12 generates answer sentences indicating the presence of a pedestrian, a question and answer sentence about the pedestrian's location, a question and answer sentence about the pedestrian's color, and a question and answer sentence about the pedestrian's action (S313) as questions and answers about the pedestrian's detailed items (S214 in FIG. 9). The explanation generating unit 12 generates a reply sentence indicating that there is no pedestrian (S314).

[0032] The caption generating unit 12 asks whether or not there is a car in the image (S315) as the determination process (S213) in FIG. 9, and if there is, the process proceeds to S316, and if there is no car, the process proceeds to S317. The explanation generation unit 12 generates, as answers to questions about the detailed items of the car (S214 in FIG. 9), answer sentences indicating the existence of the car, a question and answer sentence about the location of the car, a question and answer sentence about the car's model, a question and answer sentence about the car's color, and a question and answer sentence about the car's operation (S316). The caption generating unit 12 asks whether or not a bicycle is present in the image (S317) as the determination process (S213) in FIG. 9, and if present, the process proceeds to S318, and if not, the process proceeds to S319. The explanation generation unit 12 generates, as answers to questions about the details of the bicycle (S214 in FIG. 9), an answer sentence indicating the existence of a bicycle, a question and answer sentence about the location of the bicycle, a question and answer sentence about the color of the bicycle, and a question and answer sentence about the operation of the bicycle (S318).

[0033] 9 (S213), the explanatory text generation unit 12 asks whether there is a pedestrian crossing in the image (S319), and if there is, the process proceeds to S320, and if there is no pedestrian crossing, the process ends. The explanatory text generation unit 12 generates a response text indicating the presence of a pedestrian crossing (S320). Although the process of generating an image caption for a general road has been described above with reference to FIG. 10, the process of generating an image caption for other traffic scenes (S303 to S305) is similar.

[0034] FIG. 12 is a table showing intermediate data resulting from the image caption generation process (S21) performed on the still image data 111 in FIG. In this table, each row contains an image description consisting of a question and an answer to that question. For example, row A01 is the result of the process of classifying traffic scenes (S211). Rows A02 to A04 are the results of the process of determining whether or not the object to be selected exists in the still image data 111 (S213). Lines A05 to A13 are the results of the process (S214) for acquiring detailed items about the selected object.

[0035] Fig. 13 is a table showing output data resulting from the explanation generation unit 12 deleting unnecessary data from the intermediate data of Fig. 12. The difference from Fig. 12 is that the explanation for pedestrians (row A02), bicycles (row A03), and toll booths (row A13), which are deemed not to require explanation when not recognized in the necessity table 14A of the explanation target setting unit 14, have been deleted in Fig. 13.

[0036] FIG. 14 is a diagram showing details of the GPS description generation process (S22). The explanation generator 12 generates an explanation classifying the capture times of the still image data 111 into morning, afternoon, evening, night, etc. according to the time period of the GPS information obtained from the vehicle driving GPS data 11B (S221). The description generator 12 generates a description that specifies the location where the still image data 111 was taken as a main road or a city name according to the latitude and longitude of the GPS information obtained from the vehicle driving GPS data 11B (S222). This makes it possible to generate traffic condition descriptions that include information about the time period and location of vehicle travel, thereby improving the search accuracy for user search queries.

[0037] FIG. 15 is a diagram showing details of the process of generating a control explanation sentence (S23). The explanatory text generating unit 12 generates explanatory text regarding the driving speed control from the vehicle driving control data 11C (S231). The explanatory text may be, for example, that a speed of 20 km / h or less is a low speed, that of 20 km / h to 60 km / h is a medium speed, and that of 60 km / h or more is a high speed. The explanatory text generator 12 generates explanatory text regarding the acceleration / deceleration control from the vehicle driving control data 11C (S232). The explanatory text may be, for example, that the acceleration is equal to or greater than a certain level, that the vehicle is in an accelerating state, that the deceleration is equal to or greater than a certain level, and that the vehicle is in a constant speed state otherwise. The explanatory text generator 12 generates explanatory text on steering control from the vehicle driving control data 11C (S233). The explanatory text may indicate, for example, that a steering angle of a certain degree or more indicates a left turn or a right turn, and that the vehicle is otherwise traveling straight. This makes it possible to generate more natural traffic situation descriptions that include descriptions of the vehicle's control state and behavior, thereby improving the accuracy of searches performed by users.

[0038] FIG. 16 is a table showing an example of the explanatory text generated in FIG. 14 and FIG. In line B01, an explanation of the time period is generated in S221. In line B02, a description of the driving location is generated in S222. In line B03, an explanation of the driving speed is generated in S231. In line B04, an explanation of the acceleration / deceleration state is generated in S232. In line B05, a description of the steering state is generated in S233.

[0039] FIG. 17 is a table showing instruction sentences to the large-scale language model unit 13 used in the process of generating a driving situation explanation sentence (S24). Line C01 contains an instruction to generate a traffic situation description. Line C02 describes information to be included in the traffic situation description. Line C03 describes information to be excluded from the traffic situation description. Line C04 contains an example of a traffic situation description.

[0040] In the process of generating a driving situation description (S24), the description generator 12 generates a prompt by combining the following texts in order from (item 1) to (item 4). Note that at least one of (item 2) and (item 3) may be omitted. (Item 1) Instructions to the large-scale language model unit 13 (Figure 17) (Item 2) GPS description (lines B01 and B02 in Figure 16) (Item 3) Control description (lines B03, B04, and B05 in Figure 16) (Item 4) Image description (Figure 13)

[0041] The description generation unit 12 obtains a traffic situation description written in natural language by inputting this generated prompt to the large-scale language model unit 13. Below, an example of a traffic situation description (corresponding to situation description 732 in FIG. 21 described later) generated by the large-scale language model unit 13 from the prompt is shown. Traffic description = "You are driving straight ahead on Metropolitan Expressway Route 5 at midday, slowing down at high speed. In this scene, the highway has solid orange lanes. Additionally, a white truck is driving ahead of you on the road ahead." This allows for the generation of natural traffic situation descriptions that correspond to the intended use and the user's preferences, thereby improving the search accuracy for user search queries.

[0042] FIG. 18 is a table showing an example generated from an image other than the image caption of FIG. 18 is for an image captured from a vehicle traveling on a general road. Therefore, since the combination of a general road and a pedestrian in the necessity table 14A is "needs" explanation, the information "pedestrian was not recognized" is written in row D07. The traffic situation description generated by the large scale language model unit 13 from the prompt including the image description of FIG. 18 through the driving situation description generation process (S24) is as follows: Traffic description="You are currently driving at a slow and steady speed on a city road in Mito city at night, preparing to turn left. There are no pedestrians in the scene."

[0043] The database creation of the driving situation explanation (the process up to storing it in the explanation DB 15) has been described above with reference to Fig. 18. An example of the use of the information created in the database will be described below. FIG. 19 is a flowchart showing the process of the search unit 16. The search unit 16 searches for an image matching the input search statement from the images stored in the description DB 15. Specifically, the search unit 16 outputs an image having a high similarity between the input search statement and the situation description stored in the description DB 15 as a search result. Note that the search unit 16 may, for example, list the images in the search results in order of similarity and output the top X images as "images with high similarity" (judgment of similarity based on relative rank), or may output images in the search results whose similarity is higher than a preset reference value (threshold value Y) (judgment of absolute similarity). The process of the search unit 16 will now be described in detail.

[0044] The search unit 16 receives a search statement input by a user from the input / output unit 17 (S61). The user here is, for example, a commentator at a traffic control center that manages expressways and the like, and the search statement is, for example, a statement to collect from a database images of situations similar to an accident that occurred at a specific time and place on the expressway. The commentator plans to edit the image materials acquired from the database and produce a news program about the accident that occurred. The search unit 16 evaluates the similarity between the search sentence from the user and the sentence stored in the sentence DB 15 (S62), and acquires video information (image information) associated with the sentence having a high similarity from the sentence DB 15. The similarity evaluation may use a cosine similarity search based on document vectorization, or the like.

[0045] The search unit 16 searches for and acquires the driving video and driving situation description acquired in S62 as well as related information about the driving situation (such as weather information that is not in the description DB 15 but can be acquired from the weather database by specifying the location and date of the driving situation description), and generates an answer based on these results (S63). That is, the search unit 16 may output, as a search result, related information acquired from a database other than the description DB 15 based on information included in the situation description corresponding to the image with high similarity. The search unit 16 transmits the reply text of S63 to the input / output unit 17 (S64). This makes it possible to quickly search for a desired video when searching for video data having a traffic condition description similar to a search statement in natural language entered by a user.

[0046] FIG. 20 is a diagram showing the image search interface 71 of the input / output unit 17. As shown in FIG. The image search interface 71 is composed of a search query input section 72 which is an input field for the search query of S61, and a search result display section 73 which is a display field for the answer query of S64. In the search result display section 73, a plurality of search results (scenes 1 to 4) are displayed as icons or thumbnail images.

[0047] Figure 21 shows the playback screen when the search result (scene 1) in Figure 20 is clicked. The following information is displayed on this playback screen from top to bottom: · 731 images from search results. The situation description 732 of the image 731 was extracted as having a high similarity to the search sentence. Additional information such as GPS description (time, location), control description (vehicle type, driving speed), and weather 733.

[0048] Fig. 22 shows the playback screen when the search result (scene 2) in Fig. 20 is clicked. As in Fig. 21, this playback screen displays image 741 as a search result, its situation description 742, and additional information 743, in the same manner as in Fig. 21. This allows a user to input a search statement in natural language, and then search for videos by referring to traffic condition descriptions, videos, and related information that are similar to the search statement, thereby improving search efficiency.

[0049] According to the present embodiment described above, when generating a description of a vehicle image, the description generator 12 refers to the necessity table 14A to generate a natural description including the presence or absence of peripheral objects that should be noted based on the traffic scene the vehicle is in. This allows natural descriptions that mention necessary peripheral objects but do not mention unnecessary peripheral objects to be compiled into a database, improving the accuracy of database searches from search queries entered by humans.

[0050] Furthermore, the present invention is not limited to the above-described embodiments, and of course, various other applications and modifications are possible without departing from the gist of the present invention described in the claims. For example, the above-described embodiments are detailed and specific descriptions of the configuration of the vehicle image analyzing device 1 in order to easily explain the present invention, and are not necessarily limited to those including all of the components described. Also, it is possible to replace a part of the configuration of one embodiment with a component of another embodiment. Also, it is possible to add a component of another embodiment to the configuration of one embodiment. Also, it is possible to add, replace, or delete other components to, part of the configuration of each embodiment.

[0051] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. As the hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used. Furthermore, each component of the vehicle image analyzing device 1 according to the above-described embodiment may be implemented in any hardware as long as the respective hardware can transmit and receive information to each other via a network. Furthermore, the processing executed by a certain processing unit may be realized by a single piece of hardware, or may be realized by distributed processing by multiple pieces of hardware. [Explanation of symbols]

[0052] 1 Vehicle image analysis device 8. Communication Lines 11 Driving log DB 11A Vehicle driving video data 11B Vehicle driving GPS data 11C Vehicle driving control data 12 Description Generation Unit 13 Large-scale language model section 13A VQA section 13B Summary generator 14 Explanation target setting section 14A Necessity table 14B Recognition Table 15 Description DB (image database) 16 Search section 17 Input / output section 91-93 Vehicles 100 Image Description System 111 Still image data 732,742 Situation Description

Claims

1. An image description system having a camera mounted on a vehicle for taking an image, and a vehicle image analysis device for generating a situation description of the image taken by the camera, The vehicle image analysis device includes an explanatory sentence generation unit, The explanation generation unit, Identifying a scene appearing in an image received from the camera, and for each identified scene, reading recognition necessity information for an object in the image from a necessity table; Recognizing from within the image the objects designated as requiring recognition in the read recognition necessity information, and generating an explanatory text for each object from the recognition result; A situation description of the image is generated based on the specified scene and a description for each object, and the situation description of the image and the image are associated with each other and stored in an image database. Image description system.

2. The explanation generation unit reads explanation necessity information about an object in the image from the necessity table, and when an object specified as requiring an explanation in the read explanation necessity information cannot be recognized from within the image, generates an explanation for each object to the effect that the object does not exist in the image.

2. The image description system of claim 1.

3. The description generating unit reads detailed items of the object in the image from the recognition table, and generates a description of the read detailed items of the object in the image as a description for each object.

2. The image description system of claim 1.

4. The description generator adds a description regarding at least one of information regarding a time period when the image was captured and information regarding a driving position when the image was captured to a description of a situation of the image based on positioning data read from an on-board satellite positioning system of the vehicle.

2. The image description system of claim 1.

5. The description generator adds a description regarding at least one of information regarding speed, information regarding acceleration / deceleration, and information regarding steering angle to a situation description of the image based on vehicle driving control data read from an on-board ECU (Electronic Control Unit) of the vehicle.

2. The image description system of claim 1.

6. The vehicle image analysis device has a summary generation unit that generates a summary sentence according to an input prompt, The description generation unit generates the situation description of the image by inputting to the summary generation unit a prompt including an instruction for generating a situation description based on information to be included in the situation description, information to be excluded from the situation description, and information on example sentences of the situation description, and a description for each object.

2. The image description system of claim 1.

7. The vehicle image analysis device has a search unit that searches for an image that matches an input search statement from among images stored in the image database, The search unit is characterized in that it outputs, as a search result, images having a high similarity between the input search sentence and the situation description stored in the image database.

2. The image description system of claim 1.

8. The search unit is characterized in that it also outputs related information obtained from a database other than the image database based on information included in a situation description corresponding to an image with a high similarity as a search result.

8. The image description system of claim 7.

9. Identifying the scenes in the image, and for each identified scene, reading the recognition necessity information for the object in the image from the necessity table; Recognizing from within the image the objects designated as requiring recognition in the read recognition necessity information, and generating an explanatory text for each object from the recognition result; The present invention is characterized in that the present invention has a description generating unit that generates a description of a situation of an image based on the specified scene and a description for each object, and stores the description of the situation of the image and the image in an image database in association with each other. Image analysis equipment.

10. The image analysis device Identifying the scenes in the image, and for each identified scene, reading the recognition necessity information for the object in the image from the necessity table; Recognizing from within the image the objects designated as requiring recognition in the read recognition necessity information, and generating an explanatory text for each object from the recognition result; A situation description of the image is generated based on the specified scene and the description of each object, and the situation description of the image and the image are associated with each other and stored in an image database. Image description method.

Images