Image description system, image analysis apparatus, and image description method

Abstract

The invention provides an image description system, an image analysis device, and an image description method, which can database images associated with natural persuasive and plaintext conforming to a search text input by a person. A speaking text generation unit (12) of a vehicle image analysis device (1) identifies scenes presented in an image received from a camera, reads identification requirement information on an object in the image from a requirement table (14A) for each identified scene, identifies an object specified as required to be identified by the read identification requirement information from the image, and analyzes the identified object in the image. A persuasion plaintext for each object is generated on the basis of the identified scene and the persuasion plaintext for each object, and a status persuasion plaintext for the image is generated on the basis of the identified scene and the persuasion plaintext for each object, and is stored in a persuasion plaintext database (15) in association with the image.

Description

Technical Field

[0001] The present invention relates to an image description system, an image analysis device and an image description method. Background Art

[0002] Conventionally, there has been disclosed a technique for generating image captions that recognizes an image and a moving picture and generates a description of the image (for example, Patent Document 1).

[0003] In the technology described in Patent Document 1, surrounding objects moving in an image of a vehicle-mounted camera are recognized, and text including the positional relationship between the vehicle and the surrounding objects is output to provide as a surrounding situation description text.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-214320 Summary of the invention

[0007] Problems to be solved by the invention

[0008] However, the prior art such as Patent Document 1 does not provide a system suitable for the purpose of making a database of vehicle images collected from connected cars and retrieving images that meet the user's natural language instruction conditions from the database. When constructing such an image database, it is preferable to associate a description of the image in advance with each image in the database and attach a description in the database that is similar to the search text input by the user.

[0009] On the other hand, in the prior art such as Patent Document 1, no special consideration is given to generating natural surrounding condition description text similar to the search text input by the user. For example, when only the object in the image is detected and the name of the object is directly used as the description text, there is a case where the description text does not mention whether there are surrounding objects of interest corresponding to the traffic scene in which the vehicle is located, and as a result, it does not match the search text input by the user.

[0010] The present invention has been made in consideration of such a situation, and a main object of the present invention is to form a database of images to which natural explanatory texts matching search texts input by a person are associated.

[0011] Technical solutions to solve problems

[0012] In order to solve the above-mentioned problems, the image description system of the present invention has the following features.

[0013] The present invention is an image description system, comprising a camera mounted on a vehicle for capturing images, and a vehicle image analysis device for generating a condition description of the image captured by the camera, wherein the image description system is characterized in that:

[0014] The vehicle image analysis device includes a description text generating unit.

[0015] The description text generating unit,

[0016] determining a scene presented in an image received from the camera, and reading recognition necessity information about an object in the image from the necessity table for each determined scene,

[0017] Recognize the objects specified as requiring recognition by the read recognition necessity information from within the image, and generate a description of each object based on the recognition result.

[0018] A description of the situation of the image is generated based on the determined scene and the description of each object, and the description of the situation of the image is stored in an image database in association with the image.

[0019] Other features are described later.

[0020] Effects of the Invention

[0021] According to the present invention, it is possible to form a database of images associated with natural explanatory texts that match search texts input by a person. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a block diagram of the image description system according to the present embodiment.

[0023] Figure 2 It is a diagram showing the details of the travel log DB according to the present embodiment.

[0024] Figure 3 This is a diagram showing an example of still image data of a part of vehicle running dynamic image data according to the present embodiment.

[0025] Figure 4 It is a configuration diagram of the vehicle image analysis device according to this embodiment.

[0026] Figure 5 This is a diagram showing the structure of the necessity table according to this embodiment.

[0027] Figure 6 This is a diagram showing the structure of the identification table according to this embodiment.

[0028] Figure 7 This is a flowchart of the explanatory text generating unit according to this embodiment.

[0029] Figure 8 4 is a hardware configuration diagram of the vehicle image analysis device according to the present embodiment.

[0030] Fig. 9 This is a detailed flowchart of the image description text generation process according to the present embodiment.

[0031] Fig.10 This embodiment shows Fig. 9 A flowchart of an example of specific operations of the image description text generation process described in .

[0032] Fig.11 This is a detailed flowchart of the process of generating image description text for general roads according to the present embodiment.

[0033] Fig.12 This embodiment shows Figure 3 A table of intermediate data of the result of executing the image description text generation process for the still image data.

[0034] Fig.13 This embodiment shows Fig.12 The intermediate data is a table of output data resulting from the deletion of unnecessary data by the description text generating unit.

[0035] Fig.14 It is a diagram showing details of the GPS description text generation process according to the present embodiment.

[0036] Fig.15 It is a diagram showing details of the control description text generation process according to the present embodiment.

[0037] Fig.16 This embodiment shows Fig.14 and Fig.15 A table showing an example of the description text generated in .

[0038] Fig.17 This is a table showing instruction texts for the large-scale language model unit used in the generation process of the driving status description text in this embodiment.

[0039] Fig.18 This is the basis and Fig.13 A table showing an example of image descriptions generated for different images.

[0040] Fig.19 This is a flowchart showing the processing of the search unit according to this embodiment.

[0041] Fig. 20 It is a diagram showing an image search interface of the input / output unit according to the present embodiment.

[0042] Fig.21 This is the click of the present embodiment. Fig. 20The playback screen when searching for the search results (Scene 1).

[0043] Fig. 22 This is the click of the present embodiment. Fig. 20 The playback screen when searching for the search results (Scene 2). DETAILED DESCRIPTION

[0044] Hereinafter, one embodiment of the present invention will be described using the drawings.

[0045] Figure 1 is a block diagram of the image captioning system 100 .

[0046] In the image interpretation system 100, there are vehicles 91-93 as vehicles (networked vehicles) capable of communicating with the vehicle image analysis device 1 via the communication line 8. Each vehicle 91-93 transmits various measurement data measured in the vehicle to the travel log DB 11 of the vehicle image analysis device 1 via the communication line 8.

[0047] Each of the vehicles 91 to 93 is equipped with a vehicle-mounted camera for capturing images. The vehicle image analysis device 1 generates a situation description text of the image captured by the vehicle-mounted camera.

[0048] In addition, the communication line 8 is a general public line network of wired or wireless, and for example, the fifth generation mobile communication system, so-called 5G (5th Generation), which can realize "large number of simultaneous connections" and "ultra-low latency" can be used. Furthermore, by taking advantage of the characteristics of new mobile phone systems after 5G, it is also possible to expect effects such as generating explanatory texts online (in real time while driving).

[0049] Figure 2 It is a diagram showing the details of the travel log DB 11 .

[0050] The travel log DB 11 stores the measurement data from each of the vehicles 91 to 93 by data type as described below.

[0051] The vehicle running dynamic image data 11A is dynamic image data captured by a vehicle-mounted camera (not shown) while the vehicle is running or parked.

[0052] The vehicle-mounted GPS (Global Positioning System) data is the vehicle driving GPS data 11B.

[0053] Vehicle travel control data 11C, which is vehicle travel log data obtained from an on-vehicle ECU (Electronic Control Unit) or the like, and is vehicle control data such as speed, acceleration / deceleration, and steering angle.

[0054] Among them, GPS is an example of a satellite positioning system.

[0055] Figure 3 1 is a diagram showing an example of still image data 111 which is a part of vehicle running dynamic image data 11A.

[0056] The still image data 111 is an example of data extracted from the vehicle travel dynamic image data 11A captured from each of the vehicles 91 to 93 traveling on the expressway.

[0057] Figure 4 2 is a structural diagram of the vehicle image analysis device 1 .

[0058] Vehicle image analysis device 1 Figure 2 In addition to the driving log DB 11 described in the description, it also has a description text generation unit 12, a large-scale language model unit 13, a description object setting unit 14, a description text DB (image database) 15, a search unit 16, and an input / output unit 17. The description object setting unit 14 has a necessity table 14A and a recognition table 14B. The large-scale language model unit 13 has a VQA (Visual Question Answer) unit 13A and a summary generation unit 13B.

[0059] In addition, various data stored in the vehicle image analysis device 1 (driving log DB 11, necessity table 14A and identification table 14B) can also be configured to be stored in a storage device (not shown) outside the vehicle image analysis device 1, and the storage device can be accessed from the vehicle image analysis device 1 via a network.

[0060] The description text generation unit 12 analyzes the driving condition of each vehicle based on the data in the driving log DB 11 using the large-scale language model unit 13 and the description object setting unit 14, and generates a natural text describing the driving condition of the vehicle according to the analysis result. To this end, the description text generation unit 12 analyzes which of the pre-classified traffic scenes the driving condition of the vehicle corresponds to.

[0061] The large-scale language model unit 13 is called by the explanatory text generation unit 12. The large-scale language model unit 13 is implemented by LAVIS (Language Vision), a natural language conversational interface for communicating between question texts and answer texts, and includes the following processing units.

[0062] The VQA unit 13A answers natural text queries about images in natural text. To this end, the VQA unit 13A prepares an image recognition model such as a CNN (Convolutional Neural Network) using learning data, etc., and inputs the query text to the image recognition model to obtain the corresponding image description text.

[0063] The summary generation unit 13B summarizes the content of the input natural text (prompt) (combines multiple articles) to obtain a summary text as a traffic condition description text. The summary generation unit 13B can use existing services such as GPT-4, CLIP, BLIP, BLIP-2 as text generation AI services that perform summary processing and translation processing.

[0064] The explanation object setting unit 14 sets the explanation object corresponding to the traffic scene to the explanation text generating unit 12 by referring to the following table, and stores the object in the explanation text DB 15 .

[0065] ·Do you need Form 14A ( Figure 5 ) Associate a description object with each traffic scene, and define the importance of whether the individual description object is mentioned in the description text.

[0066] · Identification Table 14B ( Figure 6 ) For each object of description, define the details mentioned in the description.

[0067] The description text DB 15 stores the driving status description text generated by the description text generating unit 12 .

[0068] In this way, the explanatory text generating unit 12 executes the following (processing 1) to (processing 3).

[0069] (Process 1) The scene represented in the image received from the camera is determined, and recognition necessity information about the object in the image is read from necessity table 14A for each determined scene.

[0070] (Processing 2) Objects designated as necessary by the read recognition necessity information are recognized from within the image, and a description of each object is generated based on the recognition result.

[0071] (Process 3) A description text of the situation of the image is generated based on the specified scene and the description text of each object, and the description text of the situation of the image is stored in the description text DB 15 in association with the image.

[0072] Figure 5 The explanation object setting unit 14 sets the explanation object for each traffic scene such as an expressway, a general road, and a parking lot.

[0073] For example, the combination of the traffic scene "highway" and the description object "pedestrian" is "must / must". The recognition necessity information "must" on the left side of the "must / must" indicates that the description object needs to be recognized from the image. In addition, the description necessity information "must" on the right side of the "must / must" indicates that the description object recognized or not recognized from the image needs to be described with a description.

[0074] Thus, in the traffic scene "general road", it is set that pedestrian recognition is "required", and even if it is not recognized, it is "required". On the other hand, in the traffic scene "general road", crosswalks are "required" to be recognized, but there is no need to explain.

[0075] In this way, the descriptive text generating unit 12 reads the description necessity information about the object in the image from the description necessity table 14A, and if an object designated as requiring description by the read description necessity information cannot be identified from the image, a descriptive text indicating that the object does not exist in the image is generated as the descriptive text for each object.

[0076] Thus, it is possible to generate a description text only when an object that is usually absent exists, and to generate a description text even when an object that is usually absent does not exist, in accordance with the traffic scene in which the vehicle is traveling. Therefore, it is possible to generate a more natural description text of the traffic situation, which has the effect of improving the search accuracy of the user's search text.

[0077] Figure 6 14B is a diagram showing the structure of the identification table 14B. The identification table 14B defines, as detailed items, items to be analyzed by the VQA unit 13A for the identified description object and items to be included in the description text with the analysis results.

[0078] For example, when the VQA unit 13A recognizes a pedestrian, it analyzes the location, color of clothing, and behavior of the pedestrian.

[0079] In this way, the description text generation unit 12 reads the detailed items about the object in the image from the recognition table 14B, and generates a description text about the read detailed items about the object recognized from the image as a description text for each object. Thus, it is possible to set the information to be added individually according to the description object, and generate a natural traffic condition description text. Therefore, the search accuracy of the user's search text is improved.

[0080] Figure 7 This is a flowchart of the explanatory text generating unit 12 .

[0081] In the description text generating unit 12, as a generation process of image description text (S21), the VQA unit 13A performs image analysis on the still image data 111 extracted from the vehicle driving dynamic image data 11A, thereby generating image description text. The extraction process of the still image data 111 is, for example, a process of extracting continuously captured images at a certain interval such as every 10 seconds, or a process of extracting 10 images at equal time intervals from a dynamic image file.

[0082] The description text generating unit 12 generates GPS description text based on the vehicle travel GPS data 11B as a GPS description text generating process (S22). At this time, the target GPS data (positioning data) is the GPS data at the same time as the still image data 111 of S21 or the GPS data closest in time.

[0083] That is, the description text generating unit 12 adds description text on at least one of the time period when the image was captured and the driving position when the image was captured to the image situation description text based on the GPS data read from the vehicle-mounted GPS of the vehicle 91 .

[0084] The description text generating unit 12 generates a control description text based on the vehicle travel control data 11C as a control description text generating process (S23). At this time, the control data used as the target is the same time as or the closest time to the still image data 111 of S21.

[0085] That is, the description text generating unit 12 adds description text on at least one of the speed information, the acceleration / deceleration information, and the steering angle information to the image status description text based on the vehicle travel control data read from the vehicle 91 onboard ECU (Electronic Control Unit).

[0086] In the description text generation unit 12, as a process of generating a driving condition description text (S24), the summary generation unit 13B summarizes the image description text generated in S21, the GPS description text generated in S22, and the control description text generated in S23, thereby generating a driving condition description text. The description text generation unit 12 stores the generated driving condition description text in the description text DB 15 in association with the still image data 111 that is the description object of the driving condition description text.

[0087] That is, the summary generating unit 13B generates a summary text that conforms to the input prompt. Then, the description text generating unit 12 generates a situation description text of the image by inputting a generation instruction text including information included in the situation description text, information to be excluded from the situation description text, and information of an example text of the situation description text and a prompt of a description text for each object to the summary generating unit 13B.

[0088] Figure 8 2 is a hardware configuration diagram of the vehicle image analysis device 1 .

[0089] The vehicle image analysis device 1 is configured as a computer 900 including a CPU 901 , a RAM 902 , a ROM 903 , a HDD 904 , a communication I / F 905 , an input / output I / F 906 , and a medium I / F 907 .

[0090] The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The medium I / F 907 reads and writes data from a recording medium 917. Furthermore, the CPU 901 improves and controls each processing unit by executing a program (application program, also referred to as application) read to the RAM 902. Then, the program can be distributed via a communication line or recorded in a recording medium 917 such as a CD-ROM.

[0091] Fig. 9 is the generation process of image description text ( Figure 7 Detailed flow chart of S21).

[0092] The description text generation unit 12 classifies the traffic scene presented in the still image data 111 by querying the VQA unit 13A (S211). The VQA unit 13A receives the input of the still image data 111 and the query text of the traffic scene (e.g., "Where is this scene? For example, is it a road, a highway, or a parking lot?"), and returns the answer text (e.g., highway) to the description text generation unit 12. Fig.12 In addition, the description text generation unit 12 reads the query text of the traffic scene set by the administrator to the vehicle image analysis device 1 in advance, so the user of the vehicle image analysis device 1 does not need to generate the query text of the traffic scene by himself.

[0093] Next, the description text generating unit 12 sequentially selects and executes description objects (pedestrians, bicycles, motor vehicles, etc.) registered in the necessity table 14A for the loop processing of S212 to S217. Hereinafter, the objects selected in this loop processing are referred to as selected objects.

[0094] The description text generating unit 12 refers to the necessity table 14A to determine whether the selected object needs to be recognized in the traffic scene determined in S221 (S212). If the answer in S212 is "yes" (necessary), the process proceeds to S213, and if the answer is "no", the process proceeds to S217.

[0095] The description text generation unit 12 inquires the VQA unit 13A of the large-scale language model unit 13 whether there is a selection object in the still image data 111. The inquiry text is, for example, "Is there a pedestrian in this scene?" Fig.12 The description text generation unit 12 recognizes the object specified as the object to be recognized from the image based on the answer text (the recognition result of the object) to the inquiry text. As the recognition result, the description text generation unit 12 determines whether the selected object exists in the still image data 111 (S213). If the answer in S213 is "yes" (existent), the process proceeds to S214, and if the answer is "no", the process proceeds to S215.

[0096] The description text generation unit 12 obtains the detailed items about the selected object existing in the still image data 111 (S214). Therefore, the description text generation unit 12 refers to the recognition table 14B to obtain the detailed items corresponding to the selected object. Then, the description text generation unit 12 inquires the VQA unit 13A of the large-scale language model unit 13 about the detailed items of the selected object in the still image data 111 one by one. For example, in the description text generation unit 12, because the selected object = motor vehicle, the detailed items corresponding to the motor vehicle in the recognition table 14B include "color", so a query text such as "What color is the car?" is generated.

[0097] The explanation text generating unit 12 refers to the necessity table 14A to determine whether the selected object needs to be explained (mentioned) (S215). If the answer is "yes" (necessary) in S215, the process proceeds to S216, and if the answer is "no", the process proceeds to S217.

[0098] The description text generating unit 12 generates a description text of the selected object according to any of the following items (S216).

[0099] If the answer is YES in S213 , a description text of the selected object is generated based on a combination of the inquiry text and the answer text regarding the detailed items of the selected object acquired in S214 .

[0100] If the answer in S213 is "No", a description text of the selected object indicating that the selected object is not recognized in the still image data 111 is generated ( Fig.18 D07, etc.).

[0101] The description text generating unit 12 ends the processing of the selected object this time, and determines whether the processing of all the selected objects has been completed (S217). If S217 is "yes" (completed), the processing is ended, and if it is "no", it switches to the unprocessed selected object and returns to S212.

[0102] As a result, a natural traffic condition description text including descriptions of surrounding objects to be paid attention to and confirmed can be generated according to the traffic scene in which the vehicle is traveling, thereby improving the search accuracy of the user's search text.

[0103] Fig.10 Yes means Fig. 9 A flowchart of an example of specific operations of the image description text generation process described in .

[0104] The description text generating unit 12 and the processing for classifying the traffic scene presented in the still image data 111 ( Fig. 9 According to the result of S211), the process branches to the processing of each traffic scene (S301) as described below.

[0105] If the classification result is a general road, an image description for the general road is generated (S302).

[0106] If the classification result is a highway, an image description text for the highway is generated (S303).

[0107] If the classification result is a parking lot, an image description for the parking lot is generated (S304).

[0108] If the classification result is other, a predetermined image description for other use is generated (S305).

[0109] Fig.11 This is the process of generating image description text for general roads ( Fig.10 Detailed flow chart of S302).

[0110] The description text generation unit 12 generates a traffic scene question and answer text ( Fig. 9 S211), question and answer text on whether there is a signal (S311).

[0111] In the description text generating unit 12, as Fig. 9 The judgment process (S213) asks whether there is a pedestrian in the image (S312). If yes, proceed to S313, and if not, proceed to S314.

[0112] In the explanatory text generating unit 12, as the question answering of the detailed items about the pedestrian ( Fig. 9 Then, a question and answer text indicating the presence of a pedestrian, a question and answer text about the location of the pedestrian, a question and answer text about the color of the pedestrian, and a question and answer text about the movement of the pedestrian are generated (S313).

[0113] The explanatory text generating unit 12 generates a reply text indicating that there is no pedestrian ( S314 ).

[0114] In the description text generating unit 12, as Fig. 9 The judgment process (S213) asks whether there is a motor vehicle in the image (S315). If yes, proceed to S316, and if not, proceed to S317.

[0115] In the description text generating unit 12, as a question answer about the detailed items of the motor vehicle ( Fig. 9 ), generate a response text indicating the existence of a motor vehicle, a question and response text about the location of the motor vehicle, a question and response text about the model of the motor vehicle, a question and response text about the color of the motor vehicle, and a question and response text about the action of the motor vehicle (S316).

[0116] In the description text generating unit 12, as Fig. 9The judgment process (S213) asks whether there is a bicycle in the image (S317). If yes, proceed to S318, and if not, proceed to S319.

[0117] In the description text generating unit 12, as a question answering of detailed items about the bicycle ( Fig. 9 Then, a question and answer text indicating the existence of a bicycle, a question and answer text about the location of the bicycle, a question and answer text about the color of the bicycle, and a question and answer text about the movement of the bicycle are generated (S318).

[0118] In the description text generating unit 12, as Fig. 9 The judgment process (S213) is performed, and a question is asked whether there is a crosswalk in the image (S319). If there is, the process proceeds to S320, and if not, the process ends. The explanatory text generation unit 12 generates a reply text indicating that there is a crosswalk (S320).

[0119] Used above Fig.10 The process of generating image descriptions for general roads has been described, but the process of generating image descriptions for other traffic scenes ( S303 to S305 ) is the same process.

[0120] Fig.12 Yes Figure 3 The table of intermediate data of the result of executing the image description text generation process (S21) for the still image data 111.

[0121] In this table, image description text is composed of a combination of a question text and an answer text to the question text in each row.

[0122] For example, row A01 is the result of the process ( S211 ) of classifying the traffic scene.

[0123] Lines A02 to A04 are the results of the process ( S213 ) of determining whether or not the selection target object exists in the still image data 111 .

[0124] Lines A05 to A13 are the results of the process ( S214 ) of acquiring detailed items about the selected object.

[0125] Fig.13 Yes Fig.12 The intermediate data is a table of output data resulting from the deletion of unnecessary data by the description text generating unit 12. Fig.12 In the necessity table 14A of the description object setting unit 14, the description text of pedestrians (row A02), the description text of bicycles (row A03), and the description text of toll booths (row A13) are set as not requiring description when not recognized. Fig.13 was deleted.

[0126] Fig.14It is a figure which shows the detail of the generation process (S22) of a GPS description text.

[0127] The description text generating unit 12 generates a description that assigns the shooting time of the still image data 111 to morning, daytime, evening, night, etc., according to the time period of the GPS information obtained from the vehicle travel GPS data 11B ( S221 ).

[0128] The description text generating unit 12 generates a description that identifies the shooting location of the still image data 111 as a main arterial road or a city name, according to the longitude and latitude of the GPS information obtained from the vehicle travel GPS data 11B ( S222 ).

[0129] This makes it possible to generate a traffic condition description text including information on the travel time and travel location of the vehicle. Therefore, the search accuracy of the user's search text is improved.

[0130] Fig.15 This is a diagram showing details of the control description text generation process (S23).

[0131] The description text generating unit 12 generates description text on the travel speed control based on the vehicle travel control data 11C (S231). The description text includes, for example, that 20 km / h or less is a low speed, 20 km / h to 60 km / h is a medium speed, and 60 km / h or more is a high speed.

[0132] The description text generation unit 12 generates description text on acceleration / deceleration control based on the vehicle travel control data 11C (S232). The description text includes, for example, acceleration at a certain acceleration or above, deceleration at a certain deceleration or above, and constant speed at other speeds.

[0133] The description text generating unit 12 generates a description text on the steering control based on the vehicle travel control data 11C (S233). The description text includes, for example, a left turn or a right turn when the steering angle is above a certain level, and a straight run when the steering angle is otherwise.

[0134] This makes it possible to generate a more natural traffic condition description text including descriptions of the control state and operation of the vehicle, thereby achieving an effect of improving the search accuracy of the user's search text.

[0135] Fig.16 Yes means Fig.14 and Fig.15 A table showing an example of the description text generated in .

[0136] In row B01, a description text of the time period is generated by S221.

[0137] In row B02, a description of the travel location is generated in S222.

[0138] In row B03, a description text of the travel speed is generated in S231.

[0139] In row B04, a description of the acceleration and deceleration state is generated in S232.

[0140] In row B05, a description of the steering state is generated in S233.

[0141] Fig.17 This is a table showing instruction texts for the large-scale language model unit 13 used in the process of generating the driving status description text (S24).

[0142] In row C01, an instruction to generate a traffic condition description text is described.

[0143] In row C02, information included in the traffic condition description is recorded.

[0144] In row C03, information to be excluded from the traffic condition description is recorded.

[0145] In row C04, an example of a traffic condition description text is recorded.

[0146] Then, in the driving state description text generation process (S24), the description text generation unit 12 generates a presentation in which the following texts (item 1) to (item 4) are sequentially combined. In addition, at least one of (item 2) and (item 3) may be omitted.

[0147] (Item 1) Instruction document for the large-scale language model unit 13 ( Fig.17 )

[0148] (Item 2) GPS Description Fig.16 Line B01, Line B02)

[0149] (Item 3) Control Description ( Fig.16 Row B03, Row B04, Row B05)

[0150] (Item 4) Image description ( Fig.13 )

[0151] The description text generation unit 12 inputs the generated prompt to the large-scale language model unit 13 to obtain a traffic condition description text described in natural language. The following is an example of a traffic condition description text (described later) generated by the large-scale language model unit 13 based on the prompt. Fig.21 (equivalent to status description text 732).

[0152] Traffic situation description = "You are driving straight at noon on Metropolitan Expressway Route 5 at a very high speed and then slowing down. In this scene, there are solid orange lane lines on the highway. Also, a white truck is moving forward on the road ahead of you."

[0153] As a result, a natural traffic condition description text is generated according to the usage purpose and the user's preference, so the search accuracy of the user's search text is improved.

[0154] Fig.18 It means according to Fig.13 A table showing an example of generating other images with different image descriptions.

[0155] Fig.18 The image description text is based on images taken from vehicles traveling on general roads. Therefore, since the combination of general roads and pedestrians in the necessity table 14A is "required", the information "no pedestrians are recognized" is recorded in row D07.

[0156] According to the Fig.18 In order to provide a prompt for an image description, the traffic condition description generated by the large-scale language model unit 13 using the generation process (S24) of the driving condition description is as follows.

[0157] Traffic situation description = "You are driving at a slow and steady speed on a street in Mito City at night, preparing to turn left. There are no pedestrians in this scene."

[0158] Above, refer to Fig.18 , the databaseization of the driving condition description text (processing up to storage in the description text DB 15) is explained. Next, an application example of the databaseized information is explained.

[0159] Fig.19 It is a flowchart showing the processing of the search unit 16.

[0160] The search unit 16 searches for images that match the input search text from among the images stored in the description text DB 15. Specifically, the search unit 16 outputs images having a high degree of similarity between the input search text and the situation description text stored in the description text DB 15 as search results.

[0161] In addition, in the retrieval unit 16, as "images with high similarity", for example, images can be listed in order of similarity from high to low in the retrieval results and the highest ones from 1st to Xth can be output (relative sequence similarity judgment), or images of retrieval results whose similarity is higher than a pre-set reference value (threshold Y) can be output (absolute similarity judgment).

[0162] The following describes the details of the processing of the search unit 16.

[0163] The search unit 16 receives a search text input by a user from the input / output unit 17 (S61). The user here is, for example, a commentator at a traffic control center that manages a highway, and the search text is, for example, a content indicating that images of a situation similar to an accident that occurred at a predetermined time at a predetermined location on a highway are collected from a database. The commentator plans to edit the image material obtained from the database and produce a report program about the accident that occurred.

[0164] The search unit 16 evaluates the similarity between the search text obtained from the user and the explanatory text stored in the explanatory text DB 15 (S62), and acquires video information (image information) associated with the explanatory text with high similarity from the explanatory text DB 15. The similarity evaluation may use cosine similarity search based on document vectorization or the like.

[0165] In addition to the driving dynamic image and driving condition description text obtained in S62, the search unit 16 also searches for related information of the driving condition (although it does not exist in the description text DB 15, it can specify the location, date and time of the driving condition description text, such as weather information obtained from the meteorological database), and generates a reply text based on these results (S63).

[0166] That is, the search unit 16 may output, as the search result, related information acquired from a database other than the description text DB 15 based on information included in the situation description text corresponding to the image with a high similarity.

[0167] The search unit 16 sends the reply text of S63 to the input / output unit 17 ( S64 ).

[0168] Thus, when searching for video data having a traffic condition description text similar to a natural language search text input by a user, it is possible to search for a target video in a short time.

[0169] Fig. 20 2 is a diagram showing an image search interface 71 of the input / output unit 17 .

[0170] The image search interface 71 is composed of a search text input section 72 which is a search text input field in S61 and a search result display section 73 which is a reply text display field in S64. The search result display section 73 displays a plurality of search results (scenes 1 to 4) as icons or thumbnail images.

[0171] Fig.21 Yes Click Fig. 20 The playback screen when the search result (scene 1) is displayed. The following information is displayed in sequence from the top on this playback screen.

[0172] · Image 731 of search results.

[0173] The situation description text 732 of the image 731 is extracted as a situation description text having a high degree of similarity with the search text.

[0174] Additional information 733 such as GPS description (time, location), control description (vehicle type, driving speed), and weather.

[0175] Fig. 22 Yes Click Fig. 20 The playback screen when the search results (Scene 2) are displayed. Fig.21 Similarly, in this playback screen, Fig.21 Similarly, an image 741 of the search result, a status description 742 thereof, and additional information 743 are displayed.

[0176] Thus, after the user inputs a search text in natural language, the user can search for dynamic images by referring to traffic condition description texts and dynamic images and related information similar to the search text, thereby improving search efficiency.

[0177] According to the present embodiment described above, when generating a description of a vehicle image, the description text generating unit 12 refers to the necessity table 14A and generates a natural description text including whether there are surrounding objects of interest based on the traffic scene the vehicle is in. Thus, natural description texts that mention necessary surrounding objects and do not mention unnecessary surrounding objects are stored in a database, so that the accuracy of database searches based on search texts input by people is improved.

[0178] Furthermore, the present invention is not limited to the above-mentioned embodiments, and various other application examples and modified examples may be adopted as long as they do not deviate from the main purpose of the present invention as described in the claimed scope of rights. For example, the above-mentioned embodiments describe the structure of the vehicle image analysis device 1 in detail and specifically in order to explain the present invention in an easy-to-understand manner, and are not limited to all the components that must be described. In addition, a part of the structure of a certain embodiment can be replaced with the components of other embodiments. In addition, the components of other embodiments can be added to the structure of a certain embodiment. In addition, for a part of the structure of each embodiment, other components can be added, replaced, or deleted.

[0179] In addition, the above-mentioned structures, functions, processing units, etc. may be partially or entirely implemented in hardware by, for example, designing in an integrated circuit, etc. As hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used.

[0180] In addition, the components of the vehicle image analysis device 1 of the above embodiment can be implemented in any hardware as long as the hardware can send and receive information to each other via a network. In addition, the processing performed by a certain processing unit can be implemented by a single hardware or by distributed processing performed by multiple hardware.

[0181] Description of Reference Numerals

[0182] 1Vehicle Image Analysis Device

[0183] 8 Communication lines

[0184] 11 Driving log DB

[0185] 11A Vehicle driving dynamic image data

[0186] 11B Vehicle driving GPS data

[0187] 11C Vehicle driving control data

[0188] 12. Explanatory text generation unit

[0189] 13 Large-Scale Language Model Department

[0190] 13A VQA Department

[0191] 13B Summary Generation Unit

[0192] 14 Description object setting section

[0193] 14A Whether to declare

[0194] 14B Identification Table

[0195] 15Description DB (image database)

[0196] 16 Search Department

[0197] 17 Input and output section

[0198] 91-93 vehicles

[0199] 100 Image Captioning System

[0200] 111 Still image data

[0201] 732, 742 Status description.

Claims

1. An image description system, comprising a camera mounted on a vehicle for capturing images, and a vehicle image analysis device for generating a condition description of the image captured by the camera, wherein: The vehicle image analysis device includes a description text generating unit. The description text generating unit, determining a scene presented in an image received from the camera, and reading recognition necessity information about an object in the image from the necessity table for each determined scene, Recognize the objects specified as requiring recognition by the read recognition necessity information from within the image, and generate a description of each object based on the recognition result. A description of the situation of the image is generated based on the determined scene and the description of each object, and the description of the situation of the image is stored in an image database in association with the image.

2. The image description system according to claim 1, wherein: The description text generating unit reads description necessity information about the object in the image from the necessity table, and if the object specified as requiring description by the read description necessity information cannot be identified from the image, generates description text indicating that the object does not exist in the image as description text for each object.

3. The image description system according to claim 1, wherein: The descriptive text generating unit reads detailed items about an object in an image from a recognition table, and generates a descriptive text about the read detailed items about an object recognized from the image as a descriptive text for each object.

4. The image description system according to claim 1, wherein: The description text generating unit adds description text on at least one of information on a time period when the image was captured and information on a driving position when the image was captured to the condition description text of the image based on positioning data read from a satellite positioning system mounted on the vehicle.

5. The image description system according to claim 1, wherein: The description text generating unit adds description text on at least one of speed information, acceleration / deceleration information, and steering angle information to the condition description text of the image based on vehicle travel control data read from an onboard ECU (Electronic Control Unit) of the vehicle.

6. The image description system according to claim 1, wherein: The vehicle image analysis device includes a summary generating unit for generating a summary text according to an input prompt. The description text generation unit generates a condition description text of the image by inputting a condition description text generation instruction text and a description text prompt for each object into the summary generation unit, wherein the condition description text generation instruction text includes information included in the condition description text, information to be excluded from the condition description text, and information of an example of the condition description text.

7. The image description system according to claim 1, wherein: The vehicle image analysis device includes a search unit for searching images matching an input search term from images stored in the image database. The search unit outputs, as a search result, an image having a high degree of similarity between an input search text and a situation description text stored in the image database.

8. The image description system according to claim 7, wherein: The search unit also outputs, as a search result, related information acquired from another database different from the image database based on information included in the situation description text corresponding to the image with a high similarity.

9. An image analysis device, characterized in that: Composition: The scene presented in the image is determined, and for each determined scene, recognition necessity information about an object in the image is read from the necessity table. Recognize the objects specified as requiring recognition by the read recognition necessity information from within the image, and generate a description of each object based on the recognition result. A description of the situation of the image is generated based on the determined scene and the description of each object, and the description of the situation of the image is stored in an image database in association with the image.

10. An image description method, characterized in that: Using an image analysis device, The scene presented in the image is determined, and for each determined scene, recognition necessity information about an object in the image is read from the necessity table. Recognize the objects specified as requiring recognition by the read recognition necessity information from within the image, and generate a description of each object based on the recognition result. A description of the situation of the image is generated based on the determined scene and the description of each object, and the description of the situation of the image is stored in an image database in association with the image.

Images