Device for storing data in database

Abstract

To extract each piece of information in a standardized format for easy use as training data and to prevent duplicate storage of similar information when storing information from a document 3 composed of images of traffic safety records and related text into a database 2.SOLUTION: A device 1 stores information from a document composed of images and text into a database, converts the images into scene graph data, and generates summarized text data from the text using a summary generation AI. In the database, the combination of scene graph data converted from the images and summarized text data generated from the text is stored in the database only if the same or similar combination does not already exist in a predetermined number. If it exists in the predetermined number, the device ensures that the same or similar combination is not stored beyond the predetermined number in the database.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an apparatus for storing data in a database, and more particularly to an apparatus for storing information extracted from documents consisting of images and text in a database as data for archival. [Background technology]

[0002] With the spread of artificial intelligence (AI) technology using machine learning, various technologies related to the accumulation of data for learning have been proposed. For example, Patent Document 1 proposes a method including the steps of creating a uniformly defined data representation for traffic scene data, constructing a graph depicting temporal and / or spatial relationships between entities in a traffic scene based on the traffic scene data having the uniformly defined data representation, and using the constructed graph as an input for a graph neural network, training the graph neural network, and extracting features from the traffic scene data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2023-98864 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, within companies, organizations, and various other places, documents that combine images and text (hereinafter referred to as "data documents") are created and used as materials on various situations, such as traffic scenes. If the information contained in such data documents can be collected using an appropriate method, digitized, and stored in a database, it can be used as data for learning AI technology (learning data). For example, with regard to the above traffic scene, automobile manufacturers and other related companies may hold meetings to analyze the situation and consider safety regarding various photos, videos, and illustrations (hereinafter referred to as "images") of traffic scenes, such as camera footage of the area around the vehicle taken with a dashcam, footage from surveillance cameras installed on the road, or illustrations depicting traffic conditions, and may create data documents (referred to as "traffic safety records") that are sets of images and related text. Many such data documents consisting of images and text are stored in various formats, and it is difficult to obtain information from such data documents. Information obtained from source documents is also useful as training data. In this regard, since the image and text formats are generally not consistent across document sources, it is difficult to use the data as training data as is. Therefore, to use such document sources as training data, it is preferable to extract the image and text features into an appropriately set format. Furthermore, information may be similar between document sources. In such cases, if all information from similar document sources were to be stored as data in a database, the database may contain a biased data set. Therefore, when storing document information as data in a database, it is preferable to avoid unnecessary duplication of information with similar content. In this regard, if the image and text formats are not consistent across documents, it becomes difficult to determine whether the information between documents is similar or dissimilar. Therefore, it is preferable to extract image and text information in an appropriately set, unified format, which makes it easier to select and discard similar information.

[0005] In view of the above circumstances, the main object of the present invention is to provide an apparatus for storing data in a database that, when digitizing information from document data sets that include images such as traffic safety records and related text and storing it in a database, extracts the image and text information from the document data in a unified format, making it easy to use as learning data, and prevents data of similar information from being stored more than necessary. [Means for solving the problem]

[0006] According to the present invention, the above problem is solved by an apparatus for converting information of document documents consisting of images and text into data and storing the data in a database, a scene graph data conversion means configured to recognize each of the images and convert it into scene graph data; a summary generation means configured to generate summary text data of the sentence using a summary generation AI from the sentence; a scene graph data searching means configured to search the database for scene graph data identical or similar to the scene graph data converted from the image; a summary search means configured to search the database for whether summary text data identical to or similar to the summary text data generated from the sentence is stored therein; a storage data selection means configured to store in the database a combination of the scene graph data converted from the image and the summary text data generated from the sentence when a predetermined number of combinations of scene graph data identical or similar to the scene graph data converted from the image and summary text data identical or similar to the summary text data generated from the sentence do not exist in the database, and to prevent the number of combinations identical or similar to the combination of the scene graph data converted from the image and the summary text data generated from the sentence from exceeding the predetermined number when the database contains the predetermined number of combinations of scene graph data identical or similar to the combination of the scene graph data converted from the image and the summary text data generated from the sentence; an apparatus including This is achieved by:

[0007] In the above configuration, the image may be any image, whether still or moving, and may be a photograph or drawing, as long as each of the images depicted therein can be converted into scene graph data. When converting an image into scene graph data, any image depicted in the image is represented as a node, and when multiple images exist, the relationships between those images, such as their positional relationships, may be represented as edges. The text may be any text related to the image. The "scene graph data search means" may use a technology that generates a scene graph from any image. The "summary search means" may use a summary generation AI technology that generates summary text data from any text ("summary text data" is a summary of the text in a document converted into text data that can be processed and saved by a computer). The "predetermined number" may be a natural number. Each of the above means is realized by the operation of a computer device in accordance with a program.

[0008] In the device of the present invention, when document information consisting of images and text is digitized and stored in a database, the images are converted into scene graph data and the text into summary text data, thereby extracting each piece of information in a unified format. This is advantageous in that the amount of data is smaller than when the images and text themselves are digitized and stored. Furthermore, in the device of the present invention, the scene graph data and summary text data are not stored in the database as is. Instead, the device searches the database for identical or similar combinations of scene graph data and summary text data. If the number of identical or similar combinations of scene graph data and summary text data stored in the database has already reached a predetermined number, the device either does not store the newly acquired combination of scene graph data and summary text data, or deletes one of the previously stored combinations in place of storing the newly acquired combination, so that the number of identical or similar combinations of scene graph data and summary text data does not exceed the predetermined number. This prevents imbalance in the types of data stored in the database.

[0009] In the above configuration, the scene graph data search means may be configured to determine, as identical or similar scene graph data, data that have the same graph structure and whose node categories match. A node category is a classification based on the type of image or other characteristics assigned to the node. This allows data that are not exactly the same to be determined to be similar, advantageously preventing bias in the scene graph data stored in the database. Furthermore, the summary search means may be configured to determine, as identical or similar summary text data, summary text data whose text similarity is equal to or greater than a predetermined, appropriately set threshold. The text similarity may be detected using any algorithm. This allows data that are not exactly the same to be determined to be similar, advantageously preventing bias in the summary text data stored in the database.

[0010] In the above configuration, the image may be an image of a traffic scene, the text may be text related to the traffic scene, the image recognized in the image may be an image of a moving object (person, vehicle), and the image of the object may be represented as a node in the scene graph data converted from the image, and the front-to-back, left-to-right positional relationships between the objects corresponding to the image may be represented as edges. In this way, people and vehicles in the traffic scene image are represented as nodes, and their positional relationships are represented as edges, simplifying the information, significantly reducing the amount of data, and making it easier to understand the positional relationships of the objects, which is expected to facilitate various data processing. Furthermore, when the text is text related to a traffic scene, the text may include a hazard prediction text identifying potential hazards in the situation shown in the image and a countermeasure text explaining countermeasures for avoiding the hazards, and the summary generation means may be configured to generate summary text data for each of the hazard prediction text and the countermeasure text. This allows the potential hazards in the traffic scene and the countermeasures to be summarized separately, making it easier to understand the content of the text and facilitating various data processing. [Effects of the Invention]

[0011] Thus, according to the device of the present invention, when information on document sets containing images and related text, such as traffic safety records, is digitized and stored in a database, the images are converted into scene graph data and the text is converted into summary text data, so that each piece of information is extracted in a unified format, eliminating differences in document formats and making it easy to use as learning data. Furthermore, the device of the present invention is configured to prevent data of similar information from being stored redundantly (beyond a predetermined number), thereby preventing bias due to type in the data stored in the database. Furthermore, since the data stored by the device of the present invention includes scene graph data, it is expected to be useful in any machine learning using graph neural networks.

[0012] Other objects and advantages of the present invention will become apparent from the following description of preferred embodiments of the invention. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram of the device of this embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of the device of this embodiment. [Figure 3] Fig. 3(A) is a diagram illustrating the process of extracting images and text from a document in the device of this embodiment. Fig. 3(B) is a diagram illustrating the process of preparing scene graph data from an image. Fig. 3(C) is a diagram illustrating the process of preparing summary text data from text. [Explanation of symbols]

[0014] 1...data storage device, 2...database, 3...document, 11...image extraction unit, 12...scene graph generation unit, 21...text extraction unit, 22, 23...specific text part extraction unit, 24, 25...summary text data generation unit, 30...previously registered similar data search unit, 31...new data registration unit BEST MODE FOR CARRYING OUT THE INVENTION

[0015] The present invention will now be described in detail with reference to some preferred embodiments thereof with reference to the accompanying drawings, in which like reference numerals indicate like parts.

[0016] Data storage device configuration As shown schematically in Fig. 1, an apparatus 1 for storing data in a database 2 according to this embodiment is configured to digitize information from document 3 containing images and text, and store the data in database 2. As shown in Fig. 2, data storing apparatus 1 is generally provided with an image extraction unit 11, a scene graph generation unit 12, a text extraction unit 21, specific text portion extraction units 22 and 23, summary text data generation units 24 and 25, a previously registered similar data search unit 30, and a new data registration unit 31. Data storing apparatus 1 may be a computer, and each of the above units is realized by the operation of the computer in accordance with a program.

[0017] Operation of the data storage device (a) Overview In brief, the device 1 of this embodiment first constructs a set of data by combining scene graph data generated from image information in the document 3 and summary text data generated from text information. The device 1 then searches for data already registered in the database 2 to determine whether data identical to or similar to the newly constructed data exists among the data previously registered in the database 2. If a predetermined number of data identical to or similar to the newly constructed data exists, the data is selected and stored in the database so that the total number of data identical to or similar to the newly constructed data does not exceed the predetermined number. The operation of each component will now be described in detail.

[0018] (b) Image scene graph data In this embodiment, as shown in Figure 3(A), in the image extraction unit 11, if the source document 3 is a document recorded on paper or film, etc. and is not digital data, the image 3im of the source document 3 may be digitized in any manner using a digital scanner or the like and extracted in a state that allows image processing by a computer. If the source document 3 is a computer file, image data is extracted. Thereafter, in the scene graph generation unit 12, using any scene graph generation technology, each image present in the image 3im is recognized, and each image is further specified as a node in a graph representation, and the relationships between the images are specified as edges in the graph representation, thereby generating scene graph data. For example, when the document 3 is a document consisting of images and text related to traffic scenes, such as a traffic safety record, and the images are images of traffic scenes, such as camera footage of the area around a vehicle captured by a dashcam, footage from a surveillance camera installed on the road, or illustrations depicting traffic conditions, as shown on the left side of Figure 3(B), images of moving objects such as people and vehicles are recognized in the images, and a scene graph 3g may be generated, as shown on the right side of Figure 3(B), in which these are used as nodes and the positional relationships of the front, back, left, and right of the images are expressed as edges. Note that the exact distances between the entities corresponding to the images do not need to be detected (although they may be detected). By converting the images into a scene graph that expresses the positional relationships, the amount of data can be reduced.

[0019] (b) Summarizing text and converting it into text data In this embodiment, as shown in FIG. 3A, if the document 3 is a document recorded on paper or film, or a computer file document that is not yet in text data form, the text extraction unit 21 may convert the text of the document 3 into text data using any optical character recognition technology and extract it in a computer-processable form. Furthermore, if the text can be segmented based on its content, the specific text portion extraction units 22 and 23 may extract each of the segmented texts as separate text data 3t1 and 3t2. For example, if the document 3 is a document consisting of images and text related to traffic scenes, such as a traffic safety record, and the document contains a hazard prediction sentence and a countermeasure sentence explaining how to avoid the hazard, each of these sentences is extracted as separate text data 3t1 and 3t2. Thereafter, in summary text data generators 24 and 25, summary text data 3a1 and 3a2 of text data 3t1 and 3t2 are generated using any summary generation AI technology that generates a summary from any sentence, as shown in Figure 3(C).The summary text data 3a1 and 3a2 are then associated with scene graph data 3g generated from the above image.

[0020] (c) Data selection and storage As described above, once a new dataset of scene graph data 3g and summary text data 3a1, 3a2 is constructed from information in document 3, the pre-registered similar data search unit 30 first checks whether a dataset identical or similar to the newly constructed dataset is already stored in database 2. In determining whether scene graph data are identical or similar, data having the same graph structure and matching node categories may be determined to be identical or similar scene graph data. As previously mentioned, the node category refers to a classification based on the type of image or other common features assigned to the node. For example, in the case of a scene graph 3g of a traffic scene image, categories such as people, bicycles, and vehicles including cars and trucks may be set. In determining whether summary text data are identical or similar, the text similarity between the summary text data may be detected using an arbitrary algorithm, and summary text data whose text similarity is equal to or exceeds a predetermined threshold value set as appropriate may be determined to be identical or similar. When both the scene graph data and the summary text data of the newly constructed datasets are identical or similar, the datasets are determined to be identical or similar.

[0021] If the previously registered similar data search unit 30 does not find a predetermined number of datasets that are identical or similar to the newly constructed dataset, the newly constructed dataset is stored in the database 2 by the new data registration unit 31. As already mentioned, the predetermined number is a natural number (an integer equal to or greater than 1) that may be set as appropriate. On the other hand, if a predetermined number of datasets that are identical to or similar to the newly constructed dataset are found, the newly constructed dataset is not stored, or the newly constructed dataset is stored and one of the datasets that was stored in the database 2 and is identical to or similar to the newly constructed dataset is deleted.

[0022] In the above configuration, scene graph data is judged to be similar even if it is not exactly the same graph data, which advantageously prevents bias in the scene graph data stored in the database. Also, summary text data is judged to be similar even if it is not exactly the same text data, which advantageously prevents bias in the summary text data stored in the database.

[0023] Thus, according to the configuration of the present embodiment, images of document documents are converted into scene graph data, and sentences are converted into summary text data, and each piece of information is extracted in a unified format, making it easy to use as learning data. Furthermore, by representing document information as scene graph data and summary text data, it becomes easy to determine whether data is identical or similar, and this prevents data with similar information from being stored redundantly (beyond a predetermined number), thereby preventing bias in the type of data stored in the database.

[0024] The above description has been made in relation to the embodiments of the present invention, but it will be apparent that many modifications and changes will be readily apparent to those skilled in the art, and the present invention is not limited to the above-exemplified embodiments, but can be applied to various devices without departing from the concept of the present invention.

Claims

1. A device that converts information about document data consisting of images and text into data and stores it in a database, a scene graph data conversion means configured to recognize each of the images and convert it into scene graph data; A summary generation means configured to generate summary text data of the sentence using a summary generation AI from the sentence; a scene graph data searching means configured to search the database for scene graph data identical or similar to the scene graph data converted from the image; a summary search means configured to search the database for whether summary text data identical to or similar to the summary text data generated from the sentence is stored therein; a storage data selection means configured to store in the database a combination of the scene graph data converted from the image and the summary text data generated from the sentence when a predetermined number of combinations of scene graph data identical or similar to the scene graph data converted from the image and summary text data identical or similar to the summary text data generated from the sentence do not exist in the database, and to prevent the number of combinations identical or similar to the combination of the scene graph data converted from the image and the summary text data generated from the sentence from exceeding the predetermined number when the database contains the predetermined number of combinations of scene graph data identical or similar to the combination of the scene graph data converted from the image and the summary text data generated from the sentence; An apparatus comprising:

2. 2. The device of claim 1, wherein the image is an image of a traffic scene, the text is text related to the traffic scene, the images recognized in the image are images of moving objects, the images of the objects are nodes in scene graph data converted from the image, and the front-to-back, left-to-right positional relationships between the objects corresponding to the images are represented by edges.

3. 3. The device of claim 2, wherein the device includes a danger prediction sentence that identifies a danger that is anticipated in the situation in which the sentence appears in the image, and a countermeasure sentence that explains a countermeasure method for avoiding the danger, and the summary generation means is configured to generate summary text data for each of the danger prediction sentence and the countermeasure sentence.

4. 2. The device of claim 1, wherein the scene graph data search means is configured to determine that data having the same graph structure and having matching node categories is the same or similar scene graph data.

5. 2. The apparatus according to claim 1, wherein said summary search means is configured to determine that summary text data having a sentence similarity equal to or greater than a predetermined threshold value are identical or similar to each other.

Images