Information processing device

Abstract

To enable proper searching of image data. [Solution] The information processing device comprises: storage means for storing image data and graph structures extracted from the image data in association; acquisition means for acquiring a graph structure input by the user as a search key; search means for searching for graph structures stored in the storage means using the search key; and output means for outputting image data associated with graph structures similar to the search key as a search result.

Description

Technical Field

[0001] This disclosure relates to the technical field of information processing devices.

Background Art

[0002] As this type of device, for example, a system has been proposed in which a large language model (LLM) generates query data based on a document, and a pair of the document and the query data is used for learning a search model for a chatbot (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As one of the machine learning models, a GNN (Graph Neural Network) that handles data having a graph structure is known. The GNN has been proposed to be applied to, for example, classification and regression. On the other hand, regarding the search for images using a graph structure, no specific proposal has been made yet, and there is room for improvement.

[0005] This disclosure has been made in view of the above problems, and an object thereof is to provide an information processing device capable of realizing appropriate search of image data.

Means for Solving the Problems

[0006] An information processing device according to one aspect of this disclosure includes: storage means for storing image data and graph structures extracted from the image data in association; acquisition means for acquiring the graph structure input by a user as a search key; search means for searching the graph structures stored in the storage means using the search key; and output means for outputting image data associated with graph structures similar to the search key as a search result. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram showing the hardware configuration of an information processing device according to an embodiment. [Figure 2] This is a block diagram showing the functional configuration of an information processing apparatus according to an embodiment. [Figure 3] This is a flowchart showing the flow of the storage operation by the information processing device according to the embodiment. [Figure 4] This is a plan view illustrating an example of the accumulated image data and graph structure. [Figure 5] This is a flowchart showing the flow of the search operation by the information processing device according to the embodiment. [Modes for carrying out the invention]

[0008] The following describes an embodiment of the information processing device with reference to the drawings.

[0009] (Hardware configuration) First, the hardware configuration of the information processing device according to the embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing the hardware configuration of the information processing device according to the embodiment.

[0010] In Figure 1, the information processing device 10 according to this embodiment comprises an arithmetic unit 110, a storage device 120, a communication device 130, an input device 140, and an output device 150. The arithmetic unit 110, the storage device 120, the communication device 130, the input device 140, and the output device 150 are connected to each other via a data bus.

[0011] The arithmetic unit 110 is configured to perform various arithmetic operations in the information processing device 10. The arithmetic unit 110 may have a processor. The arithmetic unit 110 may have a single processor or may have multiple processors. In other words, the arithmetic unit 110 may have one or more processors. The processor may be a multi-core processor. If the arithmetic unit 110 has a single processor that is a multi-core processor, then logically, the arithmetic unit 110 can be said to have multiple processors.

[0012] The processor in the arithmetic unit 110 may be at least one of the following: CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and TPU (Tensor Processing Unit).

[0013] The storage device 120 may be at least one of the following: RAM (Random Access Memory), ROM (Read Only Memory), hard disk drive, magneto-optical disk drive, SSD (Solid State Drive), and optical disk array. In other words, the storage device 120 may be implemented by a single device or by multiple devices.

[0014] The storage device 120 is capable of storing desired data. The storage device 120 may store the computer program CP that the arithmetic unit 110 will execute. The storage device 120 may temporarily store data that the arithmetic unit 110 will use temporarily when the arithmetic unit 110 is executing the computer program CP.

[0015] The computer program CP may be recorded on a non-temporary recording medium that is readable by a computer. In this case, the computer program CP may be stored in the storage device 120 by reading the recording medium using a recording medium reading device (not shown) provided by the information processing device 10. At least one of the following may be used as the recording medium: an optical disc, a magnetic medium, a magneto-optical disc, a semiconductor memory, and any other medium capable of storing a program. The computer program CP may also be obtained from an external device (not shown) of the information processing device 10 via a communication device 130. In other words, the computer program CP may be downloaded from an external device to the storage device 120 of the information processing device 10.

[0016] The arithmetic unit 110 (for example, a processor) may execute the processing that the information processing device 10 should perform together with the memory device 120 in which the computer program CP is stored (in other words, together with the memory device 120 and the computer program CP stored in the memory device 120). For example, by the arithmetic unit 110 executing the computer program CP, a logical functional block for executing the processing that the information processing device 10 should perform may be realized within the arithmetic unit 110 (for example, within the processor).

[0017] The communication device 130 is configured to communicate with devices outside the information processing device 10. The communication device 130 may use wired communication or wireless communication.

[0018] The input device 140 is a device capable of receiving input of information to the information processing device 10 from the outside. The input device 140 may include an operating device (for example, a keyboard, a mouse, a touch panel, etc.) that can be operated by the user of the information processing device 10. The input device 140 may include a recording medium reading device capable of reading information recorded on a recording medium detachable from the information processing device 10, such as a USB (Universal Serial Bus) memory. When information is input to the information processing device 10 via the communication device 130 (in other words, when the information processing device 10 acquires information via the communication device 130), the communication device 130 may function as an input device.

[0019] The output device 150 is a device capable of outputting information to the outside of the information processing device 10. The output device 150 may have a display device capable of outputting visual information such as characters and images as the above information. The output device 150 may also have a speaker capable of outputting auditory information such as sound as the above information. The output device 150 may be configured to output the above information (for example, control information of other devices, etc.) to other devices. The output device 150 may be capable of outputting information to a recording medium detachable from the information processing device 10, such as a USB memory. When the information processing device 10 outputs information via the communication device 130, the communication device 130 may function as an output device.

[0020] <Functional Configuration> Next, the functional configuration of the information processing device 10 according to the embodiment will be described while referring to FIG. 2. FIG. 2 is a block diagram showing the functional configuration of the information processing device according to the embodiment.

[0021] In FIG. 2, the information processing apparatus 10 is configured as an apparatus that searches for an image based on a graph structure input by a user. The information processing apparatus 10 includes, as components for realizing its functions, an image acquisition unit 210, a graph structure extraction unit 220, an image database (DB) 230, a search key acquisition unit 240, an image search unit 250, and an image output unit 260. Each of the image acquisition unit 210, the graph structure extraction unit 220, the search key acquisition unit 240, the image search unit 250, and the image output unit 260 may be a processing block realized by the arithmetic unit 110 described above. Also, the image DB 230 may be a database realized by the storage device 120 described above.

[0022] The image acquisition unit 210 is configured to be able to acquire image data. The image acquisition unit 210 may be configured to acquire, for example, image data input by a user. Alternatively, the image acquisition unit 210 may be configured to automatically collect and acquire image data.

[0023] The graph structure extraction unit 220 is configured to be able to extract a graph structure from the image data acquired by the image acquisition unit 210. Note that the method for extracting a graph structure from image data is not particularly limited, but the graph structure extraction unit 220 may extract a graph structure from an image using, for example, a model that has been machine-learned in advance. This model may be a model that has been learned to output a graph structure with image data as an input.

[0024] The image DB 230 is configured to be able to store the image acquired by the image acquisition unit 210 and the graph structure extracted by the graph structure extraction unit 220. More specifically, the image DB 230 is configured to be able to store the image data and the graph structure extracted from the image data in association with each other. The image DB may store the image data in association with the feature amount indicating the graph structure.

[0025] The search key acquisition unit 240 is configured to acquire a search key (i.e., a condition for searching for image data) entered by the user. The search key acquisition unit 240 acquires a graph structure as the search key.

[0026] The image search unit 250 searches for image data stored in the image database using the search key acquired by the search key acquisition unit 240. Specifically, the image search unit 250 compares the graph structure acquired as the search key with the graph structures stored in the image database 230 to identify graph structures with high similarity, and outputs the image data stored in association with that graph structure as the search result. For example, the image search unit 250 may output the image data associated with the graph structure with the highest similarity to the search key as the search result. Alternatively, the image search unit 250 may output the image data associated with a graph structure whose similarity to the search key is above a predetermined threshold as the search result. The image search unit 250 may also perform the search using a feature quantity that represents the graph structure acquired as the search key.

[0027] The image output unit 260 is configured to output image data, which is output as a search result by the image search unit 250, to the user. For example, the image output unit 260 may be configured to display the image data, which is the search result, on a display. In addition to the image data, the image output unit 260 may also output the graph structure associated with that image. For example, the image output unit 260 may display the image data and the graph structure side by side on the display.

[0028] (Accumulation action) Next, with reference to Figures 3 and 4, the flow of the storage operation by the information processing device 10 according to the embodiment (i.e., the operation of linking and storing image data and graph structures) will be described. Figure 3 is a flowchart showing the flow of the storage operation by the information processing device according to the embodiment. Figure 4 is a plan view showing an example of the stored image data and graph structure.

[0029] As shown in Figure 3, when the storage operation by the information processing device 10 according to the embodiment is started, the image acquisition unit 210 first acquires an image (step S101). Then, the graph structure extraction unit 220 extracts the graph structure from the image acquired by the image acquisition unit 210 (step S102).

[0030] Next, the image database 230 stores the images acquired by the image acquisition unit 210 and the graph structures extracted by the graph structure extraction unit 220, linking them together (step S103). As the processes described in steps S101 to S103 are repeatedly executed, multiple pairs of image data and graph structures are accumulated in the image database 230.

[0031] As shown in Figure 4, suppose image data containing, for example, a car and two people is acquired. In this case, a graph structure containing information about the positional relationship between the car and the two people (an adult and a child), as well as information about the shape and color of the car and people, is extracted from the image data. The image DB230 stores such pairs of image data and graph structures.

[0032] (Searching action) Next, with reference to Figure 5, the flow of the search operation by the information processing device 10 according to the embodiment (i.e., the operation when searching for image data using a search key) will be described. Figure 5 is a flowchart showing the flow of the search operation by the information processing device according to the embodiment.

[0033] As shown in Figure 5, when the storage operation by the information processing device 10 according to the embodiment is started, first the search key acquisition unit 240 acquires a graph structure, which is the search key, from the user (step S201). Then, the image search unit 250 uses the graph structure acquired by the search key acquisition unit 240 to search for image data stored in the image DB 230 (step S202).

[0034] Next, the image output unit 260 outputs the search result from the image search unit 250 (step S203). That is, the image output unit 260 outputs image data that matches the search key to the user. If the search result includes multiple image data, the image output unit 260 may output multiple image data.

[0035] (Technical effects) Next, the technical effects obtained by the information processing device 10 according to this embodiment will be described.

[0036] As explained in Figures 1 to 5, in the information processing device 10 according to this embodiment, image data and graph structures extracted from the image data are linked and stored. Then, the stored image data is searched using the graph structure input by the user. By searching for images using the graph structure in this way, the desired image data can be extracted with high accuracy from the stored image data. For example, images can be searched by considering various features such as the positional relationships of multiple objects included in the graph structure, as well as the shapes and colors of the objects.

[0037] The embodiments of the invention derived from the above-described embodiments are described below.

[0038] An information processing device according to one aspect of this disclosure includes: storage means for storing image data and graph structures extracted from the image data in association; acquisition means for acquiring the graph structure input by a user as a search key; search means for searching the graph structures stored in the storage means using the search key; and output means for outputting image data associated with graph structures similar to the search key as a search result. In the above embodiment, "image DB 230" corresponds to an example of "storage means," "search key acquisition unit 240" corresponds to an example of "acquisition means," "image search unit 250" corresponds to an example of "search means," and "image output unit 260" corresponds to an example of "output means."

[0039] In the information processing apparatus according to the above embodiment, the search means may perform the search based on the positional relationships of multiple objects shown in the graph structure. In this way, it becomes possible to search for images more appropriately based on the positional relationships of objects included in the graph structure.

[0040] In the information processing apparatus according to the above embodiment, the search means may perform the search based on at least one of the shape and color of the object shown in the graph structure. In this way, it becomes possible to search for images more appropriately based on at least one of the shape or color of the object included in the graph structure.

[0041] This disclosure is not limited to the embodiments described above and can be modified as appropriate without contradicting the gist or idea of ​​the invention as can be inferred from the claims and the specification as a whole. Information processing devices with such modifications are also included within the technical scope of the present invention. [Explanation of symbols]

[0042] 10...Information processing unit, 110...Calculation unit, 120...Storage device, 130...Communication device, 140...Input device, 150...Output device, 210...Image acquisition unit, 220...Graph structure extraction unit, 230...Image DB, 240...Search key acquisition unit, 250...Image search unit, 260...Image output unit

Claims

1. A storage means for linking and storing image data and a graph structure extracted from the image data, A means for obtaining the graph structure input by the user as a search key, A search means that uses the search key to search for the graph structure stored in the storage means, An output means that outputs image data linked to the graph structure similar to the search key as a search result, An information processing device equipped with the following features.

2. The search means performs the search based on the positional relationships of multiple objects shown in the graph structure. The information processing apparatus according to claim 1.

3. The search means performs the search based on at least one of the shape and color of the object shown in the graph structure. The information processing apparatus according to claim 1 or 2.

Images